Coding Agents Need Experiments. Your Test Suite Needs Regression Tests.

Coding agents write tests. Lots of them.

They reproduce bugs, explore edge cases, challenge assumptions, and occasionally write tests to verify that their other tests actually test something.

That's useful.

The problem isn't that agents write too many tests.

It's that temporary experiments often become permanent maintenance obligations.

A check that helped debug something today doesn't necessarily deserve a place in CI for the next five years.

And adding more tests isn't automatically the same as improving test quality.

So here's a simple idea:

Let coding agents experiment. Make regression tests earn their place.

1. Same Framework. Different Responsibilities.

Not every executable check serves the same purpose.

Type| Purpose| Lifetime
Evidence check| Investigate a question or assumption| Temporary
Regression test| Protect established behavior or a contract| Maintained
Execution evidence| Record what ran, what happened, and what remains unknown| Retain when useful

An evidence check helps an agent learn something.

It might deliberately fail. It might test a hypothesis that turns out to be wrong. It might explore an edge case that doesn't justify permanent coverage.

That's normal engineering work.

A regression test makes a stronger promise:

«This behavior matters, and future changes must preserve it.»

Its expected result should come from a requirement, contract, or established invariant, not simply mirror whatever the current implementation happens to do.

Both can use PHPUnit.

Different responsibilities don't require different frameworks.

And no, this doesn't require inventing "AIEnhancedEvidenceDrivenTestingFrameworkFactory".

The existing tools are probably enough.

1. An Ignored Directory Is Surprisingly Effective

For a PHP project using PHPUnit, temporary checks can live outside the maintained test suite:

build/evidence/TASK-123/probes/

Run a check explicitly:

vendor/bin/phpunit \
--no-configuration \
--bootstrap vendor/autoload.php \
build/evidence/TASK-123/probes/QuestionTest.php

Assuming:

• "build/" is ignored by Git.
• The regular PHPUnit suite discovers tests under "tests/".
• Composer and CI do not independently discover the probe directory.

The temporary checks stay out of normal validation.

No additional dependency.

No custom runner.

No new CI stage.

Just PHPUnit doing what PHPUnit already does.

The important part is the execution boundary, not the directory name.

And there's one subtle detail:

Scratch-space permission and implementation scope are not the same thing.

A task-local scratch directory can stay outside the validated source snapshot when explicitly permitted by the task or repository.

That permission doesn't authorize unrelated source edits or override sandbox restrictions.

Temporary checks should be disposable without making implementation validation stale.

1. A Real Experiment That Became a Regression Test

During a Git candidate-export investigation, a coding agent needed to answer a specific question:

«Does exporting a candidate preserve symbolic links and executable permissions?»

The agent created a temporary PHPUnit check.

It constructed a small Git repository containing a symlink and an executable file, then exercised the export helper.

The check passed.

Now came the interesting decision:

Should this test stay?

In this case, yes.

Preserving those metadata was part of the established candidate-export contract.

Losing an executable bit or changing a symlink would change the exported candidate.

So the meaningful assertions became a maintained regression test.

The original experiment was removed.

The resulting regression test protects the export helper's behavior.

It does not establish that every gateway selection, submodule operation, or interrupted cleanup path is correct.

That's the difference between useful evidence and exaggerated confidence.

The process was simple:

Question
↓
Experiment
↓
Observation
↓
Decision
├── Promote to regression test
├── Record consequential evidence
└── Discard temporary check

Not every experiment needs to become a regression test.

Sometimes its job is simply finished.

1. What About Keeping Evidence for Code Reviews?

This raises another question:

Could retaining an agent's experiments help an independent reviewer find more defects?

It sounds reasonable.

A reviewer might benefit from knowing which assumptions were challenged, which checks failed, and what the implementing agent never examined.

But plausible benefits aren't measured benefits.

A small experiment compared four fresh reviewer sessions:

• Two received ordinary review materials.
• Two additionally received a recorded probe observation.

Unfortunately, the reviewers receiving the additional observation couldn't access the actual probe source.

They could read about the experiment.

They couldn't reproduce it.

The comparison also lacked an independently established defect oracle. All four sessions used the same model family, without independent adjudication of their qualitative conclusions.

Result: INCONCLUSIVE.

No demonstrated improvement in defect detection.

But also no evidence that reproducible probe histories are inherently useless.

The experiment simply couldn't answer that question.

It did establish one practical lesson:

«If reproducibility matters, provide the actual candidate, executable check, and relevant inputs. A hash or summary isn't enough.»

A record saying an experiment passed isn't the same as making that experiment reproducible.

And neither proves the implementation correct.

1. More Tests Don't Automatically Mean Better Tests

Earlier coding-agent experiments revealed another interesting pattern.

Extended testing instructions generated more test code without detecting more of the selected defects than the existing guidance.

That doesn't prove additional tests are useless.

It doesn't mean comprehensive testing is bad.

It means more generated test code didn't produce a measured improvement in those experiments.

A few things worth remembering:

• Coverage measures execution reach, not correctness.
• Assertion counts don't measure behavioral protection.
• Mutation testing can expose weak assertions, but selected mutants aren't a complete correctness oracle.
• A meaningful integration test may be expensive and still worth maintaining.
• A temporary experiment may be valuable even if it never becomes a regression test.

The objective isn't minimizing the test suite.

It's improving the relationship between feedback, confidence, and maintenance cost.

An evidence check should answer a useful question.

A regression test should protect meaningful behavior.

Both need a reason to exist.

1. The Practical Rule: Promote, Record, or Discard

At the end of an implementation task, classify the temporary checks.

Promote when a check protects an established requirement or invariant and justifies its maintenance cost.

Record when an observation explains a consequential decision, unexpected failure, or unresolved uncertainty.

Discard when an experiment has answered its question and provides no lasting value.

Don't delete existing regression tests just because an agent generated new experiments.

Don't retain every passing probe simply because it ran.

And don't introduce a new testing framework unless the current one demonstrably cannot support the work.

The goal isn't fewer tests.

It's faster feedback during development and less unnecessary maintenance afterward.

Conclusion

Coding agents should experiment.

They should reproduce failures, challenge assumptions, explore edge cases, and occasionally discover that their first implementation was completely wrong.

That's engineering.

But temporary investigation and permanent regression protection are different responsibilities.

They can use the same framework.

They don't need the same lifetime.

Let coding agents experiment. Make permanent tests earn their place.

Your CI pipeline already has enough responsibilities.

It doesn't need to remember every question an LLM asked itself while debugging.

TL;DR

Use your existing test framework for three different purposes:

• Evidence checks help agents investigate.
• Regression tests protect behavior worth maintaining.
• Execution evidence records meaningful observations and limitations.

Keep temporary checks in an explicitly permitted workspace outside the normal test suite.

Promote meaningful checks. Record consequential findings. Discard disposable experiments.

No new framework required.