How to regression test AI agents before deployment

What an Agent Regression Test Is, and Why It Misleads

A regression test for an agent runs the same fixed inputs through the version in production and the version you want to ship, scores both runs the same way, and asks one question: did anything that used to work stop working? It is the same idea as a unit test suite. What makes it harder is that neither the agent nor the scorer is deterministic.

Run the same case twice and an agent can take a different path, call tools in a different order and land on a different answer. Score the same transcript twice with an LLM judge and the verdict can change. Put those together and a single run per case produces red builds that pass on retry, and green builds that hide a real regression behind a lucky sample. Teams then learn to click "re-run", and the suite stops meaning anything.

Aggregates add a third problem. An overall pass rate that holds at 88% can conceal twelve cases that broke and twelve that got fixed. If the twelve that broke include the refund-limit case, the aggregate is the wrong number to gate on.

The Pieces You Need

• A frozen dataset. Real cases from production, labelled, hashed and versioned, so the inputs cannot change between the baseline run and the candidate run. Building one is its own job.
• A safe candidate environment. Run the candidate against staging tools or recorded tool responses. A regression test that can issue a real refund or modify production data is itself a risk.
• The same evaluations as production. Run the candidate sessions through the code-based and LLM-based evaluations already used for live traffic, so pre-deploy and post-deploy scores mean the same thing.
• A baseline. Keep the current version's sessions and results for the same dataset version, with agent version and environment attached so the cohorts cannot be mixed up.
• A gate. Thresholds written down before the run, not chosen after looking at the numbers.

Run the Regression Set Through Failproof AI

1. Capture the baseline. Select production sessions that cover ordinary traffic, important workflows and failures you never want to repeat. Save the user inputs, expected outcomes and any tool responses needed for a safe rerun.
2. Run the current and candidate agents. Execute the same cases in staging with side-effecting tools stubbed or pointed at test systems. Tag every resulting session with the agent version, dataset version and trial number.
3. Evaluate both cohorts. Let Failproof AI Cloud run the same code checks, LLM-based evaluations and existing evaluation suite on every session. Keep the evaluation versions fixed throughout the comparison.
4. Compare case by case. Look first at cases that passed on the current version and failed on the candidate. Aggregate scores are useful, but they can hide an equal number of fixed and broken cases.
5. Open regressions in session replay. Step through the trace to see where tool choice, arguments, recovery or the final answer diverged. The score identifies the regression; the trace explains it.

Setting the Gate

The flip test is the one that handles noise honestly. Only cases that changed outcome carry information about the difference between versions. If b cases broke and c got fixed, then under "no real difference" each changed case is a coin flip, and an exact binomial test on b out of b + c - McNemar's exact test - tells you how surprising the split is.

Floors should come from history, not hope. Replay the current production version against the dataset three or four times without changing anything, and look at the spread of each score: that spread is the noise floor of your suite. A floor set inside that spread will fail the production version on a bad day. A floor set well below it will pass real regressions. A workable rule is to set each floor just under the lowest pass rate the production version produced across those repeat runs, then tighten it as the suite grows. Write the floors, the must-pass list and the judge version into the repository, so a change to any of them is reviewed like code.

Record the gate as a release checklist or CI decision before the candidate results arrive: required cases, score floors, the maximum allowed evaluation errors and the owner who can approve an exception. With a few hundred cases, a statistical flip test can still be weak for small effects: four broken and one fixed may not be significant, yet one of those four may be the exact refund or deletion failure you care about. Statistics control noise; must-pass cases protect specific promises.

Handling Flaky Agents and Flaky Judges

For the agent, decide what "passes" means across trials and write it down. Majority of trials, as in the gate above, measures typical behaviour. All trials passing is the stricter reliability bar, and it is the right one for must-pass cases, where one bad run in three is one bad run in three in production too. More trials shrink the noise at a linear cost in agent and judge tokens, so spend them on the cases that flip, not on the whole set.

For the judge, remove every source of variation you control. Pin the judge model to an exact version, not an alias that moves. Version the rubric and store the version beside every result. Measure the judge's own noise by re-judging the baseline transcripts without re-running the agent: any case that changes verdict there is judge noise, not agent regression, and it tells you how many flips to expect from the judge alone. When that number is high, the rubric is ambiguous. Fix the wording before adding more trials.

Pitfalls, and How to Check Your Work

• A stale baseline. Regenerate it whenever the dataset version changes. Comparing a candidate on v4 with a baseline on v3 compares datasets, not agents.
• Changing two things. A prompt change and a judge change in one run cannot be told apart. Change one, gate, then change the other.
• Live side effects. Grep the replay harness for real credentials. Stub or record every tool that writes.
• Thresholds after the fact. Floors picked after seeing the candidate's numbers always pass. Commit them first.
• A smoke set with no ordinary successes. Known failures belong in the gate, but so do representative happy paths. Remove redundant low-value cases rather than moving every stable success to the nightly run; otherwise a broad regression can break normal behavior without touching a historical failure case.
• Suites nobody reads. Print the broken case ids and make someone open three of them before each release. The transcript is where the regression is actually understood.

From Regression Result to Failure Finding

Failproof AI keeps the regression score beside the full session trace. Re-evaluate one stored session after changing an evaluation, or backfill the new evaluation across recent sessions to confirm that the scoring change behaves as expected. Keep the same evaluations running after deployment so staging and production share one definition of success.

Automatic failure analysis can group related candidate failures into findings, show the traces behind the pattern and recommend what to change. That is more useful than treating every red case as an unrelated test failure. After release, the same process finds production failure modes the frozen dataset missed; add representative, scrubbed cases to the next dataset version after human review.

Use alerts and findings to assign the regression to an owner and validate the recommended fix. If the evidence identifies a known high-risk action, a behavioral policy can prevent that action in production; backtest it against historical calls and include policy denials in regression results. Keep the original evaluation running so you can confirm the behavior improved rather than simply moving around the rule.

When You Do Not Need a Regression Suite

If the agent is a prototype with a handful of users, reading twenty transcripts after each change is cheaper and more informative than a suite. If a change cannot affect agent behaviour - a logging tweak, a dashboard - skip the run with a path filter. And if you can safely roll back in minutes and each bad session costs little, a careful canary with production evals may catch more than a small offline suite ever will. Regression suites earn their keep when a bad session is expensive and rollbacks are slow.