How to run agent evals in CI with Failproof AI

What an Eval Job in CI Can and Cannot Tell You

A CI eval job answers one narrow question well: did this change make the agent worse on cases we already know about? It catches the prompt edit that breaks refund handling, the tool schema change that confuses argument formatting, the refactor that drops a system instruction. It cannot tell you how the agent behaves on requests nobody has seen yet, which is what production evaluation is for. Treat the two as a pair.

CI eval jobs fail in their own ways. They are slow, because every case is a full agent run plus a model-based evaluation, so developers stop waiting for them. They are expensive, because each push can rerun everything. They are flaky, because agents and LLM evaluations are nondeterministic, so a red build gets rerun until it goes green. And pull requests from forks do not receive repository secrets. The workflow here accounts for all four.

The Simpler Failproof AI Workflow

1. Keep a reviewed smoke set. Choose 30-60 representative cases and named failures that must never recur. Store the user input, expected outcome and safe tool fixtures with a dataset version.
2. Run the candidate in staging. CI invokes your agent on those cases with side effects stubbed or directed to test systems. Tag the emitted sessions with the candidate version, dataset version and environment.
3. Let Failproof AI score the sessions. Cloud runs the same code-based and LLM-based evaluations used on production traffic, including your existing evaluation suite. You do not need a second scoring implementation just for CI.
4. Compare and inspect. Filter results to the candidate run, compare them with the accepted baseline and open failed cases in session replay. The evaluation says what regressed; the trace shows where behavior diverged.
5. Return one release decision. CI should fail if a must-pass case is absent or fails, a critical score falls below its pre-agreed floor, or the evaluation run is incomplete. Keep the threshold configuration reviewed alongside the agent.

Failproof AI simplifies the evaluation layer, not the execution of your agent. CI still needs one command that runs the candidate against the selected inputs, and a small final step that turns the returned evaluation results into a pass or fail exit code. The platform handles evaluation execution, trace storage, result history, session replay, alerts and failure analysis instead of making you rebuild those around a custom script.

Get the paths list right, because it decides what the gate protects. Agent behaviour changes when any of these change: the system prompt and prompt templates, tool definitions and their descriptions, the model id, retrieval settings, and the code that assembles context. If the model id lives in a shared config file or an environment file checked into the repository, that file belongs in the filter. A model bump that skips the eval job because it touched config/ rather than agent/ is the change you most wanted tested. When in doubt, widen the filter and let cancel-in-progress absorb the extra runs. A job that runs slightly too often costs tokens; a job that misses the model change costs an incident.

Use a separate API key for CI, with its own spend limit at your model provider, so a runaway job cannot eat the production budget. On public repositories, remember that pull_request runs from forks get no secrets: the eval step will fail for outside contributors. Have a maintainer run the evals through workflow_dispatch on a branch they control, rather than switching to a trigger that exposes secrets to untrusted code.

Choose a Gate That Matches the Size of the Set

A small pull-request set should gate on named must-pass cases and absolute floors. It is too small to reliably detect a change of only a few percentage points. Use the larger nightly or pre-release set for version-to-version drop checks, and inspect pass-to-fail case flips even when the overall average is unchanged.

Sixty independent cases with a 90% observed pass rate produce a wide confidence interval, roughly 79% to 96%. A five-point difference is therefore not a defensible pull-request gate on that set. Use the smoke run to catch large breakage and protect specific cases; use a properly sized comparison to decide whether a small aggregate movement is real.

Keep It Fast

• An explicit smoke subset. Mark reviewed cases with "smoke": true so the same high-value cases run on every pull request. Do not rely on the first hashed filenames in a directory; that is stable but arbitrary.
• Parallel cases. Agent runs are mostly waiting on the network. A thread pool of 8-16 workers cuts wall-clock time sharply; stay within your provider's rate limits.
• Stubbed tools. Recorded tool results are faster than live calls and do not flake when a third-party API has a bad afternoon.
• One trial on pull requests. Repeated trials belong in the nightly run and the pre-release gate.
• Fail early on infrastructure. If the first three cases all raise an exception, stop and report a setup error instead of burning through the rest.

Guard the Cost, with the Math Written Out

Assumptions for this example: 60 cases, 1 trial each on pull requests; each agent run reads 40,000 input tokens and writes 2,000 across all its model calls; each judge call reads 5,000 and writes 200. A_in and A_out are your agent model's prices and J_in and J_out your judge model's, in dollars per million tokens.

Two things stand out. The agent, not the judge, is usually most of a CI eval bill, because the agent makes many model calls per case and the judge makes one. So the cheapest lever is often a smaller smoke set, not a cheaper judge. And the multiplier that surprises teams is completed runs per day: twenty pushes across ten open pull requests is twenty runs without cancel-in-progress, and closer to ten with it.

Pitfalls, and How to Check Your Work

• Baseline drift. When a change intentionally shifts behaviour, update the accepted baseline in the same pull request so the review shows it. A baseline silently regenerated from the candidate always passes.
• A moving judge. Pin the LLM evaluation to an exact model and rubric version. A moving model alias can change every score at once, and the first unrelated pull request after the change gets blamed.
• Production data in artifacts. Uploaded results are readable by everyone with repository access. The dataset should already be scrubbed; check before the first upload, not after.
• Retry culture. If people re-run red eval jobs until they pass, the job has become noise. Look at which cases flip on re-runs and fix those cases, the rubric, or the trial count.
• Stale datasets. A smoke set frozen a year ago tests last year's traffic. Add new production failures to the next dataset version, and point the workflow at it in a reviewed change.
• Test the gate itself. Break the agent on purpose in a branch - delete a key instruction - and confirm the job goes red.

Keep Learning from Production After the Gate Passes

The same Failproof AI evaluations can continue running on real sessions after deployment. That removes a common source of confusion: when CI passes but production declines, the likely gap is the frozen dataset or traffic mix, not two unrelated scoring implementations.

After deployment, compare production evaluation results with the CI baseline. When production declines but the offline suite did not, inspect the affected traces and add representative, scrubbed cases to the next dataset version. Keep the new cases labeled and reviewed rather than automatically copying every failed session into the suite.

Failproof AI also analyzes evaluation results and trace evidence across sessions to find recurring failure modes, group them into findings and recommend a fix. Those findings help select regression cases without letting one repeated incident dominate the dataset. Export the chosen sessions, scrub sensitive data and freeze them into the next version used by CI.

When You Do Not Need Evals in CI

If prompts, tools and models change rarely and deployments are manual, a pre-release run of the full suite may be enough. With only a handful of cases, percentage thresholds are too noisy for a broad gate, though named must-pass cases can still block known regressions. And if behavior changes mainly through a model provider you do not control, production evaluation and alerts remain essential because a repository-only trigger cannot see every upstream change.