Evaluate an agent after a model upgrade

the short answer

Run the current and upgraded models in separate environments with distinct agent IDs. Send complete sessions from both to Failproof AI, apply identical code-based and LLM-based evaluations, and run audits with the same goals and windows. Compare completion, behavior, cost, latency and failure findings through saved queries and dashboards. Inspect the affected traces, then promote the new model gradually while alerts and audits continue watching for regressions.

Separate: Give current and candidate versions distinct agent IDs and environments.
Keep fixed: Use the same tasks, evaluation versions and audit goals.
Compare: Quality, behavior, cost, latency and recurring failure modes.
Roll out: Increase traffic gradually while the same monitoring remains active.

A Model Upgrade Changes More than Answer Quality

A newer model can improve benchmark quality and still make an agent worse. It may choose different tools, call them in a different order, retry more often, ignore a stopping condition, become slower or use enough additional tokens to change the economics of the task.

Evaluate the complete agent behavior, not only a sample of final answers. The decision should answer whether the candidate completes more real tasks, introduces new failure modes and stays within the product’s cost and latency limits.

1. Run the Models as Separate Cohorts

Keep the current model as the baseline and run the candidate in a separate environment. Give each version a distinct agent_id and session.environment value so evaluations, queries, dashboards and audits can filter them reliably.

Cohort	Agent ID	Environment
Current model	`support-agent-current`	`baseline`
Upgraded model	`support-agent-candidate`	`candidate`

Start with the same offline tasks or replayed production cases. If the candidate proceeds to live traffic, route only a controlled share through it and preserve the cohort identifiers throughout the test.

2. Apply the Same Evaluations to Both Models

Use the same evaluation versions for both cohorts. Code-based evaluations should cover exact requirements such as required tool calls, tool errors, retries and budgets. LLM-based evaluations should cover task completion, instruction following and criteria that require the full trace.

Do not edit a rubric midway through the comparison. If the measuring instrument changes with the model, you cannot tell which change produced the score difference.

fp evals --agent-id support-agent-current --aggregate
fp evals --agent-id support-agent-candidate --aggregate
fp --json evals --aggregate --env candidate

3. Look for New Failure Modes

Evaluation scores tell you whether known criteria changed. Failproof audits show how behavior changed beyond those criteria. Run matching audits against both cohorts with the same goal, cadence, sensitivity and time window.

Compare the findings, not only their count. A candidate may improve completion while introducing a smaller but more severe pattern, such as using the wrong account, retrying a write after an uncertain response or skipping a verification step.

fp audits findings --status open --limit 20

Open the affected traces and confirm the pattern before deciding on a fix. Each finding includes analysis, severity, affected sessions, evidence and a recommendation.

4. Compare Behavior, Cost and Latency

Use Analyze → Queries to compare the cohorts by task completion, evaluation pass rate, tool errors, retries, cost per successful task, p50 latency and p95 latency. Save the useful queries as one rollout dashboard.

fp agent ask "Compare the current and candidate support agents on completion, cost, p95 latency and recurring failures"
fp query run <query-id>

Read quality and efficiency together. A cheaper candidate is not better if it completes fewer tasks, and a higher-scoring candidate may still be unsuitable if its slow tail violates the product experience.

5. Make a Staged Rollout Decision

Promote the candidate only when it meets the task-level quality threshold, introduces no unacceptable finding and stays within cost and latency budgets. Keep the decision criteria explicit instead of relying on one blended score.

Increase traffic in stages and keep the same evaluations, audits, dashboard and alerts active. Production traffic may reveal task types and tool conditions absent from the offline set. Pause or roll back when a severe finding appears or a sustained metric crosses its limit.

After full rollout, retain the previous cohort long enough to show that the improvement persists. Then use the same workflow for the next prompt, tool or model change.

FAQ

What should I test after upgrading an agent model?

Test task completion, instruction following, tool selection, tool errors, retries, cost per successful task and p50 and p95 latency. Also compare audit findings to catch new behavior outside the predefined checks.

Can I compare two agent models in Failproof AI?

Yes. Identify the versions with distinct agent IDs and environments, apply the same evaluations and audit goals, and compare them with native queries, saved dashboards and trace evidence.

How much traffic should a new model receive?

Start with offline or replayed sessions, then use a small controlled production cohort. Increase traffic only after quality, behavior, cost and latency remain acceptable, with alerts and audits active throughout.

Get Started

Failproof AI is free to start. It finds recurring failure modes across agent sessions using code-based and LLM-based evaluations, groups the evidence into findings, and recommends fixes. Bring the eval suite you already have, alert the right owner when behavior drifts, and turn a tested fix into a policy that prevents the failure from recurring. See pricing for the tiers.

start free get a demo →

Sources

Products change; if a detail here is out of date, tell us at support@befailproof.ai.