How to compare two agent versions

1. Run the Two Versions in Separate Environments

Keep the current agent and the candidate isolated so their tool calls, state and side effects cannot influence each other. Point both at equivalent test data and dependencies. If tools can write, use separate sandboxes or test accounts and reset them between runs.

Send the same task set through both environments for an offline comparison. For a production comparison, route a controlled share of traffic to the candidate while the current version remains the baseline. Traffic allocation stays in your application or feature-flag system; Failproof AI observes and evaluates the sessions produced by each route.

2. Identify Every Session by Agent and Environment

Give the two versions distinct agent_id values, such as checkout-agent-current and checkout-agent-candidate. Set session.environment to the environment where each one ran, such as baseline and candidate, or your existing staging and production names. Keep the naming stable for the full comparison.

Do not reuse one identifier and rely on memory about when the deployment changed. Explicit identifiers let evaluations, queries, dashboards, alerts and audits filter the same cohorts without rebuilding the comparison in every view.

3. Run the Same Evaluations on Both Versions

Deploy the same evaluation versions to the current and candidate agents. Use code-based checks for exact outcomes such as required tool calls, tool errors, step counts and budget limits. Use LLM-based evaluations for task completion, instruction following and other criteria that require reading the full trace. Failproof AI can also run your existing evaluation suite in the cloud.

Keep the evaluation version fixed during the comparison. If the rubric or code changes halfway through, a score difference may come from the measuring instrument rather than the agent. Create a new comparison after changing an evaluation.

fp evals --agent-id checkout-agent-current --aggregate
fp evals --agent-id checkout-agent-candidate --aggregate
fp --json evals --aggregate --env candidate

4. Run Matching Audits on Both Cohorts

Evaluation averages show whether known criteria moved. Audits show how the behavior changed. Create audits with the same goal, cadence, sensitivity and window for both cohorts so their findings are comparable.

For example, run the goal "find sessions abandoned after a tool failure" against both agent IDs. Failproof AI analyzes the traces, evaluation results and policy decisions, then groups related failures into findings with severity, affected sessions, evidence and a recommended fix.

Compare the failure modes as well as their counts. A candidate can improve the headline completion score while introducing a smaller but more severe pattern, such as acting on the wrong account or retrying a write after an uncertain response.

5. Analyze Agent Behavior Over Time

Open Analyze → Queries and filter by session.agent_id and session.environment. Compare evaluation results, error counts, tool behavior, latency, token use and cost across the same time window. Use the visual query builder for common questions or read-only SQL when the comparison needs a custom cohort.

Save the queries that answer the rollout questions. Useful views include completion rate by version, failed tool calls by agent, cost per completed session, latency percentiles, policy decisions and the number and severity of audit findings over time.

fp agent ask "Compare checkout-agent-current and checkout-agent-candidate in the last 7 days. Show evaluation results, tool errors, cost and latency by environment."
fp query run <saved-query-id>

6. Build the Comparison Dashboard

Build a dashboard from the saved queries so the current and candidate versions stay visible side by side throughout the test. Keep the same cohort filters and time window on every chart. A dashboard that mixes environments or compares different dates can make traffic changes look like an agent improvement.

• Quality: pass rate for the primary evaluations and the must-not-regress checks.
• Failure intelligence: recurring findings, severity and affected-session count for each version.
• Reliability: tool errors, repeated calls, incomplete sessions and escalations.
• Efficiency: cost, tokens, steps and latency per completed task.
• Runtime behavior: policy hits or denials, when policies are enabled for supported actions.

Use the dashboard for the trend and the trace for the explanation. Open the sessions behind every important regression. The score tells you which cohort changed; session replay shows whether the cause was tool selection, arguments, recovery, context or the final response.

7. Decide, Roll Out and Keep Watching

Write the release rule before reading the final dashboard. Name the primary evaluation, the regressions that block promotion, the acceptable cost and latency change, and the audit findings that require investigation. A higher average score should not outweigh a new high-severity failure mode.

If the candidate passes, increase its production share in stages while keeping the same evaluations, audits, queries and alerts running. If it fails, assign the relevant findings, make the change and rerun the same comparison. Preserve the current version as the rollback until the candidate has remained healthy on real traffic.

Failproof AI has already built the evaluation execution, trace storage, session replay, population-level failure analysis, native querying, dashboards, alerts and issue workflow around this comparison. Your team runs the two agent versions, identifies the sessions correctly and makes the product decision. You do not need to assemble a separate experiment database, scoring pipeline and investigation dashboard.