How to evaluate agents without ground truth

the short answer

Define an agent contract describing the task, required behavior and unacceptable outcomes. Send complete sessions to Failproof AI, then use code-based evaluations for exact trace evidence and LLM-based evaluations for criteria such as task completion, instruction following and groundedness. Run audits to discover recurring failures outside those checks, review the supporting sessions, give feedback on findings and update the evaluations. Track the accepted criteria with queries, dashboards and alerts to verify improvement in production.

Reference: Use the agent contract, instructions, tool evidence and user outcome instead of one expected answer.
Known criteria: Code-based and LLM-based evaluations score each production session.
Unknown failures: Audits find recurring patterns that the initial criteria missed.
Learning loop: Reviewer feedback improves findings and evaluations over time.

No Ground Truth Does Not Mean No Evidence

Production agents rarely have one exact correct answer. A research agent can produce several useful reports, a support agent can resolve the same issue in different ways and a coding agent can implement more than one valid fix. Comparing each response with a reference string would reject good work and still miss bad behavior.

The session still contains evidence: what the user asked, what the agent was allowed to do, which tools it called, what those tools returned, whether errors occurred and what happened next. Reference-free evaluation asks whether the agent behaved correctly given that evidence.

1. Write the Agent Contract

Start with a short contract that states the agent’s job, required evidence, boundaries and definition of completion. This becomes the shared reference for evaluations and audits even when no expected answer exists.

Contract part	Example question
Purpose	What outcome is this agent responsible for?
Required behavior	Which tools or verification steps must it use?
Evidence	What must support a claim of success?
Boundaries	Which actions, data or decisions are outside its authority?
Failure handling	When should it retry, stop or escalate?

Keep the contract specific enough that a reviewer can point to evidence in a trace. “Be helpful” is not testable. “Use the account tool before stating a balance, and escalate when identity cannot be verified” is.

2. Capture the Complete Session

Send prompts, model turns, tool calls, tool results, errors and the final response to Failproof AI in one session trace. Set stable agent and environment identifiers so production behavior can be separated from staging and compared across versions.

The trace is the evidence base. A final answer may claim that a refund was issued or a test passed; the tool result shows whether that actually happened. Without the trajectory, a reference-free evaluation is forced to judge confidence and writing style instead of behavior.

3. Evaluate Known Requirements

Use code-based evaluations for exact facts and LLM-based evaluations for contextual criteria. Keep the dimensions separate so a strong tone score cannot hide an unsupported action or incomplete task.

Question	Evaluation type
Did the required tool run successfully?	Code-based
Did the agent exceed its retry or step budget?	Code-based
Are claims supported by tool results in the trace?	LLM-based
Did the agent complete the requested task?	LLM-based
Did it follow the contract and escalate when required?	LLM-based, with exact trace checks where possible

Results appear beside the session under Observe → evaluations. Test a new evaluation against known sessions, re-evaluate individual traces while refining it and backfill recent sessions once the version is ready.

fp evals --since 24h --aggregate
fp evals --agent-id support-agent --aggregate
fp --json evals --aggregate --env production

4. Discover What the Initial Evaluations Missed

A team cannot write a check for every production failure in advance. Run Failproof audits across sessions with a goal based on the agent contract, such as “find recurring ways the support agent fails to resolve or safely escalate requests.”

Audits analyze traces, evaluation results and related evidence, then group similar failures into findings with severity, affected sessions and a recommended fix. This surfaces patterns outside the original checklist, such as one tool result being routinely misunderstood or a class of requests that sends the agent into repeated retries.

fp audits list
fp audits findings --status open --limit 20

5. Use Reviewer Feedback to Improve the System

Review the traces behind each finding and mark whether the pattern is real, overstated or missing important context. That feedback improves the interpretation of future sessions and shows where an existing evaluation needs a clearer condition or rubric.

When reviewers confirm a new recurring failure, add or update an evaluation so it can be measured continuously. This is more useful than maintaining a static labeled dataset as the only source of truth: production findings keep the evaluation set aligned with how the agent is actually failing now.

Human review remains important, but it is focused on validating discovered patterns and difficult edge cases rather than manually labeling a random stream of sessions forever.

6. Query, Alert and Verify

Use Analyze → Queries to compare contract adherence, task completion and failure patterns by agent, environment and time period. Save the useful queries as dashboards so the team can see whether a fix holds.

fp agent ask "Which production sessions failed the agent contract most often this week, and what patterns do they share?"
fp query run <query-id>

Create alerts for evaluation-score drops, compound conditions, metrics or custom SQL and route them to email, Slack, a webhook or the dashboard. Assign the finding, fix the prompt, model, tool, data or workflow, then keep the same evaluations and audit running to verify recovery.

Use a policy only when the evidence identifies a specific high-risk action that must not occur. Backtest and observe it before enforcement; most reference-free failures require improving the agent rather than blocking it.

FAQ

Can you evaluate an AI agent without a correct answer?

Yes. Evaluate whether it followed its contract, used required tools, supported claims with trace evidence, completed the task and handled uncertainty or escalation correctly. None of those requires one reference answer.

What is reference-free agent evaluation?

Reference-free evaluation scores an agent against criteria and evidence available in the session rather than comparing its response with a predefined answer. It is useful when several outputs can be valid.

How does Failproof AI find failures without ground truth?

Code-based and LLM-based evaluations measure known requirements, while audits analyze sessions together to discover recurring patterns outside those checks. Reviewers validate findings and use the evidence to improve evaluations and the agent.

Do I still need human review?

Yes, for validating findings, difficult edge cases and important evaluation changes. The goal is to focus human judgment where it improves the system, not require people to label every production session.

Get Started

Failproof AI is free to start. It finds recurring failure modes across agent sessions using code-based and LLM-based evaluations, groups the evidence into findings, and recommends fixes. Bring the eval suite you already have, alert the right owner when behavior drifts, and turn a tested fix into a policy that prevents the failure from recurring. See pricing for the tiers.

start free get a demo →

Sources

Products change; if a detail here is out of date, tell us at support@befailproof.ai.