Why do you need to calibrate an LLM judge?

Why an LLM Judge Needs Calibration

An LLM judge applies written criteria to a session, but it does not know your product as well as the people operating it. A rubric may leave an edge case undefined. The trace may omit the evidence needed for a fair verdict. A correct escalation for one customer may be a failure for another. The model can also change how it interprets the same instruction.

Calibration is the feedback loop that catches those gaps. A person reviews the session and decides whether the evaluation identified a real failure, missed one or described the wrong cause. That decision should improve the evaluation used on future sessions.

Without calibration, false positives create alerts nobody trusts and false negatives leave expensive failures invisible. The problem gets worse as the agent gains tools, new workflows and new kinds of users.

Why the Traditional Workflow Is so Manual

Older evaluation tools treat calibration as a separate data project. A team exports sessions, labels 50 to 200 examples, runs the judge over the same set, calculates agreement and failure precision or recall, reads every disagreement, rewrites the rubric and repeats the test on a held-out set.

1. Find representative sessions. Include normal traffic, known failures and hard edge cases.
2. Label them independently. Reviewers decide PASS or FAIL without seeing the model verdict.
3. Compare the two sets of decisions. Calculate where the judge raises false alarms and where it misses real failures.
4. Rewrite the rubric. Turn every resolved disagreement into another instruction or example.
5. Run the full set again. Repeat after every rubric, model or agent change.

That process can produce a reliable benchmark, but the team has to curate the dataset and keep it representative of production. It also starts with failures people already know to label. New behavior can remain invisible until somebody notices it elsewhere.

1. Let Failproof Audits Find the Patterns First

Failproof AI audits analyze a population of production sessions using trace evidence, code-based and LLM-based evaluation results and prior policy decisions. Instead of asking reviewers to label a random file from scratch, the audit groups related failures into findings with severity, affected sessions, evidence and a recommended fix.

Set an audit goal that reflects the agent's job, such as abandoned tasks after a tool failure or behavior changes after a deployment. An agent contract can supply the stable definition of purpose, expected output, completion and prohibited behavior. Run broad audits as well, because a fixed evaluator can only measure the failures it was written to recognize.

fp audits list
fp audits findings --status open --limit 20

2. Give Feedback on the Found Failure Pattern

Open the affected sessions and check the cited evidence. Confirm the finding when the pattern is real. Dismiss it when the sessions do not support the conclusion. Mute it when the behavior is expected and should not keep returning as a finding. Failproof records that feedback by fingerprint across the organization.

The useful feedback is specific. Explain which evidence makes the behavior acceptable or harmful, what condition separates a pass from a failure, and which edge case the current evaluation misunderstood. This is the product knowledge a generic judge prompt is missing.

You are reviewing a discovered pattern rather than labeling hundreds of unrelated sessions. The human decision stays important, but the work is concentrated on behavior that is already recurring in production.

3. Let the Evaluation Update from That Feedback

Failproof AI uses the accepted and rejected pattern feedback to improve the evaluation that recognizes the behavior. The updated evaluation captures the clarified boundary, including the evidence that distinguishes a true failure from a harmless lookalike.

Keep the update versioned. Test it against the sessions that produced the finding, then re-evaluate or backfill it across historical sessions before enabling it broadly. The previous version remains available if the new one introduces false positives or hides failures the earlier check caught.

This turns calibration into part of the failure-analysis workflow. The evaluation evolves from reviewed production evidence instead of relying on a label spreadsheet that somebody has to refresh by hand.

4. Verify the Updated Evaluation on Real Sessions

Check three groups separately: the sessions that originally supported the finding, similar sessions the reviewer marked as acceptable, and new production sessions after the update. The evaluation should still catch the confirmed failures without absorbing the acceptable cases.

Keep the result linked to its trace evidence and alert when the failure rate changes. The next audit should show whether the same pattern disappeared, narrowed to one cohort or returned in a new form. If the score changes immediately after an evaluator update, treat that as a measurement change until the trace evidence proves the agent itself changed.

fp evals --since 24h --aggregate
fp evals --agent-id checkout-agent --aggregate

5. Fix the Agent, Then Keep Learning

A calibrated evaluation improves measurement. The product fix still belongs in the prompt, tool, model, data or workflow. Assign the finding as an issue, make the change and keep both the evaluation and audit running to verify that the behavior improves.

When a confirmed failure maps to a recognizable high-risk action in a supported harness, Failproof AI can draft a behavioral policy from the issue. Backtest it against historical calls and use observe mode before enforcement. The evaluation remains the measure of whether the agent learned to complete the task safely.

Failproof AI has already built the recurring-failure discovery, trace-backed findings, reviewer feedback loop, evaluation updates, version history, backfill, alerts and verification workflow. Your team contributes product judgment at the point where it matters. Older tools leave the team to assemble and maintain that loop from exports, label files, scripts and dashboards.

When a Separate Human-Labeled Benchmark Still Matters

Keep a frozen external benchmark when an evaluation gates a regulated decision, publishes a formal quality claim or must be compared independently across vendors. That dataset provides an outside measurement of the system. For day-to-day production improvement, feedback on real audit findings keeps the evaluation closer to current agent behavior with much less labeling work.