How to set up LLM judges for AI agents

What a Judge Is, and Why First Judges Go Wrong

An LLM judge is a model call that reads what your agent did and grades it against written criteria. For an agent, "what it did" is a session: the user's request, every tool call and tool result, and the final reply. The judge returns a verdict and a reason, and you aggregate verdicts into a pass rate you can chart and alert on. If the idea is new, what is LLM-as-a-judge covers it from the top.

First judges fail for three boring reasons. The criteria are vague ("was the agent helpful?"), so the judge fills the gap with its own taste and everything scores a seven. The judge sees the wrong input: the final message, not the tool calls where the agent actually went wrong. And nobody checked the judge against a human, so a judge that passes 95% of sessions is indistinguishable from one that has stopped reading. Every step below closes one of those gaps.

Step 1: Decide What You Are Judging

Pick the failure first and the judge second. Agent failures show up in four places, and each wants a differently shaped judge:

Start with the target that costs you most when it fails. For most agents that is outcome, because a session can look busy and healthy while doing nothing useful. Trajectories get their own treatment in how to evaluate agent trajectories. And if a criterion can be settled with an if statement, write the if statement: LLM judge vs code judge explains where each belongs.

Step 2: Write the Rubric

Start with one criterion per result. A judge asked for one blended score across completion, tone and safety can trade those concerns off in ways you cannot inspect. You may return several independent results from one model call to reduce cost, but calibrate each result separately and compare its accuracy with separate calls.

• PASS/FAIL or three points, not 1-10. A binary scale forces whoever writes the rubric to decide where the line is. A ten-point scale lets the judge avoid deciding, and the scores bunch in the middle.
• Observable evidence. "A tool result confirms the refund" is checkable in a transcript; "was thorough" is not.
• A rule for missing evidence. Tell the judge what to answer when the transcript cannot settle the question. Left alone, it guesses, and usually guesses generously.
• A real pass and a real fail, pasted from your own sessions once you have them.

Criterion: task completion.

PASS if the session did what the user asked in their first message, and the
transcript shows evidence of it: a tool result, a file written, a confirmed reply.

FAIL if the agent stopped early, claimed success with no evidence in the
transcript, or completed a different task from the one requested.

If the transcript does not contain enough evidence to decide, answer FAIL and
say what evidence is missing.

Scales, anchors and a copyable template are in how to write a rubric for an LLM judge; ready-made prompts for six common criteria are in judge prompt templates.

Step 3: Choose the Model and Build the Judge

The judge model is configuration, not code. Read it from the environment, pin a specific version, and re-check the evaluation whenever it changes; choosing a judge model covers the trade-offs. The example below takes a session as a list of event dicts and returns a verdict with concise evidence a reviewer can verify. Do not ask for private chain-of-thought; ask for quotations or event references that support the score.

# judge.py - one criterion, one judge, JSON out
import json
import os
import re
import sys

import anthropic

JUDGE_MODEL = os.environ["JUDGE_MODEL"]

RUBRIC = open("rubrics/task_completion.txt").read()

PROMPT = """You are grading one AI agent session against one criterion.

{rubric}

The session transcript, one JSON event per line, oldest first:
<transcript>
{transcript}
</transcript>

Reply with JSON only, no text around it, in this shape:
{{"reasoning": "two or three sentences that cite specific events", "verdict": "PASS"}}
The verdict is exactly PASS or FAIL."""

client = anthropic.Anthropic()                    # reads ANTHROPIC_API_KEY


def render(events, limit_chars=60_000):
    text = "\n".join(json.dumps(e, default=str) for e in events)
    if len(text) > limit_chars:
        # keep the start (the request) and the end (the outcome)
        half = limit_chars // 2
        text = text[:half] + "\n[... middle of session truncated ...]\n" + text[-half:]
    return text


def parse(text):
    match = re.search(r"\{.*\}", text, re.DOTALL)
    if not match:
        return None
    try:
        data = json.loads(match.group(0))
    except json.JSONDecodeError:
        return None
    if data.get("verdict") not in ("PASS", "FAIL"):
        return None
    return data


def judge(events):
    prompt = PROMPT.format(rubric=RUBRIC, transcript=render(events))
    msg = client.messages.create(
        model=JUDGE_MODEL, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = msg.content[0].text
    data = parse(text)
    if data is None:
        # a judge failure, not an agent failure: never count it as PASS or FAIL
        return {"verdict": "ERROR", "reasoning": f"unparsable judge reply: {text[:200]!r}"}
    return data


if __name__ == "__main__":
    with open(sys.argv[1]) as f:                  # one session, one JSON event per line
        events = [json.loads(line) for line in f if line.strip()]
    print(json.dumps(judge(events), indent=2))

Three details do real work. The transcript is serialized whole rather than picked apart by field name, because the payload shape is whatever your framework emits. Long transcripts keep the start and the end, where the request and the outcome live; if your failures happen in the middle, summarize instead of truncating. And when the reply is not valid JSON, the function returns ERROR rather than guessing - an unparsable reply is a judge failure, and it must not land in either column. If your judge runs on OpenAI, swap in OpenAI() and read resp.choices[0].message.content; nothing else changes.

Step 4: Calibrate on 50-100 Labeled Sessions

Before the judge's numbers mean anything, find out whether it agrees with you. Pull 50 to 100 real sessions, deliberately including failures you already know about, and label each one PASS or FAIL against the same rubric without looking at the judge's answer. Run the judge on the same sessions, write one line per session, and compare.

# agreement.py - does the judge agree with you on the sessions you labeled?
# labels.jsonl lines look like: {"session": "s-104", "human": "FAIL", "judge": "PASS"}
import json
import sys

rows = [json.loads(line) for line in open(sys.argv[1]) if line.strip()]
errors = sum(r["judge"] == "ERROR" for r in rows)
rows = [r for r in rows if r["judge"] in ("PASS", "FAIL")]

agree = sum(r["human"] == r["judge"] for r in rows) / len(rows)
tp = sum(r["human"] == "FAIL" and r["judge"] == "FAIL" for r in rows)
fp = sum(r["human"] == "PASS" and r["judge"] == "FAIL" for r in rows)
fn = sum(r["human"] == "FAIL" and r["judge"] == "PASS" for r in rows)
precision = tp / (tp + fp) if tp + fp else float("nan")
recall = tp / (tp + fn) if tp + fn else float("nan")

print(f"n={len(rows)} judge_errors={errors} agreement={agree:.2f}")
print(f"FAIL precision={precision:.2f} FAIL recall={recall:.2f}")
for r in rows:
    if r["human"] != r["judge"]:
        print("disagree:", r["session"], "human", r["human"], "judge", r["judge"])

Look past raw agreement. If 90% of sessions pass, a judge that always says PASS scores 90% agreement and catches nothing. Precision and recall on the FAIL class answer the questions you care about: when the judge flags a session, is it right, and of the sessions that really failed, how many did it catch.

Then read every disagreement. Most come from the rubric, not the model: a line you thought was clear that the judge read differently, or a case the rubric never covered. Fix the rubric, re-run, repeat. Expect your own criteria to move while you label; Shankar et al. call this criteria drift - grading outputs is how you find out what your criteria were. Cohen's kappa and the rest of the method are in calibrating an LLM judge against human labels.

Steps 5-7: Run It, Alert on It, Act on It

Run code checks on every session, because they cost nothing, and the judge on every session you can afford. Scoring everything gives per-agent and per-environment rates without sampling error; when volume or cost rules that out, sample randomly for a baseline and oversample the agents and paths that fail most (sampling production traffic for evals, reducing judge cost). Keep judging off the agent's hot path. A judge that runs after the session ends can take thirty seconds without slowing anyone down.

Alert on the rate, not on single verdicts. One FAIL is a data point; the production pass rate for one agent falling well below its usual level over a day is an incident. Alerting on eval score drops covers thresholds and trends.

The response depends on the pattern. A prompt or tool-description problem needs a change and a rerun on the failed sessions. A pattern spread across many sessions needs population-level analysis to find the shared cause. Once a high-risk action is understood and the fix is tested, it can become a behavioral policy that steers or blocks the same failure on a later run. See turning judge findings into runtime policies.

Pitfalls, and How to Check Your Work

• Judging the final message only. Agents fail in the middle. A judge that never sees the tool calls is grading the agent's own account of what it did.
• Counting parse failures as scores. Track the ERROR rate on its own. When it climbs, transcripts got longer or the judge model changed.
• Silent judge changes. A new judge model or a reworded rubric changes what the number means. Re-run the labeled set first, and treat the result as a new series.
• Bias nobody measured. Judges tend to favour longer answers and their own model family, and to lean lenient. LLM judge bias shows how to test for each.
• A judge doing arithmetic. "Was the refund above the order total" is a comparison of two numbers. A judge will occasionally get it wrong; code will not.

Running This Workflow in Failproof AI

Failproof AI runs code-based and LLM-based evaluations in the cloud and can use an existing evaluation suite. Results stay connected to the session and tool calls they scored, so disagreements can be investigated against the original evidence.

1. Create the evaluation. Use a deterministic check for facts available in the event stream and an LLM judge for behavior that needs interpretation.
2. Test and version it. Run the evaluation against labeled sessions before enabling it, and recalibrate whenever the rubric or judge model changes.
3. Monitor production behavior. Alert on meaningful score changes and use automated audits to find related failures across sessions, including patterns outside the evaluator you wrote.
4. Validate the fix. Rerun failed sessions after changing the prompt, tool or agent logic. For repeatable high-risk actions, test a behavioral policy before enabling it.

The point is to connect the judge to a fixing loop. Scores identify known problems. Failure analysis groups the evidence and helps explain recurrence. Alerts establish ownership, and a tested change or policy closes the finding.

When You Do Not Need Any of This

If every criterion you care about is checkable in code - the tests passed, the file exists, no tool returned an error - you do not need a judge yet. Write the checks. They are free and exact, and they do not drift when a model is updated.

If the agent runs a handful of sessions a week, read them. The 50 sessions you would label for calibration are already the evaluation, and a spreadsheet holds them fine.

If you want a large catalog of ready-made judges, compare products built around that model. Judgment Labs focuses on rubric-based judges for long trajectories, while Galileo lists more than 20 built-in evaluations and offers Luna-2 evaluation models on Enterprise. Failproof AI combines code and LLM evaluations with automated failure analysis, findings, alerts and a workflow for validating fixes.