Human review for agent evals

What Human Review Is For

An LLM judge produces a number on every session, and nothing in that number tells you whether it is right. Human labels are the reference that answers that question. They are also the slowest, most expensive and least consistent part of an eval program, which is why the goal is not "humans review everything" but "humans review the few sessions that make everything else trustworthy".

That gives human review four jobs, in rough order of value:

1. Calibration. Before anyone relies on a judge's scores, label a set of sessions and measure how often the judge agrees.
2. Drift detection. After it goes live, label a small random sample on a regular cadence. Score behavior can shift when the judge model or rubric changes, and the production distribution changes when agents, tools or traffic change.
3. Disagreement triage. When a code check and a judge disagree on the same session, one of them is wrong. A person decides which, and the answer usually improves the rubric.
4. Discovery. Reading sessions with no score in mind is still the best way to find failure modes nobody wrote a check for.

How Many Sessions to Label

The honest answer is "enough to make the interval narrow enough for your decision", and it is worth seeing the arithmetic once. If a judge agrees with humans on 45 of 50 sessions, the observed agreement is 90 percent, but the 95 percent confidence interval runs from about 79 to 96 percent. At 180 of 200 it narrows to about 85 to 93 percent.

import math


def wilson(successes, n, z=1.96):
    """95% Wilson score interval for a proportion."""
    p = successes / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return round(centre - half, 3), round(centre + half, 3)


print(wilson(45, 50))    # (0.786, 0.957)
print(wilson(180, 200))  # (0.851, 0.934)

Fifty labels tell you whether a judge is roughly usable. They do not tell you whether it is 85 or 95 percent right. For most teams starting out, 50 to 100 is the right first batch, with more added over time as the drift sample accumulates.

Running the Review Queue

A review queue can be a spreadsheet. What matters is what the reviewer sees and what they are asked.

• Show the task, the transcript and the outcome. Not the judge's score or reasoning. A reviewer who sees "judge: pass" agrees with it more often, and the labels stop being independent.
• One criterion per question, binary, with "can't tell". "Did the agent complete the task? yes / no / can't tell" labels faster and more consistently than a 1-5 scale. Count the can't-tells; a lot of them means the transcript is missing information, or the criterion is vague.
• Ask for a one-line reason on every "no". The reasons are how you find out what reviewers mean, and they become examples in the judge's rubric.
• Time-box it. Twenty sessions in forty minutes, once a week, sustains. A two-day labeling sprint happens once.
• Use people who know the domain. A support lead labels support conversations; an engineer who knows the repository labels coding sessions. Their disagreement with you is information.

Store every label with the session ID, the criterion, the reviewer, the date and the rubric version. The rubric version is the field people forget and then need, because a label is only comparable with judge verdicts produced under the same definition.

Once the judge is live, the weekly batch changes shape. A mix that works for twenty sessions: ten drawn at random, which keep the agreement estimate honest; five the judge failed, which check its precision; and five where the judge and a code check disagreed, which are the cheapest way to find rubric gaps. Keep the random ten separate when you compute agreement - the other ten were chosen because something looked wrong, and folding them in makes the judge look worse than it is on ordinary traffic.

Inter-Rater Agreement, Then Judge Agreement

Before asking whether the judge agrees with humans, check whether the humans agree with each other. Put two reviewers on an overlapping 20 to 30 percent of the batch and compute Cohen's kappa, which corrects raw agreement for the agreement you would get by chance. If two people who know the domain cannot agree, the rubric is ambiguous, and no judge will do better than they do.

from collections import Counter


def cohen_kappa(a, b):
    """a, b: labels from two raters for the same sessions, in the same order."""
    if len(a) != len(b) or not a:
        return None
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    expected = sum(ca[k] * cb[k] for k in set(a) | set(b)) / (n * n)
    return None if expected == 1 else (observed - expected) / (1 - expected)


def judge_vs_humans(human, judge, fail="fail"):
    pairs = [(h, j) for h, j in zip(human, judge) if h != "cant_tell"]
    if not pairs:
        return {"agreement": None, "kappa": None, "fail_precision": None, "fail_recall": None}
    tp = sum(h == fail and j == fail for h, j in pairs)
    fp = sum(h != fail and j == fail for h, j in pairs)
    fn = sum(h == fail and j != fail for h, j in pairs)
    return {
        "agreement": sum(h == j for h, j in pairs) / len(pairs),
        "kappa": cohen_kappa([h for h, _ in pairs], [j for _, j in pairs]),
        "fail_precision": tp / (tp + fp) if tp + fp else None,
        "fail_recall": tp / (tp + fn) if tp + fn else None,
    }

Read kappa as a diagnostic rather than a universal pass mark. Values near zero mean the raters agree little beyond what their label frequencies would predict; higher values indicate stronger agreement, but the acceptable level depends on the decision and the class balance. For the judge, inspect failure precision and recall separately. High precision with low recall misses failures; high recall with low precision creates review and alert fatigue.

Feeding Labels Back into the Judge

1. Read every disagreement

   Sort the sessions where the judge and the humans differ, and read the reviewer's reason next to the judge's. Most disagreements fall into two or three patterns: a criterion the rubric never mentions, an edge case with no example, or a judge that is too lenient on polite text.

2. Change the rubric, bump the version

   Add the missing clause or a pass and fail example drawn from a real disagreement. Bump the rubric version so old and new verdicts are never mixed in one chart.

3. Re-run on the same labels

   Run the new rubric on the full labeled set, not only the disagreements, and compare fail recall and kappa before and after. A fix for one pattern regularly breaks another.

4. Freeze the set as a regression test

   The labeled sessions become the judge's own test suite. Re-run them whenever the judge model, the rubric or the prompt changes, and relabel a fresh sample after any agent change big enough to change behavior.

The calibration guide goes deeper on the iteration loop, and debugging a failing judge covers what to do when the numbers will not move.

Using Human Review with Failproof AI

Failproof AI does not currently include a dedicated human-annotation queue. Latitude includes human annotation and an eval-to-human alignment check, while Galileo offers a workflow for adapting judge metrics from annotated records. Those may be better fits when managed labeling inside the platform is the primary requirement.

Failproof AI provides the session traces, code-based and LLM-based evaluation results, and exports needed for an external review workflow. Keep the rubric version with each human label, and create a new evaluation version when the rubric or judge changes so old and new measurements are not mixed.

Use a random sample to estimate judge quality, then add targeted cases from low scores, evaluator disagreements and automated failure findings. Failproof AI groups recurring failures with their affected sessions and evidence, which gives reviewers a higher-value queue than simply sorting every trace by score. Keep the random and targeted samples separate in reported agreement so selection bias does not distort the result.

When You Do Not Need Any of This

Deterministic checks do not need judge-to-human calibration, but they still need unit tests and spot checks against real sessions to confirm the implementation matches the intended rule. If every score is deterministic, human review is mainly for discovering failures no check covers yet. The full labeling and agreement workflow becomes important when people will act on an LLM judge without reading every session.