LLM-as-a-judge bias

What Judge Bias Is, and Why Averages Do Not Fix It

A judge can make both inconsistent and systematic errors. Run-to-run variation may move verdicts in either direction; bias pushes results in a recurring direction associated with position, length, model family or another feature. More samples can narrow the uncertainty around a biased estimate without making the estimate more accurate, which is why aggregate agreement alone is not enough.

Several studies have documented these effects under different evaluation settings. Zheng et al. (2023) reported position, verbosity and self-enhancement biases in LLM judges. Ye et al. (2024) catalogued 12 potential biases with an automated framework called CALM. The size of each effect depends on the model, prompt and data, so this page focuses on four you can test against your own labels: position, verbosity, self-preference and leniency.

Agent evals make these worse in specific ways. Sessions vary enormously in length, so a length preference has a lot to act on. The agent and the judge often come from the same provider. And a lenient judge on a production dashboard hides exactly the regression you set it up to catch.

Position Bias

What it is. In a pairwise comparison, the judge favours whichever candidate appears in one particular slot. Wang et al. (2023) showed that the ranking of candidate responses could be changed simply by swapping their order in the prompt.

How to measure it. Run every pair twice with the order swapped and count how often the two verdicts agree. A judge with no position bias agrees with itself on every pair except genuine ties. Also count which slot wins when the verdicts disagree; a heavy skew to one slot is the bias itself.

How to reduce it. Always judge both orders and treat a pair whose verdict flips as a tie, the approach Wang et al. call Balanced Position Calibration. If the flip rate stays high, the criterion is too vague for a pairwise call; a pointwise rubric is often the better tool. The code is on pairwise vs pointwise LLM judges. Pointwise judges have no candidate order to exploit, but the same habit applies to anything you list in the prompt, such as few-shot examples: rotate them and check the verdicts hold.

Verbosity Bias

What it is. Judges prefer longer outputs, independent of quality. Zheng et al. named it verbosity bias; Dubois et al. (2024) found an automatic evaluator that correlated well with human preferences still favoured models that generate longer outputs, and built a length-controlled version of AlpacaEval that uses regression to ask what the preference would be if both outputs were the same length.

How to measure it on agents. Sort your labeled sessions by length and compare the judge's pass rate with yours in each length band. If the judge is about as strict as you on short sessions and more lenient than you on long ones, length is buying passes. The script below does this.

How to reduce it. Say in the rubric that length and effort are not evidence, and make every PASS condition something a reader can point at. Hand the judge the facts code extracted - the refund succeeded, the tests ran - rather than asking it to be impressed by a long transcript. In pairwise judging, compare pairs of similar length, or adjust for length the way Dubois et al. do.

Self-Preference

What it is. Judges score output from their own model family higher than other output that humans rate as equal. Panickssery et al. (2024) found LLM evaluators recognise their own generations with non-trivial accuracy, and that the stronger the self-recognition, the stronger the self-preference. The G-Eval authors (Liu et al., 2023) flagged a related concern: LLM-based evaluators may be biased toward LLM-generated text in general.

How to measure it. If your agents run on more than one model family, split the labeled sessions by whether the agent shares the judge's family and compare the judge-minus-human gap in each group. If all agents share the judge's family, run a second judge from a different family on the same labels and compare their gaps.

How to reduce it. Test a judge from a different family than the agent it grades rather than assuming it will be neutral. A panel can also help: Verga et al. (2024) found that a panel of smaller models from disjoint families showed less intra-model bias than a single large judge in their experiments. Validate either approach on your labels; more on the choice in choosing a judge model.

Leniency

What it is. Judges pass things a careful human would fail. Thakur et al. (2024) found a tendency toward leniency among LLM judges, along with sensitivity to the complexity and length of the prompt, and noted that high percent agreement can hide very different scores.

How to measure it. Compare the judge's overall pass rate with yours on the same labeled sessions, and look at recall on the FAIL class: of the sessions you failed, how many did the judge fail? Leniency shows up as a judge pass rate above yours and a FAIL recall well below 1.

How to reduce it. Use PASS/FAIL rather than an unanchored long scale, where leniency can show up as scores creeping upward. Define how to handle missing evidence instead of letting the judge resolve doubt generously. Require concise citations to specific events, and include a subtle real failure among the rubric examples. Writing a rubric for an LLM judge covers these choices.

A Bias Report for a Pointwise Judge

All of the pointwise checks run off one labeled file: your verdict, the judge's verdict, the session length, whether the agent shares the judge's model family, and a second judge run on the same session. The last field measures run-to-run consistency, which is not a bias but sits underneath all of them.

# bias_report.py - leniency, length, family and consistency checks; no dependencies
# labels.jsonl lines: {"session": "s-1", "human": "FAIL", "judge": "PASS",
#                      "chars": 18231, "same_family": true, "judge_rerun": "PASS"}
import json
import sys
from statistics import mean

rows = [json.loads(line) for line in open(sys.argv[1]) if line.strip()]
rows = [r for r in rows if r["judge"] in ("PASS", "FAIL") and r["human"] in ("PASS", "FAIL")]


def pass_rate(rs, key):
    return mean(r[key] == "PASS" for r in rs) if rs else float("nan")


def gap(rs):
    # positive: the judge passes more than you do on these sessions
    return pass_rate(rs, "judge") - pass_rate(rs, "human")


fails = [r for r in rows if r["human"] == "FAIL"]
recall = mean(r["judge"] == "FAIL" for r in fails) if fails else float("nan")
print(f"leniency: judge-minus-human pass rate {gap(rows):+.2f}, FAIL recall {recall:.2f}, n={len(rows)}")

rows.sort(key=lambda r: r["chars"])
band = max(1, len(rows) // 4)
for i in range(0, len(rows), band):
    b = rows[i:i + band]
    print(f"length {b[0]['chars']:>7} to {b[-1]['chars']:>7} chars: gap {gap(b):+.2f} (n={len(b)})")

for same in (True, False):
    group = [r for r in rows if r.get("same_family") is same]
    if group:
        print(f"agent shares judge family = {same}: gap {gap(group):+.2f} (n={len(group)})")

reruns = [r for r in rows if r.get("judge_rerun") in ("PASS", "FAIL")]
if reruns:
    print(f"consistency: same verdict on rerun {mean(r['judge'] == r['judge_rerun'] for r in reruns):.2f}")

Read the bands and groups as a pattern, not as precise numbers. With 50 to 100 labels, each band has a dozen or two sessions, so a difference of a few points is noise. A gap that grows steadily from the shortest band to the longest, or a clearly larger gap for same-family sessions, is the signal. When you see one, change one thing - the rubric line, the input format, the judge model - and run the report again.

Measuring Judge Bias with Failproof AI

Failproof AI runs your LLM-based and code-based evaluations and keeps their results linked to each session trace. You can also bring an existing evaluation suite. Bias testing still depends on reference labels and controlled comparisons: the platform should make those comparisons inspectable, not replace them with an unexplained correction.

• Compare judge families. Run the same criterion with judges from different model families and compare each one with the same human labels. A gap that appears only for agents from one family is evidence worth investigating.
• Slice by relevant attributes. Attach labels for length bands, agent model or other suspected factors, then compare judge-human disagreement across those groups. Inspect the cited trace evidence when a gap appears.

Version any change to the rubric, prompt or judge model and re-run it on the frozen label set before comparing the new score series with the old one. Once deployed, Failproof AI can alert on score changes and analyze recurring failed evaluations across sessions, but the bias report remains the check that tells you whether the measurement itself has shifted.

When Bias Matters Less

Bias is a problem for numbers people trust without reading. If the judge only pre-sorts sessions for a human who reads every flagged one, a lenient judge costs you missed flags, not wrong conclusions; measure recall and move on. If your checks are code - error counts, budgets, schemas - none of this applies. And a judge with a steady, known bias still shows a real drop on a trend chart; bias hurts most when you compare across judges, rubrics or agents of different lengths and families.