pairwise vs pointwise llm judges

Side by Side

The two approaches answer different questions. Pointwise asks "is this good enough?" and pairwise asks "which of these is better?". Most teams need the first question answered every day and the second whenever they change the agent.

What Each One Is

Pointwise judging shows the judge one output - one agent session - with a rubric and asks for a verdict: PASS or FAIL, or a point on a short anchored scale. Every session is judged independently, so scores can be aggregated into a rate per agent, per environment, per week. That rate is what dashboards chart and alerts watch.

Pairwise judging shows the judge two outputs produced for the same input and asks which one better meets the criterion, with "tie" allowed. It never says whether either output was acceptable; two bad sessions still produce a winner. What it gains is sensitivity. Deciding which of two answers is better is often easier to do consistently than placing one answer on an absolute scale, so pairwise picks up differences a PASS/FAIL rubric rounds away. Both modes can also take a reference answer, which helps when a correct answer exists.

Position Bias, and Why You Always Swap

Pairwise judges are sensitive to which output comes first. Wang et al. (2023) showed that the ranking of candidate responses could be changed simply by altering their order in the prompt, and proposed Balanced Position Calibration: evaluate both orders and combine the results. That is the minimum for any pairwise judge you rely on.

# pairwise.py - judge A vs B in both orders; if the order changes the answer, it is a tie
import json
import os
import re

from openai import OpenAI

JUDGE_MODEL = os.environ["JUDGE_MODEL"]
client = OpenAI()                                 # reads OPENAI_API_KEY

PROMPT = """Two AI agent sessions attempted the same task. Decide which did it better.

Criterion: {criterion}

Judge only against the criterion. Length and position are not evidence of quality.

<task>{task}</task>
<session_1>{first}</session_1>
<session_2>{second}</session_2>

Reply with JSON only: {{"reasoning": "...", "winner": "1"}}
winner is exactly "1", "2" or "tie"."""


def ask(criterion, task, first, second):
    prompt = PROMPT.format(criterion=criterion, task=task, first=first, second=second)
    resp = client.chat.completions.create(
        model=JUDGE_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    text = resp.choices[0].message.content or ""
    match = re.search(r"\{.*\}", text, re.DOTALL)
    try:
        data = json.loads(match.group(0)) if match else {}
    except json.JSONDecodeError:
        data = {}
    winner = data.get("winner") if isinstance(data, dict) else None
    return winner if winner in ("1", "2", "tie") else None


def compare(criterion, task, a, b):
    """Returns "A", "B", "tie", or None if either judge call failed."""
    a_first = ask(criterion, task, a, b)
    b_first = ask(criterion, task, b, a)
    if a_first is None or b_first is None:
        return None
    one = {"1": "A", "2": "B", "tie": "tie"}[a_first]
    two = {"1": "B", "2": "A", "tie": "tie"}[b_first]
    return one if one == two else "tie"

Track the consistency rate - the share of pairs where both orders agree - as a health number for the judge. If the judge changes its mind whenever the order flips on a large share of pairs, it is not detecting a real difference between the versions, and the headline win rate is mostly noise. A sharper criterion, a stronger judge model or a pointwise rubric usually does better.

Length is the other bias to control for in pairwise judging. Judges tend to prefer the longer of two answers - Dubois et al. (2024) built a length-controlled variant of AlpacaEval to correct for it - so when one version of your agent is simply wordier, check whether its wins survive when you compare pairs of similar length. LLM judge bias covers the other biases.

Pairwise for Agent Version A/B Tests

Pairwise judging earns its keep when you change the agent - a new prompt, a new model under it, a different tool set - and want to know whether the change is better. The method:

1. Freeze a set of inputs. 50 to 200 real requests from production, covering the agents and paths you care about, with personal data removed.
2. Run both versions on every input. Agent runs are not deterministic, so for important decisions run each version more than once per input and compare matched runs.
3. Judge every pair in both orders with the code above, on one criterion at a time.
4. Count wins, losses and ties, then test whether the wins and losses differ by more than chance.
5. Run the pointwise checks too. A version can win most head-to-heads and still introduce a new hard failure in a few sessions. A preference score will not show that; a pointwise safety or policy check will.

# ab_verdict.py - is the challenger really better? a two-sided sign test on wins vs losses
# usage: python ab_verdict.py 31 19 10   (wins, losses, ties)
import sys
from math import comb


def sign_test(wins, losses):
    n = wins + losses                             # ties say nothing about direction
    k = max(wins, losses)
    tail = sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n
    return min(1.0, 2 * tail)


wins, losses, ties = (int(x) for x in sys.argv[1:4])
print(f"challenger: {wins} wins, {losses} losses, {ties} ties; two-sided p = {sign_test(wins, losses):.3f}")

The numbers are sobering at typical sizes. On 60 inputs, a challenger that wins 31, loses 19 and ties 10 looks clearly better - but the sign test on 31 against 19 gives a two-sided p of roughly 0.12, suggestive rather than conclusive. At 36 wins to 14 losses it drops well below 0.01. If the decision matters, grow the input set before you trust a small margin. Comparing two agent versions covers the rest of the champion-and-challenger setup.

One Prompt Change, Judged Both Ways

Say a support agent gets a rewritten system prompt meant to make its answers more specific. The numbers below are illustrative; the shape of the result is common.

1. Pointwise: did anything break?

   Run both versions over the same 80 frozen requests and grade every session with the task-completion and policy rubrics. Task completion is nearly identical, two sessions apart, which is inside the noise at this size. The policy rubric, though, fails three challenger sessions that promised a refund timeline the policy does not allow. That is a hard failure the change introduced, and only the pointwise check could see it.

2. Pairwise: is it better?

   Judge the 80 pairs for specificity, in both orders. The challenger wins 44, loses 16, and 20 pairs flip with the order and count as ties. The sign test on 44 against 16 gives a p far below 0.01: the new prompt really does produce more specific answers.

3. The decision

   Better answers and a new policy break. Fix the prompt line that produced the refund promise, re-run both checks, and ship when the pairwise win holds and the policy failures are gone. Either approach alone would have given the wrong answer: pointwise alone says the change made no difference, and pairwise alone says ship it.

Why Production Judging Is Pointwise

Live traffic produces one output per input. There is no second session to compare it with unless you deliberately run two versions on the same request, which doubles cost and exposes users to both. So the everyday question in production - is this agent working, and is it getting worse - is answered pointwise: judge each session against a rubric, aggregate to a rate, alert when the rate moves.

A pointwise rate also survives changes a pairwise score cannot. Last month's pass rate and this month's are comparable as long as the rubric and judge model are unchanged. A pairwise win rate is only meaningful against the specific baseline it was measured against, and it resets every time the baseline changes.

Where Each Is Stronger

Pointwise Is Stronger For

• Monitoring and alerting, which need an absolute level.
• Hard requirements. "Never promised a refund it could not give" is PASS/FAIL on each session, not a preference.
• Cost. One judge call per output, rather than two per pair.
• Many candidates. Ten prompt variants are ten pointwise runs; pairwise needs a tournament or a fixed baseline.

Pairwise Is Stronger For

• Close calls between versions, where both mostly pass a pointwise rubric and the difference is in quality.
• Criteria that resist absolute scales, such as helpfulness or clarity, where "better than" is easier to agree on than "a 4 out of 5".
• Ship decisions. A win rate with a significance test answers "should we switch?" more directly than two pass rates a few points apart.

Where Failproof AI Fits

Failproof AI's evaluations are pointwise by construction: each one scores a single finished session and writes its result - a score, a metric or an assertion, with reasoning - beside that session's trace. That is the monitoring half. Results appear under Observe → evaluations, and fp evals --agent-id checkout-agent --aggregate pulls one agent's aggregate from the terminal, so running a challenger under a different agent_id or environment gives you two pointwise rates side by side. A hosted evaluation can also be scoped with a condition on session.agent_id or session.environment, such as session.agent_id == "checkout-agent".

The pairwise half - running two versions over a frozen input set and judging the pairs - is an offline job in your own harness, with the code on this page. Failproof AI does not run it for you. If you would rather compare versions on real production sessions than on a frozen set, Raindrop's Experiments compare a baseline cohort and an experiment cohort of events already logged in production, defined by model, feature flag, property, tool or date range: you ship the change, for example behind a feature flag, and Experiments compares the two cohorts. Per Raindrop's docs, Experiments read existing traffic rather than running or replaying your agent, and creating one requires the Pro plan.