How to reduce LLM judge cost

Reduce the Cost of Online Evaluations First

Online evaluations run continuously on production sessions, so their cost grows with traffic. Do not treat every evaluation as an all-or-nothing choice between full coverage and no coverage. Run inexpensive code checks broadly, then reserve model-based evaluations for a representative sample and for sessions carrying signals that deserve closer inspection.

Useful signals include tool errors, repeated calls, unusually long sessions, missing completion evidence, escalations and markers your application already attaches to important workflows. These filters do two different jobs: the random sample preserves an unbiased view of overall quality, while triggered sessions concentrate spend where failures are more likely. Report the two groups separately or reweight the sample; a raw average over triggered sessions will make the fleet look worse than it is.

Keep full model-based coverage for genuinely high-stakes workflows. Sampling is a cost control for broad monitoring, not a reason to leave consequential actions unexamined. You can still reduce their cost with code checks, shorter evidence, combined criteria and a calibrated smaller model.

Where Judge Spend Comes From

An LLM judge bill has four factors: how many sessions you judge, how many judge calls each session gets, how many tokens each call reads and writes, and what the judge model charges per token. Written out, the daily cost is sessions judged × calls per session × (input tokens × P_in + output tokens × P_out) / 1,000,000, with prices in dollars per million tokens.

For agent evaluation, input tokens usually dominate because a judge reads a rubric plus a transcript that may run to tens of thousands of tokens and returns a short verdict with evidence. That is why the largest savings often come from judging fewer sessions and trimming irrelevant context before changing models.

Costs also creep for structural reasons. Teams often add a separate model call for every criterion, even when several independently scored results could share one call. They judge every session because the first version did, pass the whole transcript because trimming feels risky, and re-judge unchanged cases in CI without caching identical inputs.

The Levers, in the Order to Pull Them

1. Do Not Ask a Model What Code Can Check

Did a tool return an error? Was the output valid JSON? Did the agent call the refund tool more than once? Did the session take more than 40 steps? These are exact checks that need no additional model call. Move every reliably computable criterion out of the judge prompt. What remains - "was the answer correct", "was the tone appropriate", "should it have escalated" - is where a judge adds value.

2. Gate the Judge on the Code Checks

If the code checks already say a session failed, the judge adds reasoning but not a verdict. Decide which you want. If the judge is there to find new failures, skip it on sessions that already failed a hard check. If it is there to explain known ones, judge only those.

3. Judge a Sample

Judge a stratified random sample plus every session a cheap signal flags, and reweight when you report. The sampling guide has the sample sizes and the code. This is usually the single largest cut, because it divides the whole bill.

4. Show the Judge Less

Cut the transcript to what the criterion needs. A task-completion judge needs the user request, the final answer and the tool results that bear on it; it does not need twelve file reads of the same config. Truncate large tool outputs to their first and last few hundred characters, drop consecutive duplicates, and put the rubric first so it can be cached. Then check agreement with your labels again, because over-trimming removes the evidence the judge needed.

5. Make Fewer Calls

Scoring three criteria in one call reads the transcript once instead of three times. The trade is real: one-criterion-per-judge prompts are easier to write and calibrate, and combined prompts let criteria bleed into each other. Combine only criteria that read the same evidence, and measure agreement per criterion afterwards.

6. Cache Verdicts, and Let the Provider Cache the Prefix

Key a verdict cache on a hash of the judge model, the rubric version and the trimmed transcript. Identical inputs - a retried evaluation request, an unchanged case in a frozen CI dataset - then never pay twice. Separately, providers cache repeated prompt prefixes: Anthropic bills cache reads at a fraction of the normal input price, but prompts shorter than a model-specific minimum are silently not cached. A 600-token rubric can fall under that minimum on some models, so check before counting on it.

7. Batch What Is Not Urgent

Anthropic's Message Batches API and OpenAI's Batch API both document a 50% discount against synchronous requests in exchange for asynchronous results. Anthropic says most batches finish in under an hour, while OpenAI provides a 24-hour completion window. Nightly scoring, backfills and offline dataset runs rarely need a verdict in seconds.

8. Then, and Only Then, a Cheaper Judge

A smaller judge model can be a large saving, and it is the lever most likely to quietly change your scores. Run it against the same 50-200 human-labelled sessions you used to calibrate the original, compare agreement on the fail class, and switch only if it holds. When it does, treat it as a new judge: new score history, new thresholds.

A Worked Estimate

Assumptions, all of them illustrative: 10,000 completed sessions a day; three criteria, each judged in its own call; a 600-token rubric; transcripts averaging 12,000 tokens; 250 output tokens per call; 2% of sessions trip a trigger. P_in and P_out are your judge model's dollar prices per million tokens. No real model price is used.

Input cost falls from 378 to 5.43 units of P_in, about 70 times less, and output cost from 7.5 to 0.295 units of P_out. The order of the rows matters for your own estimate: code checks and trimming change what every remaining judgment costs, so do them before you decide how much to sample. Caching is left out of the table because its effect depends on how often inputs repeat, which in production is rarely, and in CI can be most of the time.

A Cost-Aware Judge in Code

This judge trims the transcript, checks a local verdict cache first, parses the reply defensively, and logs the tokens it actually used so the estimate above can be replaced with real numbers.

# judge.py - trimmed input, cached verdicts, logged token usage
import hashlib
import json
import os
import sqlite3

import anthropic

client = anthropic.Anthropic()                      # reads ANTHROPIC_API_KEY
JUDGE_MODEL = os.environ["JUDGE_MODEL"]
RUBRIC_VERSION = "task-completion-v4"
RUBRIC = (
    "You grade one AI agent session. Pass if the agent completed the user's task "
    "or clearly said it could not. Fail otherwise.\n"
    'Reply with JSON only: {"pass": true or false, "evidence": "one sentence"}\n'
)
db = sqlite3.connect(os.environ.get("VERDICT_CACHE", "verdicts.sqlite"))
db.execute("CREATE TABLE IF NOT EXISTS v (k TEXT PRIMARY KEY, verdict TEXT)")

def trim(events: list[dict], max_chars: int = 1500) -> str:
    lines, last = [], None
    for e in events:
        body = json.dumps(e["payload"], default=str)
        if body == last:
            continue                                 # drop consecutive duplicates
        last = body
        if len(body) > max_chars:
            body = body[: max_chars // 2] + " ...[trimmed]... " + body[-max_chars // 2 :]
        lines.append(f"[{e['event_type']}] {body}")
    return "\n".join(lines)

def judge(events: list[dict]) -> dict:
    transcript = trim(events)
    key = hashlib.sha256(f"{JUDGE_MODEL}|{RUBRIC_VERSION}|{transcript}".encode()).hexdigest()
    row = db.execute("SELECT verdict FROM v WHERE k = ?", (key,)).fetchone()
    if row:
        return json.loads(row[0])                    # identical input: no new spend

    prompt = RUBRIC + "\nSession:\n" + transcript
    msg = client.messages.create(
        model=JUDGE_MODEL, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = msg.content[0].text
    print(json.dumps({"judge_in": msg.usage.input_tokens, "judge_out": msg.usage.output_tokens}))
    try:
        parsed = json.loads(text)
        verdict = {"pass": parsed["pass"] is True, "evidence": str(parsed.get("evidence", ""))}
    except (json.JSONDecodeError, KeyError, TypeError):
        return {"pass": None, "evidence": f"unparsable judge output: {text[:200]}"}  # not cached

    db.execute("INSERT OR REPLACE INTO v VALUES (?, ?)", (key, json.dumps(verdict)))
    db.commit()
    return verdict

Two details matter. The cache key includes the rubric version, so editing the rubric invalidates old verdicts instead of silently serving them. And an unparsable reply is returned as pass: None and never cached, so a bad judge response neither becomes a permanent fail nor gets counted as a pass.

What Cheaper Can Break, and How to Check

• Trimming removes evidence. A judge that never sees the failed tool result will pass the session. Re-run agreement on your labelled set after every trimming change.
• Combined prompts blur criteria. A session that fails on tone starts failing on correctness too. Check per-criterion agreement, not only the overall number.
• A smaller judge drifts differently. It may agree overall and miss the fail class. Precision and recall on failures matter more than percent agreement.
• Sampling hides small agents. Stratify, or a low-volume agent vanishes from your numbers.
• Stale caches. Key on judge model and rubric version, or a changed rubric keeps returning yesterday's verdicts.
• Batch jobs outlive their callers. A result that arrives hours later must still be attached to the right session. Key it by session id, never by position.

Set up Cost-Aware Online Evaluations in Failproof AI

Failproof AI Cloud runs code-based and LLM-based evaluations on production sessions and can run your existing evaluation suite. Start with cheap code checks across the traffic you want to monitor. They create exact signals for tool errors, excessive steps, missing actions and other mechanical failures without adding a model call.

Next, use evaluation filters to limit an expensive LLM-based evaluation to the relevant agents, environments and session markers. Combine that triggered coverage with a stable random sample of otherwise unflagged sessions. This gives likely failures more attention without losing the unbiased sample needed to detect problems that none of your existing markers know about yet.

Several related criteria can return independently scored results from one evaluation when they need the same trace evidence. That avoids rereading the transcript in a separate model call for every criterion. Test each result against human labels independently; sharing a call should not mean sharing a pass or fail decision.

Keep provider batch jobs for offline datasets and non-urgent backfills. Online evaluations are for production sessions whose results need to feed current monitoring and alerts; a provider batch that may return hours later has a different operating model. Failproof AI backfill can rerun evaluations over historical sessions when you need to test a new rubric or build a baseline.

Track model token use with the provider usage returned by each call. On the Failproof AI side, each session-and-evaluation pair is one billable evaluation, and one evaluation can return up to 25 results. Three independently calibrated criteria returned by one evaluation therefore consume one evaluation per session rather than three. Plan allowances are on the pricing page.

The selected results remain linked to their trace evidence and feed alerts and automatic failure analysis. That is important when sampling: Failproof AI can group related failed sessions into recurring findings and recommend what to change, instead of leaving you with a lower-cost score stream and no explanation of the failures behind it.

When Judge Cost Is Not Your Problem

When judge spend is small relative to the value of better coverage, spend engineering time on the rubric and calibration instead of premature optimization. If one missed failure can cost more than broad judging, optimize for recall and use the model that agrees best with your labels. And if a criterion turns out to be reliably checkable in code, remove the unnecessary model call.