How much do agent evals cost?

The Two Bills

Every eval program pays twice. The judge bill is the tokens a model reads and writes to grade your sessions. It goes to a model provider - yours, if you bring the judge, or the vendor's, if it runs its own evaluation models and prices them in. The platform bill is what the vendor charges to store traces, run the scoring and show you results, in its own unit.

The two scale differently. The judge bill grows with how much text each judge reads, so a long agent session costs far more to judge than a short chat turn. The platform bill grows with sessions, spans or scores, depending on the meter. Most surprises come from the first: teams budget per session and forget that one agent session can run to tens of thousands of tokens.

The Judge Bill, as a Formula

judge_cost_per_month = S × r × J × (T_in × P_in + T_out × P_out)

S       sessions per month
r       share of sessions judged (1.0 = every session)
J       judge calls per judged session (one per criterion, if judged separately)
T_in    input tokens per judge call (rubric + transcript)
T_out   output tokens per judge call (score + reasoning)
P_in    your provider's price per input token
P_out   your provider's price per output token

Prices differ by model and provider and change often, so this page keeps P_in and P_out as variables. Put in the numbers from your own provider's price list.

Two inputs are easy to get wrong. T_in is the whole prompt, not just the transcript: a rubric with worked examples can run to a thousand tokens, paid on every call. And J multiplies everything - judging three criteria in three calls reads the transcript three times. Measure T_in on a real sample of your sessions rather than guessing. Agent transcripts vary widely, and the long tail of sessions that loop or retry is often where most of the tokens are.

Worked Examples, with Every Assumption Stated

One mid-size deployment, evaluated three ways. The assumptions:

• 30,000 agent sessions a month.
• Three criteria: tool reliability, task completion, policy compliance.
• An untrimmed judge call reads a 1,000-token rubric plus a 7,000-token transcript, so T_in = 8,000; trimmed, T_in = 3,000.
• Each judge reply is 300 tokens, or 400 when one call covers two criteria.

Example A costs 720,000,000 × P_in + 27,000,000 × P_out a month. Whatever your provider charges, C reads one eightieth of A's input tokens. The saving reduces coverage: C judges one session in ten, so a failure that happens once in a thousand sessions appears about three times a month instead of thirty. Use sampling to discover and measure patterns. Once a high-risk failure is known, cover it with an always-on deterministic check or runtime policy. Combining two criteria into one call can also reduce accuracy, so test it against labeled examples.

The Platform Bill, by Meter

This comparison holds the work constant: 30,000 agent sessions a month, with three criteria scored for every session. That produces 90,000 scores, but vendors do not bill those scores in the same way. Failproof AI can return all three results from one evaluation, Braintrust meters each score, and Langfuse meters traces, observations and scores.

For this workload, Failproof AI Team is $99, Langfuse Core is about $127 and Braintrust Pro is $309 in platform fees. The comparison uses the same sessions and three scored criteria throughout, but the billable totals differ because the products meter different objects. These are not feature-equivalent prices: compare tracing, evaluation operations, failure analysis, alerting and retention alongside the total. Sampling can reduce evaluation and judge-model costs on every platform, but it should be applied consistently when comparing vendors.

The Levers, in Order of Impact

1. Code checks before judges. Tool errors, repeated calls, missing files and budget overruns can be computed from the event stream for free. Every criterion you move to code is a judge call you never pay for.
2. Sampling. Judge a fixed share of sessions, and oversample the ones code checks already flagged. Coverage falls in proportion, as example C shows.
3. Shorter transcripts. Trim tool results, keep the end of the session, or summarize before judging. Judge input is usually most of the bill.
4. Smaller judge models. Cheaper per token, sometimes as accurate on a narrow rubric - only your labeled sessions can tell you. Re-check agreement before switching.
5. Never judge a session twice. Deduplicate on session id, and rerun only on purpose - in Failproof AI that is re-evaluate, which runs a session's evaluations again.

Failproof AI runs code-based evaluations, LLM judges and existing evaluation suites in the cloud. A deterministic check can be drafted from a plain-English description, tested against real sessions and deployed as a version. Because it calls no model, it adds no judge-token cost.

Read the resulting charts with two things in mind. If the judge only reads sessions the code check passed, its score means "among sessions with no failed tool results", not "across all sessions". And a trimmed transcript that keeps the end of the session, where the outcome usually is, should be tuned against your labels. Code checks and judges are both billable evaluations, one per session-and-evaluation pair.

When the Judge Bill Is Not Worth Optimizing

At lower volume, optimization work can cost more than the judge calls it saves. Measure a month of real token usage before adding sampling, summarization or a smaller model. Even after a fix or behavioral policy ships, keep a lightweight evaluation running so you can detect regressions and confirm the failure stays closed.