How to sample production traffic for agent evals

Why Sample, and How Sampling Goes Wrong

An LLM judge costs tokens and time on every session it reads, and agent transcripts can be long. Judge spend therefore grows roughly in line with the number and size of sessions. Code checks avoid the additional model call and are usually much cheaper for mechanical signals, so sampling is mainly a decision about which sessions receive model-based evaluation.

The naive answer, "judge 5% at random", is right for one job and wrong for three others. It gives an honest estimate of the overall pass rate. It also gives a refund agent that handles 1% of traffic about one judged session in two thousand, which is not a measurement. It judges a real failure only 5% of the time, so the sessions you most want reasoning for mostly go unread. And if "random" quietly means "the first sessions after midnight" or "the ones short enough not to time out", it is not random at all, and the pass rate it reports belongs to a different population from the one you care about.

There is one more trap that is purely mechanical. Evaluation requests get retried. If your sampler calls random.random(), a retried session can be judged twice or skipped on the second attempt, and your counts drift. The decision has to be a pure function of the session.

Random, Stratified and Triggered Sampling

Use all three at once. Triggers first: any session with a tool error, a failed code check, an escalation or an unusually high step count goes to the judge with probability 1. Everything else is sampled at a rate that depends on its stratum - high for low-volume, high-stakes agents, low for the chatty ones. Record the probability each judged session had of being picked, because that number is what lets you turn an oversampled pile back into an honest rate.

Good triggers are cheap, computed from data you already have, and correlated with failure without being the failure itself. Tool errors in the event stream, a step count far above the agent's median, a session that never recorded an end, a user who retried the same request within minutes, and a hand-off to a human all qualify. A trigger that fires on a third of traffic is not a trigger any more; it is a second base rate, and it should be tuned or dropped. Keep a count of how often each one fires, so you can tell which triggers earn their judge calls by how often the judge agrees that something went wrong.

Strata should be the slices you actually make decisions on: usually agent_id crossed with environment, sometimes with a customer tier or task type added. Keep the list short. Every stratum needs its own minimum sample to be worth reporting, and ten strata with thirty sessions each tell you less than three strata with a hundred.

How Many Judged Sessions You Need

For a pass/fail score, the 95% margin of error on a pass rate p measured over n independent sessions is about 1.96 × sqrt(p(1 - p) / n). At a pass rate near 90%:

Set the per-stratum rate backwards from that table. If the refund agent handles 200 sessions a day and you want a weekly number within about 2 points, you need roughly 700 judged sessions a week, which is half its traffic. If the general chat agent handles 15,000 a day, 3% gives you 450 a day, which is more than enough. The rates end up very different, and that is the point of stratifying.

A Sampler That Works in Any Evaluator

The sampler below makes the decision from the session id, so it is stable across retries and across machines, and it returns the inclusion probability alongside the decision.

# sampler.py - decide which sessions the LLM judge sees
import hashlib

BASE_RATE = 0.05                               # ordinary sessions
STRATUM_RATES = {                              # (agent_id, environment) -> rate
    ("refund-agent", "production"): 0.50,      # low volume, high stakes
    ("chat-agent", "production"): 0.03,        # high volume
}

def unit_draw(session_id: str) -> float:
    """A stable number in [0, 1) for each session, so retries decide the same way."""
    digest = hashlib.sha256(session_id.encode()).digest()
    return int.from_bytes(digest[:8], "big") / 2**64

def should_judge(session_id: str, agent_id: str, environment: str,
                 code_scores: dict[str, float]) -> tuple[bool, str, float]:
    """Return (judge it?, why, probability this session had of being judged)."""
    if any(v < 1.0 for v in code_scores.values()):
        return True, "triggered", 1.0
    rate = STRATUM_RATES.get((agent_id, environment), BASE_RATE)
    return unit_draw(session_id) < rate, "sampled", rate

When you report, weight each judged session by the inverse of its inclusion probability. A triggered session stands for itself; a session sampled at 5% stands for twenty. This is the standard ratio estimator, and it is what stops a pile of triggered failures from making the whole fleet look worse than it is.

# report.py - an honest pass rate from an oversampled set
def weighted_pass_rate(rows: list[dict]) -> float:
    """rows: {"passed": bool, "p": inclusion probability} for each judged session."""
    total = sum(1 / r["p"] for r in rows)
    passed = sum(1 / r["p"] for r in rows if r["passed"])
    return passed / total if total else float("nan")

The Cost Math, with Every Assumption Stated

Assumptions for this example, none of them universal: 20,000 completed sessions a day; each judge call reads 6,000 input tokens (an 800-token rubric plus a trimmed transcript) and writes 300 output tokens; one judge per session; 3% of sessions trip a trigger. P_in and P_out are your judge model's prices in dollars per million input and output tokens - look them up for the model you use.

The sampled plan still judges 970 unbiased sessions a day, which puts the fleet-wide pass rate within about 2 points daily, and it reads every session the cheap checks flagged. What it gives up is per-session judge scores on the other 18,430 sessions. If an individual customer's session has to be scored because someone will look it up, that session belongs in a trigger, not in the random pool.

Pitfalls, and How to Check Your Sample

• Check the sample looks like the population. Compare the distribution of agent, environment, session length and hour of day between judged and all sessions. A sampler that drops long sessions is common and invisible until you look.
• Keep rates fixed within a reporting window. Changing a stratum rate mid-week breaks comparisons unless every row carries its own inclusion probability.
• Do not trigger on the judge's own output. "Judge it again if the judge said fail" makes the pass rate depend on the judge's noise.
• Watch trigger inflation. When a new code check starts failing on 40% of sessions, your judge bill jumps with it. Alert on the triggered fraction, not only on scores.
• Re-derive the rates when traffic moves. A stratum that shrinks from 2,000 to 200 sessions a day at a fixed 5% stops producing a usable number.

Sampling Evaluations with Failproof AI

Failproof AI runs code-based and LLM-based evaluations in the cloud and keeps each result beside the session trace. Run inexpensive code checks broadly to create signals such as tool errors, excessive steps or missing completion evidence. Use evaluation conditions to scope a judge to supported attributes and known triggers, such as a particular agent or environment.

EvalResult(
    score=Score(
        len([e for e in session.events_of_type("tool_result") if e.payload.get("status") == "ok"])
        / max(1, session.count("tool_result"))
    ),
    metrics={"tool_calls": Metric(session.count("tool_use"), unit="calls")},
    reasoning="Share of tool results that came back ok.",
)

For stable random or stratified sampling, make the selection in an evaluation workflow that can calculate a cryptographic hash, then send or score the selected sessions with the judge. Do not substitute Python's process-randomized hash() or an ordinary random number inside a hosted condition. Store the inclusion probability with each sampled result so the reporting layer can reweight it correctly.

Read sampled trends by agent and environment, while keeping triggered results separate from the weighted random estimate. Failproof AI can analyze failed evaluations and trace evidence across sessions to find recurring failure modes, group affected sessions into findings and recommend what to change. A tested behavioral policy is one possible fix when the finding identifies a recognizable high-risk action, not the default outcome of every failed score.

Only selected sessions carry the sampled judge result, so do not read its raw chart as the fleet-wide pass rate when triggers or unequal strata are involved. Calculate the weighted estimate in a query or reporting job, show its sample size and interval, and review current pricing when estimating how sampling changes platform and model costs.

When You Do Not Need to Sample

If traffic is low enough to judge every session within your budget, full coverage is simpler and gives every trace a result. If the criteria are fully checkable in code - the tool call succeeded, the JSON parsed, the ticket was closed - you do not need a model judge or a sampler. And if every session is high stakes, sampling is the wrong lever: reduce the cost per judgment with shorter context, fewer calls or a smaller calibrated judge while keeping full coverage.