How many examples do you need for an LLM eval?

A Pass Rate Is an Estimate with an Error Bar

Your eval set is a sample of the inputs your agent will see, and its pass rate estimates performance across that larger population. The examples must also represent that population: a precisely measured score on easy or duplicated cases is still misleading. Miller (2024) frames eval questions as draws from an unseen super-population. The consequence is that every pass rate needs an interval next to it. Without one, you cannot tell a real change from a different draw.

Use the Wilson score interval. The textbook Wald interval, p ± z × sqrt(p(1 - p) / n), has erratic coverage at small n and near 0% or 100%, which is exactly where agent evals live; Brown, Cai and DasGupta (2001) recommend Wilson for small samples. With p the observed pass rate, n the number of examples and z = 1.96 for 95%, the Wilson interval is (p + z²/2n ± z × sqrt(p(1 - p)/n + z²/4n²)) / (1 + z²/n).

from math import ceil, log, sqrt

Z = 1.96          # 95% two-sided
Z_POWER = 0.8416  # 80% power


def wilson(passes: int, n: int, z: float = Z) -> tuple[float, float]:
    p = passes / n
    denom = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return round(centre - half, 3), round(centre + half, 3)


def n_for_margin(margin: float, p: float = 0.5, z: float = Z) -> int:
    return ceil(z**2 * p * (1 - p) / margin**2)


def n_per_version(p1: float, p2: float) -> int:
    return ceil((Z + Z_POWER) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p1 - p2) ** 2)


def n_to_see_once(rate: float, confidence: float = 0.95) -> int:
    return ceil(log(1 - confidence) / log(1 - rate))


print(wilson(45, 50), wilson(180, 200))                        # (0.786, 0.957) (0.851, 0.934)
print(n_for_margin(0.05), n_for_margin(0.05, p=0.9))           # 385 139
print(n_per_version(0.80, 0.90), n_per_version(0.80, 0.85))    # 197 903
print(n_to_see_once(0.01), n_to_see_once(0.05))                # 299 59

Read the table as a warning about the top rows. A 90% pass rate on 20 examples is consistent with a true rate anywhere from 70% to 97%: a demo, not a measurement. The width shrinks with the square root of n, so halving the interval costs four times the examples. Going from 50 to 200 buys you about half the uncertainty; going from 200 to 800 buys the next half.

Sizing for a Margin of Error

To plan a set before you have data, invert the interval: n = z² × p(1 - p) / E², where E is the margin you can live with. If you do not know the pass rate, use p = 0.5, the worst case. This is the simple Wald form, fine for planning; compute the actual interval with Wilson once results are in.

Rates near the edges need fewer examples for the same margin, because there is less variance to pin down. That cuts both ways: a safety eval where the pass rate is 99% needs few examples to show the rate is high, and many to show how high.

Sizing to Detect a Change Between Versions

Comparing two versions needs more examples than estimating one, because both estimates carry noise. For two independent samples, the examples per version to detect a difference between rates p1 and p2 are n = (z + z_power)² × (p1(1 - p1) + p2(1 - p2)) / (p1 - p2)², with z = 1.96 for a 5% two-sided test and z_power = 0.8416 for 80% power.

Power is the other half of that formula. At 80% power, one in five real improvements of the stated size will fail to reach significance. If missing an improvement is expensive, raise the power to 90% (z_power = 1.2816) and every number in the table grows by about a third.

Most teams cannot label 900 examples to settle a five-point question, and they do not need to. Running both versions on the same inputs and testing the pairs uses only the examples where the versions disagree, which usually needs far fewer; comparing agent versions has the paired test with worked numbers.

Per-Slice Minimums and Rare Failures

An overall sample size does not carry over to slices. Five hundred examples spread across ten intents is fifty per intent, and each intent's pass rate carries the 17-point interval from the table above. The rule: size for the smallest slice you will make a decision about, not for the total. If you will only ever act on the overall number, the total is what matters; if refunds get their own dashboard row, refunds need their own sample size.

Stratify rather than hope. If a slice matters, sample it on purpose up to its minimum, then weight each slice back to its real share of traffic when you report the overall rate, so the oversampled slice does not drag the headline number around.

Rare failures need a different question: will the set contain the failure at all? The chance of seeing at least one instance of a failure that occurs at rate r in n examples is 1 - (1 - r)^n. To be 95% sure of seeing a 1% failure mode once, you need 299 examples; for a 5% failure mode, 59. Seeing it once is only enough to know it exists, not to measure it.

The mirror image is the rule of three: if a failure never appears in n examples, the 95% upper bound on its rate is about 3 / n. Zero failures in 50 examples still allows a failure rate near 6%. "It never failed in testing" is a statement about the size of your test set.

When 50 Is Enough, and When It Is Not

Fifty Is Enough For

• Early development, when failure rates are high. 25 of 50 is a 95% interval of 36.6% to 63.4% - wide, and still clear that the agent is not ready.
• Catching large regressions. A drop from 90% to 70% shows up at 50 examples; a drop from 90% to 87% does not.
• Smoke tests in CI, where the job is to catch breakage, not to measure quality.
• Finding failure modes. Reading the failures within 50 representative sessions can teach you more about what to fix than scoring thousands without inspecting the evidence.

Fifty Is Not Enough For

• Choosing between versions a few points apart. See the table above: that is hundreds per version, or a paired design.
• Rates of rare failures, such as a safety violation at 1%.
• Per-slice claims about anything with more than a couple of slices.
• Promises about high reliability. Showing a 99% pass rate with a tight interval takes hundreds of examples, and a single failure moves it noticeably.

Two More Error Bars: The Judge, and Repeated Runs

The Wilson interval covers sampling error only. If a judge grades the examples, its mistakes bias the rate itself. With the judge's sensitivity (the share of truly passing examples it passes) and specificity (the share of truly failing examples it fails) measured on human labels, the Rogan-Gladen correction from epidemiology estimates the true rate as (observed + specificity - 1) / (sensitivity + specificity - 1). An observed 85% from a judge with sensitivity 0.95 and specificity 0.80 corrects to 0.65 / 0.75 = 86.7%. The correction carries extra uncertainty of its own, and it is only as good as the labels behind it; calibrating a judge is how you get them.

Repeated runs of the same task are not independent examples. If you run each of n tasks m times, the effective sample size is roughly n × m / (1 + (m - 1) × ρ), where ρ is how strongly runs of the same task agree. With 100 tasks, 5 runs each and ρ = 0.6, 500 runs are worth about 500 / 3.4 = 147 independent examples. More tasks beat more runs of the same tasks; Miller (2024) covers the clustered standard errors that handle this properly.

Use Production Results Without Overreading the Chart

Failproof AI can run deployed code-based and LLM-based evaluations on production sessions, so high-volume agents accumulate evidence quickly. Volume does not remove sampling problems: a low-traffic agent, a rare workflow or a narrowly filtered customer segment can still have too few independent results to support a decision. Each session-and-evaluation pair is one billable evaluation; the pricing page lists the included monthly volume.

Filter results by agent and environment, then calculate the interval for the exact population behind the decision. Do not use the much larger all-agent total to justify a claim about one small slice:

fp evals --agent-id checkout-agent --aggregate
fp --json evals --aggregate --env production
fp evals --since 7d --score task_completed:0..0.5

The same logic applies to alerts. A threshold alert on an agent that runs twenty sessions a day can fire on sampling noise: at an observed 90% pass rate, the 95% interval is about 27 points wide. For low-volume agents, use a longer window or require a minimum number of results. When scores decline, Failproof AI can analyze the failed traces, group recurring patterns into findings and show the evidence behind them. That tells you what changed instead of adding another unsupported decimal place to the pass rate.