online vs offline agent evals

What Each One Is

Offline evaluation runs a candidate version of the agent against a dataset you control - inputs, and expected outcomes where you can write them down - in an environment you control, before the change reaches users. It is usually a CI job or a release gate. Because the inputs are fixed, it is reproducible: run the same set against two versions and the only thing that changed is the agent.

Online evaluation scores real production sessions after they complete, using the same kinds of checks - code, judges, occasionally a human - on inputs nobody chose. It runs continuously, mostly without expected answers, and it sees whatever users actually do, including the things nobody thought to test.

The names come from machine learning, and "online" misleads people in one specific way: an online eval is not in the request path. It scores a session after the session ran. Anything that has to act during the run - refuse a request, block a tool call - is a guardrail or a policy, not an eval. Evals vs guardrails covers that line.

Side by Side

The last row is there on purpose. Both approaches measure; neither intervenes. The difference between them is when the measurement happens relative to the users, and so which failures it can see at all.

Where Offline Evals Are Stronger

• Regressions before deploy. The whole point: a change that breaks what worked is caught before anyone depends on it.
• Clean comparisons. Both versions see the same inputs, so a paired test can find a real difference with far fewer examples; see comparing agent versions.
• Ground truth and end-state checks. You can write the expected outcome, reset a sandbox and check the database afterwards, which is the most reliable scoring there is.
• Rare cases on purpose. A failure that happens once in a thousand production sessions can appear fifty times in the dataset.
• Repeated runs. Running each task several times measures consistency, which production can only estimate.
• Dangerous tools, safely. A sandbox lets you test the agent against a fake production database.

The weakness is the dataset itself. It is only as good as the cases someone thought to include, it goes stale as users change, and teams tune prompts against it until the score stops meaning anything outside it. An offline pass rate is a statement about the dataset.

Where Online Evals Are Stronger

• Drift. New users, new products, a seasonal spike in one kind of request: production shows it the day it starts.
• Failures nobody wrote a test for. The dataset contains the failures you imagined; production contains the rest.
• Real integrations. Real tools, real latency, real rate limits, real malformed data from the CRM.
• Real cost and latency. Per-task spend and time on actual traffic, not on a tidy sample.
• The user's reaction. Escalations, rephrasing and repeat contacts only exist in production.

Its weaknesses mirror offline's strengths. It arrives after the fact; most sessions have no expected answer, so scores are reference-free; judging every session at volume costs real money; and comparing versions needs random assignment rather than "last week versus this week". There is also a data question: production transcripts go into your judge prompts, so check where the judge model runs and what your data policy allows before you start.

Vendors in this space lean different ways. Raindrop's own post on evals argues that "the truth increasingly lives in production". Judgment Labs documents its judges running continuously on live traces, on demand, or in offline tests (Judgment Labs docs). The disagreement is about emphasis; nobody serious argues for only one.

Sampling Online: Code on Everything, Judges on a Sample

Code checks cost nothing per session, so run them on every session. Judges cost tokens, so the question is how many sessions to judge. A worked example, with every assumption stated: 20,000 sessions a day, a judge prompt of about 6,000 input tokens per session, and 4% of sessions containing a tool error. Judging every session is 20,000 calls and about 120 million input tokens a day. Judging every error session (800) plus a 5% random sample of the rest (960) is 1,760 calls and about 10.6 million input tokens. Multiply by your judge model's price per token.

Triggered sampling has a catch: the judged sessions are no longer representative, because they over-represent failures by design. Either keep the random and triggered scores under separate keys, or weight them back: each randomly sampled session stands for 1 / 0.05 = 20 sessions, and each triggered session for one.

# sampling.py
import hashlib

RANDOM_RATE = 0.05


def in_random_sample(key: str, rate: float = RANDOM_RATE) -> bool:
    """Deterministic: the same key always gets the same answer, re-runs included."""
    bucket = int(hashlib.sha256(key.encode()).hexdigest()[:8], 16) / 0xFFFFFFFF
    return bucket < rate


def weighted_mean(scored: list[tuple[float, bool]], rate: float = RANDOM_RATE) -> float:
    """scored: (score, triggered). Triggered sessions are all judged, so each has weight 1;
    the rest are judged at `rate`, so each stands for 1 / rate sessions."""
    weights = [1.0 if triggered else 1 / rate for _, triggered in scored]
    return sum(w * s for w, (s, _) in zip(weights, scored)) / sum(weights)

Hashing a stable key, such as a session id, rather than calling random() makes the sampling decision reproducible: a re-run makes the same choice, and you can always recompute which sessions were in the sample. Sampling production traffic goes further into stratified and triggered designs.

Running Both, as One Loop

The two approaches are strongest when each feeds the other:

1. Online scores, alerts and audits surface a failure the dataset does not contain.
2. The failing sessions, scrubbed of personal data, join the offline dataset with the correct outcome written down.
3. The offline eval now fails on the current version. The fix has to make it pass, in CI, before deploy.
4. After deploy, online scores confirm the fix on real traffic.
5. If the failure is one repeatable action, a runtime policy stops it outright, and both evals become a check that the policy holds.

Use the same scoring code in both places. If the function that decides "task completed" is identical offline and online, a gap between the two numbers is about the inputs, which is the interesting question. If the scorers differ, a gap could be the scorers, and you cannot tell which.

Where Failproof AI Fits

Failproof AI's evaluations are online evaluation: they run when each production session finishes and write results beside the trace, compared across agents and environments under Observe → evaluations. Failproof AI runs them in the cloud: code checks, LLM judges, and the eval set you already have (DeepEval, Ragas, promptfoo or your own), brought in as it is, without rewriting it. That is what makes the loop above practical: the scoring code your CI runs offline - an in-house harness, or a library such as DeepEval or Ragas - is the same code that scores production, so a gap between the two numbers is about the inputs. Failproof AI does not run your offline dataset; that stays in CI, and running agent evals in CI shows a job for it. For building the dataset, GET /v1/sessions/{id}/export returns the exact bytes an evaluator receives, which is useful for running an offline suite over real sessions.

The sampling design above carries over. Run the groundedness judge as two evaluations - one on a 5% random sample, one on every other session with a failed tool result - so random and triggered results never share a key and each rate stays readable on its own. Take the random pick from a stable key in the session's own content, as sampling.py does, so a re-evaluation makes the same choice. Test both against real sessions before you deploy them, and deploy each as a version you can roll back.

Judge only the sample, and judge cost stays at the sampled volume. A code check can still cover every session; each session-and-evaluation pair counts as one billable evaluation, so size that against your plan. If staging and production report as different environments, fp --json evals --aggregate --env production keeps them apart, and the same evaluations give you a pre-production online signal from staging traffic. From there, alerts fire on evaluation scores or compound conditions, and audits review a population of sessions when a score declines.

And the last row of the table still applies. Online scores in Failproof AI arrive after the session, like anyone's. The part that acts during the run is separate: a policy at the PreToolUse hook that can deny a tool call before it executes. When an eval keeps finding the same bad action, that is where it goes; turning judge findings into runtime policies walks through one.

When One Is Enough

Before launch there is no production, so offline is all you have; build the dataset from realistic cases and internal dogfooding. After launch, a low-volume internal agent whose every output a person reviews is already evaluated online by that person, and a formal offline set can wait until changes become frequent enough that reviewers start catching regressions. The moment both are true - real traffic and regular changes - you want both.