online vs offline jev evals

Online Is Not Synonymous with Synchronous

A completed-session evaluator can process production traffic asynchronously and still be an online monitor. A policy gate is synchronous when the action waits for its answer. Use precise terms because their reliability requirements differ: blocking paths need strict deadlines and fallbacks; offline jobs need checkpointing, reproducibility and backfill control.

One Criterion Across Two Evidence Loops

Budget the Complete Synchronous Path

TypeSafe reports first-party Jev latency in the tens to hundreds of milliseconds, but its launch conditions and short dense state matter. A synchronous budget includes state retrieval, serialization, network, provider queueing, retries, response validation and fallback—not only model inference.

Set a deadline from the user or tool-call SLA, then allocate per-attempt timeout and maximum retries. If the check is not essential before action, move it asynchronous. The retry guide explains why SDK per-attempt defaults can exceed a product deadline.

An Asynchronous Evaluator Needs Durable Joins

Use an idempotency key at the evaluation-job layer so delivery retries do not create duplicate logical results. Preserve attempt count and provider failures separately. A dead-letter queue must remain visible in scorable coverage; otherwise dropped hard cases can make the dashboard look better. The drift guide defines the minimum monitoring record.

Offline Labels and Online Outcomes Answer Different Questions

Offline human labels provide quick controlled truth but can differ from actual success. Online outcomes arrive later, may be missing, and can be influenced by the action the evaluator helped choose. Preserve both. For an agent task, a reviewer’s completion label and a later customer reopening the issue are different signals.

Beware selective feedback. If only low-confidence cases receive review, labels are not representative of automated cases. Randomly audit each action band and use inverse-propensity or experimental designs when routing changes which outcomes can be observed.

Choose the Path from Consequence and Recoverability

Do not put every evaluator on the request path because Jev can be fast. Synchronous coupling adds a provider dependency and new failure mode. Jev evals versus policies explains when measurement should graduate into control.

Keep Offline and Online Measurements Comparable

• Use the same resolved model, question, criteria and state-projection versions.
• Record whether evidence was available at decision time or added afterward.
• Compare traffic mix, missingness, languages and tool versions.
• Separate provider failures from valid low-confidence results.
• Report coverage, error, latency and cost by time window and slice.

Use an Explicit Gate Before Changing the Execution Path

Passing offline tests permits observation, not automatic enforcement. A candidate should first run beside the existing workflow so online evidence can reveal traffic and instrumentation differences. Promotion criteria must be written before results are inspected; otherwise teams can move thresholds until a preferred launch decision appears. Evaluate the evaluator supplies the release scorecard.