jev benchmark methodology

Turn the Claim into a Falsifiable Research Question

“Is Jev better?” has no measurable answer. Ask whether a named Jev configuration reduces false-negative cost at 60% automation coverage for a defined criterion, compared with a pinned LLM judge or classifier on the same held-out sessions. Define the consequence of each error before running either system.

Separate model questions from system questions. Label accuracy tests the evaluator; end-to-end latency includes networking and retries; reviewer time tests the workflow. A single headline number should not merge them.

Name the Quantity the Benchmark Is Trying to Estimate

The estimand is the exact population quantity a result claims to approximate. It prevents a common benchmark error: measuring agreement on curated examples and concluding that production harm will fall. State the unit, target population, action, comparison and time window in one sentence. The Jev use-case map helps identify the actual decision before an experiment begins.

Choose one primary outcome and define a practically meaningful improvement before looking at the test set. Secondary metrics diagnose why a system won or lost; they should not become replacement objectives after the primary result disappoints.

Pre-Register Choices That Can Move the Result

Commit the protocol before inspecting final test outputs. Record later deviations and show both the planned and changed analysis. This prevents choosing a favorable threshold, metric or subset after seeing results.

{
  "question": "Does the final answer claim work that no tool result proves?",
  "unit": "completed_agent_session",
  "primary_metric": "false_negative_cost_at_60pct_coverage",
  "slices": ["tool", "agent_version", "language"],
  "split_group": "conversation_id",
  "jev_model": "jev-1.13.0",
  "comparators": ["judge_model_and_prompt_v2", "human_review_v3"],
  "exclusions": ["missing_required_trace"]
}

Build Splits Around Real Leakage Boundaries

Random rows are invalid when turns from one conversation, variants of one incident or the same source document appear across development and test. Group related examples, reserve a later time window and retain critical rare cases as named stress tests.

Document population, sampling weights, label rubric, agreement, exclusions, redaction and intended use. The dataset guide supplies a concrete schema; Datasheets for Datasets provides the broader documentation rationale.

Keep Prevalence Data and Stress Tests Separate

A benchmark often needs more rare failures than natural traffic provides. Enrichment is legitimate, but it changes class prevalence and therefore changes accuracy, predictive values and expected cost. Keep a natural-prevalence set for deployment estimates and a separately labeled stress suite for failure discovery. Do not average them into one attractive score.

Freeze the final test set before prompt, question or threshold tuning. Development errors can guide iteration; test errors are spent evidence. If the test set influences a rewrite, record that use and create a new untouched holdout for the next comparative claim.

Build Reference Labels Without Letting the Systems Grade Themselves

A model-generated reference answer is another evaluator, not ground truth. Human labels can also be inconsistent, so report reviewer agreement before adjudication and preserve unresolved ambiguity as a first-class state. For subjective criteria, a distribution of reviewer judgments may be more honest than one forced label.

Keep label production separate from threshold fitting. The evaluator scorecard explains validity and reviewer controls; the calibration guide explains why labels must match the event represented by a probability.

Pin Complete Systems While Preserving Native Interfaces

Equivalent evidence does not require identical prompt syntax. Each system should use its appropriate interface, while the decision meaning and information available remain equivalent.

Run the simplest credible baseline: an exact rule where the condition is deterministic, a trained classifier where labels already exist, and human review where judgment remains unavoidable. The alternatives guide maps those boundaries. Comparing Jev only with an unnecessarily large generative model can make both the quality and cost conclusion meaningless.

Use One Observable Harness and Randomize Execution Order

The sketch is harness logic, not a Jev SDK example. Its important properties are paired evidence, randomized system order, raw-output retention and explicit failure records. Randomizing order reduces bias from provider warm-up, rate limits or traffic changes. Interleave systems over time instead of completing every Jev request on Monday and every comparator on Friday.

Do not let one system receive more evidence through hidden history, retrieval or tool access. Hash the serialized evidence presented to each adapter, log the resolved model and provider response identifiers, and separate cold-start from steady-state latency when either matters. The HTTP architecture guide covers observable request records; error handling covers retry boundaries.

for example in frozen_test_set:
    evidence = project(example, projection_version)
    for system in randomized(active_systems):
        started = monotonic_clock()
        result = call_with_recorded_deadline(system, evidence)
        append_raw({
            "example_id": example.id,
            "system_id": system.config_hash,
            "attempt": result.attempt,
            "native_output": result.body,
            "error_class": result.error_class,
            "latency_ms": elapsed(started),
            "usage": result.usage,
        })

Report Decision Quality, Uncertainty and Operations

Match metrics to the output. Choice needs a confusion matrix and class-level errors; ordered Score needs distance-aware error; independent Nouls need per-label precision and recall. Calibration metrics apply only when the tested probability and observed event have the same meaning. scikit-learn documents standard classification and calibration measures; the primitive guide explains the Jev output shapes.

For selective automation, plot error among automatically decided cases against coverage, the share of all eligible cases decided automatically. Compare systems at the same coverage or same observed risk. Otherwise a system can appear safer simply because it sends far more work to humans. The abstention guide turns that curve into review-capacity planning.

• Confusion matrix and class-specific precision, recall or ordinal error.
• Brier score, log loss and reliability by probability band when available.
• Automation coverage, selective risk and review volume at each threshold.
• Repeated-run agreement and paired perturbation sensitivity.
• p50, p95 and p99 latency including failures and retries.
• Input/output, fallback, review and incident cost.

Attach Uncertainty to Every Comparative Claim

Report denominators, confidence intervals and paired differences, not just rounded percentages. Bootstrap or resample at the independent grouping unit; treating many turns from one conversation as independent overstates certainty.

Avoid declaring a winner from overlapping uncertainty intervals or one favorable slice. State the smallest practically meaningful difference before testing and include invalid requests as system outcomes rather than silently dropping them.

Compare Paired Outcomes, Not Two Unrelated Headline Scores

Because every system sees the same examples, analyze the within-example difference. A paired bootstrap preserves much of that comparison signal. If examples are clustered by conversation, customer or document, resample the highest independent unit that could share information. Publish the resampling unit, seed, repetitions and interval construction.

An interval describes sampling uncertainty under the benchmark design; it does not cover label error, future traffic, silent provider changes or a biased dataset. Pair it with slice tables and a qualitative error review. A small average win can conceal a serious regression for one language, tool or high-cost failure class.

# One bootstrap draw samples independent groups, not individual turns.
for b in range(10_000):
    sampled_groups = resample(group_ids, replace=True)
    rows = all_rows_for(sampled_groups)
    delta[b] = metric(rows["jev"]) - metric(rows["baseline"])

interval = percentile(delta, [2.5, 97.5])

Treat Failures and Review Work as Benchmark Outcomes

A successful-response-only benchmark rewards fragile systems. Publish the full denominator from eligible example to final action, including transport failures, exhausted retries, invalid outputs and fallback decisions. Use the same retry and deadline budget for systems unless the experiment explicitly compares operational policies.

Calculate total workflow cost at the chosen operating point: model usage, retries, fallbacks, human handling and the expected cost of wrong actions. Token price alone is not the deployment comparison. The Jev pricing guide covers request-level arithmetic; the human-review comparison covers queue economics.

Publish Enough Evidence to Reproduce the Result

A useful result bundle contains a protocol, dataset card, immutable example IDs, label and adjudication records, system manifests, raw native outputs, normalized decisions, analysis code, environment lockfile and a generated report. When evidence cannot be released, publish hashes, field definitions, aggregate counts and a runnable synthetic fixture so readers can still audit the method.

Re-run after a resolved model, question, state projection, comparator prompt or target population changes. Convert important errors into the regression suite, but keep that suite separate from the untouched evidence used for a future comparative claim. Reproducibility guidance from Pineau et al. supports exposing code, dependencies and experimental details; NIST’s AI RMF frames measurement as part of an ongoing risk process rather than a one-time leaderboard.

1. Release the protocol, configuration and code commit.
2. Release labels, examples and raw outputs where privacy and licenses permit.
3. Provide redacted substitutes and exact exclusion counts where they do not.
4. State model aliases, resolved versions, providers, regions and dates.
5. Separate vendor claims, independent results and unreplicated observations.
6. Add a correction log when reruns or model updates change the finding.