build a jev evaluation dataset

Define the Construct Before Collecting Examples

A dataset cannot rescue an ambiguous question. Write the observable criterion, evaluation unit, evidence available at decision time, allowed labels and unscorable conditions first. “Good response” is not a construct; “every factual claim is supported by the supplied retrieval passages” can be adjudicated.

Match one row to the action the result will influence. A session-completion row, individual claim row and proposed-tool-call row require different evidence and should not share labels merely because they came from one trace.

Choose the Row Boundary Before Choosing the Jev Primitive

Primitive selection follows the label geometry. Use Choice when exactly one candidate should win, Score when every valid answer lies on one defined order, and independent Nouls when several conditions may all be true. The Choice guide, Score guide and Noul guide define those contracts.

Do not label with information that the deployed evaluator will not have. If reviewers see the eventual refund outcome but the live decision occurs before that outcome exists, the dataset measures a different task. Store later outcomes separately for retrospective validation.

Combine Prevalence Sampling with Deliberate Stress Coverage

Keep source and sampling weight with every row. Do not report a stress-set pass rate as production accuracy, and do not let a traffic average hide rare critical failures.

Inventory eligible traffic before sampling: counts by agent version, customer, language, tool, outcome and time. Sample a natural-prevalence window for deployment estimates, then create separate strata for rare high-cost errors and known boundaries. Record inclusion probability so an enriched set can be reweighted for population estimates.

Deduplicate Related Examples Before Splitting

Hashing text finds exact copies but misses paraphrases. Metadata catches shared origin but can be incomplete. Use both, then inspect unusually similar examples across proposed splits. The goal is to prevent the same underlying evidence pattern from appearing as both practice and exam.

Synthetic variants inherit the group of their source example. Ten paraphrases of one incident add stress coverage, not ten independent observations. Preserve parent_example_id so analysis can cluster them correctly.

Label Blind and Preserve Disagreement

Reviewer disagreement is measurement evidence, not annotation debris. Low agreement may mean the criterion is subjective, the evidence is insufficient or the rubric is underspecified. An evaluator cannot reliably exceed an incoherent target.

Use an explicit unscorable outcome when required evidence is absent. Do not teach reviewers to infer missing tool results or reward a plausible final answer. The state-projection guide explains the evidence boundary, while agent-trajectory evaluation shows why final-message-only labels can misdiagnose a session.

1. Write a rubric with positive, negative, boundary and unscorable examples.
2. Train reviewers on a pilot set and revise the rubric before the final round.
3. Hide Jev outputs, model identity and intended threshold during labeling.
4. Double-label a meaningful subset and measure agreement by class and slice.
5. Adjudicate disagreements with reasons; preserve original labels as metadata.
6. Join delayed operational outcomes without silently replacing human judgments.

Run Annotation as a Measured Process

One agreement percentage hides direction. Review the confusion table: one reviewer may systematically call ambiguous evidence “pass” while another calls it “unscorable.” Report raw agreement and a chance-adjusted statistic where appropriate, but inspect concrete disagreements before trusting either number.

Reviewer confidence is metadata, not the target label. It can prioritize adjudication and expose difficult slices, but it should not be mixed with Jev probabilities. If a criterion remains legitimately subjective, model the reviewer distribution or narrow the claim instead of manufacturing certainty.

Split on the Unit That Could Leak

Random row splits leak when paraphrases, turns from one conversation, repeated incidents or the same customer template appear in several sets. Group by the strongest shared cause: conversation, account, document, incident, prompt template or time window. Use a later time period to test temporal generalization.

Maintain development data for question iteration, calibration data for thresholds and a final untouched test set for one-time release decisions. Once test failures guide a rewrite, those examples are no longer unseen.

Make Split Assignment Deterministic and Auditable

Store assignment code and a manifest, not only three exported files. A deterministic group rule reproduces counts, detects overlap and adds future examples without quietly reshuffling old evidence. Hash frozen labels separately from redacted evidence so privacy transformations do not conceal label edits.

Calibration is not another name for development. Use it to fit thresholds after the Jev question is frozen. Use test once for the release claim. The threshold guide explains action bands; the benchmark methodology explains paired comparison and uncertainty.

{
  "dataset_version": "claim_support_2026-09-v1",
  "group_key": "source_document_id",
  "split_seed": 1847,
  "windows": {
    "development": ["2026-06-01", "2026-07-31"],
    "calibration": ["2026-08-01", "2026-08-31"],
    "test": ["2026-09-01", "2026-09-14"]
  },
  "frozen_test_label_hash": "sha256:...",
  "stress_sets": ["prompt_injection_v2", "missing_citation_v3"]
}

Store Evidence, Lineage and Intended Use

Store a content hash or immutable reference for the exact evidence presented to reviewers and evaluators. A mutable trace URL is not sufficient lineage. Also record parent and leakage-group IDs, sampling probability, original reviewer labels, adjudicated label, rubric version and every evidence transformation.

Keep model outputs in a separate prediction table keyed by example and evaluator version. Mixing predictions into the canonical label file makes blind relabeling harder and invites target leakage into later training.

{
  "example_id": "toolcall-00418",
  "unit": "proposed_tool_call",
  "criterion_version": "authorization_scope_v2",
  "evidence_ref": "trace://session-91/event-8",
  "label": "not_allowed",
  "label_status": "adjudicated",
  "slice_tags": ["refund", "high_impact"],
  "sampling_source": "random_production_2026w34",
  "leakage_group": "account-17",
  "redaction_version": "pii-v4"
}

Minimize and Govern Retained Evidence

Redaction is part of the measurement pipeline. Compare a reviewed sample before and after transformation. If account roles, timestamps or tool arguments disappear, a previously scorable decision may become ambiguous. Record that outcome instead of asking annotators to guess.

Separate the minimum de-identified benchmark from restricted raw evidence. Define deletion propagation so removing a source session also removes or tombstones derived rows, cached projections and exported annotation tasks.

• Collect only fields needed to adjudicate the criterion.
• Redact credentials and personal data before annotation or export.
• Record consent, lawful basis, retention and deletion behavior where applicable.
• Separate raw trace access from de-identified evaluation artifacts.
• Test whether redaction removes evidence needed for a valid label.

Validate the Dataset Before Evaluating Jev

Automate structural gates in CI and review semantic gates with humans. A schema check can prove that every row has a label; it cannot prove the label follows the rubric. When a failure becomes a permanent fixture, copy it into regression testing without pretending the familiar case remains untouched test evidence.

• No example or leakage group appears in more than one split.
• All labels belong to the declared primitive and criterion version.
• Every adjudicated row retains its original reviewer decisions.
• Class and slice counts meet the pre-registered analysis minimums.
• Missing evidence, redaction loss and exclusions have explicit counts.
• A random sample can be reconstructed from immutable evidence references.
• Final test labels stay hidden from question and threshold authors until freeze.

Publish a Dataset Card for Every Version

Document motivation, owners, collection dates, population, sampling, label rubric, reviewer process, agreement, exclusions, splits, privacy handling, known gaps and intended uses. Datasheets for Datasets supplies a broader documentation framework; adapt it to the operational evaluator rather than treating a card as a checkbox.

Version the card with the examples and criterion. Regression testing consumes the frozen set; monitoring drift determines when it no longer represents production.