jev evals for ai agents

An Eval Is a Versioned Measurement, Not a Prompt

Changing any of these components changes the evaluator. A better prompt with an old version number is not the same instrument; neither is a new trace projection behind an unchanged question. Version the complete chain so historical results remain interpretable and a regression can be attributed to evidence, wording, model, provider or decision code.

Map the Eval Question to a Jev Primitive

Use a Noul for one yes/no condition, a Choice for one option from a defined set, and a Score for ordered descriptive levels. Do not hide several independent failure conditions inside one broad question. Split task completion, policy compliance, grounding, tool choice and escalation quality so each has a stable meaning and can be calibrated independently. Combine the results in code only when the operational decision requires it.

Split One Vague Quality Score into Falsifiable Criteria

Atomic does not mean context-free. Each criterion should answer one question, but its state must still contain the evidence a reviewer would need. Keep arithmetic, counts, schema validity and permissions as code evaluators. The trajectory guide, tool-call guide and RAG groundedness guide define those evidence units in detail.

Give the Evaluator the Right Trace State

An agent’s closing message is not the session. Evidence may live in tool arguments, tool results, approvals and intermediate observations. Build the smallest state that preserves the evidence needed by the criterion, identify fields explicitly, and omit unrelated transcript material. TypeSafe documents accuracy loss when Jev 1.13 receives large state full of irrelevant detail; see the full Jev limitations guide.

The state projection is part of the evaluator and must be versioned with the question. If a trace schema changes, the same words can describe a different instrument. Missing evidence should produce an explicit unscorable or review outcome in application logic rather than a guessed pass.

Jev vs an LLM Judge

Neither interface wins every eval. The detailed Jev-as-LLM-judge comparison covers the boundary. Compare them on the same labeled sessions and criteria. Report accuracy or agreement, calibration and confidence, repeatability, coverage at the chosen threshold, latency, cost and the downstream cost of false passes and false failures. An evaluator is only “better” for the decision and operating point actually measured.

Store Enough Data to Replay the Verdict

This is an application record, not a claimed Failproof wire format. Its purpose is to show the fields needed for audit and replay. Keep the raw Jev response and derive pass, fail, review or unscorable in versioned application logic. Link rather than duplicate sensitive trace evidence when retention and access controls differ.

A single Boolean loses too much information. Without the distribution and threshold version, a team cannot tell whether the verdict was clear, replay a changed threshold or measure calibration. Without the evidence reference and projection version, it cannot reconstruct what Jev actually saw.

{
  "evaluation_id": "task_completed.v3",
  "unit_id": "session_8f2",
  "evidence_ref": "trace://session_8f2#final-and-tools",
  "projection_version": "agent_outcome.v2",
  "question_version": "task_completed.v3",
  "provider": "typesafe",
  "requested_model": "jev-1.13.0",
  "resolved_model": "jev-1.13.0",
  "raw_answer": { "noul": 0.87 },
  "derived_status": "pass",
  "threshold_version": "task_completed.web.v2",
  "latency_ms": 241,
  "attempts": 1
}

Match Metrics to the Decision and Its Errors

Agreement with one reviewer is not automatically ground truth. Measure reviewer disagreement, adjudicate important cases and prefer downstream outcomes when they validly represent success. Report the denominator and selection rule: 99% accuracy on 5% of easy traffic is a different system from 90% accuracy at 95% coverage. The evaluator scorecard and calibration guide give the complete protocol.

Jev Evals in the Failproof Workflow

Failproof AI’s evaluation system runs checks on completed sessions and stores results beside traces. Those results can feed alerts and audits that group recurring evidence into findings. A Jev evaluator fits this model as a typed semantic judgment, while deterministic code remains the right evaluator for counts, schemas, permissions and arithmetic. Jev evals vs policies explains when a measured failure should - and should not - move onto the action path.

Jev evals are live in Failproof AI for beta users as of September 22, 2026. Email nikita@befailproof.ai for access.

Production Checklist

• Write one observable semantic criterion and define boundary cases.
• Pin Jev and version the question, criteria and state projection.
• Label representative production slices, including adversarial and missing-evidence cases.
• Fit thresholds away from the final held-out evaluation set.
• Store the full answer distribution and link it to trace evidence.
• Monitor calibration, false-action rate, coverage, latency, errors and cost.
• Replay the eval before changing model aliases, criteria or trace schemas.

Separate Development, Release and Production Monitoring

Repeatedly tuning against the same “test” cases turns them into development data and makes reported performance optimistic. Preserve a locked release set, then add newly discovered failures to a future regression suite. Use dataset construction, regression testing and drift monitoring as the operational sequence.