evaluate the jev evaluator

First Prove the Criterion Means What the Product Needs

Agreement is meaningless if the target label is wrong. Connect the natural-language question to an operational construct, available evidence and downstream action. “Helpful” may mix correctness, completeness and tone; separate them unless the product intentionally acts on the composite.

Ask whether reviewers can decide from the same evidence Jev receives. High disagreement or frequent missing-evidence labels signals a measurement-design problem before it signals a model problem.

Define the Unit, Label and Action as One Contract

An evaluator can be statistically accurate and operationally useless if it measures the wrong unit or arrives after the decision it is meant to influence. Write this contract before selecting a primitive. The agent-trajectory guide covers evidence boundaries; evals versus policies covers timing and authority.

Use a Scorecard That Spans Model and System Behavior

Start Error Analysis from the Confusion Matrix

Name the positive class from the action perspective. If “yes” means policy compliance on one page and policy violation on another, generic precision and recall labels invite mistakes. Publish the class definition and raw counts with every percentage.

Accuracy can be nearly perfect when the important failure is rare. For 10,000 sessions with 50 true violations, an evaluator that says “no violation” every time is 99.5% accurate and has zero violation recall. Release gates should therefore target the costly class and relevant slices.

# Example for a Noul criterion where "yes" means a harmful action occurred
                 human yes   human no
Jev yes              TP          FP
Jev no               FN          TN

recall_harm = TP / (TP + FN)      # how much harmful behavior was found
precision_harm = TP / (TP + FP)   # how many flags were valid
false_allow_rate = FN / (TP + FN) # harmful cases missed by the evaluator

Quantify Uncertainty in the Reference Labels

Use at least two independent reviewers on a meaningful subset and report agreement by class. Adjudication may create a release label, but it should not erase the original disagreement. Where real outcomes exist, report both rubric agreement and outcome prediction.

Confidence intervals matter, especially for rare failures. A perfect result on ten critical cases is weak evidence. Bootstrap at the correct grouping unit so repeated turns from one session are not treated as independent samples.

Separate Reviewer Agreement from Adjudicated Truth

Report confusion between reviewers by class, not only one kappa-like scalar. A rare but important label can have poor agreement while the overall number looks acceptable. Inspect every disagreement that changes a consequential action and revise the rubric before measuring Jev.

Evaluate the Distribution Before Choosing Thresholds

For Noul and Choice, compute proper probability scores and reliability diagrams in addition to top-answer accuracy. For Score, inspect the full ordinal distribution: two cases can share a weighted mean while expressing different uncertainty.

Choose allow, review and block thresholds on calibration data using explicit error costs and review capacity. Report risk versus automated coverage, then evaluate once on untouched test data. See Jev thresholds and calibration.

Compare Candidates Across the Whole Coverage Curve

The table describes the expected trade, not measured results. Build the real curve by replaying held-out distributions through candidate thresholds. Compare Jev, another judge and the current workflow at equal coverage or equal error; comparing each at a different operating point can manufacture a winner.

Include unscorable and provider-failure cases in the denominator. A system that drops difficult inputs can appear accurate because its coverage silently shrinks. The abstention guide explains the application states behind this curve.

Stress Invariances and Documented Weaknesses

Do not assume the model should be invariant to changes that alter evidence. Pair each perturbation with a written expectation. TypeSafe’s documented limitations should become targeted fixtures, not a generic disclaimer.

• Paraphrase the question without changing its meaning.
• Reorder irrelevant fields and remove distracting state.
• Test missing, contradictory and adversarial evidence explicitly.
• Include counting, dates, numeric comparison and multi-hop cases where relevant.
• Repeat a sample to measure instability and provider failures.
• Evaluate language and domain slices actually present in production.

Analyze Candidate Changes as Paired Decisions

When comparing evaluator A with B, retain both outputs for every same held-out session. Count A-only errors, B-only errors, shared errors and corrected errors. This paired view is more informative than subtracting two aggregate accuracies because it exposes whether the candidate fixes the failures that matter or merely exchanges them.

Estimate uncertainty by resampling the independent unit—usually session, conversation, customer or incident. SciPy documents bootstrap confidence intervals, but the grouping choice remains yours. Do not resample individual tool events when several belong to the same session. Predeclare the primary metric and smallest meaningful improvement.

Compare Against the Workflow Being Replaced

A useful evaluator need not maximize standalone accuracy if it safely automates obvious cases and routes uncertainty well. Compare it with existing rules, a generative judge or human-only review on total error cost, review minutes, latency and failure recovery.

Keep explanation requirements explicit. Jev returns typed probabilities, not a rationale. If reviewers need supporting evidence, preserve trace references or use a separate explanatory stage rather than presenting a generated explanation as Jev’s reasoning.

Store a Release Scorecard That Can Be Audited Later

This is an illustrative application artifact, not a Failproof wire format. It records enough context to understand why a candidate moved to shadow without presenting a point estimate as certainty. Store slice tables, raw paired outputs and the configuration hash alongside it.

{
  "evaluator": "refund-completion@v3",
  "model": "jev-1.13.0",
  "dataset": "refund-heldout@2026-09",
  "unique_sessions": 420,
  "primary_gate": {
    "metric": "false_allow_rate",
    "estimate": 0.012,
    "upper_confidence_bound": 0.024,
    "maximum": 0.030
  },
  "coverage": {"automated": 0.71, "review": 0.26, "unscorable": 0.03},
  "critical_fixture_failures": 0,
  "decision": "shadow"
}

Make the Release Decision Reproducible

Use explicit outcomes: reject, revise, shadow, limited release or promote. “Looks good” is not a release state. Shadow means results are recorded but cannot change the user-visible action; limited release names the slices and traffic share where evidence applies.

The release report should state what remains unsupported. A strong result on English refund sessions does not authorize multilingual safety screening. The regression guide turns the approved scorecard into CI gates, and drift monitoring tests whether its assumptions continue to hold.

1. Pre-register primary metrics, slices, gates and exclusion rules.
2. Pin the complete evaluator and dataset versions.
3. Run once on the untouched test set and retain raw outputs.
4. Document failures, uncertainty and unsupported deployment contexts.
5. Shadow on live traffic and verify review capacity and outcomes.
6. Promote with monitoring, rollback and a scheduled revalidation owner.