monitor jev evaluator drift

Name the Drift Before Choosing a Metric

These mechanisms can produce similar dashboards but require different fixes. A falling pass rate may reflect worse agent behavior, harder traffic, missing evidence or a stricter evaluator. Preserve versions and denominators so diagnosis remains possible.

Log Enough to Reconstruct the Decision

Attach latency, provider errors, retry count and cost outside the semantic result. Store evidence references under appropriate access controls. The raw distribution permits threshold replay; a Boolean alone does not.

{
  "event_id": "eval-9081",
  "criterion": "task_complete",
  "model_requested": "jev-latest",
  "model_resolved": "jev-1.13.0",
  "question_version": "v4",
  "projection_version": "v2",
  "distribution": {"true": 0.73, "false": 0.27},
  "action_band": "review",
  "state_missing": [],
  "slice_tags": ["billing", "en"],
  "outcome_ref": null
}

Pair Unsupervised Signals with Delayed Truth

Monitor answer prevalence, confidence histograms, entropy, margin, missing evidence, provider failures and action-band coverage promptly. These can reveal change without labels, but none establishes correctness. A stable answer distribution can coexist with silent error, and a shifted distribution can be an accurate response to new traffic.

Join blind audits, overrides, incidents, user corrections and downstream task outcomes when they arrive. Sample every action band, including high-confidence allows, or selective review will hide false negatives. The calibration guide explains reliability by probability bin.

Measure Label Delay and Missing Outcomes Explicitly

Track the fraction of predictions whose outcome window has matured. A recent cohort with few resolved outcomes should not be compared directly with an older complete cohort. Never treat “no recorded complaint yet” as a confirmed success. The dataset guide explains label provenance and the human-review guide covers selective-feedback bias.

Compare Compatible Windows and Meaningful Slices

• Use a stable reference window and account for seasonality and deployments.
• Require minimum sample sizes and uncertainty intervals before paging.
• Slice by criterion, agent version, tool, language, tenant and consequence where allowed.
• Track weighted overall metrics separately from deliberately oversampled audits.
• Annotate question, projection, threshold, provider and model releases on charts.

Alert on Sustained Impact, Not Dashboard Noise

Use warning and critical levels tied to operational consequences: scorable coverage loss, review-queue saturation, critical-slice false negatives, calibration error or provider failure. Require persistence across windows unless one failure is independently catastrophic.

Every alert needs an owner, evidence query, rollback path and expected label delay. Do not silently loosen a threshold to restore a preferred pass rate; that changes the policy rather than fixing drift.

Route Every Alert Through the Same Diagnosis Order

This order prevents a shifted traffic mix from being misdiagnosed as model degradation and prevents a dashboard anomaly from triggering an untested threshold change. Link the investigation, sample IDs and decision to the alert so the next recurrence begins with evidence rather than memory.

1. Confirm sample size, time window and whether the signal survived its persistence rule.
2. Check deployments, resolved model aliases, provider, question, projection and threshold versions.
3. Inspect state missingness, traffic composition and the slices contributing most to the movement.
4. Blind-label a stratified sample from the changed and reference windows.
5. Replay the current and candidate evaluator on frozen evidence and compare paired errors.
6. Roll back a known-bad configuration or keep the candidate in shadow until the cause is established.

Diagnose Through Replay Before Promotion