jev calibration

Calibration Is a Frequency Claim

If a deployed Noul emits probabilities near 0.8 for 1,000 comparable cases, a calibrated system should see the condition hold in roughly 800 of them. It does not promise that a particular 0.8 prediction is true, and it does not mean the condition is “80% severe.” Severity requires an ordered Score; independent yes/no prevalence is a Noul.

Calibration also does not imply discrimination. A system that predicts the base rate for every example can be calibrated yet useless for ranking individuals. Report calibration beside task accuracy, class-specific errors, coverage and downstream utility. The confidence-versus-probability guide explains why Choice/Score concentration is another concept again.

Define the Measurement Instrument Before Collecting Scores

Changing “the response requests a refund” to “the customer wants money back” changes the instrument. So does adding account metadata or replacing a Noul with one option in a Choice. TypeSafe’s jaggedness notes warn that superficially equivalent formulations need not preserve assumed probability identities. Calibrate the exact request you will deploy.

• Pin the Jev model and serving provider.
• Version the exact instruction, criteria and primitive.
• Version the code that projects raw records or traces into state.
• Define the outcome and adjudication procedure independently of Jev’s answer.
• Predefine population slices and the action whose risk matters.

Use Separate Fit, Calibration and Test Data

If the same examples guide prompt wording, threshold selection and the final headline metric, the estimate is optimistic. Keep a true holdout. Time-based splits are often useful when deployment drift matters; group-based splits prevent nearly identical users, documents or sessions from leaking across sets.

Build and Read a Reliability Diagram

Points on the diagonal are empirically calibrated. Points below it indicate overprediction and points above it underprediction under the chart’s convention. Aggregation can hide opposite errors in different languages, customers or task types, so the plot needs counts and slices. Scikit-learn’s calibration guide provides a conventional implementation and terminology.

1. Collect each predicted probability and its binary outcome for one fixed Noul.
2. Partition predictions into fixed-width bins or quantile bins and publish the binning rule.
3. For every bin, plot mean predicted probability against observed positive frequency.
4. Show sample count and uncertainty for each point; sparse bins should not look authoritative.
5. Repeat for important slices and time windows rather than relying only on the aggregate line.

Brier Score and ECE Answer Different Questions

For outcomes y in {0,1} and probabilities p, Brier score is the average of (p - y)^2. Predictions 0.9, 0.7, 0.4, 0.2 with outcomes 1, 0, 1, 0 produce squared errors 0.01, 0.49, 0.36 and 0.04: a Brier score of 0.225. The number becomes meaningful through a baseline, confidence interval and slice analysis—not in isolation.

Monitor Calibration Where Decisions Are Made

Log the full distribution, resolved model, question and state versions, threshold-derived action, human override and eventual outcome. Outcome delay and selective labels matter: if only reviewed cases receive truth, the observed dataset is biased by the old policy. Preserve random audit samples from automated cases.

Re-run calibration after model or alias movement, question edits, schema changes, provider changes and population drift. A calibrated offline evaluator does not automatically validate a synchronous Jev policy; the online path has different missing data, latency failures and selection effects.