how to choose a jev threshold

A Threshold Is a Product Decision, Not a Model Fact

Jev produces a probability or distribution; the application decides what that evidence is sufficient to do. A threshold that is reasonable for queue routing may be reckless for sending money or deleting data. Start with the action, enumerate the consequences of each error, and decide which cases can be reviewed or deferred.

Do not copy 0.5, 0.8 or 0.95 from another implementation. Base rates, question wording, model versions and state distributions differ. Even a calibrated probability does not encode the business cost of being wrong.

Write the Error-Cost Matrix First

Replace the qualitative cells with domain estimates or ordinal severity agreed by owners. Include reversibility, customer harm, security impact, reviewer capacity and delay. The threshold follows the matrix and empirical probabilities; it should not be chosen because a round number sounds confident.

Worked Example: Three-Way Refund-Request Routing

Suppose a Noul asks whether a message explicitly requests a refund. On the calibration set, probabilities at or above 0.88 produce 96% precision and cover 38% of messages; values from 0.55 to 0.88 cover 17% and go to review; lower values continue through ordinary support. Those figures are illustrative—not Jev benchmarks—but show what must be reported together.

If an automatic path only tags the ticket, lower precision may be acceptable. If it initiates a financial action, deterministic eligibility and amount checks still run, and the semantic signal may never be sufficient without approval. Thresholds narrow uncertainty; they do not grant authority.

Fit Thresholds Without Contaminating the Test Set

For a Choice, a useful policy may depend on both the selected option and its probability or confidence, but candidate completeness must be checked separately. For a Score, thresholds operate on a probability-weighted ordinal construct whose levels need stable meaning. The primitive guide prevents thresholds from papering over a badly chosen question type.

1. Freeze the model, question, criteria and state projection.
2. Use training/development data to design the instrument.
3. Search candidate bands on a separate calibration split using the declared cost objective.
4. Lock the bands and report once on a held-out test set, including uncertainty and slices.
5. Shadow on live traffic and retain random audits from every band.

Report Risk and Coverage Together

Raising a threshold usually reduces automated coverage and changes the composition of accepted cases. Plot risk among automatically acted cases against coverage across candidate thresholds. Also report false-action and missed-action rates, review volume, queue delay, overrides and outcomes by slice. A headline precision value without coverage can hide a system that abstains on nearly everything.

If probabilities are miscalibrated but rank cases well, a calibration mapping may improve decision use. Fit it without touching the final test set and version it as part of the policy. The calibration guide covers reliability diagrams and proper scoring rules.

Thresholds Do Not Handle Service Failure

Timeouts, 429s, malformed responses and missing state are not low-confidence predictions. Route them through explicit failure behavior chosen by consequence: fail closed, fail open, use a deterministic fallback, or require review. Log the distinction so reliability incidents do not appear as model uncertainty.

Monitor drift in band volumes as an early signal. If the review band doubles overnight, pause automatic expansion and inspect traffic, state projection, alias resolution and provider behavior. For runtime use, follow the Jev policy architecture; for completed-session measurement, follow Jev evals.