jev confidence vs probability

Read the Fields by Primitive

A Choice can be low-confidence because several acceptable alternatives split the distribution. That does not always make the decision unusable: selecting a harmless color theme may tolerate ambiguity, while routing an irreversible action should not. Confidence must be interpreted with the action and candidate set.

Probability, Concentration and Correctness Answer Different Questions

Only the third question establishes calibration, and it cannot be answered from one response. A concentrated distribution describes the model output for the supplied state and answer set. It does not validate the state, criteria, model version or application threshold.

Keep the complete distribution even when the UI displays one answer. The runner-up probability and the gap between the top two candidates can explain why two winning labels with the same top probability have different ambiguity.

A Noul Value Is Not Intensity

A Noul of 0.5 means yes and no have similar modeled probability. It does not mean the condition is present at medium strength. If the application needs severity, define an ordered Score rubric or separate semantic dimensions. If it needs several labels that can coexist, ask several Nouls.

Three Examples Show Why the Fields Are Different

The word “confidence” is tempting to read as a universal trust score. In TypeSafe’s interface it summarizes concentration for a particular Choice or Score distribution. It does not know whether the state omitted decisive evidence, whether criteria overlap, or whether the question belongs in the model at all.

Worked Choice Example: The Same Winner Can Imply Different Risk

Do not compare 0.80 across the second and third rows as if the experiments were identical. Adding, removing or rewriting candidates changes the decision being asked. Store the candidate-set version with the result and recalibrate after taxonomy changes.

If labels may coexist—billing and account security, for example—one Choice forces competition. Separate Nouls may better represent the task. The Choice-versus-Noul guide works through that modeling decision.

Worked Score Example: Preserve the Distribution Behind the Average

The expected score is a useful summary for ranking or monitoring, but it hides shape. A distribution split between routine and critical can have the same average as one concentrated on time-sensitive. Store and inspect the level probabilities, not only the weighted value.

The numeric spacing—0, 1 and 2—is chosen by the application. It assumes adjacent rubric levels are equally spaced for the calculation. Jev is not measuring minutes or dollars, and the Score guide explains why level wording must carry the semantics.

levels = {"routine": 0, "time_sensitive": 1, "critical": 2}
distribution = {"routine": 0.10, "time_sensitive": 0.65, "critical": 0.25}

expected_score = sum(
    levels[level] * probability
    for level, probability in distribution.items()
)
# 1.15 on this application-defined ordinal scale

A Concentrated Wrong Answer Can Still Have High Confidence

Suppose a routing Choice contains only billing and technical, but the request actually needs an account-security team. Jev can strongly prefer billing over technical because it must compare the supplied candidates. The distribution may be concentrated even though the candidate set is wrong. Add a no-match path or separate fit test.

The same issue appears with missing state. If the relevant tool result was never supplied, a clean-looking distribution does not recover it. Validate evidence completeness before interpreting model numbers. See Choice and state design.

Turn Distributions into Actions Carefully

Thresholds belong to application code and depend on consequences. Fit them against representative labeled outcomes, then report both coverage and errors. A high threshold may reduce false actions while sending more traffic to review. A model update, question rewrite or population shift can invalidate the old operating point.

• Log raw distributions instead of only the winning label.
• Do not compare confidence across unrelated questions as if it were one quality score.
• Do not use confidence as authorization for a protected action.
• Keep a review or fallback path for uncertainty and service failure.

Store Model Output Separately from the Application Decision

This is an application record, not the TypeSafe response schema. Keeping raw model fields separate from derived action fields permits threshold replay and shows whether a later change came from Jev, the question or policy code. Never overwrite the original answer with review or fail; those are application states.

{
  "primitive": "choice",
  "answer": "billing",
  "distribution": {
    "billing": 0.55,
    "technical": 0.45
  },
  "confidence": "<native Jev value>",
  "question_version": "support-route-v4",
  "model_resolved": "jev-1.13.0",
  "threshold_version": "support-route-bands-v2",
  "application_action": "human_review"
}

Calibration Connects Probabilities to Observed Outcomes

To test a Noul probability, group many comparable predictions into ranges and compare each range with observed yes rates. If cases near 0.8 are positive only half the time, that deployment is overconfident even though individual answers may look plausible. Choice and Score need equivalent task-specific analysis of their distributions.

Calibration belongs to the full evaluator: model, question, criteria, state projection and traffic. Changing any part can invalidate the old threshold. The calibration guide provides reliability diagrams and scoring metrics; thresholds converts validated probabilities into actions.

Use Different Diagnostics for Different Failure Claims

scikit-learn’s calibration documentation illustrates reliability diagrams and proper scoring rules, but those methods must be applied to the exact Jev question and target population. A good Brier score can combine calibration and discrimination effects; inspect the reliability curve and task errors rather than treating one scalar as the whole evaluation.

NIST’s measurement guidance emphasizes documenting metrics, uncertainty and limitations. Record denominators and confidence intervals, especially for rare high-consequence slices. The evaluate-the-evaluator guide turns these diagnostics into release evidence.

Monitor the Operating Point, Not a Confidence Average

A higher average confidence after a release is not automatically an improvement; the system may simply have become more concentrated while accuracy stayed flat or worsened. Confirm changes with labels or outcomes and replay the old and new instruments on the same evidence. The drift guide defines that workflow.

• Probability and confidence distributions by criterion, answer and important slice.
• Automation, review, refusal and unscorable coverage with stable denominators.
• Delayed errors, overrides and downstream outcomes for every action band.
• Resolved model, provider, question, candidate-set and projection versions.
• Missing evidence, invalid responses, retries and latency separately from semantic uncertainty.