jev score vs choice

Use Score for Levels and Choice for Categories

Use Score when the answers are levels on the same scale and can be put in a clear order. For example, support-ticket urgency can be routine, time_sensitive or critical. Critical is more urgent than time-sensitive, and time-sensitive is more urgent than routine. Each level still needs a written definition based on evidence in the ticket.

Use Choice when the answers are different categories rather than higher and lower levels. For example, billing, technical_support and account_access are different destinations. Billing is not above or below technical support, so turning those categories into a Score would give their numeric order a meaning that does not exist.

Score is also not the right tool merely because the input contains numbers. If the question is “How many failed payments occurred?” or “Is the balance greater than $500?”, calculate that exactly in code. Use Jev Score only when text evidence must be mapped to ordered descriptive levels, such as low, medium and high urgency. TypeSafe documents weaknesses around counting, arithmetic and numeric representations.

A Useful Score Rubric Defines Observable Transitions

The operational wording gives labelers and the model common evidence. Adjacent levels should differ by one coherent dimension. If “high” mixes urgency, customer tier and emotional tone, the expected score has no stable meaning. Ask separate questions and combine them in code when the action truly depends on all three.

Interpret the Weighted Score Without Inventing Precision

Suppose levels are encoded 0, 1 and 2 and the returned probabilities are 0.10, 0.60 and 0.30. The weighted score is 0×0.10 + 1×0.60 + 2×0.30 = 1.20. That summarizes the distribution for ranking or thresholding. It does not establish that the case has exactly 1.2 units of urgency or that the distance from low to medium equals medium to high.

Retain level probabilities. Two cases can share an expected score while one is concentrated around the middle and another splits between extremes. Those distributions may warrant different review decisions. The confidence guide explains this distinction.

When Choice Is the Better Model

Do not assign arbitrary numbers to nominal classes just to obtain a Score. A weighted average between “billing” and “security” is meaningless. Conversely, a Choice among low, medium and high discards the intended ordinal structure even if the labels look familiar.

• Routing among teams, models or tools whose names have no intrinsic order.
• Selecting a policy category where severity is a separate dimension.
• Choosing the passage that best answers a query from a bounded shortlist.
• Selecting a workflow state when transitions are not a single monotonic scale.

Evaluate Ordinal Errors Differently from Nominal Errors

For Choice, use a confusion matrix and class-specific costs. For Score, preserve the full level distribution and report ordinal errors: how often the prediction crosses a consequential boundary and how far it moves from the adjudicated level. Calibration can be examined for each level or for cumulative events such as “at least high.”

Thresholds belong to the downstream action. A Score above 1.4 might route a case to review only after that operating point is validated on held-out outcomes. Follow the threshold guide, and use separate veto questions for risks that must not be averaged away.