composite jev scoring

Composite Scoring Is Application Logic

A workflow may care about task completion, answer grounding, policy compliance and user impact. One broad “quality” prompt hides which dimension failed and makes remediation difficult. TypeSafe’s composite-scoring pattern instead asks atomic questions and combines their outputs in ordinary code.

The formula is a product policy, not a Jev primitive. It expresses tradeoffs and must be reviewable independently from the model. Version weights, transformations, vetoes and thresholds just as you version the questions and state projection.

Define What the Composite Represents Before Writing a Formula

A number cannot be validated until its claim is named. “Quality score” often mixes a descriptive index, a prediction and a policy decision. Write the target population, outcome, time horizon and downstream action first. If no real-world event corresponds to the number, do not label it confidence or probability.

Keep measurement separate from preference. Jev components estimate bounded semantic propositions or ordered rubrics; weights express how the product trades those dimensions. Changing a weight is a policy change even when every model output remains identical.

A Transparent Weighted Calculation

This formula is illustrative, not a recommended universal weighting. Completion and grounding receive more weight because the hypothetical product owners assigned them higher cost. A policy violation bypasses the average: excellent clarity must not cancel a serious compliance signal.

components = {
    "completion": completion_noul,        # 0..1
    "grounding": grounding_noul,          # 0..1
    "clarity": clarity_score / 2,         # rubric levels 0..2 normalized
}

weights = {"completion": 0.45, "grounding": 0.40, "clarity": 0.15}

if policy_violation_noul >= POLICY_REVIEW_THRESHOLD:
    decision = "review"  # veto: never averaged away
else:
    composite = sum(weights[k] * components[k] for k in weights)
    decision = band_for(composite)

Design Components That Are Atomic and Non-Duplicative

Each component should have its own evidence boundary, label rubric and reason to change an action. Correlated components are not automatically invalid, but weights no longer represent independent contributions. Inspect correlation, disagreement and slice errors before claiming that adding dimensions adds information.

Run an ablation: compare the full composite with each component alone and with each component removed. If a component never changes ranking or action on held-out data, it adds latency and maintenance without measured value. The evaluate-the-evaluator guide supplies component validity gates.

Normalization Creates Assumptions

A Noul already lies between zero and one, but its probability is tied to one event. A three-level Score with values 0, 1 and 2 can be divided by two, yet that assumes the ordinal level spacing is useful for this formula. Choice confidence measures concentration and should not be inserted as though it were probability of correctness.

Document directionality, scale and missingness for every component. If one dimension is unavailable, decide whether to fail the composite, renormalize remaining weights, impute conservatively or route to review. Each choice changes the meaning of the score.

Do Not Combine Values Merely Because They Fit Between Zero and One

The confidence-versus-probability guide explains why concentration cannot stand in for correctness. The Score-versus-Choice guide explains why an ordered expected value and a categorical distribution answer different questions. Preserve these meanings through normalization.

Do Not Average Away Hard Constraints

Weighted sums are compensatory: a high value in one component can offset a low value in another. That is appropriate only when the product genuinely accepts that trade. Use conjunctions, minimums or vetoes when one failure is independently unacceptable.

Make Missingness a Branch in the Decision Graph

This application pseudocode makes the order visible: exact constraints, evidence sufficiency, semantic vetoes, completeness and only then a compensatory score. A model timeout, truncated state and genuinely inapplicable criterion are different missingness mechanisms and may require different actions.

Renormalizing remaining weights changes the construct. A response scored only on clarity and style is not a lower-information version of completion-plus-grounding quality; it is a different score. If renormalization is allowed, version and validate each missingness pattern separately.

def decide(measurements):
    if measurements.authorization == "deny":
        return "deny"
    if measurements.required_evidence_missing:
        return "review"
    if measurements.semantic_veto >= VETO_THRESHOLD:
        return "review"

    available = required_components(measurements)
    if not available.complete:
        return "unscorable"

    value = weighted_sum(available.normalized)
    return action_band(value)

Compare the Weighted Sum with Simpler Decision Rules

A learned combiner is not automatically more objective. It encodes the labels, sampling process and loss function supplied to it, and it can exploit spurious correlations between components. Start with an interpretable baseline and adopt a learned layer only when held-out outcomes show a practically meaningful improvement.

Sculley et al. describe how entanglement and hidden consumers create technical debt in ML systems. Composite evaluators have the same risk: one convenient aggregate can spread into dashboards, alerts and release gates until its original assumptions are invisible. Track every consumer and version the semantic contract.

Fit and Validate the Complete Decision Rule

A composite value is not automatically calibrated merely because its inputs are. Test its reliability against the outcome it claims to predict. Report raw component errors and the cases where the formula changes the action. Follow Jev calibration and thresholds.

1. Define the downstream outcome and error-cost objective.
2. Evaluate each atomic question for validity, calibration and slice behavior.
3. Choose weights on development/calibration data with constraints set by domain owners.
4. Compare the composite with simple baselines, including the best single component.
5. Lock weights and thresholds, then report on untouched data and shadow traffic.

Evaluate the Score at Three Levels

Use grouped development, calibration and test partitions. Tune questions and candidate components on development data, fit transformations and action bands on calibration data, then evaluate the locked pipeline once on test. The dataset guide covers leakage boundaries; the benchmark protocol covers paired uncertainty.

Report monotonicity checks and counterfactuals. Improving grounding while holding everything else fixed should not lower an intended quality index. Triggering a veto should never be cancelled by another component. These properties can be unit-tested even before statistical validation.

Store the Ingredients, Recipe and Result

Without components, an aggregate regression cannot be diagnosed and a new formula cannot be replayed. Preserve immutable inputs or governed references, then compute candidate recipes offline before changing production.

• Raw typed answers and full distributions.
• Resolved model, provider, state and question versions per component.
• Normalization transform, weight set, veto and missing-value policy versions.
• Composite value, selected action and threshold band.
• Override, reviewer decision and eventual outcome.

Monitor Components Before the Aggregate

A stable average can hide opposing movement: grounding may fall while clarity rises enough to offset it. Track each component’s availability, distribution, calibration, action-band contribution and slice errors alongside the composite. Alert on veto rate and missingness separately because their operational effects are discontinuous.

When a model, question, projection, transform, weight or threshold changes, replay the same frozen cases and attribute movement by layer. The drift-monitoring guide defines the production event; regression testing defines promotion and rollback evidence.