jev regression testing

Freeze the Evaluator, Not Only the Input Text

A Jev result depends on the state projection, question wording, criteria, primitive, model version and provider path. Store those fields in a manifest beside each fixture. If an alias resolves to a new model, that is an evaluator change even when application code is unchanged.

Fixtures should preserve the semantic evidence needed for the criterion while removing secrets and unstable fields. The state-design guide explains projection; the question guide explains why wording is part of the instrument.

{
  "suite": "refund_completion_v3",
  "model": "jev-1.13.0",
  "projection_version": "refund_trace_v2",
  "question_version": "completion_v3",
  "thresholds": {"pass": 0.82, "review": 0.55},
  "fixtures": [{"id": "completed-017", "expected_band": "pass"}]
}

Use Four Test Layers with Different Promises

The contract suite should not call a provider. It validates projection code, question serialization, response parsing and action logic against hand-built fakes. Live smoke tests prove that credentials, route and current model still work, but a handful of cases cannot establish evaluator quality. The release suite is the evidence-bearing gate.

Keep the scheduled alias replay informational until a candidate is intentionally promoted. If jev-latest moves, compare it with the pinned production model on the same fixtures rather than letting an alias silently redefine the release baseline. The model-alias guide covers that separation.

Split CI by Trust Boundary and Expense

Do not expose a production API key to untrusted fork pull requests. Put live jobs behind protected environments or an equivalent secret boundary, give the credential the smallest available scope and use synthetic state. GitHub Actions matrix jobs can separate pinned models, provider routes and language slices while preserving one manifest.

Assert Decisions and Distributions at the Right Precision

Exact probability snapshots are brittle and imply a determinism guarantee the public contract does not make. Assert the expected answer, an approved action band, or bounded movement from a recorded baseline. For a Choice, inspect the full distribution and margin; for a Score, test ordinal behavior as well as the weighted mean.

A fixture near a threshold is valuable, but mark it as a boundary case rather than forcing it to pass forever. Use paraphrases and repeated live runs to estimate instability. The threshold guide separates model probability from application action.

def assert_jev_case(case, result):
    assert result.model == case.expected_model
    assert set(result.distribution) == set(case.expected_labels)
    assert abs(sum(result.distribution.values()) - 1.0) < 1e-6

    action = apply_thresholds(result, case.threshold_version)
    assert action in case.allowed_action_bands

    if case.maximum_probability_shift is not None:
        assert (
            max_abs_delta(result.distribution, case.baseline)
            <= case.maximum_probability_shift
        )

Test Invariants and Metamorphic Relationships

These are not universal promises made by TypeSafe; they are application requirements to test. Define exceptions before running them. For example, a paraphrase can legitimately alter an ambiguous criterion, which is itself a reason to rewrite the criterion rather than bless whichever wording scores best.

TypeSafe’s jaggedness page supplies version-specific failure patterns. Convert every relevant pattern and every production incident into a named stress fixture, while keeping a separate untouched set for estimating generalization.

Make Failures Diagnostic, Not Merely Red

• Include ordinary positives, ordinary negatives, boundary cases and documented jaggedness cases.
• Tag fixtures by criterion, language, source, severity and evidence availability.
• Print prior and current distributions, resolved models and configuration hashes on failure.
• Distinguish provider errors, invalid responses, missing evidence and semantic regressions.
• Require a reason and reviewer for baseline updates; never auto-bless changed outputs.

Gate Releases on Aggregate Risk and Critical Cases

One changed fixture does not necessarily mean a harmful regression, and unchanged aggregate accuracy can hide a critical failure. Combine hard gates for high-consequence fixtures with confidence intervals for aggregate metrics and slice-specific limits. Select the gates before observing the candidate result.

Keep the test set separate from question development. If a failure informs a rewrite, move that example into development and obtain fresh held-out evidence. The dataset guide defines leakage controls and split lineage.

Represent Release Policy as Data

This is an application manifest, not a TypeSafe or Failproof configuration format. Its purpose is to make the release decision reviewable. Store the calculation code, denominator and confidence interval beside every gate; a threshold name without a formula is not reproducible.

Use one-sided confidence bounds when the gate limits harmful error, and resample at the independent unit such as session or customer—not every event from one trace. When a critical slice is too small for a stable rate, use hard fixtures plus mandatory review rather than presenting a noisy estimate as proof of safety.

suite: refund_completion_v3
candidate: jev-1.13.0
gates:
  - metric: critical_false_allow_rate
    slice: high_value_refund
    maximum: 0.005
  - metric: selective_risk
    at_coverage: 0.70
    maximum: 0.03
  - metric: scorable_coverage
    minimum: 0.98
  - metric: p95_latency_ms
    maximum: 750
  - fixture_tag: must_block
    failures_allowed: 0

Classify a Red Build Before Changing the Baseline

Never solve a red build by repeatedly calling the model until it returns the preferred answer. That selects an outcome after observation and conceals instability. Preserve every attempt, compute the repeated-run flip rate where needed and decide whether the production workflow can tolerate it.

Control CI Time, Spend and External Flakiness

Run deterministic contract tests freely. Put live tests behind explicit markers, use protected secrets, bound concurrency and record rate-limit failures separately. A small smoke set can run often; a representative suite belongs at release or on a schedule.

Do not retry until a preferred semantic answer appears. Retries are for documented transient transport failures and must remain visible in cost and latency. See errors, retries and limits.

Make Every Release Produce an Evaluator Change Report

The ML Test Score argues for production readiness across data, model, infrastructure and monitoring rather than one offline metric. Apply the same discipline here: a Jev regression suite must test the evaluator artifact and the software path that turns it into a decision.

After promotion, link the report to live drift dashboards and sampled outcomes. Regression evidence describes the frozen set at release time; it does not replace production drift monitoring.

• Base and candidate model, provider, question, projection, thresholds and code commits.
• Dataset version, unique-case count, slice counts and label provenance.
• Paired changes in class errors, calibration, selective risk and coverage.
• Every critical fixture outcome and the largest probability movements.
• Invalid responses, retries, p50/p95 latency and observed total cost.
• Approval, rationale, rollout scope and exact rollback target.