test jev evaluators with pytest

This Is a Harness Pattern, Not a Claimed Integration

pytest can call any Python evaluator wrapper, but neither this article nor its sample names establishes an official TypeSafe pytest plugin or a Failproof setup command. Adapt the pattern to the current official TypeSafe SDK and to beta onboarding details supplied by Failproof.

Keep one application function responsible for building state and questions and normalizing the response. Tests should target that boundary so provider code does not leak across the suite. The Python guide covers the source-verified SDK shape.

Parametrize Semantic Fixtures with Explicit Bands

The fake proves fixture wiring, parsing and action-band logic. It does not prove semantic quality. Give every case a stable ID, expected-label reason and leakage-group ID in the dataset source.

import pytest

CASES = [
    pytest.param(
        {"request": "Refund the duplicate charge", "refund_created": True},
        "pass",
        id="completed-refund",
    ),
    pytest.param(
        {"request": "Refund the duplicate charge", "refund_created": False},
        "fail",
        id="claimed-but-not-completed",
    ),
]

@pytest.mark.parametrize(("state", "expected_band"), CASES)
def test_completion_contract(fake_evaluator, state, expected_band):
    result = fake_evaluator.evaluate_completion(state)
    assert result.band == expected_band
    assert set(result.distribution) == {"true", "false"}

Make Live Tests Opt-in and Visibly External

The wrapper and field names are deliberately application-owned pseudocode, not undocumented TypeSafe SDK syntax. Run the live marker only where a protected credential and network access exist. Fail the live job on missing configuration rather than silently substituting a fake.

@pytest.mark.live_jev
@pytest.mark.asyncio
@pytest.mark.parametrize("case", RELEASE_CASES, ids=lambda case: case.id)
async def test_live_jev_release(case, jev_evaluator):
    result = await jev_evaluator.evaluate(case.state)
    assert result.resolved_model == "jev-1.13.0"
    assert result.band in case.acceptable_bands, result.debug_summary()
    assert abs(sum(result.distribution.values()) - 1.0) < 1e-6

Avoid Exact Semantic Snapshots

Keep Transport Failures Separate from Semantic Failures

Report authentication, timeout, rate-limit and invalid-response failures as infrastructure outcomes. Do not turn them into a semantic false label, and do not retry until the model returns the expected answer. Bound retries at the wrapper and include attempt count in diagnostics.

Use pytest markers to separate contract, smoke and release suites. Limit parallel live workers to current provider constraints and avoid sharing mutable client fixtures unless the SDK documents that behavior.

Protect CI Credentials and Production Evidence

• Use the CI secret store; never place an API key in a fixture or recorded cassette.
• Run forked pull requests without privileged live-test secrets.
• Redact traces before committing fixtures and scan artifacts for credentials.
• Set an explicit spend and request cap for the live job.
• Retain only the evidence required for failure diagnosis under the project policy.

Print the Context Needed to Reproduce a Failure

A useful failure includes fixture ID, question and projection versions, requested and resolved model, provider, full distribution, action threshold, latency and attempt count. It should not dump unredacted customer state into a public CI log.

Promote passing fixtures into the broader regression-testing architecture, and use the dataset guide to manage provenance and splits.