the short answer
A Jev eval scores completed behavior and helps discover or measure failure patterns. A Jev-backed policy uses a typed semantic judgment before an action, then explicit policy logic decides whether to allow, observe, review or block. Start with the eval, validate it against outcomes, identify a narrow preventable action, backtest the proposed policy and observe it before enforcement.
- Eval timing
- After or outside the action path
- Policy timing
- Before a proposed action executes
- Eval output
- Measurement linked to evidence
- Policy output
- Allow, observe, review or block from explicit code
The Same Model Can Serve Different Jobs
| Dimension | Jev eval | Jev-backed policy |
|---|---|---|
| Question | Did completed behavior meet a criterion? | May this proposed action proceed? |
| Latency pressure | Usually off the agent’s critical path | Synchronous with the action path |
| Failure behavior | Record error and retry/re-evaluate | Must choose fail-open, review or fail-closed |
| Primary risk | Misleading measurement | Wrongly allowing or blocking a real action |
| Rollout | Shadow against labels | Backtest, observe, canary, enforce |
When an Eval Should Become a Policy
A finding is a policy candidate when the failure repeats, appears before a specific action, has enough evidence for a narrow question, and has a safe response. “The agent gives weak answers” is not a policy. “This outbound message contains confidential project context” can be, when deterministic secret scanning, authorization and human review remain around the semantic check.
Do not graduate merely because an evaluator’s aggregate accuracy looks high. Examine false allows at the intended threshold, slice by action and customer, test adversarial state, define outage behavior and confirm that blocking does less harm than allowing.
The Failproof Loop
Failproof’s public product loop is session to audit to finding to issue to policy. Jev can supply a semantic measurement or policy signal within that loop. Jev evals and policies are live for beta users; email nikita@befailproof.ai for access.
- Trace completed sessions and run code, LLM or Jev evaluations.
- Group repeated evidence into a finding and identify the narrow failure mechanism.
- Draft a policy whose inputs and action are explicit.
- Backtest against recorded calls, then run in observe mode.
- Canary enforcement, monitor outcomes and retain rollback.
FAQ
Should I start with a Jev eval or policy?
Start with an eval unless an existing requirement and labeled evidence already establish the decision. Measurement reveals error modes and a safe threshold before the model can affect live actions.
Can the same Jev question power both?
It can be a starting point, but timing and available state may differ. Re-evaluate the exact pre-action state projection and threshold before using a post-session evaluator for enforcement.
Does a Jev policy replace deterministic guardrails?
No. Keep permissions, schema checks, paths, amounts, allowlists and other exact invariants in code. Jev handles the semantic remainder.
Does backtesting rerun the agent?
No. Policy backtesting evaluates the proposed decision over recorded calls or evidence. It does not claim to reproduce the full agent execution or counterfactual outcome.
Sources
Checked against the sources below on September 22, 2026. Model versions, prices and limits change.
- TypeSafe AI docs: Primitives
- TypeSafe AI docs: Confidence
- TypeSafe AI docs: Jev 1.13 jaggedness
- Failproof AI docs: Evaluations overview
- Failproof AI docs: Policies overview