the short answer
Jev 1.13 is designed for focused semantic judgments, not generation or exact computation. TypeSafe documents weaknesses with literal wording, counting, arithmetic, numeric representations, date comparison, multi-hop indirection, long irrelevant state, adversarial content, contradictory criteria and assumed probability identities. Keep deterministic work in code, minimize state, write direct criteria and test every consequential decision on representative data.
- Applies to
jev-1.13, reviewed by TypeSafe September 17, 2026- Strongest fit
- Focused common-sense semantic judgments
- Weakest fit
- Generation, exact math/counting and multi-hop reasoning
- Security warning
- Adversarial state can steer the answer
Documented Jev 1.13 Failure Modes
| Failure mode | Safer design |
|---|---|
| Literal reading and complex negation | Write the exact direct condition and explicit boundaries |
| Counting, arithmetic and numeric comparison | Compute in deterministic code |
| Date ordering and duration | Extract bounded parts, then use a date library |
| Several layers of indirection | Split into focused steps or use a reasoning model |
| Large state with irrelevant detail | Retrieve/filter first and send only needed evidence |
| Adversarial instructions inside state | Defense in depth, narrow criteria and adversarial tests |
| Contradictory instructions and criteria | Make criteria an aligned extension of the instruction |
| Text generation | Use a generative model; Jev only selects bounded candidates |
Do Not Assume Probability Identities
TypeSafe documents that equivalent-looking formulations can return values that do not obey an expected arithmetic identity. A Noul for a label is not interchangeable with that label’s Choice probability. A positive Noul and a separately asked negated Noul need not sum to one. They are separate model questions with different framing.
Do not transfer a threshold fitted for one primitive or wording to another. Treat the exact question, criteria, primitive and model version as the instrument you evaluated. When any of them changes, replay the labeled set.
Turn Each Limitation into a Regression Fixture
A warning becomes operational only when the team can reproduce it and detect its return. Store expected behavior, exact state, question, resolved model and result distribution. The regression-testing guide explains how to avoid brittle exact-probability snapshots.
| Weakness | Test pair | Expected control |
|---|---|---|
| Literal wording | Direct criterion versus idiomatic paraphrase | Prefer the clearest validated wording |
| Counting | Zero, one and many repeated events | Compute event count in code |
| Dates | Before/after and ambiguous locale formats | Parse normalized timestamps in code |
| Indirection | Evidence stated directly versus several references away | Resolve dependencies before the call |
| Irrelevant state | Minimal evidence versus full transcript | Project only decision-relevant fields |
| Adversarial state | Benign content versus embedded evaluator instructions | Keep deterministic defenses and least privilege |
Limitations That Matter for Guardrails
The adversarial-state warning is especially important for policies. A webpage, tool argument or user message may contain text designed to move a model decision. Jev cannot turn untrusted text into authorization. Keep identity, permission, path, amount and command restrictions outside the model; grant the runtime the least privilege it needs; and define safe outage behavior.
A policy should be evaluated as a whole system. Measure false allows and false blocks after deterministic rules, state projection, Jev judgment, thresholds and fallbacks have all run. Model-only benchmark accuracy does not establish policy safety.
Mitigation Changes the System Under Evaluation
Shortening state, rewriting a question, adding a no-match option or computing a date in code can reduce a known failure. Each change also creates a new evaluator configuration. Re-run held-out labels instead of assuming that an intuitively cleaner design improved accuracy.
Some mitigations trade one error for another. Aggressive filtering can remove decisive evidence; a broad no-match class can absorb valid rare categories; a stricter threshold can overload review. Measure class errors, calibration, coverage and downstream cost together.
Diagnose the Layer That Actually Failed
This decomposition keeps fixes targeted. More context cannot solve a bad outcome label, and a new model cannot repair incorrect authorization code. The evaluator scorecard treats the configured evaluator as one measurable system.
Limitations Are Versioned
This page reflects jev-1.13 documentation checked September 22, 2026. A later release may fix some weaknesses and introduce others. Pin a version for evaluated production behavior, log the resolved model from every response, and re-run the target-domain suite before moving an alias.
FAQ
Can Jev do math?
Jev 1.13 is not a calculator. TypeSafe recommends implementing mathematical logic and counting in code.
Can Jev resist prompt injection?
Do not assume so. TypeSafe warns that adversarial state can steer Jev 1.13. Use deterministic security boundaries, least privilege, narrow state and adversarial evaluation.
Does Jev suffer from context rot?
TypeSafe says accuracy falls when state grows with irrelevant detail. Retrieve and filter before the call, and send only evidence the question needs.
Can Jev generate text by chaining choices?
TypeSafe says Jev is not trained to generate text and that forcing generation through chained choices works poorly and slowly. Use a generative model when output is open-ended.
Sources
Checked against the sources below on September 22, 2026. Model versions, prices and limits change.
- TypeSafe AI docs: Jev 1.13 jaggedness
- TypeSafe AI docs: System One
- TypeSafe AI docs: Models
- TypeSafe AI docs: Confidence