how to write good jev questions

A Question Is a Measurement Instrument

The wording defines what the returned probability is about. “Is this good?” has no stable outcome; “Does assistant_response resolve every explicitly requested task using evidence in tool_results?” can be labeled and disputed. Before calling Jev, write how two human reviewers would decide the criterion and what evidence they may use.

Question IDs are application identifiers and TypeSafe says they are not sent to the underlying model. Naming a field is_safe does not define safety. Put the operational meaning into instructions and criteria, then version it alongside state projection and model.

Split Broad Quality into Decisions That Lead Somewhere

Decomposition improves diagnosis and lets each error carry its own threshold. Do not decompose dependent reasoning into one parallel call: questions share state but cannot consume each other’s answers. Use the multiple-question architecture when later candidates depend on earlier results.

Turn Implied Intent into Literal Criteria

TypeSafe says Jev 1.13 reads instructions literally and struggles with indirection. When a wrong result makes the author say “what I really meant was…,” that missing explanation belongs in the criterion. Keep wording direct enough for an average reviewer to apply consistently.

Criteria Extend the Instruction; They Must Not Contradict It

Choice descriptions should separate options by operational evidence rather than synonyms. Score levels should form one ordered dimension. Noul true/false descriptions, when supplied, should reinforce the proposition—not reverse it. TypeSafe explicitly documents degraded behavior when criteria and instructions conflict.

Include a no-match Choice when the taxonomy can be incomplete, or ask a separate fit Noul. Avoid an other bucket with no review destination. The Choice-versus-Noul guide shows why exclusive and independent label structures need different tests.

Point to Evidence Without Embedding a Second Workflow

Named fields make the evidence relationship explicit. Do not ask Jev to count tool calls, calculate amounts or compare timestamps; compute those facts first. Keep untrusted text visibly separate from policy and validate permissions outside the model. See state design.

{
  "state": {
    "assistant_response": "Your refund has been submitted.",
    "tool_results": [{"tool": "refund", "status": "not_called"}],
    "requested_tasks": ["Refund the duplicate charge"]
  },
  "questions": {
    "completion": {
      "type": "noul",
      "instructions": "Using the requested_tasks, assistant_response, and tool_results fields, did the agent complete every requested task? A claim without matching tool evidence is not completion."
    }
  }
}

Treat Every Wording Edit as a Model Change

Do not select the wording that performs best on the final test set. Use development data for iteration, a calibration split for thresholds and an untouched holdout for reporting. A question can become more accurate overall while regressing a critical slice.

1. Build fixtures from real traffic, including edge, adversarial and disagreement cases.
2. Collect blind labels using the written criterion and preserve reviewer disagreement.
3. Run old and new questions on the same pinned model and state projection.
4. Compare errors, calibration, coverage, distribution shifts, latency and tokens.
5. Inspect changed decisions, approve a new question version and recalibrate thresholds.

Pre-Publication Question Checklist

• One criterion with an independently useful answer.
• Primitive matches exclusive, ordinal or independent label semantics.
• Instructions define the condition without relying on the ID.
• Criteria are aligned, distinct and cover intended boundary cases.
• Exact computation and authorization remain in code.
• State fields and trust levels are named explicitly.
• Version, labels, thresholds and rollback path are recorded.