jev speculative fan-out

The Pattern Speculates Across Branches

Suppose support triage first selects bug, billing, account or feature. A bug branch needs severity and reproducibility; billing needs refund intent; every branch may need frustration. A sequential design calls Jev for category, waits, and then calls it for branch questions. Fan-out includes all independent branch questions with the original ticket and discards those that do not apply.

TypeSafe recommends this pattern because its questions are evaluated independently and in parallel. That is not a promise of zero marginal latency or cost. Every question consumes input, enlarges the response and creates an answer to version and observe.

A Batched Question Set

The application reads bug_severity only when category is bug, and reads refund_requested only for the billing workflow. The speculative answers are not inputs to category and cannot alter it through a hidden chain. The multiple-question guide explains this independence guarantee.

{
  "questions": {
    "category": {
      "type": "choice",
      "instructions": "What is the primary support category?",
      "criteria": {
        "bug": "Broken or unexpected product behavior",
        "billing": "Charges, invoices or refunds",
        "account": "Login, permissions or profile",
        "feature": "Request for new capability"
      }
    },
    "bug_severity": {
      "type": "score",
      "instructions": "If a product bug is described, how severe is it?",
      "criteria": ["No functional impact", "Degraded with workaround", "Blocking without workaround"]
    },
    "refund_requested": {
      "type": "noul",
      "instructions": "Does the customer explicitly request a refund?"
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": ["Calm", "Frustrated", "Very angry"]
    }
  }
}

Do Not Fan Out Questions with Changed Evidence or Candidates

Sending unavailable candidates “just in case” can create results that should never exist. Filter permissions and availability before inference. If branch A changes what B means, issue a second request and make the dependency visible in code and telemetry.

Measure the Trade: Saved Round Trips Versus Wasted Inference

Compare sequential and fan-out designs on end-to-end p50, p95 and p99 latency, serialized input tokens, output tokens, request rate, error rate and downstream quality. Shared state may reduce duplicate tokens, while many unused questions increase question and output tokens. A single large call also expands retry scope when transport fails.

TypeSafe says adding questions usually has little latency effect, but “usually” is an attributed architecture claim, not a capacity guarantee. Benchmark representative state and batch sizes through the intended provider. The pricing guide shows why model-token cost alone is not total system cost.

Keep Discarded Answers Observable Without Letting Them Act

• Store every answer with question and model version even if its branch was not selected.
• Mark whether an answer was consumed, ignored or invalidated by deterministic code.
• Do not attach downstream labels to unused answers as though their action occurred.
• Monitor each question independently for distribution and error drift.
• Delete speculative questions whose information never changes a measured decision.

When Sequential Requests Are Better

Use stages when the first result changes available evidence, when later state is expensive or sensitive, when most branches are rare and costly, or when the first result often causes an early deterministic exit. Sequential design is also easier when each branch has a different timeout or provider requirement.

Fan-out is an optimization after correctness, not a default. Establish the decision graph, evaluate each question and compare the deployed paths. The state guide helps decide whether questions truly share one evidence boundary.