build a jev model router

A Router Has Two Decisions, Not One

Do not let Jev choose a route that the caller is not permitted to use. Candidate discovery and hard capability filters happen first. If no candidate is adequate, a forced Choice still returns a winner, so include an explicit no-fit design as explained in Choice versus Noul.

Build the Candidate Set from a Versioned Capability Registry

Code should remove routes that fail capability, residency, tenant, context, health or budget requirements before Jev sees their descriptions. The registry is operational truth; the natural-language descriptions are a projection of eligible entries, not the source of authorization.

Version the exact ordered candidate set because Choice probabilities are relative to the supplied options. Adding a new route changes the question even when the task and Jev model remain unchanged.

{
  "registry_version": "routes-2026-09-22.3",
  "routes": {
    "fast": {
      "model": "<pinned-provider-model>",
      "capabilities": ["classification", "short_answer"],
      "max_input_tokens": 32000,
      "allowed_data_classes": ["public", "internal"],
      "enabled": true
    },
    "coding": {
      "model": "<pinned-coding-model>",
      "capabilities": ["repository_tools"],
      "allowed_data_classes": ["public", "internal"],
      "enabled": true
    }
  }
}

Give the Router Task Evidence and Stable Route Semantics

Route descriptions in state must reflect current deployed capabilities, not marketing labels. Version them and avoid injecting dynamic price or availability as raw prose when code can filter exactly. The route Choice and fit Noul are separate formulations and require separate calibration.

{
  "state": {
    "task": "Refactor the payment retry worker and add regression tests",
    "language": "python",
    "repository_size": "large",
    "allowed_routes": {
      "fast": "Low-latency model for simple edits and classification",
      "coding": "Coding model with repository tools",
      "reasoning": "Higher-latency model for difficult dependent analysis"
    }
  },
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which allowed route best matches the task requirements?",
      "criteria": {
        "fast": null,
        "coding": null,
        "reasoning": null
      }
    },
    "any_route_fits": {
      "type": "noul",
      "instructions": "Can at least one allowed route complete the task with the stated capabilities?"
    }
  }
}

Use a No-Fit Gate to Avoid Forced Routing

A Choice always ranks the candidates it receives. The winner can be a poor fit in absolute terms, so top probability alone is not enough. Evaluate the no-fit mechanism on out-of-scope tasks, unavailable capabilities and deliberately incomplete registries.

Optimize for the Completed Task

Route-label accuracy is only a proxy. A human may call a task “reasoning” while the cheaper route still succeeds. Compare the router with simple baselines such as one default model, rules based on task metadata, or a small classifier. Evaluate on future or held-out traffic to avoid memorizing the current workload.

Join the Route Decision to Execution and Outcome

This is an application record, not a TypeSafe response format. Keep router inference, selected-model execution and delayed task outcome separate. A route can be semantically appropriate but fail in transport; a poor route can occasionally succeed. Those cases require different fixes.

{
  "request_id": "task-1842",
  "router": {
    "model": "jev-1.13.0",
    "question_version": "route-v4",
    "registry_version": "routes-2026-09-22.3",
    "distribution": {"fast": 0.16, "coding": 0.78, "reasoning": 0.06}
  },
  "selected_route": "coding",
  "execution": {"status": "ok", "latency_ms": 1840, "cost_usd": 0.012},
  "outcome": {"task_success": null, "label_matures_at": "2026-09-29T00:00:00Z"}
}

Budget the Serial Path and Fallback Tail

Measure p95 and p99 end-to-end latency, not the mean Jev call alone. A router that saves model cost but adds enough tail latency or cache misses can make the product worse. Record whether the selected provider reused prompt or tool context, because route switching may sacrifice cache locality.

Fallback Must Not Create an Unbounded Agent Loop

Define a maximum attempt count and route order. If the selected model fails, fallback can use one known alternate or human review; it should not repeatedly ask Jev to choose until a different label appears. Preserve the first route decision, execution error and fallback outcome as separate events.

A Jev timeout is a router outage, not evidence that the expensive model is required. Choose a deterministic default based on risk and service objectives. The errors and retry guide provides the failure budget.

Historical Replay Cannot Reveal Every Route Outcome

A production log normally contains the response from the model that was chosen, not the responses every alternative would have produced. Relabeling the historical task with a preferred route therefore measures routing opinion, not counterfactual task success. Where cost and data policy permit, run each candidate on a frozen offline set and score all outputs independently.

Online exploration can estimate route outcomes on current traffic, but it changes which models users receive. Contextual-bandit research formalizes learning from actions and rewards under partial feedback; adopting that design requires controlled exploration, propensity logging and stronger safety review. A Jev Choice alone does not implement a bandit or remove selection bias.

Evaluate Offline, Shadow Online and Then Canary

Distribution drift in the route answer can be useful, but outcome drift is decisive. Pin the router model and question bundle, then follow the model-version upgrade process.

1. Replay historical tasks through every eligible route where cost permits and score final outcomes blind.
2. Fit the router question and action bands on development/calibration data.
3. Shadow live traffic while the existing route remains authoritative.
4. Canary a small reversible segment and monitor task, latency and cost metrics.
5. Expand only when end-to-end utility improves; retain a rollback and audit trail.

Make the Router Beat Simple Baselines

Report regret relative to the best candidate observed for each offline task, as well as success and cost against the fixed default. The oracle is not deployable—it uses outcomes unavailable at route time—but it shows whether routing headroom exists. If every candidate performs alike, no sophisticated router can create much value.

Watch for Route Collapse and Self-Reinforcing Feedback

If one route receives nearly all traffic, the others stop collecting outcome evidence and become harder to compare. That can be a correct result or a feedback artifact. Audit a safe sample across alternatives, preserve offline test coverage and investigate selection shifts after registry or model changes. Hidden feedback loops are a classic source of ML-system technical debt.

• Candidate eligibility and health by route before the Jev call.
• Choice distribution, no-fit rate and selected-route share by task slice.
• Execution success, latency, cost and fallback rate by selected route.
• Delayed task quality and reviewer outcomes for every route, including low-volume ones.
• Registry, question, projection, Jev and downstream model versions.