evaluate agent tool calls with jev

A Tool Call Has Four Distinct Correctness Layers

Collapsing these layers into “correct tool call” makes diagnosis impossible. A semantically sensible refund call can be unauthorized; an authorized lookup can use the wrong order; a correct call can fail due to the provider. Each result needs its own owner and remediation.

Place Semantic Evaluation Beside—Not Inside—the Authorization Boundary

Jev is a probabilistic signal, not an authorization service. A high probability cannot grant a capability the caller lacks, override a spending limit or make malformed arguments valid. Conversely, a low semantic-fit score should not erase the exact reason a deterministic check denied the call. Keep both records.

This separation also defines outage behavior. If the semantic evaluator times out, application code follows an explicit fail-closed, fail-open, queue or fallback path based on the action’s consequence. The Jev policies overview discusses runtime placement; the errors and retries guide covers bounded failure handling.

Evaluate Proposals with Pre-Call Evidence Only

Code should first ensure cancel_order is permitted and the argument matches its schema. A Noul can then ask whether the proposed call fulfills the stated request given the pre-call observations. Do not include the successful result when judging whether the proposal was reasonable; that leaks future information.

Construct the state from trusted fields, not by concatenating an entire transcript. Tool descriptions, retrieved text and prior tool output can contain instructions that compete with the evaluator question. Delimit their roles, strip irrelevant fields and test injected examples. OWASP identifies prompt injection and excessive agency as separate LLM-application risks; a tool-evaluation design must address both.

{
  "user_request": "Cancel order A-104 before it ships",
  "pre_call_observations": {
    "order_id": "A-104",
    "status": "processing"
  },
  "permitted_tools": {
    "cancel_order": "Cancel an unshipped order",
    "lookup_order": "Read order state without changing it"
  },
  "proposed_call": {
    "tool": "cancel_order",
    "arguments": {"order_id": "A-104"}
  }
}

Ask One Question Whose Answer Maps to One Action

Use separate Nouls when several independent conditions may all hold. Use Choice only when one member of a complete candidate set should win—for example, the primary intent among billing, technical support, account access and explicit no-match. Use Score only for a defensible ordered construct such as recovery quality. The Choice-versus-Noul guide covers the statistical difference.

Write positive, negative, boundary and missing-evidence examples before selecting a threshold. A threshold is fitted against labeled consequences, not chosen because 0.8 sounds confident. See Jev thresholds and confidence versus probability.

Execution and Outcome Need Separate Records

A 200 response does not prove the intended effect occurred, and a denied call is not necessarily a bad proposal if policy changed after planning. Use stable event IDs and timestamps to attribute each conclusion.

Use a Failure Taxonomy That Points to the Owning System

A single session can contain several rows from this table. Preserve them independently so a provider outage is not counted as a semantic-selection error and a misleading final answer is not hidden by a successful API response. The RAG groundedness method applies the same claim-to-evidence discipline to final responses.

Useful Semantic Criteria

Ask these separately. Avoid a broad question that mixes tool selection, arguments, execution and final response. Use code for exact equality, counts and schemas, and Jev Score only for a genuinely ordered qualitative rubric.

• The selected tool’s documented purpose matches the user’s requested outcome.
• The proposed target refers to the entity identified in available evidence.
• The tool result supports one specific claim in the final response.
• The agent recovered appropriately after a particular tool error.
• A handoff was appropriate because no permitted tool could complete the task.

Build Labels Around Realistic Failure Modes

Sample real tool distributions rather than a balanced toy set alone. Report evaluator error and calibration by tool, agent version and consequence. Rare destructive tools deserve explicit stress tests even when they contribute little to average accuracy.

Create one immutable row from the evidence available at the evaluated stage. Group calls from the same session, incident and synthetic template before splitting so related examples do not leak across development and test. Keep natural-prevalence traffic separate from destructive-action stress tests; the dataset construction guide gives the complete lineage and adjudication workflow.

Measure the Evaluator and the Guarded Workflow Separately

Offline agreement proves only the frozen evaluator task. A runtime gate can still fail through missing traces, stale permissions, timeouts or a costly review queue. Shadow the decision on live traffic first, retain the action that would have occurred, and compare it with eventual outcomes before enforcement.

Compare alternatives at the same evidence and operating point: deterministic rules, Jev, a pinned generative judge, or a layered combination. Report error at equal coverage rather than comparing one system that decides everything with another that defers difficult cases. The benchmark protocol defines paired analysis; Jev versus LLM judge covers interface differences.

Store Enough Evidence for an Incident Review

OpenTelemetry’s GenAI semantic conventions provide a useful vocabulary for model and agent events, but applications still need domain-specific tool and outcome fields. The agent-trajectory guide shows how tool evidence fits the larger session.

• Session, span and tool-call IDs plus event timestamps.
• Pre-call observations and permission decision reference.
• Proposed name and arguments, validation output and execution status.
• Jev question/state/model versions and full distribution.
• Derived finding, reviewer override and observed side effect.

Keep a Decision Record That Can Be Replayed

This is an application-level audit shape, not a TypeSafe or Failproof wire format. It shows the joins required to reconstruct a decision. Store the complete distribution and versioned inputs rather than only allow; future threshold changes can then be replayed without rerunning the model.

Avoid copying sensitive arguments into every analytics table. Keep controlled evidence references and purpose-specific projections, with retention and access rules appropriate to the underlying data. The evaluation architecture shows how these records connect to audits and recurring findings.

{
  "decision_id": "tool-decision-921",
  "session_id": "session-91",
  "proposal_event_id": "event-8",
  "tool_registry_version": "tools-42",
  "authorization": {"result": "allow", "policy_version": "authz-17"},
  "semantic_eval": {
    "criterion": "intent_tool_fit_v3",
    "model": "jev-1.13.0",
    "projection": "precall-v2",
    "distribution": {"true": 0.94, "false": 0.06}
  },
  "decision": "allow",
  "execution_event_id": "event-9",
  "outcome_ref": "order:A-104:state-change-3"
}