evaluate agent trajectories with jev

Choose the Unit Before Writing the Criterion

“Evaluate this agent” is not an evaluation unit. A trajectory can contain several attempts, tools, approvals and outcomes. Decide whether one row represents a complete session, one proposed tool call, one response, or one recovery episode. The label and available evidence must match that unit.

Session-level completion may require the original request, final response and decisive tool results. Tool appropriateness should use the state visible before the call, not a later result that leaks success. Temporal scope is part of the evaluator contract.

Normalize the Trace as Events and Edges Before Projecting Text

A flattened transcript loses causality. The same tool result can appear before or after the agent’s decision in text order, and concurrent branches can interleave. Preserve the graph first, then render the bounded evidence needed by Jev. OpenTelemetry’s GenAI semantic conventions provide shared span and event vocabulary, while domain-specific outcome fields remain application-owned.

Normalization should be deterministic and independently tested. If two trace exporters encode the same call differently, the projection layer should produce the same evaluation unit or record why it cannot.

Project Traces Without Flattening Provenance

This projection preserves event IDs and exposes the mismatch between a claimed action and tool evidence. It omits unrelated conversation. A reviewer can return to the original trace through the references. The state guide covers trust, redaction and ablation.

{
  "task": "Refund the duplicate captured charge",
  "events": [
    {"id": "e12", "type": "tool_call", "tool": "lookup_order", "arguments": {"order_id": "A-104"}},
    {"id": "e13", "type": "tool_result", "status": "ok", "duplicate_charge": true},
    {"id": "e14", "type": "assistant_message", "text": "Your refund has been submitted."}
  ],
  "outcome": {"refund_tool_called": false},
  "missing_fields": []
}

Give Each Criterion an Explicit Evidence Window

Windowing prevents hindsight leakage and ambiguous attribution. A later success cannot make an earlier unauthorized proposal appropriate; a later refund reversal can matter to long-term outcome without changing whether the original refund tool executed successfully.

Store the window rule and cutoff timestamp with each result. When late events arrive, create a new outcome record linked to the original evaluation rather than mutating what the evaluator could see at the time.

Decompose Trajectory Quality into Actionable Judgments

Do not combine these into one “trajectory quality” score unless the product truly needs a composite and its weights are explicit. Separate results reveal whether the fix belongs in prompting, tool descriptions, permissions, recovery logic or user communication.

One Trajectory Should Produce Several Independent Records

This is an application record, not a TypeSafe or Failproof schema. It illustrates why one session can contain an appropriate choice, a failed completion and an outcome that is not yet observable. Collapsing them into one pass/fail result destroys the diagnosis.

Store the exact question, projection and resolved model with each scored record. The same state can support several independent questions, but their answers are not a causal chain. The multiple-questions guide explains batching without implying dependence.

[
  {
    "criterion": "tool_choice_appropriate",
    "unit": "event:e12",
    "evidence_window": ["e01", "e11"],
    "primitive": "noul",
    "distribution": {"yes": 0.91, "no": 0.09}
  },
  {
    "criterion": "refund_completed",
    "unit": "session:s9",
    "evidence_window": ["e01", "e14"],
    "status": "scored",
    "distribution": {"yes": 0.04, "no": 0.96}
  },
  {
    "criterion": "customer_outcome",
    "unit": "session:s9",
    "status": "pending_outcome"
  }
]

Missing Evidence Is Not a Negative Label

A trace can be truncated, a tool result can be dropped or an external outcome may not have arrived. In those cases the criterion may be unscorable. Represent missingness in state and application result status; do not ask Jev to guess and record the guess as failure.

Track scorable coverage by agent version, tool and environment. A sudden improvement in pass rate can result from dropping difficult events. The evaluation pipeline should fail visibly when required evidence disappears.

Model Partial Completion as Observed Subgoals

Multi-step tasks often end between “nothing happened” and “complete.” Define the requested subgoals and evidence for each before evaluating. A travel agent may find flights, fail to reserve a hotel and correctly ask the user for missing passport information. One global completion score cannot distinguish useful progress from a false claim of success.

Use deterministic checks for exact artifacts—reservation ID exists, file was written, payment status is captured—and Jev for bounded semantic judgments such as whether the handoff explains the unresolved step. If the product needs an overall status, compose it in code from the atomic records and retain every component. The composite-scoring guide covers weights and vetoes.

Separate Proposed Action, Execution and Outcome

An appropriate tool choice can fail due to infrastructure; an inappropriate call can accidentally succeed. Labeling only the final outcome confounds policy, agent and tool reliability. Tool-call evaluation goes deeper on this separation.

Assign Each Failure to the Layer That Can Fix It

Evaluation is useful when it shortens the path to a fix. Attach a failure to the smallest responsible layer and retain evidence for that attribution. A generic “agent quality declined” alert gives the owner little to act on and can mix unrelated regressions.

Concurrent Branches Require Causal Ordering, Not Transcript Order

Agents can launch searches or tools in parallel. Wall-clock order alone may place one result between another call and its result even though they are unrelated. Use parent span IDs, tool-call IDs and branch IDs to reconstruct causal neighborhoods before producing Jev state.

For a branch-level criterion, include the branch objective, observations available when it started, its calls and its terminal result. For session completion, merge branch outcomes only after all required branches reach a terminal or explicitly cancelled state. This avoids scoring an in-flight trajectory as failed merely because one result arrived later.

Evaluate the Evaluator and the Agent

LangChain’s Jev evaluation experiment is useful independent evidence for one harness and task. It does not establish universal superiority over LLM judges. Use the same labels and complete trajectory projection for every comparator; see Jev as an LLM judge.

• Evaluator agreement, per-class errors and calibration against blind labels.
• Scorable coverage and missing-evidence rate.
• Agent task success, recovery rate, tool errors and user outcomes.
• Results by agent/model/tool/version and consequential slices.
• Evaluation latency, token cost, overrides and disagreement reasons.

Sample Trajectories Without Hiding Rare Failures

Do not evaluate only failed sessions or only traces selected by another detector. That can measure diagnostic usefulness but cannot estimate production error. The dataset guide covers provenance and weighting; drift monitoring covers ongoing slice and missingness changes.