How to evaluate multi-agent systems

Why Multi-Agent Failures Are Hard to Score

In a single agent, a failure and its cause live in the same transcript. In a multi-agent system they drift apart. The planner writes a sub-task that drops the budget constraint; the worker executes that sub-task faithfully; the reviewer checks the work against the sub-task and passes it; the system returns an answer the user cannot afford. Every agent did its job as it was given. Per-agent scores are green and the system failed.

Research on real multi-agent traces agrees that this is the normal case, not an edge case. Cemri et al. annotated more than 1,600 traces across seven multi-agent frameworks and grouped the failures into 14 modes in three categories: system design issues, inter-agent misalignment, and task verification (Cemri et al., 2025). Two of those three categories are about how agents pass work to each other and check it, which is exactly what per-agent scoring does not look at.

Score at Three Levels

The phrase that matters is "given what it received". Score a worker against the input it was handed, not the user's original request; otherwise every upstream mistake lands on the worker's report card, and you will spend a week tuning the one agent that was behaving correctly. The user's request belongs to the system score and to the handoff checks, which is where you find out that the constraint went missing.

Treat the orchestrator or planner as an agent in its own right, with its own score. A plan that splits the work badly, assigns the wrong specialist or forgets a step is the most common root cause that never shows up as an error, because nothing downstream can know the plan was wrong.

Check Handoffs in Code First

Give every handoff a type. If the researcher passes findings to the writer, define what a findings message must contain and validate it on the way through. A message that fails validation is a failure you can attribute with certainty, at zero judge cost.

from pydantic import BaseModel, Field, ValidationError   # pip install pydantic


class ResearchHandoff(BaseModel):
    question: str = Field(min_length=10)
    constraints: list[str]
    sources: list[str] = Field(min_length=1)
    findings: list[str] = Field(min_length=1)


def check_handoff(raw_json: str) -> list[str]:
    try:
        ResearchHandoff.model_validate_json(raw_json)
        return []
    except ValidationError as err:
        return [f"{'.'.join(map(str, e['loc']))}: {e['msg']}" for e in err.errors()]


print(check_handoff('{"question": "Which EU carriers ship lithium batteries?", '
                    '"constraints": [], "sources": [], "findings": ["DHL does"]}'))
# ['sources: List should have at least 1 item after validation, not 0']

Log every handoff as its own record - sender, receiver, message - rather than leaving it buried inside the receiving agent's prompt. Attribution, replay and the handoff judge all start from that record, and it is much harder to reconstruct afterwards from prompts that were assembled at runtime.

A schema catches missing fields, not missing meaning. The constraints list above is valid when empty, even if the user said "EU carriers only" three turns earlier. The semantic check compares the user's request with the handoff and lists what was dropped or changed:

# mas_judge.py
import json
import os

import anthropic

JUDGE_MODEL = os.environ["JUDGE_MODEL"]
client = anthropic.Anthropic()                    # reads ANTHROPIC_API_KEY

HANDOFF_RUBRIC = """Compare a user's request with the message one agent passed to the next.
List every constraint in the request (dates, amounts, names, exclusions, formats) that the
handoff drops or changes. Reply with JSON only, for example:
{"preserved": false, "dropped_or_changed": ["user said EU carriers only; handoff says any carrier"]}"""

BLAME_RUBRIC = """A multi-agent system failed the task below. Each numbered log line names its agent.
Find the FIRST step whose mistake made the failure inevitable, and the agent that made it.
Reply with JSON only, for example: {"agent": "planner", "step": 4, "reason": "split the task without the budget"}"""


def ask(prompt: str) -> str:
    msg = client.messages.create(
        model=JUDGE_MODEL, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = msg.content[0].text
    return text


def parse_json(text: str) -> dict | None:
    start, end = text.find("{"), text.rfind("}")
    if start == -1 or end <= start:
        return None
    try:
        return json.loads(text[start:end + 1])
    except json.JSONDecodeError:
        return None


def judge_handoff(request: str, handoff: str) -> dict | None:
    verdict = parse_json(ask(f"{HANDOFF_RUBRIC}\n\nREQUEST:\n{request}\n\nHANDOFF:\n{handoff}"))
    return verdict if verdict and isinstance(verdict.get("preserved"), bool) else None


def first_decisive_error(task: str, log: str) -> dict | None:
    verdict = parse_json(ask(f"{BLAME_RUBRIC}\n\nTASK:\n{task}\n\nLOG:\n{log}"))
    return verdict if verdict and isinstance(verdict.get("agent"), str) else None

Attributing a Failure to the Agent That Caused It

Once the system fails, you want a name: which agent, at which step. There are three ways to get one, in increasing order of cost and reliability.

1. Heuristics in code. The first tool error, the first handoff that fails validation, the first agent output that breaks its contract. Cheap and often right, but only for failures that leave a mechanical trace.
2. A judge reading the log. first_decisive_error above asks for the first step that made failure inevitable. Useful for triage, and weaker than it looks.
3. Counterfactual replay. Replace one agent's output with a corrected version, re-run everything downstream, and see whether the system now succeeds. If it does, that agent's step was decisive. The strongest evidence available, and the most expensive.

On the judge route, the numbers are sobering. Zhang et al. built the Who&When benchmark from failure logs of 127 multi-agent systems and found the best automated method identified the responsible agent 53.5% of the time and the decisive step 14.2% of the time (Zhang et al., 2025). Reading the whole log at once did better at naming the agent; reading it step by step did better at finding the step. Use judge attribution to decide which sessions a person looks at first, not to close the ticket.

def decisive_agents(run_system, task, corrections: dict, trials: int = 3) -> list[str]:
    """run_system(task, overrides) re-runs the pipeline, using overrides[agent] in place of
    that agent's output, and returns an object with a boolean .success.
    corrections: {agent_name: corrected_output}, written or approved by a person."""
    decisive = []
    for agent, fixed in corrections.items():
        wins = sum(run_system(task, overrides={agent: fixed}).success for _ in range(trials))
        if wins > trials / 2:          # downstream agents are not deterministic either
            decisive.append(agent)
    return decisive

Replay needs a pipeline that accepts an injected output for any agent, which is worth building into a multi-agent system from the start. It also answers a question the judge cannot: whether fixing one agent is enough, or whether two agents each contributed half the failure.

Pitfalls, and How to Check Your Work

• Blaming the last agent. The agent that produced the final answer is where the failure became visible, not necessarily where it started. Check handoffs upstream before tuning it.
• Scoring workers against the user's request. See above: score each agent on what it received.
• No score for the plan. If the orchestrator is not scored, its mistakes get distributed across everyone else's numbers.
• Judge cost multiplied by agent count. Five agents with three judges each is fifteen judge calls per task. Run code checks on everything and judges on system failures plus a random sample.
• Single replays. Downstream agents are nondeterministic, so one successful replay proves little. Replay several times and take the majority, as the function above does.
• Shared blame recorded as no blame. Sometimes two agents each contribute: a vague plan and a worker that never asked for clarification. Replay each correction alone and both together; if only the combination fixes the run, record both agents.

Find Multi-Agent Failure Patterns with Failproof AI

Failproof AI Cloud runs code-based and LLM-based evaluations on finished sessions and keeps every result linked to the trace. If each agent emits its own session and agent_id, apply a contract and evaluations to each role. If the whole system appears as one session, record agent identity on handoffs and tool events so the system-level evaluation can cite the step and agent associated with a failure.

Use separate evaluations for the system outcome, each agent's responsibility and the handoffs between them. Deterministic checks catch malformed or incomplete handoffs; LLM-based evaluations assess whether constraints or intent were lost. Your existing evaluation suite can run in the same Cloud workflow, avoiding a second scoring pipeline for multi-agent traces.

When the system fails, automatic failure analysis can group similar traces into findings and show whether the recurring pattern starts in planning, a handoff, one worker or final verification. Treat model-generated attribution as a lead, not proof: the reviewer can open session replay and inspect the cited events before assigning the fix.

Alerts can be scoped to the system or a specific agent. A system-level decline without a matching worker decline points toward planning, handoff or verification problems. A concentrated decline for one agent_id points the owner toward that role and its trace evidence.

Define an agent contract for each role—its purpose, expected outputs, completion condition and prohibited behavior—so failure analysis evaluates each agent against the job it was actually given. When a recurring finding identifies a narrow high-risk action, a tested behavioral policy can prevent that action. Counterfactual replay still belongs in your own pipeline when you need stronger causal proof.

When You Do Not Need Multi-Agent Evaluation

Many "multi-agent" systems are one agent calling a few specialised prompts through fixed, typed functions. If the handoffs are deterministic code and the sub-agents are stateless, test the sub-agents like functions and score the whole thing as one agent with an outcome check. The three-level approach earns its keep when agents decide what to pass each other, when a planner chooses who does what, and when a failure report that says "the system got it wrong" is no longer enough to know who should fix it.

There is also a staging case. If one team owns every agent and every fix is "change the prompt of whatever looks wrong", start with the system score and handoff validation, which are cheap, and add per-agent scores when ownership splits or when the system score stops pointing at a fix.