How to judge multi-turn agent conversations

Turn-Level and Session-Level Scores Answer Different Questions

A turn-level score asks whether one reply was good given what came before it. A session-level score asks whether the conversation got the user where they were going. The second is not the average of the first. A session can contain ten acceptable turns and still fail, because the agent assumed something in turn two and built everything after on it.

That failure is common enough to have been measured. In simulations covering more than 200,000 conversations, Laban et al. found an average 39% drop in performance across six generation tasks when the same task was spread over several turns instead of given in one, mostly from a rise in inconsistency; in their words, when models "take a wrong turn in a conversation, they get lost and do not recover" (Laban et al., 2025). A turn-level judge sees each of those turns as locally reasonable. Only a judge reading the session sees the wrong turn.

Judging the Whole Session

Number the turns, give the judge binary criteria that only make sense across turns, and ask for turn numbers as evidence. The four below cover most conversational failures: the goal, consistency, re-asking for information the user already gave, and whether a correction stuck.

# conversation_judge.py
import json
import os

import anthropic

JUDGE_MODEL = os.environ["JUDGE_MODEL"]
client = anthropic.Anthropic()                    # reads ANTHROPIC_API_KEY
SESSION_CRITERIA = ("goal_met", "consistent", "no_reasking", "correction_present", "correction_honoured")

SESSION_RUBRIC = """You grade a whole conversation between a user and an AI agent.
Mark each criterion true (pass) or false (fail):
- goal_met: by the end, the user's request was resolved or correctly handed off.
- consistent: the agent never contradicted something it said or did earlier.
- no_reasking: the agent never asked for information the user had already given.
- correction_present: the user corrected the agent at least once.
- correction_honoured: after a correction, later turns reflect it; true when there was no correction,
  but exclude those sessions when reporting the correction-honoured rate.
Cite turn numbers as evidence.
Reply with JSON only, for example:
{"goal_met": true, "consistent": false, "no_reasking": true, "correction_present": false, "correction_honoured": true, "evidence": "turn 9 quotes a price that contradicts turn 4"}"""

TURN_RUBRIC = """You grade the LAST assistant turn below, using only the conversation before it.
pass is true if that turn is accurate, answers what the user last asked, and fits earlier turns.
Reply with JSON only, for example: {"pass": false, "evidence": "ignores the delivery date given in turn 3"}"""


def ask(prompt: str) -> str:
    msg = client.messages.create(
        model=JUDGE_MODEL, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = msg.content[0].text
    return text


def parse_json(text: str) -> dict | None:
    start, end = text.find("{"), text.rfind("}")
    if start == -1 or end <= start:
        return None
    try:
        return json.loads(text[start:end + 1])
    except json.JSONDecodeError:
        return None


def render_turns(turns: list[dict], start: int = 1) -> str:
    return "\n".join(f"[{i}] {t['role'].upper()}: {t['content']}" for i, t in enumerate(turns, start))


def judge_session(transcript: str) -> dict | None:
    verdict = parse_json(ask(f"{SESSION_RUBRIC}\n\nCONVERSATION:\n{transcript}"))
    if verdict is None or not all(isinstance(verdict.get(k), bool) for k in SESSION_CRITERIA):
        return None   # not scored; count parse failures separately
    return verdict

Keep the criteria separate in what you store. "Consistent" failing and "goal met" failing are different bugs: the first may point to context handling or memory, while the second may point to capability or tooling. Report correction handling only among sessions where correction_present is true; otherwise conversations with no correction inflate the rate.

Give the judge the goal when you have it. In many conversations the user's real goal only becomes clear a few turns in, and a judge left to infer it will sometimes infer a different one than you would. If the session started from a ticket, a form or a task description, put that text above the transcript as the goal, and tell the judge to grade against it rather than its own reading of the first message.

Turn-Level Scoring Without Peeking Ahead

The rule for turn-level judging is that the judge sees the history up to and including the turn it grades, and nothing after. Show it the whole session and it grades with hindsight: if the user complains in turn eight, turn five suddenly looks wrong, whether or not it was. Hindsight is useful for finding where things broke; it is not a fair grade of the turn.

from conversation_judge import TURN_RUBRIC, ask, parse_json, render_turns


def judge_turns(turns: list[dict]) -> list[dict]:
    results = []
    for i, turn in enumerate(turns):
        if turn["role"] != "assistant":
            continue
        history = render_turns(turns[: i + 1])          # up to this turn, never after
        verdict = parse_json(ask(f"{TURN_RUBRIC}\n\nCONVERSATION:\n{history}")) or {}
        ok = verdict.get("pass")
        results.append({
            "turn": i + 1,
            "pass": ok if isinstance(ok, bool) else None,   # None = not scored
            "evidence": verdict.get("evidence", "reply did not parse"),
        })
    return results


def summarize_turns(results: list[dict]) -> dict:
    scored = [r for r in results if r["pass"] is not None]
    return {
        "first_failing_turn": next((r["turn"] for r in scored if not r["pass"]), None),
        "failing_share": sum(not r["pass"] for r in scored) / len(scored) if scored else None,
        "unscored": len(results) - len(scored),
    }

The aggregate to look at is the first failing turn, not the average. A session whose first failure is in turn two and a session whose only failure is the last reply have the same failing share and need very different fixes. Note the cost, too: each assistant turn re-sends the whole history, so a 40-turn session sends the opening turns 20 times. Turn-level scoring is a debugging tool for sessions that failed at the session level, not something to run on everything.

When the Session Does Not Fit

Long sessions break judges in two ways. Past the context limit they do not fit at all. Well before that, they fit badly: models use information at the start and end of a long input more reliably than information in the middle (Liu et al., 2023), which is exactly where a long session's turning point tends to sit. The options, roughly in order of preference:

1. Extract with code first. Pull out what code can find anywhere in the session - tool errors, repeated calls, the turns where the user said "no" or "that's wrong" - and state them in the prompt. Code reads every event, whatever the length.
2. Keep the head and the tail verbatim. The opening turns hold the request and constraints; the closing turns hold the outcome. Most session criteria hinge on those.
3. Summarize the middle, with instructions about what to keep. A generic summary drops exactly the evidence a grader needs. Tell the summarizer to preserve corrections, user-supplied facts, commitments, errors and refusals, with their turn numbers.
4. Grade in chunks, then grade the chunks. For very long sessions, score each chunk against the session criteria and give the session judge the chunk verdicts plus head and tail. More calls, less loss.

from conversation_judge import ask, render_turns

SUMMARY_PROMPT = """Summarize these conversation turns for a grader, in under 200 words.
Keep verbatim, with turn numbers: every user correction, every fact or constraint the user gave,
every commitment the agent made, every tool error and every refusal.
Turns:
"""


def fit_session(turns: list[dict], max_chars=40_000, head=4, tail=8, chunk=20) -> str:
    full = render_turns(turns)
    if len(full) <= max_chars or len(turns) <= head + tail:
        return full
    middle = turns[head:-tail]
    notes = [
        ask(SUMMARY_PROMPT + render_turns(middle[k:k + chunk], start=head + k + 1))
        for k in range(0, len(middle), chunk)
    ]
    return "\n".join([
        render_turns(turns[:head]),
        f"[SUMMARY OF TURNS {head + 1}-{len(turns) - tail}]",
        *notes,
        render_turns(turns[-tail:], start=len(turns) - tail + 1),
    ])

Pitfalls, and How to Check Your Work

• Replaying a recorded conversation against a new agent. Once the new agent's turn three differs, the recorded user's turn four no longer fits it. Offline multi-turn evals need a simulated user or scripts written to tolerate different replies; recorded transcripts are for judging the agent that produced them.
• Length rewarded as thoroughness. Judges tend to favour longer answers (Zheng et al., 2023 call it verbosity bias). In a long conversation that compounds. Add "a longer reply is not a better reply" to the rubric, and check on labels whether the judge agrees with you on short, correct turns.
• Treating a quiet ending as success. Users often leave without saying whether it worked. Unless the goal is checkable, "no complaint" is not "goal met".
• Summaries that erase the failure. See the warning above; the check is cheap.
• A judge from the agent's own model family. Zheng et al. also report self-enhancement bias, judges favouring answers like their own. If the agent and the judge share a model, check agreement with human labels on exactly the sessions the judge passes.
• Grading sessions that are still going. A conversation paused overnight is not finished. Decide what ends a session for scoring - an explicit end, a resolution event, a stretch of inactivity - and apply it consistently, or you will grade half-conversations as failures.

Evaluating Multi-Turn Conversations with Failproof AI

Failproof AI runs code-based and LLM-based evaluations in the cloud against finished session traces, and it can use the evaluation suite you already have. A session-level judge can grade the full conversation against the rubric above. For turn-level scoring, preserve exact turn boundaries and ensure each result sees only the history available at that turn, whether you prepare those inputs in your existing evaluation workflow or an offline harness.

Keep goal completion, consistency, repeated questions and correction handling as separate results, so a session can complete the goal while still exposing a behavior worth fixing. Before enabling the judge, test it against your longest real sessions and labeled edge cases; that is where trimming or summarization is most likely to remove the evidence that mattered.

Each result stays linked to the conversation trace and cited turn evidence. Failproof AI can alert when a score changes and analyze failures across many sessions, grouping recurring problems into findings with affected conversations and a recommended fix. Version rubric or summarization changes and re-check them against human labels before treating a new pattern as a change in agent behavior.

When You Do Not Need Session-Level Judging

If most of your sessions are one or two exchanges - a question and an answer, a command and a confirmation - there is no "across turns" to judge, and turn-level scoring is the whole job. The same holds when each turn is independent by design, such as an agent that handles unrelated requests in one long-lived chat. Session-level judging earns its cost when conversations carry state: a booking assembled over several messages, a debugging session, a support case where the user supplies details as they go.