How to measure hallucination in AI agent runs

Two Kinds of Hallucination, Two Ways to Measure

The word covers two failures that behave very differently in an agent. A chat model that invents a citation produces a wrong sentence. An agent that invents a file path, a customer ID or a CLI flag produces a wrong action - an error at best, and at worst a call that succeeds against the wrong target.

The event stream lets code detect unknown tool names exactly and surface strong signals such as not-found results, identifiers with no observed provenance and action claims with no matching call. Those signals are not always proof of hallucination: exploration can produce a legitimate not-found result, and an identifier may come from trusted context outside the captured trace. Use an LLM-based check for factual prose and human review to calibrate the ambiguous code signals.

The practical requirement is the same for both kinds: compare the agent's claim or action with an external source of evidence. For tool use, that evidence is the registered tool set, prior messages, tool outputs and the target system. For prose, it is the retrieved or observed information available in the session. Without that comparison, plausibility is easily mistaken for correctness.

Invented References, Checked in Code

Normalize the session into a list of events - user messages, tool calls with their input, tool results with their output - and run four checks. The ID pattern below matches Stripe-style prefixed IDs; replace it with whatever your system's identifiers look like.

import re

NOT_FOUND = re.compile(
    r"no such file|enoent|command not found|modulenotfounderror|cannot find module"
    r"|unknown (tool|option|flag)|unrecognized arguments",
    re.I,
)
ID = re.compile(r"\b(?:ord|cus|inv)_[A-Za-z0-9]{6,}\b")
TEST_CLAIM = re.compile(r"\b(tests? (now )?pass(es|ed)?|all tests pass|ran the tests)\b", re.I)
TEST_CMD = re.compile(r"\b(pytest|npm (run )?test|go test|cargo test|jest|vitest)\b")


def invented_references(events, registered_tools, final_reply):
    calls = [e for e in events if e["kind"] == "tool_call"]
    unknown_tools = [c["tool"] for c in calls if c["tool"] not in registered_tools]
    not_found = [str(e["output"])[:120] for e in events
                 if e["kind"] == "tool_result" and NOT_FOUND.search(str(e["output"]))]

    seen, invented_ids = set(), []
    for e in events:
        if e["kind"] == "tool_call":
            invented_ids += [i for i in ID.findall(str(e["input"])) if i not in seen]
        else:
            seen.update(ID.findall(str(e.get("output", "")) + str(e.get("text", ""))))

    commands = [str(c["input"].get("command", "")) for c in calls if c["tool"] == "Bash"]
    unbacked_test_claim = bool(TEST_CLAIM.search(final_reply)) and not any(
        TEST_CMD.search(c) for c in commands)

    return {
        "tool_calls": len(calls),
        "unknown_tools": unknown_tools,
        "not_found_results": not_found,
        "invented_ids": invented_ids,
        "unbacked_test_claim": unbacked_test_claim,
    }

ID provenance is the check most teams have never run and most should. An identifier the agent passes to a tool should have come from somewhere: the user's message or an earlier tool result. An ID that appears for the first time in a tool call was generated by the model. Sometimes the call fails harmlessly. Sometimes the ID exists and belongs to another customer.

The action-claim check generalizes. "I committed the change" with no git commit in the session, "I emailed the customer" with no call to the email tool, "I updated the config" with no edit to it. Each is a pair of patterns - the claim in the reply, the evidence in the tool calls - and a mismatch is a hallucinated action. These are the ones that hurt most, because a human reading the summary believes them.

Check Invented Facts Against Session Evidence

For facts in prose, the useful question is not "is this true" - a model may answer from its own knowledge, which can introduce the same failure you are trying to measure - but "is this backed by something the agent saw in this session". Ask the check to list specific claims and mark each one backed or unbacked by the tool outputs.

import json
import os

from openai import OpenAI

JUDGE_MODEL = os.environ["JUDGE_MODEL"]


def unbacked_facts(tool_outputs, final_reply):
    prompt = (
        "Tool outputs the agent saw in this session:\n" + tool_outputs[-50000:]
        + "\n\nThe agent's final reply:\n" + final_reply[-8000:]
        + "\n\nList every specific factual claim in the reply (names, numbers, versions, "
        "paths, function names, dates). Mark each 'backed' if a tool output states it, "
        "otherwise 'unbacked'. Do not use your own knowledge.\n"
        'Reply with JSON only: {"claims": [{"claim": "...", "status": "backed"}]}'
    )
    client = OpenAI()                                 # reads OPENAI_API_KEY
    resp = client.chat.completions.create(
        model=JUDGE_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    text = resp.choices[0].message.content or ""
    try:
        claims = json.loads(text[text.index("{"): text.rindex("}") + 1])["claims"]
        return [c["claim"] for c in claims if c.get("status") != "backed"], len(claims)
    except (ValueError, KeyError, TypeError):
        return None, 0

Report the unbacked claims, not only their count. "Unbacked" is not the same as false: the agent may know that Python 3.12 exists without having looked it up. The value of the list is that a person can scan five claims in ten seconds and see which ones matter. For agents that answer from retrieved documents, the stricter claim-by-claim method in the groundedness guide fits better.

From Counts to Rates You Can Track

A single session with an invented file path tells you nothing. The same number across a thousand sessions, split by agent and by prompt or model version, tells you whether last week's change made things better. Three numbers cover most of what a team needs to watch:

• Invented references per hundred tool calls. The most stable number, because it normalizes for session length. Track each kind - unknown tools, not-found errors, invented IDs - as its own line.
• Share of sessions with at least one invented action claim. Low in absolute terms and high in cost, because each one is a summary a person may have acted on. This is the one worth an alert.
• Unbacked claims per reply, from the LLM-based check, on a sample. The noisiest of the three. Watch its trend rather than its level, and read the claims whenever it moves.

Set a baseline from a week of normal traffic before deciding what counts as bad. Every agent has some rate of not-found errors from reasonable exploration, and the useful signal is a change against that baseline, not the raw level. When a rate jumps after a deploy, compare a handful of sessions from before and after side by side; the difference is usually visible in the first three you open, often as a tool that was renamed or a directory the prompt still mentions.

Pitfalls, and How to Check Your Work

• Not every not-found is a hallucination. An agent that runs ls on a path to check whether it exists is doing the right thing. Count not-found results that are followed by a read or edit of the same path, or weight exploratory commands down.
• Rates need a denominator. "Twelve invented references" means nothing without the number of tool calls. Report invented references per hundred tool calls, and the share of sessions with at least one.
• Keep the kinds apart. A prompt change that cuts invented files may raise invented facts. One blended hallucination score would show no change at all.
• Error text differs by tool and harness. "No such file", "ENOENT", "does not exist" and a dozen localized variants all mean the same thing. Collect the not-found messages your own agents actually produce from a week of sessions and build the pattern from those, rather than trusting a generic list like the one above.
• Plant a known failure. Take a session, add an ID to a tool call that never appeared earlier, and confirm the provenance check flags it. Do the same for the LLM-based check with a made-up version number in the reply.

Finding Hallucination Patterns with Failproof AI

Failproof AI runs code-based and LLM-based evaluations against production sessions, including the evaluation suite you already have. Use code checks for unknown tools, missing evidence and unsupported action claims, and a calibrated LLM check for factual claims in prose. Each result stays linked to the trace evidence behind it.

Treat these as separate evaluations rather than one blended hallucination score. A rise in unknown tools points to tool definitions or prompting; unsupported IDs may point to missing provenance; false claims of completed actions need a completion or honesty check. Keeping the categories separate makes the eventual fix clear.

Failproof AI analyzes evaluation results and trace evidence across sessions to find recurring failure modes automatically. It can group, for example, repeated guesses at renamed tools or unsupported completion claims after a particular error into findings with affected sessions, severity and a recommended fix. Alerts route changes to the owning team through email, Slack, webhooks or the dashboard.

Fix the prompt, tool description, context assembly or integration based on that evidence, then keep the evaluations running to confirm the rate falls. When a finding identifies a recognizable high-risk action, a tested behavioral policy can prevent that action at runtime. That policy closes the loop on a known pattern; it does not replace evaluation for new hallucinations.

When You Do Not Need Any of This

For a coding agent whose output is compiled and tested in CI, the compiler and test suite already catch many invented functions, imports and flags more reliably than transcript heuristics. They do not catch every unsupported summary or wrong external identifier. Add hallucination-specific evaluation when actions touch real systems, summaries go to people who will not verify them, or the agent answers factual questions in prose.

If you do only one thing from this page, make it the action-claim check. It is twenty lines, it needs no model, and it catches the hallucination that most often reaches a human as fact: the summary that says the work was done when the transcript shows it was not.