How to evaluate tool selection and tool calls

What a Good Tool Call Means

A tool call can go wrong in four distinct places, and each needs a different check. Lumping them into one "tool accuracy" number is how teams end up with a metric that moves without telling them why.

The order matters for cost too. Much of argument validation and unknown-tool detection is mechanical and can run without a model call. Whether a tool was necessary and whether the agent used its result correctly depend on context, which is where a judge is useful.

Checks You Can Write in Code

Your agent already describes its tools to the model with JSON schemas. Use the same schemas to validate what the model sent back. Add rules that schemas cannot express - such as the order some calls must happen in - and you have a fast checker that can run on every call without invoking another model.

from jsonschema import Draft202012Validator   # pip install jsonschema

TOOLS = {
    "lookup_order": {
        "type": "object",
        "properties": {"order_id": {"type": "string", "pattern": "^[A-Z0-9]{8}$"}},
        "required": ["order_id"],
        "additionalProperties": False,
    },
    "issue_refund": {
        "type": "object",
        "properties": {
            "order_id": {"type": "string"},
            "amount": {"type": "number", "exclusiveMinimum": 0},
        },
        "required": ["order_id", "amount"],
        "additionalProperties": False,
    },
}
MUST_COME_AFTER = {"issue_refund": ["lookup_order"]}


def check_calls(calls: list[dict]) -> list[str]:
    problems, seen = [], []
    for i, call in enumerate(calls, 1):
        name, args = call["name"], call.get("args", {})
        schema = TOOLS.get(name)
        if schema is None:
            problems.append(f"step {i}: unknown tool {name!r}")
            continue
        for err in Draft202012Validator(schema).iter_errors(args):
            problems.append(f"step {i}: {name}: {err.message}")
        for earlier in MUST_COME_AFTER.get(name, []):
            if earlier not in seen:
                problems.append(f"step {i}: {name} called before {earlier}")
        seen.append(name)
    return problems


calls = [
    {"name": "issue_refund", "args": {"order_id": "7Q2ZK9XA", "amount": -5}},
    {"name": "refund_order", "args": {"order_id": "7Q2ZK9XA"}},
]
for problem in check_calls(calls):
    print(problem)
# step 1: issue_refund: -5 is less than or equal to the minimum of 0
# step 1: issue_refund called before lookup_order
# step 2: unknown tool 'refund_order'

An unknown tool name is the cheapest hallucination you will ever catch: the model invented refund_order because it sounded right. A schema check catches the shape of bad arguments but not their truth - "7Q2ZK9XA" is a valid order id and may still be the wrong customer's. Whether the arguments came from the request or from thin air is a question for the judge below.

Two more checks are worth adding once the basics run. Count retries with identical arguments straight after an error: repeating a call that just failed the same way is almost never a strategy. And count tool errors per session, so a run with seven failed calls and a cheerful final answer does not read as a success.

Where these checks run is a choice with consequences. Inside the agent loop, before the tool executes, a failed check can reject the call and hand the model the error message, which it will usually correct on the next step. Afterwards, over logs, the same check becomes a metric: how often the model produced a bad call in the first place. You want both numbers. A validator that quietly repairs arguments in the loop makes the agent look better in the logs than it is, so record the rejection even when the retry succeeds.

A Judge for the Questions Code Cannot Answer

Whether a call was needed, and whether the reply reflects what the tool returned, depend on meaning. Give the judge three things: the tool list the agent had, the user's request, and what the agent did. Ask for one pass or fail per criterion and a sentence of evidence.

# tool_judge.py
import json
import os

import anthropic

JUDGE_MODEL = os.environ["JUDGE_MODEL"]
client = anthropic.Anthropic()                    # reads ANTHROPIC_API_KEY
CRITERIA = ("call_decision", "right_tool", "args_grounded", "result_used")

RUBRIC = """You grade how an AI agent used its tools on one task.
Mark each criterion true (pass) or false (fail):
- call_decision: it called a tool when it needed information or an action it did not have, and not otherwise.
- right_tool: each tool it called was the most suitable available tool for its purpose.
- args_grounded: argument values come from the request or earlier tool results; nothing is invented.
- result_used: its reply reflects what the tools returned, including any errors.
Reply with JSON only, for example:
{"call_decision": true, "right_tool": false, "args_grounded": true, "result_used": true, "evidence": "used search_docs where lookup_order returns the order directly"}"""


def parse_json(text: str) -> dict | None:
    start, end = text.find("{"), text.rfind("}")
    if start == -1 or end <= start:
        return None
    try:
        return json.loads(text[start:end + 1])
    except json.JSONDecodeError:
        return None


def judge_tool_use(tools: str, request: str, actions: str) -> dict | None:
    prompt = f"{RUBRIC}\n\nTOOLS:\n{tools}\n\nREQUEST:\n{request}\n\nWHAT THE AGENT DID:\n{actions}"
    msg = client.messages.create(
        model=JUDGE_MODEL, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = msg.content[0].text
    verdict = parse_json(text)
    if verdict is None or not all(isinstance(verdict.get(k), bool) for k in CRITERIA):
        return None   # not scored - keep a count of these, do not fold them into pass or fail
    return verdict

Include the tool descriptions exactly as the agent saw them. A judge that does not know lookup_order exists cannot tell that a search was the wrong choice, and a judge given better descriptions than the agent had will blame the agent for your documentation.

Result handling is where the most expensive tool failures hide, because the call itself looks perfect. The agent calls issue_refund, the tool returns an error, and the agent tells the customer the refund is on its way. Code can catch the crude version - an error result followed by a final message with no retry in between - and flag the session. The judge then answers the real question: does the reply match what the tools said? Put both the tool result and the reply in front of it; a judge that sees only the conversation cannot know the tool failed.

Measuring Selection Against Labels

For tool choice specifically, labels beat a judge. Take 100 or so real turns, write down which tool should have been called (or "none"), and compare with what the agent did. Per-tool precision and recall tell you which way it errs: low precision on a tool means the agent reaches for it too often; low recall means it misses cases where the tool was needed.

from collections import Counter


def selection_report(pairs: list[tuple[str, str]]) -> None:
    """pairs: (expected_tool, actual_tool) per labeled turn; "none" means no call."""
    tp, fp, fn = Counter(), Counter(), Counter()
    for expected, actual in pairs:
        if expected == actual:
            tp[expected] += 1
        else:
            fp[actual] += 1
            fn[expected] += 1
    for tool in sorted(set(tp) | set(fp) | set(fn)):
        precision = tp[tool] / (tp[tool] + fp[tool]) if tp[tool] + fp[tool] else float("nan")
        recall = tp[tool] / (tp[tool] + fn[tool]) if tp[tool] + fn[tool] else float("nan")
        print(f"{tool:14} precision {precision:.2f}  recall {recall:.2f}")


selection_report([
    ("lookup_order", "lookup_order"), ("lookup_order", "search_docs"),
    ("none", "search_docs"), ("search_docs", "search_docs"),
    ("none", "none"), ("issue_refund", "issue_refund"),
])
# issue_refund   precision 1.00  recall 1.00
# lookup_order   precision 1.00  recall 0.50
# none           precision 1.00  recall 0.50
# search_docs    precision 0.33  recall 1.00

Read the example as a diagnosis: search_docs has precision 0.33, so two of the three times the agent searched, it should have looked the order up or not called anything. "none" has recall 0.50, so half the turns that needed no tool got one anyway. That is an over-caller, and the fix is usually in the tool descriptions, not the model. Six turns is only a demonstration; with real labels, check the counts per tool before you believe a ratio (see how many eval examples you need).

Pitfalls, and How to Check Your Work

• Two right answers. Sometimes two tools both work. Label the set of acceptable tools, not one, or your precision will punish reasonable choices.
• Scoring the call, not the task. An agent can make every call correctly and still pursue the wrong goal. Pair tool scores with a task completion score.
• Errors that are the environment's fault. A timeout from a flaky API is not the agent's mistake; ignoring it is. Score how the agent handled an error, and track raw error rates separately so infrastructure noise does not land on the agent's report card.
• Stale schemas. If the checker's schemas drift from the ones the agent is given, you get false failures. Load both from the same source.
• Per-call rates that hide sessions. A per-call success rate weights long sessions heavily: one session with 200 calls outweighs fifty short ones. Report per-call and per-session numbers, and read the sessions with the most failures, not only the average.
• An uncalibrated judge. Before trusting call_decision, label 50 turns yourself and compare. Necessity is the criterion judges and humans disagree on most, because "needed" depends on what the agent already knew.

Evaluating Tool Use with Failproof AI

Failproof AI runs code-based and LLM-based evaluations against complete session traces. Use code checks for schema validity, ordering, error rates and repeated calls; use an LLM judge for necessity, argument grounding and whether the final response reflects the tool result. Each result remains linked to the underlying call and output, so a failed score has evidence a reviewer can inspect.

EvalResult(
    score=Score(
        len([e for e in session.events_of_type("tool_result") if e.payload.get("status") == "ok"])
        / max(1, session.count("tool_result"))
    ),
    metrics={"tool_calls": Metric(session.count("tool_use"), unit="calls")},
    reasoning="Share of tool results that came back ok.",
)

If you already have tool-use evaluations, bring the existing evaluation suite rather than rewriting it. Keep tool availability, selection, argument validity and result handling as separate results; otherwise one combined score cannot tell you whether the fix belongs in a schema, tool description, prompt, integration or model.

Test each evaluation against labeled sessions and version material changes. Then query the sessions where tool selection scored poorly or compare results by agent:

fp evals --since 7d --score tool_choice:0..0.5
fp evals --agent-id checkout-agent --aggregate

Alert separately on tool-result reliability and tool-selection quality because they usually have different owners. A rise in tool errors often belongs to the integration team; a decline in selection quality may follow a prompt, tool-description or model change.

Across many sessions, Failproof AI analyzes these evaluation results with trace evidence to identify recurring failure modes and group them into findings with a recommended fix. The right fix may be a clearer tool description, stricter schema, prompt change or integration repair. If the evidence identifies a recognizable high-risk action, a tested behavioral policy can prevent it at runtime; keep the evaluation running afterward to confirm the underlying behavior improves.

When You Do Not Need a Tool-Call Eval

If your agent has two or three tools with clearly separate jobs, selection errors are rare and schema validation in the agent itself - rejecting a bad call before it executes - covers most of the rest. Many frameworks already validate arguments against the schema; check yours before writing your own. A task completion score will catch the remaining failures, and you can add a tool-level eval later, when it stops telling you why.

The signal that you need one is a growing tool list with overlapping descriptions, or repeated incidents where the final answer hides a bad call. At that point, the four-part breakdown above tells you where to look first.