What a hallucinated call looks like
The agent decides it needs to do something and reaches for a name that sounds right: a helper function that was never written, a config file at a path that doesn't exist, an MCP tool it was not given, an API endpoint it half-remembers from a different library. It calls utils/formatDate when the file is lib/date.js. It imports a package that isn't in package.json. The call is syntactically perfect and refers to nothing.
This isn't carelessness - it's how the model works. It predicts the most plausible next token, and a plausible-sounding tool name is exactly what it's good at producing. It has no built-in step that checks the name against the set of things that actually exist. That check has to come from outside the model.
Why it's expensive to let it through
Left alone, a hallucinated call does one of two things, both bad. It errors - and the agent gets a bare stack trace, guesses again, and often invents a second fabrication to paper over the first, sliding toward a loop. Or worse, it succeeds against the wrong target: it writes to a real file that happened to match the invented path, or calls a real endpoint with the wrong shape. Now you have a confident, silent, wrong side effect - the hardest kind of agent failure to catch after the fact.
How to block it at the hook layer
Every tool call an agent makes passes through the harness PreToolUse hook before it executes. That is the point to validate it. failproof runs there, and for each call it checks the target against reality:
- Tools against the set actually registered with the harness and its MCP servers.
- Files and paths against what exists in the working tree.
- Commands and binaries against what is installed and on the path.
If the target is real, the call passes untouched. If it was invented, the policy blocks the call and returns the closest real matches - the tools that do exist, the files near that path, the command it probably meant. The agent reads that in the same turn and corrects course, instead of running a fabrication or spiralling on a stack trace. No agent code changes; the check lives in a local policy at the hook, one of the thirty-nine built in.
Concretely, the policy is a few lines. It runs at the PreToolUse hook, and for a file call it checks the path against the working tree before the call can run:
// block-hallucinated-files.js — deny reads/edits to files that don't exist
import { customPolicies, allow, deny } from "failproofai";
import { existsSync } from "fs";
import { resolve } from "path";
customPolicies.add({
name: "block-hallucinated-files",
description: "Block file calls the agent invented",
match: { events: ["PreToolUse"] },
fn: async (ctx) => {
if (!["Edit", "Read"].includes(ctx.toolName)) return allow();
const target = ctx.toolInput?.file_path;
const root = ctx.session?.cwd ?? ".";
if (target && !existsSync(resolve(root, target))) {
return deny("No file at " + target + " — check the path, it may not exist.");
}
return allow();
},
});Install it with failproofai policies --install --custom ./block-hallucinated-files.js, or run failproofai policies --install to enable the built-in checks for invented tools, files, and commands.
See every hallucination, tighten the net
agenteye records every blocked call, so hallucinated tools and phantom paths show up as a pattern you can query across every run - which fabricated names recur, which real targets the agent keeps missing. That tells you what to add to the allowlist or surface more prominently, so the agent stops reaching for the wrong name in the first place. Find the pattern in the trace, fix it in the policy.
Get started
failproof is free, open-source, and fully local. Install the CLI, enable the built-in policies, and invented tool and file calls get caught at the hook layer.