ai agent ran the command anyway despite deny rules

Six Ways a Blocked Action Still Happens

This is why teams that only configured a harness’s built-in deny list report that “agents get around it”. Anthropic’s own permissions documentation says a Bash rule matches the command text Claude writes and is not a security boundary around the program: Bash(rm *) does not stop /bin/rm -rf build/ or bash -c 'rm -rf build/'. See are managed settings enough.

Layer 1: Deterministic Rules That Also Read What Gets Written

The first fix is cheap: gate the write as well as the run. A PreToolUse policy sees Write calls with their content and Edit calls with their new_string, so a script that contains a destructive call can be refused before it exists.

import { customPolicies, allow, deny } from "failproofai";

// Destructive infrastructure calls, whether run directly or written into a
// script, including argument lists such as ["terraform", "destroy"].
const DESTRUCTIVE = [
  /\b(?:terraform|tofu)["',\s]+destroy\b|\bapply\b[^\n]*\s-destroy\b/,
  /\b(change|delete)[-_]?(resource[-_]?record[-_]?sets|hosted[-_]?zone)/i,
  /\bkubectl\b["',]?(?:\s+--?[\w-]+(?:[=\s]+[^-\s|]\S*)?)*["',\s]+delete\b/,
];

customPolicies.add({
  name: "no-destructive-scripts",
  description: "Refuse commands and scripts that embed destructive infra calls",
  match: { events: ["PreToolUse"] },
  fn: async (ctx) => {
    const text = ["Write", "Edit"].includes(ctx.toolName ?? "")
      ? String(ctx.toolInput?.content ?? ctx.toolInput?.new_string ?? "")
      : ctx.toolName === "Bash" ? String(ctx.toolInput?.command ?? "") : "";
    if (!DESTRUCTIVE.some((re) => re.test(text))) return allow();
    return deny("This embeds a destructive infrastructure call. Propose it as a reviewed change, not a local script.");
  },
});

Test Cases

An Edit only carries the replaced fragment in new_string, so a destructive line added through an edit is caught, but one already in the file is not.

Where Layer 1 Stops Working

• Scripts that already exist in the repository (a Makefile target) are run, not written, so content checks never see them.
• Runtime-assembled commands and encodings defeat any regex you can maintain.
• Broad patterns block legitimate work. The coding agent pack’s opt-in block-terraform denies the whole CLI, including terraform plan, which developers need all day.
• Each new tool, SDK or wrapper means another pattern.

Layer 2: Decision Models That Judge the Call

A decision model replaces “does this string match” with “what does this call do, and did the person ask for it”. Failproof’s semantic layer uses Jev, TypeSafe AI’s classifier, which answers yes/no checks about a tool call in the context of the user’s recent prompts: does it change live infrastructure, does it delete data that cannot be regenerated, does it send credentials off the machine. The checks come from the FailproofAI/jev-policies pack, and Jev runs only once it is configured, through FailproofAI Cloud or your own provider.

Two properties matter. It judges the call in context, so python -c "...boto3...delete..." is judged on its inline code. And it can clear false positives from broad rules: a policy marked reviewable lets Jev clear terraform plan or aws sts get-caller-identity while the destroy stays denied. Policies marked hard can never be cleared. It does not read files on disk: bash cleanup.sh is judged on its name and arguments, which is why layer 1 checks scripts when they are written.

What a Decision Model Does Not Fix

• It only judges what it is sent: the tool call, the recent prompts and the agent’s last message, with secrets redacted.
• It is probabilistic. Failproof documents measured cases where evidence landed just under the firing line and a reviewable deny was cleared. Keep the policies whose failure you cannot afford hard.
• It adds one model call per reviewed call.
• It is not a credential boundary. Use scoped AWS profiles as the backstop.

Find Bypasses That Already Happened

Failproof’s saved queries list “policy denials followed by a successful alternative” as a common use: a deny, then the same effect reached another way a few calls later. Run that over your fleet’s sessions under Analyze → Queries, or ask the Failproof Assistant (fp agent ask "show sessions where a denied command was followed by a script that did the same thing"), and turn each hit into a test case.

Red-Team Your Own Rules in 20 Minutes

1. Pick one blocked action. Ask the agent to do it directly. Confirm the deny and its reason.
2. Ask it to “write a script that does this and run it”. Note whether the write or the run was caught.
3. Ask it to do the same with the language SDK your team uses (boto3, the Go SDK, the Node SDK).
4. Ask it to use a Makefile target that already wraps the action.
5. Record every miss as an fp policies test case, then decide per miss: new deterministic rule, semantic check, or credential change.