ai agent deleted route 53 dns records

What Happened

This is an anonymized reconstruction of an incident a platform team described to us. The team had no visibility into its coding agents’ tool calls. During a CDN migration, an engineer asked an agent to clean up the old CDN configuration. The agent interpreted that as removing the old DNS records too, and deleted endpoints from several non-production Route 53 hosted zones.

The team noticed broken environments, pulled CloudTrail, and found ChangeResourceRecordSets calls with DELETE actions. The role session in the events led back to the engineer’s SSO session. Nobody had typed those calls by hand; the agent had. The records were restorable because it was a test environment. The engineer had production access in the same shell.

The Failure Chain

Why Existing Controls Did Not Stop It

• IAM allowed it. The engineer’s role was allowed to change records. IAM evaluates the principal, not the intent.
• Harness settings did not cover it. Global agent settings existed, but a deny rule on one command string does not catch Terraform, SDK calls or a script that wraps the same API.
• CloudTrail is after the fact. It is the right forensic record and the wrong prevention tool.
• Nobody could see agent tool calls. Without session telemetry the team could not tell how often agents were already touching DNS.

The Policy That Would Have Stopped It

This policy denies Route 53 record and zone changes by operation name, so the CLI, the Python, Go and JavaScript SDKs and scripts the agent writes are all caught, and leaves read-only calls alone. It also denies a command that only mentions the operation, such as aws route53 help change-resource-record-sets; that false positive is the price of matching every SDK spelling. The deny reason tells the agent where DNS changes go instead, so it proposes a reviewed change rather than looking for another route.

import { customPolicies, allow, deny } from "failproofai";

// Route 53 record and zone changes by operation name, in any spelling: CLI
// (change-resource-record-sets), Python (change_resource_record_sets), Go and
// JS SDKs (ChangeResourceRecordSets, changeResourceRecordSets), in a command
// or in a script being written.
const R53_MUTATION = /\b(change|delete|create)[-_]?(resource[-_]?record[-_]?sets|hosted[-_]?zone)/i;
const R53_TAGS = /\broute53\b.*\bchange-tags-for-resource\b/;

customPolicies.add({
  name: "protect-route53",
  description: "Agents never change Route 53 directly; DNS changes go through review",
  match: { events: ["PreToolUse"] },
  fn: async (ctx) => {
    const t = ctx.toolName ?? "";
    const text = t === "Bash" ? String(ctx.toolInput?.command ?? "")
      : ["Write", "Edit"].includes(t) ? String(ctx.toolInput?.content ?? ctx.toolInput?.new_string ?? "")
      : "";
    if (!R53_MUTATION.test(text) && !R53_TAGS.test(text)) return allow();
    return deny("Agents may not change Route 53 directly. Put the change in the dns/ Terraform module and open a PR; CI applies it after review.");
  },
});

Test Cases

Run each row with fp policies test ./protect-route53.policy.mjs --command "<command>" --expect <decision> before you publish. A Terraform plan that removes records is a different shape; the Terraform policy covers it.

Catching the Shapes a Regex Misses

A script that already exists in the repository (python scripts/cleanup_dns.py) is run, not written, so its content never passes through the policy. Two additions close part of that gap. The coding agent pack’s opt-in block-aws-cli is marked reviewable by the production-infra-change semantic check: with Jev on and the FailproofAI/jev-policies pack installed, read-only calls are cleared while mutations against live infrastructure stay denied. And Jev judges the tool call in the context of the user’s request, which helps with inline code; it does not read script files from disk. See how agents get around deny rules.

Make It Robust with a Decision-Model Check

A pattern only catches the spellings it lists, so every new tool, wrapper or script needs another pattern. A decision-model check asks what the call does and whether the person asked for it, so it holds up when the wording changes. Keep the regex policy as a hard floor and add the check on top.

This is a custom check for this one rule. The regex policy keeps denying what it matches; marking it reviewable by the check lets Jev clear only the matches that change nothing.

We write custom checks like this one for your own rules: talk to us and tell us what should never happen on your machines. Jev comes with Failproof; turn it on whenever you choose, with failproofai jev setup or a Cloud machine key that carries jev:evaluate.

import { customPolicies, semanticPolicies } from "failproofai";

// A Jev check: yes/no questions the decision model answers about each call.
// Every probe must hold for it to fire.
semanticPolicies.add({
  name: "dns-change-outside-review",
  title: "Changed DNS outside the reviewed Terraform module",
  appliesTo: ["shell", "write"],
  mode: "deny",
  userCanOverride: false,
  probes: [
    {
      id: "changes_dns",
      instructions: "Does this call create, change or delete DNS records or hosted zones, directly or through a script, SDK, CloudFormation, CDK or another infrastructure tool?",
      criteria: { true: "It changes DNS.", false: "It only reads DNS, or does not touch DNS." },
    },
    {
      id: "bypasses_review",
      instructions: "Does it make the change itself, rather than editing the dns/ Terraform module for a pull request?",
    },
  ],
  guidance: "DNS changes go through the dns/ Terraform module and a pull request. Propose the change there instead of running it.",
});

// The regex policy above, now reviewable: Jev may clear its deny only through
// this check, so "aws route53 help change-resource-record-sets" is let through
// while a real change stays denied.
customPolicies.add({
  name: "protect-route53",
  description: "Agents never change Route 53 directly; DNS changes go through review",
  match: { events: ["PreToolUse"] },
  authority: "reviewable",
  reviewedBy: ["dns-change-outside-review"],
  fn: async (ctx) => { /* unchanged from the policy above */ },
});

• cdk destroy EdgeDnsStack or aws cloudformation delete-stack --stack-name edge-dns: DNS removed without the operation name ever appearing in the command.
• A one-off script the agent writes in a language or SDK the pattern does not spell out.
• aws route53 help change-resource-record-sets: matched by the pattern, cleared by the check because nothing changes.

The IAM Backstop

Give agent work a profile that cannot change records in production hosted zones, and enforce that agents use it with an AWS profile policy. If you run AWS Organizations, a service control policy can deny route53:DeleteHostedZone outside a break-glass role. Policies make the wrong action unlikely; IAM makes it impossible.

Roll It Out Without Breaking DNS Work

1. Publish the policy as a version: fp policies publish protect-route53 ./protect-route53.policy.mjs (needs fp login with policies:write).
2. Deploy it in observe mode to the machines with AWS access, from Admin → enforcement or with fp fleet deploy <machine-id> --add protect-route53:observe.
3. After a week, read Observe → policy. Every match is a DNS change an agent attempted.
4. Switch to enforce and confirm each machine reports the new deployment under Admin → enforcement.
5. Create an alert for future attempts: a per_event or custom_sql alert over policy decisions for protect-route53, with a Slack channel. Check the field names with fp query schema first.