policy-as-code for ai coding agents

Why Code Instead of a Settings File

• One rule, every harness. Harness settings have different schemas and capabilities. A Failproof policy is written once against normalized tool names (Bash, Read, Write, Edit) and evaluated the same way on every supported harness.
• Logic, not patterns. Code can check the working directory, combine conditions, and return a tailored reason.
• Reviewable and testable. Pull requests, unit tests and backtests work the way they do for any other code.

The Lifecycle

Drafting from Evidence

You do not have to start from a blank file. When an audit finding becomes an issue, generate policy first says whether a policy can express the problem at all (a no policy result means the fix is an alert, a workflow change or a person), then drafts source into Admin → policy editor from the issue’s finding, root cause and recommendation. From a terminal, fp policies compose "Block git push --force on release branches" prints a draft and publishes nothing. Either way, review it before publishing.

Test and Backtest

fp policies test runs the file locally against a synthetic event; a policy whose match does not cover the event reports skipped. Backtesting is a dashboard feature: the editor replays the draft against calls your fleet already made (every agent, 30 days by default) and counts the working calls it would have interrupted. Keep an explicit failproofai policies --install --custom <file> step in CI, because convention files that fail to load are skipped at enforcement time rather than failing loudly.

fp policies test ./checkout.policy.mjs --command "git push --force" --expect deny
fp policies test ./checkout.policy.mjs --command "git push" --expect allow
failproofai policies --install --custom ./checkout.policy.mjs --scope project   # enforce on this machine only

Rules the Runtime Enforces

• A policy function has a 10-second deadline, and module loading has its own 10-second deadline.
• A thrown exception or timeout is logged and treated as allow(). Choose deliberately how your own dependency failures should resolve.
• Built-in policies run before custom ones. The first deny stops evaluation.
• Convention files must end in policies.js, policies.mjs or policies.ts.
• Separately, an unreachable daemon fails closed, and a declared pack that will not load denies the events its policies covered.

Share Policies as a Pack

The pack version is the commit you publish from. The digest is recorded at install and re-verified before every load, so a pack cannot change under a machine. SHA256SUMS proves the bytes match the release, not who published it; publish from a repository whose write access you control. Adding the failproofai-policies topic puts the repository up for consideration; the policy hub lists it once it has a release whose manifest verifies against its own SHA256SUMS.

failproofai publish --init           # scaffold a working policy file
failproofai policies -i -c ./infra.mjs   # enforce it on this machine first
failproofai publish                  # build manifest + bundle + SHA256SUMS, create the GitHub release

# on any other machine
failproofai policies show acme/infra-guardrails   # read the manifest; runs no code
failproofai policies add acme/infra-guardrails    # install, pinned to the resolved release

Make It Robust with a Decision-Model Check

Policies decide with code. Jev checks decide with yes/no questions the decision model answers about each call, declared with semanticPolicies.add() beside your customPolicies.add() policies. Checks block or warn on their own, so they catch calls your patterns never anticipated. A policy marked authority: "reviewable" with a reviewedBy list lets Jev clear its verdict only through the checks it names; every other policy is hard.

We write custom checks like this one for your own rules: talk to us and tell us what should never happen on your machines. Jev comes with Failproof; turn it on whenever you choose, with failproofai jev setup or a Cloud machine key that carries jev:evaluate.

import { customPolicies, semanticPolicies } from "failproofai";

// A Jev check: yes/no questions the decision model answers about each call.
// Every probe must hold for it to fire.
semanticPolicies.add({
  name: "dns-change-outside-review",
  title: "Changed DNS outside the reviewed Terraform module",
  appliesTo: ["shell", "write"],
  mode: "deny",
  userCanOverride: false,
  probes: [
    {
      id: "changes_dns",
      instructions: "Does this call create, change or delete DNS records or hosted zones, directly or through a script, SDK, CloudFormation, CDK or another infrastructure tool?",
      criteria: { true: "It changes DNS.", false: "It only reads DNS, or does not touch DNS." },
    },
    {
      id: "bypasses_review",
      instructions: "Does it make the change itself, rather than editing the dns/ Terraform module for a pull request?",
    },
  ],
  guidance: "DNS changes go through the dns/ Terraform module and a pull request. Propose the change there instead of running it.",
});

// The regex policy above, now reviewable: Jev may clear its deny only through
// this check, so "aws route53 help change-resource-record-sets" is let through
// while a real change stays denied.
customPolicies.add({
  name: "protect-route53",
  description: "Agents never change Route 53 directly; DNS changes go through review",
  match: { events: ["PreToolUse"] },
  authority: "reviewable",
  reviewedBy: ["dns-change-outside-review"],
  fn: async (ctx) => { /* unchanged from the policy above */ },
});

• A reviewedBy list must name checks that exist; failproofai publish refuses a pack that names one it does not declare.
• Keep the policies whose failure you cannot afford hard: the model is probabilistic, and a reviewable deny can be cleared.

Roll Back Exactly

A rollback mints a new generation carrying the old set, so history stays append-only and the decisions that explain what went wrong are kept. These commands need a signed-in session (fp login); they exit with an error under an API key.

fp fleet history <machine-id>
fp fleet rollback <machine-id> <generation>
fp policies disable <policy-id>    # take one policy off every deployment carrying it
fp policies enable <policy-id>     # and put it back