security review checklist for ai coding agents

1. Reach

1. Which harnesses do developers actually use? (Survey, or run failproofai audit on pilot machines; do not assume.)
2. Which cloud profiles, kubeconfig contexts and database URLs are present in a typical developer shell? An admin’s?
3. Is there a scoped profile for agent work, with a short session duration?
4. Which on-call or break-glass roles are long-lived, and on whose machines?

2. Enforcement

1. Are destructive infrastructure actions denied before they run (destroy, DNS deletes, prod kubectl delete)?
2. Are .env reads and environment dumps denied?
3. Are force-pushes and direct pushes to protected branches denied?
4. Do the rules catch the same action via a script, an SDK call and a Makefile target? (Test it.)
5. Is ambiguous work steered with guidance rather than blocked, so developers do not route around the tool?

3. Tamper Resistance

1. Can the agent uninstall, pause or reconfigure the policy client? (Ask it to.)
2. What happens when the evaluator is down: fail open or fail closed?
3. Can the agent edit policy files or packs, and would the change take effect?
4. Does the agent have sudo? Does the evaluator run as a separate user?

4. Audit

1. Can you list every enrolled machine and the policies it runs?
2. Can you find every tool call an agent made in a given hour on a given machine?
3. Can you link a CloudTrail event to the agent session that produced it?
4. Are transcripts collected, and is that decision documented?

5. Operations

1. Can a policy run in observe mode before enforcing?
2. Is every policy version immutable, and is rollback exact?
3. Who gets alerted on a production deny or a detected secret?

How Failproof Answers the Tamper and Operations Questions

• block-sudo is on by default in the coding agent pack.
• block-failproofai-commands is always on: agents cannot change Failproof’s files or use the local pause. A human’s pause covers one session, always expires, and never pauses Cloud-managed policies.
• Packs are pinned by SHA-256 digest at install and re-verified before every load, so they cannot change under the machine.
• On a machine configured for the daemon, an unreachable daemon fails closed; a declared pack that will not load denies only what its policies covered.
• Cloud deployments support observe mode, immutable versions and generation rollback.

How This Maps to Frameworks

OWASP LLM06:2025 Excessive Agency lists excessive functionality, permissions and autonomy as root causes, and recommends human approval for high-impact actions and authorization in downstream systems. Areas 1 and 2 address permissions and functionality; deny decisions with a redirect implement human-in-the-loop for high-impact actions. NIST’s AI RMF organizes the same work into Govern, Map, Measure and Manage; this checklist is mostly Map (reach) and Manage (enforcement, tamper resistance), with audit feeding Measure.