Do guardrails slow down AI coding agents? Benchmark results

the short answer

Not when the policies are targeted. In Failproof’s FIRE study on 87 Terminal-Bench 2.1 tasks, failure-derived runtime policies raised the share of tasks solved on both attempts for all three models tested, by 6.1 points pooled. On the tasks where a policy recognized its failure pattern, Luna and Terra succeeded 69% of the time versus 37% without. Broad, noisy blocks are what slow developers down; narrow rules mostly stay out of the way.

Benchmark: Terminal-Bench 2.1, 87 tasks, two attempts each, with and without policies
Models: Three GPT-5.6 tiers: Luna, Terra and Sol
Repeated success: +6.1 points pooled (p = 0.025)
When a policy recognized its pattern: 37% → 69% of Luna and Terra attempts (p = 0.003)
Cost: Luna −5.2%, Terra +0.9%, Sol +47.7%
Main limit: Text-matching recognized 2 of 18 reworded tasks

Why Admins Ask This First

Every rollout conversation hits the same objection: developers chose agents to go faster, and guardrails sound like friction. The honest answer depends on what the guardrails do. A rule that denies the whole Terraform CLI will slow people down. A rule that only fires on terraform destroy, or that tells the agent one missing fact at the moment it is about to make a known mistake, mostly does not, and the data below says it can make agents finish more work.

Repeated Success Rose for All Three Models

Repeated success, passing both attempts, is the number that matters for a team: it is the difference between an agent that can sometimes do the task and one that does it every time. Pooled across models it rose 6.1 points (p = 0.025).

Tasks passed on both attemptswithout policieswith policies

Terminal-Bench 2.1, 87 tasks, two attempts per task. Axis zoomed to 45–90%. Source: FIRE (arXiv:2609.26048).

Best-of-Two Moved Less

Tasks solved at least once in two attemptswithout policieswith policies

Same runs as above. Policies mostly turned solutions the model could sometimes reach into ones it delivered reliably.

Where the Gain Comes From

Across the other 148 Luna and Terra model-and-task combinations, where no policy recognized its pattern, the total did not move: 197 of 296 attempts passed before and after. That is the productivity answer in one line. A narrow policy changes outcomes where it applies and leaves everything else alone.

Model	Tasks a policy recognized	Without	With policies
Terra	14	10/28 · 36%	20/28 · 71%
Luna	12	9/24 · 38%	16/24 · 67%
Combined	26	19/52 · 37%	36/52 · 69%

The Specific Instruction Matters, Not the Interruption

Generic interruptions barely moved results: 36% to 43% against a 39% baseline. The targeted policy reached 61%, though with 14 tasks the comparison with the timing-matched generic message is not significant at the usual 5% level (p = 0.061). The deployed instruction for one failure reads: “Make it survive this shell: start it in a new session with all streams redirected, then probe it from a separate, later command.” That is why the reason text you write matters; see block or warn.

Success on the 14 policy-covered tasks, by messagebaseline (no message)with that message

“Generic” is a message at the same moment as the policy. Targeted policy vs that timing-matched generic message: +25 points, 95% CI [7, 46], p = 0.061.

Cost Depends on How Often Policies Step In

Terra’s policies stepped in on 15 of 87 tasks and cost almost nothing extra. Sol’s broader set intervened on 81 tasks and raised cost by nearly half. A policy that fires on most calls is a tax; one that fires on the risky few is close to free.

Model	Change in successful attempts	Cost change
Luna	+3.4 points	−5.2%
Terra	+6.3 points	+0.9%
Sol	+5.2 points	+47.7%

What This Does and Does Not Show

These are runtime policies derived from observed failures on a benchmark, mostly instructions with some action denials. They are not the security denies an admin writes for terraform destroy, which are rare by design and do not touch most work.
Recognition is the weak point: when 18 tasks were reworded, text-matching rules recognized only 2. Decision models are how Failproof addresses this, and they need their own measurement.
Three models, one benchmark. Measure on your own traffic in observe mode before enforcing.

How to Keep Guardrails from Slowing Your Team

Deny only irreversible or security-boundary actions. Everything else is instruct (the call continues with guidance) or nothing.
Write reasons that tell the agent what to do instead.
Backtest a draft in Admin → policy editor against the last 30 days of fleet traffic; it counts the working calls the draft would have interrupted.
Deploy in observe mode and count how often each policy would fire under Observe → policy. A policy firing on a large share of calls needs narrowing.
For broad rules, Jev review can clear false positives such as terraform plan under a Terraform block, once the FailproofAI/jev-policies pack is installed and Jev is on.

FAQ

Does policy evaluation add latency?

Deterministic policies run locally in the Failproof daemon with no network call; our engineering write-up measured the daemon path at about 0.7 ms p50. Jev checks add a model call, and failproofai jev test reports the latency on your machine.

Will developers notice the policies?

Mostly when a deny fires, and the agent explains why and what it will do instead. Commands developers type themselves are not gated.

Can I reproduce these numbers?

The FIRE paper publishes the method, the frozen policies and the analysis; the research page links to the artifact.

Sources

Checked against the sources below on October 21, 2026. Harness settings, hook capabilities and cloud features change between versions.