Without policy
$ python3 server.py & $ curl localhost:PORT/health 200 OK # session ends connection refused
Failproof research · How to control coding agents
37% → 69%on the failures a policy recognized.Same model. Half the failures.
Telling a coding agent to "verify your work" changed nothing. We read its failed runs, wrote a short rule for each mistake that kept coming back, and fired it only when the mistake was about to repeat. The agent followed the rule. No fine-tuning, no model change.
01 · Recovery
Where a policy recognized the failure, it cut failures in half.
Each policy targets one known mistake. Everywhere else it stayed silent, and the pass rate held at 67%.
Failure rate
63% → 31%
02 · Example
Almost done. Then the server died with the session.
With policy
policy make the process survive this shell; redirect streams, then probe again $ setsid -f python3 server.py … $ curl localhost:PORT/health 200 OK
48% → 83%
of service-task attempts passed with this one policy, across three model tiers. The largest model made the same mistake.
The whole policy
customPolicies.add({
name: "service-persistence",
match: { events: ["Stop"] },
fn: async (ctx) => {
const s = load(ctx);
if (!s.persistentService || !s.launched) return allow();
if (s.detached && s.probedAfter) return allow();
return instruct("Make the process survive this shell; redirect streams, then probe again.");
},
});
Destroying the evidence
Opened a damaged SQLite database before copying it, wiping the log it was asked to recover. A deny rule now blocks the open until a full copy exists.
Right meaning, wrong byte
A 38 MB transformation was correct except for one literal character. The policy asks for a mechanical diff before stopping.
A check that checks nothing
Declared success after its verification command failed with python3: command not found.
03 · T-Bench 2.1
Every model scored higher. Same weights.
All 87 tasks, two attempts each, with and without policies. The score rose for all three tiers, and the share of tasks passed on both tries rose more.
T-Bench 2.1 score
without with policies
Luna59.8 → 63.2%
Terra64.4 → 70.7%
Sol75.3 → 80.5%
Passed both tries
+6.1 pts pooled · p = 0.025
Luna50.6 → 54.0%
Terra55.2 → 60.9%
Sol64.4 → 73.6%
04 · Cost
Cheaper than upgrading the model.
On the tasks its policies cover, the mid-tier model with policies passed more often than the flagship without them, for half the spend. Narrow policies that fire rarely add almost nothing to the bill.
Terra + policies
71.4%
$7.19
Sol, no policies
64.3%
$14.48
Full-suite cost change
Luna −5.2%
Terra +0.9%
Sol +47.7% (broad set)
14 covered tasks, so the gap is imprecise (95% CI [−21.4, 35.7]). Policies and model upgrades fix different failures; together they cost less than upgrading alone.
Inspect the evidence.
Control your agents.
Policies protect delivery; they don't add capability. Results cover these models on these tasks with small samples. Agarwal, N. and Jain, N. (2026). FIRE: Failure-Informed Runtime Engineering for Reliable Language-Model Agents. arXiv:2609.26048