50.6%→54.0%
Failproof research · T-Bench 2.1
64% → 71%with the same Terra modelJust by policies.derived from failures
We read failed coding-agent runs, found mistakes that kept recurring, and wrote short instructions that appear when the agent is about to repeat one. Pass rate rose for all three models, with no fine-tuning or model change.
The idea
We read every failed run. The same few mistakes kept coming back.
The agent often knew how to do the task but missed one fact about its environment: a server dies with its shell, opening a database can destroy recovery evidence, or one extra byte fails an exact grader. For each recurring mistake, we wrote a rule that spots the risky moment and gives the missing instruction.
01 · Results
All three models completed more tasks reliably.
Each model attempted the same 87 T-Bench 2.1 tasks twice, with and without policies. The charts show how often it succeeded at least once and how often it succeeded both times.
55.2%→60.9%
64.4%→73.6%
withoutwith policies
withoutwith policies
+6.1 points pooledRepeated success · p = 0.025
Best-of-two success moved less. Policies chiefly converted solutions the model could sometimes reach into solutions it delivered repeatedly.
Where the gain comes from
When a policy spotted its failure pattern, success rose from 37% to 69%.
These policies each target one specific mistake, so they intervene on only a small part of the benchmark.
ModelRecognized tasksWithoutWith policies
Terra1410/28 · 36%20/28 · 71%
Luna129/24 · 38%16/24 · 67%
Combined2619/52 · 37%36/52 · 69%
Across the other 148 model-and-task combinations, the total did not move: 197 of 296 attempts passed both before and after.
02 · Controlled experiment
The specific instruction made the difference.
We interrupted the agent at the same moments on the same tasks, but changed what the message said. A targeted policy beat generic prompts to reconsider or verify the work.
Success on the 14 tasks covered by these policies, with two attempts per setup. Targeted policy versus the generic message: +25 points, 95% CI [7, 46], p = 0.061.
03 · Method
Turn a recurring failure into a timely instruction.
- 01Observe
Read failed runs and identify the exact mistake.
- 02Specify
Write the fact or instruction the agent was missing.
- 03Trigger
Watch the task and the agent's actions. Show the instruction only when that mistake is about to happen.
- 04Freeze
Lock the policies, matching rules, tasks, and analysis before the final run.
- 05Test
Compare them with no policy, generic advice, and the same interruption timing.
04 · Cost
Policies that intervene rarely add little cost.
Terra's policies stepped in on 15 of 87 tasks and increased cost by 0.9%. Sol's broader set intervened on 81 tasks and increased cost by 47.7%.
The figures show the change in successful attempts across all 87 tasks. With policies, Terra finished 4.6 points below Sol without policies, at 45% of Sol's cost per attempt.
05 · Scope
What these results show, and where they stop.
These policies repair mistakes found in these models on these tasks. When 18 new task descriptions referred to the same situations in different words, the text-matching rules recognized only two.
Better recognition is needed before the policies can transfer reliably to unfamiliar task wording.
06 · Example
Almost done. Then the server died with the session.
Without policy
$ python3 server.py &
$ curl localhost:PORT/health
200 OK
# session ends
connection refusedWith policy
policy make the process survive this shell;
redirect streams, then probe again
$ setsid -f python3 server.py …
$ curl localhost:PORT/health
200 OKSimplified from real sessions. The instruction is the deployed policy text, trimmed.
Across three model tiers22 → 38
of 46 service-task attempts passed after adding the policy. The largest model made the same mistake.
Destroying the evidence
The agent opened a damaged SQLite database before copying it. The read removed the write-ahead log it was supposed to recover.
Right meaning, wrong byte
A 38 MB transformation was correct everywhere except for one colon that the prompt specified literally.
A check that checks nothing
The agent declared success after its verification command failed with python3: command not found.
customPolicies.add({
name: "service-persistence",
match: { events: ["Stop"] },
fn: async (ctx) => {
const s = load(ctx);
if (!s.needsPersistentService || !s.launchedService) return allow();
if (s.detached && s.probedFromLaterCommand) return allow();
return instruct("Make it survive this shell: start it in a new session with all " +
"streams redirected, then probe it from a separate, later command.");
},
});Paper + policies
Inspect the evidence.
Implement it for your agent.
Agarwal, N. and Jain, N. (2026). FIRE: Failure-Informed Runtime Engineering for Reliable Language-Model Agents.