A runtime policy sees all events in the harness, including the tool calls an agent is about to make. It can veto a call with a reason, or add a fact to what comes back. The question is whether that actually changes how the agent plays out a task. To measure it, we put the same agent in five real games and ran each one twice: once bare, once with failproof ai policies on its tool calls.
Same model, same prompt, same tools, same harness. The only difference between the two arms is a hooks config.
| Game | Bare agent | + failproof ai |
|---|---|---|
| PokémonWin rate | 18% | 42% |
| CivilizationGame score | 224 | 244 |
| PokerBig blinds per 100 hands | −10.8 | +1.5 |
| Yu-Gi-Oh!Misplays per duel | 1.27 | 0.45 |
Why games
Every decision in a game is a tool call with a checkable context, and every game ends in a number: a win, a score, chips won or lost. And the agent's mistakes are the ordinary agent mistakes: forgetting a step before ending its turn, guessing at arithmetic it could have computed, retrying an action the environment already rejected.
We picked turn-based games with open engines, so every decision has a public fact a policy can check:
| Game | Engine | Opponent |
|---|---|---|
| Pokémon, Gen 9 random battles | Pokémon Showdown server, offline | Foul Play, an open-source tree-search bot |
| Civilization, 60 turns | Unciv, the open-source Civ V remake | Unciv's own AI civilizations |
| Heads-up no-limit hold'em | PokerTH | PokerTH's computer player |
| Yu-Gi-Oh! | EDOPro duel server | WindBot |
| DOOM | ViZDoom, Freedoom maps | The monsters |
The setup
Each run gets its own container and its own failproof ai daemon. In the policy arm the run directory carries a hooks config, so every tool call goes through the policies before it executes, and every result comes back through them. The bare arm has no config. A run is voided if the policy arm records no hook events, or the bare arm records any.
two arms · one difference
Both arms run the same agent with the same model, prompt, tools and harness. In the bare arm every tool call goes straight to the game. In the failproofai arm the same call first passes through the hooks of a frozen policy pack, which can veto a bad call with a reason before it runs, or add an exact fact to what the game returns.
A policy can add a fact after a tool call returns, warn before an action, or veto it. It never picks the move. Every veto carries a reason and the agent chooses again; most vetoes also let the agent override by repeating the exact same call, so a wrong policy can't lock it out.
One real call from the held-out Pokémon batch:
one veto, from a real held-out battle
From a real report battle. The agent tries to switch out while the opponent's Venomoth is boosted. The pk-free-turn policy denies the call before it reaches the server and tells the agent that Magnezone can hit Venomoth now, with Thunderbolt doing 415 to 488 percent, a knockout. The agent chooses again and uses Thunderbolt, which is allowed. The new state comes back with the next turn's damage table added as a fact.
Under the hood
None of these games has a seat for an agent. For each one we kept the engine and the opponent bot as their authors ship them, and replaced only the human seat with a bridge.
inside one game container
Inside one container per game: the agent session sends each tool call through failproofai's daemon and policies, then to the game tools, a Python environment that talks to a small bridge over a socket. The bridge drives the real game engine, which runs the unmodified rules and the open-source opponent bot. The environment writes exactly what it showed the agent to visible_state.json, and that file is the only game state a policy may read. The two arms differ only in whether the tool calls pass through failproofai.
- Game tools. A Python environment per game exposes a few tools to the agent over MCP:
battle_state,use_move,switch;table_state,act;end_turnand unit orders. Each returns a text view of what a human in that seat would see, and nothing from the engine's hidden state. - Bridge. Where the game has no API, a small patch on the human seat sends its view as newline-delimited JSON over a local socket and plays the reply through the game's own action functions. Each patch lives in the repo under the licence of the project it patches.
- Observation file. Whatever a tool shows the agent is also written to
visible_state.jsonin the run directory. That file is the only game state a policy may read, so "no hidden state" holds by construction.
A policy is a module that registers handlers on the hook events. This is the poker free-check veto from the repo:
customPolicies.add({
name: "poker-free-check",
description: "Hard-veto folding when checking costs nothing.",
match: { events: ["PreToolUse"] },
fn: safe((ctx) => {
if (!isTool(ctx, "act") || String(ctx.toolInput?.action).toLowerCase() !== "fold") return;
const st = visible(ctx)?.state; // reads visible_state.json, nothing else
if (st && st.to_call === 0)
return deny("Nothing to call here — checking is free and keeps your cards. " +
"Folding now only gives up the pot. Choose again.");
}),
});
How each game is wired:
- Pokémon. The official Showdown server runs offline at a pinned commit. The agent's seat is a poke-env player whose move choice waits on our tools. Foul Play runs a Monte Carlo tree search over a Rust battle engine, unchanged apart from logging in to a local server; its search time per move sets its strength. Results are re-scored from the server's own replay files.
- Civilization. Two new Kotlin files in Unciv start a headless game in which one civ is a human player driven over a socket and every other civ is the stock AI. The bridge exposes that player's empire, the tiles and units it can see, and the game's own battle previews and city-site suggestions. Four small guards stop the engine crashing without a window or sound system.
- Poker. PokerTH is built from the Debian source with a patch and run under a virtual display. The human seat's view goes out as JSON, and the reply is played through the same functions the buttons call. Two environment variables seed the deck and swap the seats, which is what makes duplicate poker possible.
- Yu-Gi-Oh! The Multirole duel server hosts two WindBot clients: the stock opponent, and ours, patched to send every decision plus the duel state the server sent that seat to our environment. The server enforces the rules and saves a replay of every duel.
- DOOM. ViZDoom runs the DOOM engine as a library. After every action the agent gets a 320×240 screenshot plus text: health, ammo, its own position, the objects drawn on that frame with bearing and distance, and a seven-column wall-distance scan from the depth buffer. Kills, damage and exits come from engine counters the agent never sees.
Keeping it honest
The rules were written down before any report run:
- No hidden state. Policies read only what the agent was shown.
- Policies never choose. Facts, warnings and vetoes only.
- Tune on dev, report on held-out. Dev rounds are the practice games we tuned the policies on; held-out games are fresh ones they never saw. After the dev rounds, each pack was frozen and hashed, and only held-out results are reported.
- Pair what can be paired. Both arms play the same Civilization maps and DOOM levels. Poker uses duplicate decks: each deck is played twice with the seats swapped, so card luck cancels.
- Score from the game. Metrics come from the engine, never from the agent's text. Crashed or interrupted games are voided and re-run, never dropped.
how a number in this post gets made
Each game goes through the same pipeline. First the opponent is calibrated so the bare agent loses a fair share of games. Then dev rounds: watch the agent play, find a repeated mistake, write or change one policy, and run again. Then the pack is frozen and hashed. Then a held-out report batch runs both arms on fresh seeds; any interrupted game is voided and re-run on its own. Finally scores are taken from the game's own records. Every number in the post is from the report step.
Results
All numbers are from held-out games.
Pokémon
Every turn is a damage calculation, and the bare agent was guessing: it knows roughly what is super effective, not whether a hit does 30% or 90%. The main policy runs the public damage calculator after every action and gives the agent the damage range of each legal move and switch, what Terastallizing would change, and late in the game, which of its Pokémon beats each of the opponent's. A few vetoes cover switching out of a free hit, a much weaker attack than another legal one, and setting up into a knockout.
| 100 + 100 battles vs Foul Play | Bare | + failproof ai |
|---|---|---|
| Win rate | 18% | 42% |
| Own Pokémon left at the end | 0.43 | 1.64 |
| Opponent Pokémon knocked out | 3.77 | 4.52 |
2.3 times the win rate, Fisher exact p = 0.0003.
Civilization
The bare agent's problem was bookkeeping. It kept ending turns with a city building nothing, no research queued, or a social policy it could have adopted. A check on every end_turn lists what is idle, next to the game's own score formula. Other policies reject city sites the game would refuse, and guard undefended cities and unescorted settlers.
| 100 paired maps | Bare | + failproof ai |
|---|---|---|
| Score at turn 60 | 224.3 | 244.1 (+8.8%) |
| Mistakes per turn | 0.99 | 0.04 |
| Technologies | 9.8 | 11.8 |
The policy game beat its bare twin on 64 of 99 decided pairs (p = 0.002).
Poker
Fifty hands a session, heads-up. The bare agent's leak was passivity: folding strong hands, checking hands it should bet. The policy gives hand strength, equity against a random and a strong range, pot odds, and the opponent's tendencies counted from the public table log. One hard veto: never fold when checking is free.
| Held-out sessions | Bare | + failproof ai |
|---|---|---|
| Big blinds per 100 hands | −10.8 | +1.5 |
| Strong hands folded, per 100 decisions | 0.50 | 0.21 |
Deck for deck over 102 duplicate decks, the edge is +9.7 big blinds per 100 hands, 95% interval −10.3 to +28.3. That crosses zero: a lean, not a result.
Yu-Gi-Oh!
The policies build checklists from the engine's list of legal moves before entering battle and before ending the turn, block skipping Main Phase 2 with summons still available, show a card's text the first time the agent uses or passes on it, and show the worst-case damage the opponent can deal next turn.
| 40 + 40 duels | Bare | + failproof ai |
|---|---|---|
| Misplays per duel | 1.27 | 0.45 |
| Win rate | 57.5% | 65% |
Misplays fell by two thirds. The 7.5-point win gain is not significant at 40 duels per arm.
DOOM
The agent's real problem in DOOM is navigating 3D space from a small screenshot and text, which no policy fixes. The final pack is four vetoes: no firing with no enemy visible, aim before firing, don't repeat a move the game reported as blocked, no long bursts at low health with enemies close.
| 30 paired maps | Bare | + failproof ai |
|---|---|---|
| Shots with no enemy visible, per run | 0.87 | 0 |
| Blocked moves, per run | 12.3 | 8.7 |
Wasted shots went to zero (p = 0.001). Exits and survival didn't move.
What we learned
Every pack cut the mistake it targeted:
Lower is better. Civilization: mistakes per turn. Yu-Gi-Oh!: misplays per duel. DOOM: shots fired with no enemy visible, per run. Poker: strong hands folded, per 100 decisions.
Only some of those cuts became wins:
Pokémon: 100 + 100 battles vs Foul Play, p = 0.0003. Yu-Gi-Oh!: 40 + 40 duels vs WindBot, not significant.
- Exact facts at decisive moments win. Pokémon has many decisive decisions, each with a number the agent was guessing. That is where the win rate more than doubled.
- Fixing a mistake isn't winning. Yu-Gi-Oh! misplays and DOOM's wasted shots collapsed, but they weren't what decided those games.
- Dev gains shrink. Tune on one set of games, report on another:
dev rounds flatter you · held-out is the number
Gain of the policy arm over the bare agent. Civilization: dev round one +63.8 points, dev round two +49.7, held-out report +19.8. Poker: dev +56.8 big blinds per 100 hands, held-out +9.7. The dev bars are drawn dashed because they were measured on the same games used to tune the policies.
What broke along the way
| Problem | Effect and fix |
|---|---|
| A monster name spelled two ways in DOOM | Zombies were invisible to the agent in both arms, and the blind-fire veto fired on real targets. The first batch was voided. |
| Host suspended mid-batch | 30-minute gaps between moves in both arms. Any game with an unexplained gap is now voided and re-run. |
| 12 poker sessions played twice | Analysis kept whichever copy it read last. It now keeps the first-played copy; the paired edge moved from +10.5 to +9.7. |
The short version
A policy sees the tool call before it runs and the result before the agent does. With nothing else changed, that was enough to more than double a Pokémon win rate, add 8.8% to a Civilization score on the same maps, and cut every targeted mistake. The mistakes were the ordinary ones: skipped steps, guessed numbers, repeated failures.
The harness, every policy pack and every scored game are at github.com/FailproofAI/failproof-arcade.
failproof ai enforces your policies at runtime, on every tool call →