━━ failproof ai journal · vol. 01
blog
field writing from the team behind failproof - on agent failure, durability, and the new shape of production AI.
how to add observability to your hermes agent in 60 seconds
A Hermes agent runs behind a chat gateway, on a cron, when nobody is watching - which is exactly why the failure you care about is the one nobody saw. Two commands forward every tool call, model request and error it makes to a dashboard you can replay.
read post →━━ how to
one procedure, any agent. point failproof at what you are already running, watch every tool call it makes, then decide what to block. one walkthrough per target.
the same two commands point at any of the twelve harnesses failproof hooks — see the list
━━ archive · all posts
- №19
how we made agent enforcement sub-millisecond
Enforcement used to run as a fresh process per tool call: 700ms of runtime boot before a single policy saw the command, and no way to tell a working install from a broken one. Replacing it with a Rust daemon and a warm worker took p50 to ~0.7ms and, for the first time, made fail closed a guarantee we can state in one sentence.
→ - №18
ai workers. new buzzword?
A year ago most AI companies were building agents. Now the same companies ship AI employees, AI coworkers, digital labor. The rename tracks a real shift: an agent produces an output, a worker owns an outcome, and the two fail in very different ways.
→ - №17
hallucinations, over-persistence, and dangerous actions in frontier llms like gpt-5.6 sol
In the week after GPT-5.6 Sol shipped, agents deleted a developer's Mac, wiped a production database, and force-removed worktrees — no confirmation asked. Why hallucination and over-persistence turn destructive in agentic systems, and why runtime policy enforcement is the defense that holds.
→ - №16
how 100 rows cost us 3.4 gb of ram in clickhouse
A LIMIT 100 query was OOM-killing our 4 GiB ClickHouse pod at three concurrent requests. The cost scaled with table size, not rows returned — one wide column decompressed through FINAL on every poll. The post-mortem, and the one-line mental model that fixed it.
→ - №15
agenteye with/without skill on claude
A skill file didn't make Claude smarter — it gave Claude the operational knowledge experienced CLI users pick up over time: pagination limits, silently-skipped confirmations, hidden SQL, and knowing when to stop and ask a human.
→ - №14
modern software product engineering
Sprints and standups were built for a constraint that's gone. Here's the closed loop that observes, diagnoses, tests, and ships on its own — and where the CTO's job goes when reviewing every diff stops scaling.
→ - №13
the agent didn't fail. it was just told too much, too soon.
Why progressive disclosure is the most underrated concept in agent reliability, and why hooks are the primitive that finally makes it possible.
→ - №12
the competitor vertical ai founders refuse to name
A wake-up call for vertical AI companies. The reliability gap is the unnamed competitor - and it's eating market share quietly.
→ - №11
stop praying in prompts. start enforcing with hooks.
Git figured this out decades ago and agents are about to go further. The enforcement gap is where reliability dies.
→ - №10
the future of reliable orchestration
What does infrastructure designed from the ground up for AI agents actually look like? Pre-warmed sessions, direct context streaming, stateful routing, sandboxed isolation, and a unified control plane.
→ - №09
retry is not enough: the 3u framework for agentic reliability
In traditional software, retries fix transient errors. In AI agents, failures are cognitive. Observability after the fact doesn't help when every failure is unique. The 3U framework introduces a new model for reliability: Uncover, Understand, Utilize.
→ - №08
why temporal breaks when you put an ai agent inside it
Temporal is the industry standard for durable execution. We tried using it as the durability layer for Claude Agents. Here are the 12 specific ways it falls apart.
→ - №07
what does durability mean for agentic software?
Durability frameworks were built for deterministic workflows. Agents are probabilistic, stateful, and long-running. Here's why the old playbook breaks and what should replace it.
→ - №06
observability was built for servers. agents need oversight.
The industry has spent 18 months building increasingly sophisticated observability tooling for agents: LLM-native traces, session replays, eval pipelines. It's still not enough.
→ - №05
two env vars and done: claude code for every developer, zero api keys
Open source to allow every developer to use all possible Anthropic models and credits across clouds - without juggling API keys.
→ - №04
stop shipping code. start building factories.
The next generation of software companies will not be defined by the code they write, but by the automated assembly lines they build.
→ - №03
missing sauce for your agents
A thought piece on engineering culture, agentic systems, and what separates great products from everything else.
→ - №02
the future is saaas (subagent as a service)
A thought piece on how the most important companies of the next decade won't build software applications - they will become subagents.
→ - №01
launching the aire principles: industry standards for ai agent reliability
The first open framework translating SRE practices into actionable standards for production AI systems. Five principles, measurable metrics, and a proven path to agent reliability.
→
━━ guides
- f·1
AI agent stuck in a loop - how to break it
Why coding agents retry the same failing step forever, how to detect the loop, and the hook-level policy that breaks it, summarizes progress, and restarts clean.
→ - f·2
Agent hallucinated a tool or file - how to stop it
When the agent calls a tool, file, or API that does not exist, block the invented call at the hook layer and hand the agent what does exist nearby instead.
→ - f·3
Agent context drift - keep agents on task
Agents wander off the original goal as context fills up. Detect the drift and pull the agent back to what it was actually asked to do, at the hook layer.
→ - f·4
Agent won't stop after finishing - how to end the run
The task is done but the agent keeps going, editing more, spending more, risking more. Stop the run the moment intent is complete and surface the result.
→ - g·1
AI failure handling for production agents
The five failure categories, why retries don't cut it, and how harness hooks become the enforcement primitive.
→ - g·2
LLM agent reliability - closing the production gap
The reliability gap between demo and production, the 3U framework, and hook-level enforcement across every supported harness.
→ - g·3
Agent error recovery - detect, mitigate, continue
The retry / repair / block taxonomy and the recovery strategy mapped to every failure mode failproof catches.
→ - g·4
How to stop Claude Code from running dangerous commands
Install failproof, register the PreToolUse hooks, and the destructive command classes - rm -rf, sudo, curl | sh, force push, terraform destroy - stop at the hook layer.
→ - g·5
Claude Code safety hooks - complete setup guide
PreToolUse, PostToolUse, Stop. The 39 built-in policies grouped by category. Three configuration scopes. Custom JS policies.
→ - g·6
How to block rm -rf in Claude Code
The block-rm-rf policy in detail - every recursive flag shape it catches, the safe-path allowlist, and what the agent sees when a delete is denied.
→ - g·7
How to prevent Claude Code from accessing .env files
Two PreToolUse blockers plus five PostToolUse sanitizers - keep secrets out of the agent context window, even when the agent legitimately reads a related file.
→ - g·8
How to prevent AI agent force push
block-force-push, block-push-master, block-work-on-main. One engine, every harness - Claude Code, Codex, Cursor, Gemini CLI, Copilot, picode, opencode.
→ - g·9
How to sandbox Claude Code terminal access
Policy sandbox without a container or VM - PreToolUse confinement, PostToolUse sanitization, Stop gates. Stack a container on top only when you need a kernel boundary too.
→ - g·10
AI agent observability - see what your agents did
Why logs aren't enough for agents, the four pieces of real observability - live event stream, session replay, queries, alerts - and how failproof ai ingests every run in realtime.
→ - g·11
Agent session replay - debug every run span by span
Replay any run on the trace rail, expand a span to the raw model or tool call, and see exactly where the agent looped, drifted, or got blocked.
→ - g·12
Query agent traces in SQL
SQL and a visual builder over every event, session, and eval - find failure patterns across thousands of runs, save them to dashboards, and turn them into alerts.
→ - g·13
Agent failure alerting - catch breaks in real time
Trigger on a metric, a saved query, or an eval score; open an incident your team can triage; and pull the alert into your local Claude via the CLI and MCP.
→ - g·14
How to add observability to your Hermes agent in 60 seconds
Hermes runs behind a chat gateway when nobody is watching. Two commands - an install and failproofai config - forward every tool call, model request and error to a dashboard you can replay.
→