Dashboards are pull, alerts are push
A dashboard is a pull surface: it only tells you something is wrong when you remember to go look at it. That's fine for the weekly review and useless at 2am, when the agent that worked all week starts looping on a flaky tool and burns through tokens until someone notices the bill. The gap between “the agent broke” and “a human opened the dashboard” is where production incidents live.
Alerts close that gap by inverting the direction. Instead of you polling the data, the data pages you. In agenteye you stand up dashboards, a live event stream, session replay, and SQL when you want to investigate - and you stand up alertsfor the failures you already know you can't afford to miss. The dashboard is for the questions you didn't know to ask. The alert is for the answer you already decided you need delivered.
Agent failures are an especially good fit for push. They're often silent: the harness doesn't throw, the model returns a clean 200, and the agent confidently does the wrong thing. Nothing crashes, so nothing wakes anyone up - unless you defined, ahead of time, what “wrong” looks like in your data and asked agenteye to watch for it.
What you can trigger on
An alert in agenteye is a named condition over your agent telemetry plus where to send the notification. There are three kinds of condition, and they map cleanly to the three ways an agent failure shows up in the data.
- a metric threshold. Pick a metric, a comparison, and a window: error rate above 5% over the last 10 minutes, p95 tool-call latency over 8 seconds, cost per session over a dollar. When the metric crosses the line and stays there for the window, the alert fires. This is the right trigger for the failures that are quantitative and aggregate - the system as a whole is degrading, not one specific call.
- a saved SQL query. Write a query over your traces that returns rows only when something is wrong, save it, and have the alert fire whenever it comes back non-empty. Any failure you can express in SQL- sessions that hit the same tool five times in a row, traces where the model called a tool that isn't in the allowlist, replays that ended in a destructive shell command - becomes an alert. The returned rows are the breaching events, so you get the evidence for free.
- an eval score. If you score agent output with evals, alert when a score drops below a threshold: groundedness under 0.8, task-completion under 0.9, a safety eval that flips to fail. Eval conditions compose, too - you can require
anyof several evals to breach,allof them, orM-of-N, so a single noisy eval doesn't page anyone but a real regression across the suite does.
The three overlap on purpose. A metric threshold is the fastest to set up and the bluntest; a saved query is the most precise and gives you the offending rows; an eval score is how you catch quality regressions that never throw an error at all. Most teams run a few of each.
Incidents and triage
A firing alert that's just a message in a channel is a notification, not an incident. The difference matters: a notification scrolls away, an incident has state. When an alert fires in agenteye it opens an incident, and the incident is the unit of work from there.
The breaching events come attached. For a metric alert that's the events inside the window that pushed it over; for a saved query it's the exact rows the query returned; for an eval it's the sessions that scored below the line. You open the incident and the evidence is already there - no re-deriving which 40 traces out of 10,000 actually mattered. From any attached event you can jump straight into session replay and watch the run on the trace rail to see exactly where it went sideways.
Each incident carries the state a real triage needs:
- an assignee- the one person who owns getting it resolved, so it doesn't sit in the shared-ownership void where everyone assumes someone else is on it.
- subscribers - everyone who wants the updates without owning the fix, kept in the loop automatically.
- a conversation thread - the triage discussion lives on the incident itself, next to the evidence, instead of scattered across DMs that the next on-call engineer will never find.
- a notification log- a record of who was paged, through which channel, and when, so “did anyone actually get this?” is a question with an answer.
Routing the alert
Defining the condition is half the job; the other half is deciding who finds out and how loudly. Routing is what keeps alerting useful instead of a wall of noise everyone learns to mute.
Route by severity. A safety eval flipping to fail or the agent reaching for a destructive shell command should page a human now; a slow upward creep in p95 latency can open a quiet incident and assign it for the morning. Set the assignee per alert so the incident lands on whoever owns that surface, and let the people who care subscribe so they get the thread updates without being on the hook to fix it. The notification log means that after the fact you can prove the page went out and to whom - the difference between “the alert didn't fire” and “the alert fired and nobody was looking,” which are two very different post-mortems.
The goal is that every alert that fires is one a human actually wants to know about. An alert that pages constantly and means nothing trains your team to ignore the one that matters. Tune thresholds and compose eval conditions until firing is signal - and when an alert is chronically noisy, that's information too: either the threshold is wrong, or the failure underneath it deserves a permanent fix.
Pull alerts into local Claude
Most alerting stops at telling a human. agenteye goes one step further: it can hand the incident to the agent that's going to fix it. The agenteye CLI and MCP server pull open alerts and the underlying events into your local Claude, so your coding agent can act on a production failure without you playing copy-paste courier between the dashboard and your terminal.
From the CLI, list open alerts, read an incident, and pull the breaching traces down into your working session. Through MCP, the same alerts and events become tools your coding agent can call directly: it reads the incident, inspects the sessions that breached, replays the failing run on the trace rail, and proposes a fix - grounded in the actual events that fired the alert rather than your secondhand description of them. The built-in agent inside agenteye can help triage on the dashboard side; the CLI and MCP are how the same context reaches the agent on your machine.
This is the find→fix loop tightening. The alert finds the failure and opens the incident; the CLI and MCP carry it into the place where the fix gets written. The handoff that used to be a human reading a dashboard, summarizing it into a prompt, and hoping they got the details right becomes a direct pull of the real data.
From a repeating alert to a policy
An alert that fires once and never again did its job: you caught a failure, triaged it, shipped a fix. An alert that fires every week is telling you something else - this failure isn't an incident, it's a pattern, and triaging it by hand each time is treating a symptom. A repeating alert is the clearest possible signal to write a policy.
That's the other half of failproof ai. The failproofai CLIenforces policies at the harness hook layer - deny or warn on the agent's next move, locally. Where an alert is detection after the fact, a policy is prevention before the act. If your saved-query alert keeps firing because the agent keeps reaching for a forbidden command, you don't want to keep getting paged - you want a policy that blocks the command at the hook so the failure can't recur. failproof ships 39 built-in policies, and a chronic alert is usually pointing straight at which one to turn on, or what custom one to write.
Observability and policy are the two halves of the same loop. agenteye finds the failure and pages you; the policy layer fixes it for good so the alert goes quiet. The win condition for any alert is that it stops firing because the failure underneath it became impossible.
Get started
failproof ai is free and open-source. Install the cli to bring up agenteye and the policy layer together - the agenteye dashboard runs at localhost:7771, and the policy dashboard at localhost:8020. Create your first alert on a metric, a saved query, or an eval, and decide who it pages.