How to alert on eval score drops with Failproof AI

1. Start with an Evaluation You Trust

Create the evaluation before the alert. Use a code-based check for exact conditions such as a missing required action, a tool error or too many repeated calls. Use an LLM-based evaluation when the result depends on meaning, such as whether the agent completed the request or followed a customer policy. Failproof AI runs both in the cloud and can also run your existing evaluation suite.

Test the evaluation against real sessions, inspect both passing and failing results, and deploy a stable version. An alert amplifies whatever the evaluation says. A vague criterion or an untested threshold creates noisy incidents faster than it creates useful evidence.

fp evals --since 1h --score tool_reliability:0..1
fp evals --since 24h --aggregate
fp evals --agent-id checkout-agent --aggregate
fp --json evals --aggregate --env production

2. Choose the Alert Trigger

Use evaluation_score when one result has a clear operational meaning. Choose eval_compound when a second evaluation makes the incident more specific and reduces false alarms. Use custom_sql when the failure depends on a cohort or sequence of events that a single evaluation does not express.

3. Create and Route the Alert

1. Open Analyze → Alerts

   Create a name that identifies the agent and behavior so responders know what failed before opening the incident.

2. Set the condition and severity

   Choose the trigger kind and make the severity match the consequence. Reserve urgent delivery for failures that affect customers, security or irreversible actions.

3. Choose the destination

   Deliver the alert by email, Slack, webhook or the dashboard. Route it to the team that owns the affected agent.

4. Test before enabling it

   Confirm that the destination receives the test and that the responder can reach the resulting incident and evidence.

The Cloud CLI also manages alerts. The documented example below creates an error-count threshold. Configure evaluation-score and compound conditions in Analyze → Alerts instead of guessing an undocumented trigger-spec shape.

fp alerts create high-errors   --trigger-kind metric_threshold   --severity warning   --trigger-spec '{"metric":"error_count","op":">","value":50,"window_secs":900}'
fp alerts test high-errors
fp alerts list

4. Keep the Alert Useful

• Scope it to one agent and environment. Production and staging should not share an incident stream, and unrelated agents should not be averaged together.
• Alert on failures that have an owner. Every firing should lead to a defined investigation or product change.
• Use compound conditions when one score is ambiguous. Two relevant evaluations can describe the failure more precisely than one broad score.
• Separate evaluation errors from low scores. A failed model call or malformed result is an evaluation problem, not proof that the agent failed.
• Test every routing change. A correct trigger still fails operationally if the webhook, email or Slack destination is wrong.
• Revise alerts that produce no action. Lower the severity, narrow the condition or mute the recurring pattern rather than training the team to ignore it.

5. Investigate the Incident

A firing alert creates an incident under Analyze → Issues. Start with the affected evaluation results and open the linked sessions. The trace shows the model requests, tool calls, tool results and final outcome behind the score, so the owner can see whether the failure began in the prompt, model, tool, data or workflow.

Failproof audits can analyze failed evaluations with the trace evidence around them, group related behavior into findings and recommend what to change. This turns a stream of similar alerts into one recurring failure mode with affected sessions, severity and an owner.

fp issues list
fp issues assign <issue-id> --assignee engineer@example.com
fp issues resolve <issue-id> --yes

6. Fix the Failure and Verify Recovery

Use the finding and trace evidence to change the prompt, tool, model, data or workflow. Keep the evaluation and alert enabled after deployment. Recovery means new sessions pass and the incident stops recurring, not merely that the first issue was closed.

When the failure is a recognizable high-risk action in a supported harness, Failproof AI can draft a behavioral policy from the issue. Backtest it against historical calls and deploy it in observe mode before enforcement. Keep the evaluation running to confirm that the agent completes the task safely instead of repeatedly colliding with the policy.

Failproof AI already provides evaluation execution, result history, alert triggers, email and Slack delivery, webhooks, dashboard incidents, assignment, trace evidence and recurring-failure analysis. Your team chooses the behavior worth monitoring, owns the product fix and decides whether a high-risk action needs a runtime policy. You do not need to build and maintain a separate alerting pipeline around the agent.