How to audit agent sessions for failure patterns with Failproof AI

1. Tell Failproof AI What the Agent Should Do

Start with an agent contract when the audit needs a stable definition of success. In Failproof AI, a contract records the agent's purpose, expected outputs, completion condition, cadence and prohibited behavior. Keep it concrete and under 5,000 characters.

A useful contract says what evidence should exist in a successful trace. For a checkout agent, that might include confirming the order, checking payment state before a retry, and escalating after repeated provider failures. The audit can then compare observed sessions with the job the agent was meant to complete.

2. Create the Cloud Audit

1. Write one audit goal

   Ask a specific population-level question such as "find sessions abandoned after a failed payment lookup" or "find behavior changes after the latest deployment".

2. Choose the scope

   Select the agent, environment and any filters that define the population. Keep unrelated agents and test traffic out of a production audit.

3. Set the window

   Use since_last when each run should inspect new sessions, or a fixed window when you need a consistent period for comparison.

4. Choose cadence and sensitivity

   Audits can run from every hour to every seven days. Use a faster cadence for high-volume or recently changed agents and a weekly cadence for stable workflows.

5. Add context

   Attach a short brief and up to five reference URLs when the audit needs product rules, runbooks or policy context that is not present in the traces.

3. Let Failproof AI Analyze the Population

The Cloud audit combines complete trace evidence with code-based and LLM-based evaluation results and prior policy decisions. It looks across the selected population instead of treating every low score as an unrelated incident.

Related failures become findings. Each finding carries an analysis, recommendation, severity, affected sessions and evidence queries. This is the key difference from a dashboard of scores: the output names the recurring failure mode and keeps the sessions behind the conclusion available for review.

fp audits list
fp audits findings --status open --limit 20

4. Review and Triage Each Finding

Open the affected sessions and evidence query before acting. Confirm that the sessions share the stated failure and that the recommendation addresses the cause rather than a symptom. Severity should reflect both impact and breadth: four incorrect refunds can matter more than hundreds of harmless retries.

A finding can be acknowledged, assigned, dismissed, muted, resolved or reopened. Dismiss and mute record durable feedback by fingerprint across the organization, so the audit can stop resurfacing a pattern the team has classified as expected or irrelevant.

5. Turn a Confirmed Finding into an Owned Issue

Create an issue for a confirmed failure and assign it to the team that can change the agent. The fix may belong in the prompt, tool implementation, model, retrieval data or workflow. Keep the finding's affected sessions attached so the owner can reproduce the behavior and check the recommendation against the trace.

fp issues list
fp issues assign <issue-id> --assignee engineer@example.com
fp issues resolve <issue-id> --yes

Use Failproof AI alerts to notify the owner by email, Slack, webhook or dashboard when the relevant evaluation, metric, query or event condition fires again. The alert creates an incident under Analyze → Issues, keeping detection and ownership in the same workflow.

6. Apply the Fix and Verify the Next Audit

Ship the prompt, tool, model, data or workflow change, then leave the audit running with the same goal and scope. The next audit should show whether the recurring pattern disappeared, narrowed to a smaller cohort or returned in a different form. Resolve the issue only after the evidence confirms recovery.

When the finding identifies a recognizable high-risk action in a supported harness, Failproof AI can generate a draft behavioral policy from the issue. Nothing is published automatically. Backtest the draft against historical calls, deploy it in observe mode and review false positives before enforcement.

Failproof AI has already built the population analysis, recurring-failure grouping, trace evidence, recommendations, durable triage, issue ownership, alerts and optional policy workflow. Your team supplies the agent-specific goal, validates the finding and owns the product change. You do not need to write clustering scripts or maintain a separate pipeline that turns low scores into engineering work.

Cloud Audits and the Local CLI Audit Are Different

The open-source failproofai audit command reads session histories from supported harnesses on one machine. It is useful for reviewing a developer's local Claude Code or Codex activity. Failproof AI Cloud audits analyze stored sessions across a team, run on a schedule and produce findings that can be triaged and assigned. Use the Cloud workflow for production failure analysis.