How to build a Failproof AI evaluation in Python

What a Failproof AI Evaluation Is

A Failproof AI evaluation runs when a session finishes and writes its results beside the trace. Each result is a score (0 to 1, optionally marked passed), a metric (a value with a unit) or an assertion (passed or not), with reasoning, a summary and labels - up to 25 results per run. Evaluations belong to the organization, so every team member reads the same results.

Failproof AI runs your evaluations in the cloud: code checks, LLM judges and the evaluation suite you already have. There is no evaluation worker to deploy on your side, and existing evaluation logic can continue scoring production sessions without being rewritten as a hosted expression.

This guide builds the most common kind first, a code check: one deterministic Python expression with no imports or network access. That is right for counting events, ratios and thresholds, and it is where most teams should start because a check that can be computed should not cost a model call. The same five steps apply to an LLM-based evaluation or an evaluation you already use: decide what to check, create it, test it on real sessions, deploy a version and read the results.

Step 1: Decide What to Check

One evaluation answers one question. "Did the tools work?" is a question; "was the session good?" is several questions wearing a trench coat. Write the question down as a sentence before you open the dashboard, because that sentence is what you will hand the assistant.

• Countable? Tool errors, retries, the number of steps, whether the last event reports success: a code check.
• A judgment? Whether the agent finished what the user asked, or stayed inside a refund policy: an LLM judge, which runs in the cloud too.
• Already written? If your existing evaluation suite already measures it, bring that evaluation in rather than writing a second version of the same idea.
• Which sessions? Decide which agents and environments the question applies to. A production-only check should never score test traffic.

Step 2: Create the Evaluation

1. Open eval authoring

   Under Analyze → eval authoring, create an evaluation and describe the check in plain English - "the share of tool results that came back ok, and how many tool calls the session made". The assistant drafts the code check; you can also write the expression yourself.

2. Read the expression

   The expression evaluates to a result. This one scores the share of tool results that came back ok and records the number of tool calls as a metric. Whoever drafted it, read it before you trust it.

   EvalResult(
       score=Score(
           len([e for e in session.events_of_type("tool_result") if e.payload.get("status") == "ok"])
           / max(1, session.count("tool_result"))
       ),
       metrics={"tool_calls": Metric(session.count("tool_use"), unit="calls")},
       reasoning="Share of tool results that came back ok.",
   )

3. Add a condition

   An optional condition decides which sessions the evaluation applies to, so a production-only check never runs on test traffic, and a session that does not qualify is skipped instead of scored zero.

   session.agent_id == "code-assistant" and session.environment == "production"

A few habits keep an expression honest. Read the session through events_of_type, count, agent_id and environment, and access payload fields with .get(). Available event data includes tool_name on tool_use, status on tool_result, response on human_input, content on model_response and summary on agent_end; treat other payload keys defensively. Guard every division, as max(1, ...) does above, so a session with no tool results scores cleanly instead of failing.

If the check is a judgment rather than a count, create an LLM judge instead; it runs in the cloud like everything else here. If the check already exists in your suite, bring that eval in as it is. Either way the result has the same shape - a score, metric or assertion with reasoning - so it lands in the same place as the code check and is read the same way.

Step 3: Test It Against Real Sessions

Run the evaluation against up to 10 real sessions before enabling it, and read every result, not just the number. A check that scores all ten 1.0, or all ten 0, is usually a bug in the check rather than news about the agent. Pick sessions on purpose: two you know went well, two you know went badly, and a few ordinary ones. If the scores do not sort them the way you would, fix the check now, while it has scored nothing in production.

For an evaluation you brought in, confirm that it interprets production traces the same way it interpreted your test fixtures. Export a handful of sessions you have already labeled, run the suite over them offline and compare its verdicts with yours before enabling it broadly.

Step 4: Deploy It as a Version, and Roll Back if Needed

Enabling an evaluation deploys a version, and versions are immutable. To change the check, create a new version; to roll back, disable one version and enable another. That makes a bad change cheap: if the new version starts flagging healthy sessions, the previous one is one switch away, and the results each version wrote stay attached to it. Up to 100 hosted evaluations can be enabled per organization.

The limits say what code checks are for: a 30-second default timeout and a 60-second maximum, code up to 128 KiB, a condition up to 16 KiB. Plenty for arithmetic over events. Anything that needs a model belongs in an LLM judge.

Each session-and-evaluation pair is one billable evaluation, so a check that applies to every session consumes one evaluation for each session it scores. Conditions control scope: a check limited to production sessions for one agent consumes fewer evaluations than one that scores all traffic. Plans and allowances are on the pricing page.

Step 5: Read Results Beside Each Trace, and Alert on Scores

Results appear under Observe → evaluations, beside the trace of the session they scored, so a low score is one click from the tool calls that caused it. From the terminal, the Cloud CLI reads the same data:

fp evals --since 1h --score tool_reliability:0..1
fp evals --since 24h --aggregate
fp evals --agent-id checkout-agent --aggregate
fp --json evals --aggregate --env production

SQL queries, the assistant, dashboards and alerts read them too; an alert with the evaluation_score trigger kind under Analyze → Alerts tells the agent's owner when a score drops, by email, Slack, webhook or the dashboard. When you deploy a new version, re-evaluate reruns it on a session, and backfill covers up to 90 days, so a new check can be scored against recent history rather than waiting for new traffic. Size a 90-day backfill against your plan first.

Individual scores show which sessions failed. Failproof AI analyzes evaluation results and trace evidence across sessions to find recurring failure modes, group affected sessions into findings and recommend what to change. The fix may be a prompt, tool or workflow change; when the evidence identifies a recognizable high-risk action, a tested behavioral policy can help prevent it from recurring.

Pitfalls and How to Check Your Work

• Skip, do not zero. Use the condition for sessions an evaluation does not apply to. A zero for "not applicable" drags the average down for no reason.
• Keep every result independently interpretable. One evaluation may return several results when they share the same input and execution, but each needs its own stable key, definition and evidence.
• Version on purpose. A new rubric, a new threshold or a new release of a library you brought in is a new version, so you can tell which results came from which.
• Distrust perfect scores. A check that never fails in its first week is more often broken than the agent is perfect. Re-test it on a session you know failed.
• Include inspectable evidence. A score without an explanation is difficult to debug. Make every result say why in one concise sentence a person can verify against the trace.

When You Do Not Need This

If you only evaluate a fixed dataset before each release, a script in CI may be all you need. Cloud evaluations earn their place when you want production sessions scored continuously, results tied to traces, alerts on regressions and failure analysis across many runs. Runtime policies can complement those evaluations for known high-risk actions, but they do not replace the measurements that reveal new or changing failure modes.