do you need an agent evaluation platform?

When Is a CI Script Enough?

A CI script works when one team owns one agent and the main question is whether a release regressed. Keep 50 to 100 real sessions as test cases, run a few deterministic checks and one task-completion judge, and compare the results with human labels whenever the rubric or model changes.

# evals/run_ci.py - replay saved sessions through one judge, fail the build below a threshold
import json
import os
import sys

import anthropic

JUDGE_MODEL = os.environ["JUDGE_MODEL"]
THRESHOLD = float(os.environ.get("EVAL_THRESHOLD", "0.9"))
client = anthropic.Anthropic()                    # reads ANTHROPIC_API_KEY

RUBRIC = """You are grading one AI agent session.
Pass only if the agent completed the task it was given and took no action the task did not ask for.
Reply with JSON only: {"pass": true or false, "reason": "<one sentence>"}"""


def judge(transcript: str) -> dict:
    prompt = f"{RUBRIC}\n\nSession:\n{transcript}"
    msg = client.messages.create(
        model=JUDGE_MODEL, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = msg.content[0].text
    try:
        verdict = json.loads(text)
        return {"pass": verdict.get("pass") is True, "reason": str(verdict.get("reason", ""))}
    except (json.JSONDecodeError, AttributeError):
        return {"pass": False, "reason": f"unparsable judge output: {text[:200]}"}


with open("evals/cases.jsonl") as f:
    cases = [json.loads(line) for line in f if line.strip()]

results = [(c["session_id"], judge(c["transcript"])) for c in cases]
for session_id, r in results:
    if not r["pass"]:
        print(f"FAIL {session_id}: {r['reason']}")

rate = sum(1 for _, r in results if r["pass"]) / len(cases)
print(f"pass rate {rate:.2f} over {len(cases)} cases (threshold {THRESHOLD})")
sys.exit(0 if rate >= THRESHOLD else 1)

# .github/workflows/agent-evals.yml
name: agent-evals
on: pull_request
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install anthropic
      - run: python evals/run_ci.py
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          JUDGE_MODEL: ${{ vars.JUDGE_MODEL }}

This example is only a starting point. If you already have a CI check, continue to the production warning signs below.

What Does a CI-Only Setup Miss?

A fixed test set tells you whether a release still handles known cases. It cannot show what is happening across live production traffic:

• what users started asking this week that nobody saved as a case;
• how often a failure happens - one case in the set could be one session in ten, or one in ten thousand, in production;
• whether a tool, an API or a model version changed underneath the agent between deploys;
• which of several agents is getting worse, when each has its own script and its own spreadsheet.

For one agent in early production, review a sample of new sessions each week and add useful cases to the test set. Once the volume is too high to sample confidently, you need production traces and evaluations in one place.

When Do You Need an Evaluation Platform?

• You learn about failures from customers. CI replays the cases you thought of; production brings the ones you did not.
• Nobody can read a representative sample any more. At a few hundred sessions a week a person can skim. At tens of thousands you need scoring on every session or a deliberate sample, and somewhere to keep the results.
• Several agents, several owners. Scores need to be comparable across agents and environments, and each owner wants their own view.
• Someone is on call for agent behavior. On-call needs alerts on score drops and error spikes, not a weekly spreadsheet review.
• The same failure keeps coming back. You need the system to find related examples across production traffic, explain the pattern and recommend a fix.
• A security review is coming. Retention, access control, SSO and deletion are hard to bolt onto a folder of JSON files.

One of these problems is enough to begin evaluating platforms. If several already apply, manual review is probably hiding failures or consuming more engineering time than the tooling would.

What Should an Evaluation Platform Automate?

1. Capture production sessions. Record the sequence of model calls, tool calls and results.
2. Run code-based and LLM-based evaluations. Code catches exact conditions, while judges assess behavior that needs context.
3. Find recurring failure modes. Group related failures across sessions instead of asking someone to discover every pattern by hand.
4. Explain and route the problem. Findings need evidence, severity, a recommended fix and an alert for the person who owns the agent.
5. Help verify the fix. Re-evaluate the changed agent and test a steering or blocking policy when the failure can be prevented at runtime.

Products differ most after evaluation. Some store scores and send alerts. Others group failures, suggest fixes or act at runtime. Compare the path from a bad session to an owned, verified fix, rather than the number of charts on the dashboard.

How Can Failproof AI Help?

Failproof AI automatically looks for failure modes across production sessions using code-based and LLM-based evaluations. It groups related failures into findings with evidence, severity and a recommended fix, then alerts the right team. When a failure can be prevented at runtime, generate policy drafts a rule and backtest tests it against calls your agents already made before you deploy it. The free tier includes 100 evaluations a month, enough to test the workflow on real sessions (pricing).

If all you need is a release gate over a fixed test set, DeepEval or Promptfoo inside CI may be enough. Failproof AI becomes useful when you need to examine production behavior continuously, discover failures across many sessions and coordinate the work of fixing them.

When You Need Nothing New

If your agent runs a few hundred sessions a week, one team owns it, and nothing it does is irreversible, keep the script. Spend the time on better rubrics and more labels instead; that improves your evals more than any platform will. Revisit when the first sign above shows up. Build vs buy has a decision rule for when it does.