agent evaluation for small teams without a data team

Why Do Small Teams Struggle to Maintain Evals?

Most published examples come from companies with dedicated evaluation teams. They use curated datasets, many metrics, pairwise comparisons, custom dashboards and judges tuned against thousands of labels. A 60-person company may build a smaller version in a sprint, but it falls apart when nobody owns the weekly review. Labels go stale and the scores stop influencing product decisions.

Every check needs someone who will read the failures and change the agent. Build checks tied to decisions you already make, such as rolling back a prompt, fixing a tool or blocking a dangerous action. Leave a metric out if nobody can say what they would do when it moves.

The starting setup is ordinary engineering: a script, a judge prompt and a spreadsheet of reviewed sessions. The engineer who owns the agent can run it without specialist statistics or a separate data pipeline.

What Should a Small Team Build First?

1. Week 1: three code checks

   Tool errors per session. One rule that must never break - a destructive command for a coding agent, a refund over the order total for a support agent. And a budget: tool calls per session, or tokens, above which a session is suspicious. All three are deterministic, free and run on every session. Start here because they will find real problems before the judge exists.

2. Week 2: one judge

   Pick the single question the code checks cannot answer. For most agents it is "did it complete the task?". Make it binary, ask for a one-sentence reason, and parse the reply defensively. One judge, one question. Resist adding tone, helpfulness and conciseness in the same week.

3. Week 3: fifty labels

   The agent's owner labels fifty sessions as done or not done, without looking at the judge first. Fill half with sessions a code check flagged, so there are enough failures. Compare with the judge, read every disagreement, fix the prompt, run it again. That is calibration, and it takes an afternoon.

4. Week 4: one alert, one policy

   Alert when the judge's completion rate drops or tool errors spike, routed to the person who owns the agent. Then take the worst action any check found - the one that would cost real money or data - and block it before it runs, rather than scoring it forever.

5. Every week after: thirty minutes

   Read the sessions the checks and the judge flagged. This is where new failure modes show up, and where you decide whether the program needs a fourth check. Most weeks it will not.

Can the Evaluation Setup Start as One Script?

Yes. The example below reads sessions from a JSONL file, runs three code checks and one task-completion judge, compares the judge with available human labels, and prints the sessions that need review. Each line needs an id, a task, an events array and, when available, a human label.

"""evals.py - small-team agent evals. Usage: python evals.py sessions.jsonl"""
import json
import os
import re
import sys

import anthropic

JUDGE_MODEL = os.environ["JUDGE_MODEL"]
DESTRUCTIVE = re.compile(r"\brm\s+-[a-z]*r[a-z]*f|\bgit\s+push\b.*--force|\bDROP\s+TABLE\b", re.I)
MAX_TOOL_CALLS = 40


def code_checks(s):
    calls = [e for e in s["events"] if e["kind"] == "tool_call"]
    commands = [str(e["input"].get("command", "")) for e in calls if e["tool"] == "Bash"]
    return {
        "tool_errors": sum(1 for e in s["events"] if e["kind"] == "tool_result" and e.get("error")),
        "destructive": any(DESTRUCTIVE.search(c) for c in commands),
        "over_budget": len(calls) > MAX_TOOL_CALLS,
    }


def judge_done(s):
    prompt = (
        "Task given to the agent:\n" + s["task"]
        + "\n\nSession events:\n" + json.dumps(s["events"], default=str)[-40000:]
        + '\n\nDid the agent complete the task? Reply with JSON only: {"done": true or false, "reason": "..."}'
    )
    client = anthropic.Anthropic()                    # reads ANTHROPIC_API_KEY
    msg = client.messages.create(
        model=JUDGE_MODEL, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = msg.content[0].text
    try:
        return bool(json.loads(text[text.index("{"): text.rindex("}") + 1])["done"])
    except (ValueError, KeyError, TypeError):
        return None  # judge failed: not scored, and not counted as a fail


if __name__ == "__main__":
    with open(sys.argv[1]) as f:
        sessions = [json.loads(line) for line in f if line.strip()]
    rows = [{"id": s["id"], **code_checks(s), "done": judge_done(s), "label": s.get("label")}
            for s in sessions]

    labeled = [r for r in rows if r["label"] is not None and r["done"] is not None]
    if labeled:
        agree = sum(r["done"] == r["label"] for r in labeled) / len(labeled)
        fails = [r for r in labeled if r["label"] is False]
        caught = sum(1 for r in fails if r["done"] is False)
        print(f"judge agrees on {agree:.0%} of {len(labeled)} labels; caught {caught} of {len(fails)} labeled fails")

    for r in rows:
        if r["destructive"] or r["over_budget"] or r["done"] is False:
            print(json.dumps(r))

Two numbers from that output matter. Agreement tells you whether the judge is roughly right. "Caught N of M labeled fails" tells you whether it finds the failures, which is what you built it for. A judge that agrees 90 percent of the time but catches two of ten real failures is quiet, not good.

What Can a Small Team Skip at First?

The cost of the program is mostly judge tokens, and it is easy to estimate with your own numbers. Multiply sessions per day by the tokens in one judge prompt, add the reply, and apply your provider's price. If the total is uncomfortable, run the judge on a sample plus every session a code check flagged; the code checks stay on every session, since they cost nothing.

Who Should Own the Evals?

The person who owns the agent should also own its evals. They know what a failed session looks like, can label examples accurately and can change the prompt when completion drops. Assign an owner before adding more evaluation tooling.

Add to the setup when a specific problem appears:

• A failure keeps recurring that no check sees. Add a fourth check for that failure, in code if you can, as a second judge question if you cannot.
• Several agents share tools and failures spread between them. Individual scores stop being enough; start running audits over the population, with a short written contract for each agent.
• You ship prompt or model changes weekly. Freeze the fifty labeled sessions as a regression set and run the judge on them before each change goes out.
• The judge's verdicts start driving decisions with money attached. Label more sessions, two people on an overlap, and measure how often they agree with each other before trusting the judge further.
• The judge bill becomes a line item someone asks about. Sample the judge and keep the code checks on everything.

Let observed failures determine what you add next. This keeps the eval setup useful and gives every new check a clear owner and purpose.

How Do You Run This Setup in Failproof AI?

Failproof AI splits the program the same way this page does. Two of the three code checks - tool errors and the step budget - only count events, so they fit hosted evaluations: one Python expression each, authored under Analyze → eval authoring or drafted by the assistant from a plain-English description, tested against up to 10 real sessions and enabled, with no process to run. In an expression, session.events_of_type() replaces the list filter, tool_use payloads carry tool_name and tool_result payloads carry status. The judge runs in the cloud too, and so does the never-break rule; the evaluator tutorial walks through one. If you already run an eval library, bring that eval set in as it is rather than rewriting it. There is no pre-built metric library - the checks are the ones you choose - but the plumbing around them is done: results are stored beside the trace, and alerts and audits can use them, which is the part small teams most often never finish.

The alert and the policy map directly onto documented pieces. For tool errors, the Cloud CLI can create the alert in one command:

fp alerts create high-errors \
  --trigger-kind metric_threshold \
  --severity warning \
  --trigger-spec '{"metric":"error_count","op":">","value":50,"window_secs":900}'
fp alerts test high-errors

For the completion rate, create an evaluation-score alert in Analyze → Alerts; alerts that fire become incidents under Analyze → Issues. For the policy, npm install -g failproofai, then failproofai policies --install to wire the hooks and failproofai policies add FailproofAI/policies for the coding agent pack: 38 policies, 10 on by default, including block-push-master, block-env-files and block-sudo, with block-rm-rf and block-force-push one failproofai policies add away. That may already cover your worst action. If not, a custom policy is a few lines of JavaScript.

The weekly thirty minutes gets easier too. fp evals --since 24h --aggregate shows how the judge scored the last day, and the Failproof Assistant answers questions over your data without anyone writing a query, for example fp agent ask "Which production checkout agents had the most tool errors in the last 24 hours?". The free cloud tier includes a monthly allowance of evals; the limits per plan are on the pricing page.

When Can You Postpone Setting up Evals?

If one agent is used by the team that built it, at a volume where someone reads every session anyway, the reading is the eval program. Add the coding agent policy pack so the destructive actions are blocked, and keep reading. Build the five pieces when sessions start happening that nobody reads - the first customer-facing agent, the first agent that runs overnight, or the first week you realize you cannot say whether last Tuesday's prompt change helped.