Questions to ask an agent evaluation vendor

The Checklist

Failure Discovery and Fixing

• Can the platform find recurring failures we have not written an evaluator for? Ask to see how it groups related sessions and distinguishes a pattern from an isolated bad run.
• What evidence comes with a finding? You should be able to move from a summary to the affected sessions, tool calls and evaluation results.
• How are findings ranked, assigned and resolved? A useful workflow has severity, ownership, alerts and a way to detect recurrence after a fix ships.
• What happens after the diagnosis? Look for a recommendation, a way to test the proposed fix and, where the risk demands it, a policy or guardrail that prevents recurrence.

Scoring

• Is every session scored, or a sample? At what rate, and who sets it? A sampled monitor misses rare failures by design. Judgment Labs' monitoring, for example, reports on "traces the judge evaluated, not necessarily every trace sent to the project".
• Whose judge model runs, and who pays for its tokens? A vendor's own evaluation model, a model connection you configure, or your own code. Each changes your bill and where your transcripts go.
• Which judges are built in, and can I read their prompts? A metric you cannot read is a metric you cannot calibrate.
• How do I check a judge against human labels? Look for annotation and an agreement measure, or at least an easy export of scores to compute one.

Data

• Where is data stored, in which region, and who are the subprocessors?
• How long is it kept on each tier, and can I delete one customer's sessions?
• How do I export traces, scores and datasets, and in what format? Ask for the command or the endpoint, not a yes.
• Is PII or secret redaction done before data leaves my machines, or after it arrives?

Access and Money

• Which tier includes SSO/SAML, and which includes RBAC and audit logs? This one answer often moves the real price more than anything on the pricing page.
• What does the meter count - traces, spans, events, runs, scores, credits, gigabytes? Then ask what one of your sessions costs in that unit.
• What happens at the cap: overage, throttling, or ingestion stops? Raindrop's free Hobby tier, for example, stops ingesting at 1,000 events a month.

Runtime and Deployment

• Does anything happen during the run, or only after? If the answer is "we alert", the product is a monitor. That may be all you need; just know it.
• If it blocks, where does it sit - model I/O, a gateway, or the agent hook layer - and what happens when the enforcement point is unreachable?
• Can I self-host today, on which tier, and what do I operate? "Coming soon" deserves a date.
• Can I start without a sales call, and what does a trial need from me?

What the Public Record Already Answers

Several of these questions are answered in each vendor's docs and pricing pages. Use the table to skip them and spend demo time on the rest.

Run the Trial on Your Own Failures

Before the demo, pick five sessions where your agent failed and five where it behaved correctly. Run the same ten through each product and compare detection, evidence and setup time. Then add a larger batch of unreviewed production sessions and see whether the platform surfaces a recurring problem your team did not provide in advance. Vendor sample data cannot test either job.

Answers Worth a Follow-up

• "We sample intelligently." Ask for the rate, who sets it, and whether you can score every session for one agent.
• "SSO is available." Ask on which tier.
• "Self-hosting is on the roadmap." Ask for a date in writing, or plan as if it does not exist.
• "Unlimited." Ask what the fair-use limit is.
• "It prevents failures." Ask what runs before the tool executes. If the answer is an alert, the product detects, and detection may still be what you need.

How Failproof AI Answers the Checklist

Failproof AI separates evaluation from failure analysis. Evaluations measure known behaviors, while automated audits look across production sessions for recurring patterns and turn the supporting evidence into findings.

• Failure discovery: automated audits group related failures into findings with evidence and a recommended fix, including patterns outside the team's existing checks.
• Sampling: each evaluation can use a condition to select the sessions it scores.
• Where judges run: evaluations run in Failproof AI Cloud. Code checks call no model, while LLM judges use the configured model connection.
• Evaluation options: code-based checks, LLM judges, prebuilt policy packs and existing evaluation suites.
• Calibration: query results with read-only SQL, or pull a session with GET /v1/sessions/{id}/export - "the exact bytes an evaluator receives" - and run your agreement checks offline.
• Export: the public /v1 API with organization-scoped keys, and the fp CLI (uv tool install fp-cloud-cli); fp --json evals --aggregate --env production returns results as JSON. Not everything is scriptable: backtest is dashboard-only, and agent contracts are not in the fp CLI.
• SSO/SAML and RBAC: on the Scale and Enterprise tiers.
• Meter: runs and evals, each with a published allowance and overage per tier; each session-and-evaluation pair is one billable evaluation (pricing).
• Fixing workflow: findings carry recommendations and can lead to tested behavioral policies that allow, steer or deny actions in supported agent harnesses.
• Self-host: the MIT CLI runs locally with no server; self-hosted Cloud is Enterprise, on Kubernetes 1.27+ with Helm 3, ClickHouse and PostgreSQL.
• Trial: the free Cloud tier needs no card.
• Data region, subprocessors, deletion: not in the public docs. Ask.

When You Can Skip the Checklist

For a two-week spike on a free tier with synthetic or non-sensitive data, skip most of it: sign up, send sessions, see whether it helps. The checklist earns its time when you are about to send production transcripts, sign an annual contract, or pass a security review. For that last one, the security review checklist goes deeper.