How to evaluate agent cost and latency against a budget

Measure the Task, Not One Model Call

An agent can return the right answer and still fail economically. A support resolution that takes forty tool calls, repeated retries and several minutes may cost more than the work it replaces. Cost and latency therefore belong beside task completion and quality, not in a separate infrastructure report.

Measure the complete session. Include every model call, tool call and failed attempt used to complete the task. Report both cost per attempted task and cost per successful task because failed sessions still consume tokens and infrastructure.

1. Set a Budget for Each Task Type

Do not use one threshold for every agent session. An interactive support answer, a code change and a background research task have different acceptable costs and response times. Identify the task type in session metadata, then define a cost, latency and step budget for each one.

Start by observing the current distribution. A baseline tells you how large the gap is, but it should not define what “good” means when current performance is already unacceptable.

2. Ask Failproof Where Cost and Latency Are Going

Send complete sessions to Failproof AI and identify the agent and environment consistently. Use Analyze → Queries to calculate task-level cost, latency and success measures from production traces. Queries are read-only SQL, and saved queries can be added directly to dashboards.

You can also ask the Failproof agent to investigate in plain language. Start with a broad question, inspect the generated result and save the useful query so the team uses the same definition every time.

fp agent ask "Show cost per successful task and p95 latency by agent and environment for the last seven days"
fp agent ask "Which production sessions exceeded their latency budget, and what did they have in common?"
fp query run <query-id>

Break the results down by agent, environment, task type and time period. An overall average can hide one workflow that is becoming slower or one agent version that is consuming far more tokens than the rest.

3. Read Cost, Latency and Quality Together

A cheaper agent is not better if it completes fewer tasks, and a faster agent is not better if it skips verification. Put cost and latency beside the evaluations that measure task completion, correctness and required behavior.

Keep each measure visible rather than blending everything into one score. Separate signals make the trade-off clear and preserve the evidence needed to choose a fix.

4. Build Dashboards and Alert on Budget Breaches

Save the queries that define cost per successful task, p50 and p95 latency, over-budget session rate, tool-call count and quality. Build a dashboard from those saved queries so product and engineering teams see the same trend and can open the sessions behind a change.

Create alerts for metric thresholds, evaluation-score changes, compound conditions or custom SQL. A useful alert identifies the affected agent and environment and links to the sessions, instead of announcing only that a global average moved. Route alerts to email, Slack, a webhook or the Failproof dashboard.

Avoid paging on every slow session. Alert on sustained budget-breach rates or a meaningful increase from baseline, then retain individual sessions for investigation.

5. Find What Is Causing the Expensive Tail

A dashboard tells you that cost or latency changed. An audit helps explain why. Scope a Failproof audit to the affected agent, environment and time window with a goal such as “find recurring behaviors in sessions that exceed the cost or latency budget.”

The audit analyzes traces, evaluation results and related evidence, then groups similar sessions into findings. Common causes include repeated tool retries, oversized tool results, unnecessary model turns, growing context, a slow dependency or a workflow that sends simple tasks through an expensive path.

fp audits findings --status open --limit 20

Review the affected sessions and recommendation, assign the issue and fix the actual cause. That may mean changing the prompt, routing a task to a smaller model, truncating stale context, repairing a tool, reducing retries or splitting one workflow into clearer steps.

6. Verify the Improvement in Production

Keep the original queries, evaluations, dashboards and audit goal running after the change. Compare the same agent and task type before and after deployment. The fix is successful only if cost or latency improves without lowering completion or quality and the original failure pattern stops recurring.

Re-baseline budgets after a deliberate model or workflow change, but preserve the earlier dashboard long enough to show the effect. Otherwise a new threshold can make a regression disappear on paper.