How to write an AI agent contract

Why an Audit Needs to Know What the Agent Is For

Ask a reviewer - a person or a model - to "find problems" in a hundred agent sessions and you get generic findings: the agent was verbose, it retried a tool, it took a long time. All true, none actionable. The reviewer is not wrong; it simply has no way to know that this agent is supposed to run once a night, must never touch the ledger, and is done when every payout is matched. Without that, a session that ran twice and edited a ledger entry looks like a busy, successful session.

An agent contract is that missing context, written down. It is not a system prompt - the prompt tells the agent how to behave, the contract tells the reviewer what to hold it to - and it is not a policy, because nothing in it stops an action. It is the definition of success and failure that every later judgment is measured against.

The same document pays off outside audits. The Done when section is the first draft of a task-completion rubric. The Must not section defines evaluation criteria, and the narrow rules that can be checked safely before an action may later become runtime policies. A new engineer can also understand the agent's role without reverse-engineering its prompt.

The Five Sections, and How to Write Each

• Purpose. One or two sentences on the outcome the agent exists to produce, in business terms. "Reconcile yesterday's payouts and publish a signed summary", not "an AI assistant for finance".
• Outputs. The concrete artifacts or actions, and where they land. A reviewer should be able to look for each one in the trace: a message in a named channel, a pull request, a row in a table.
• Done when. Observable success criteria. "Every payout is matched or explicitly flagged" can be checked from the session; "the reconciliation is accurate" cannot.
• Cadence. When it runs and how often, including the upper bound. "Once daily after 02:00 UTC. Never more than once." makes a duplicate run a finding rather than a curiosity.
• Must not. Prohibited actions and unacceptable outcomes, each with what the agent should do instead. "Modify ledger entries. Report discrepancies instead." tells the reviewer what the right behaviour looked like.

Three rules make the difference between a contract that sharpens findings and one that is ignored. Write in observable terms: every line should describe something a reviewer could confirm or refute from the trace. Prefer "must" and "must not" to "should try to"; a soft obligation produces soft findings. And keep it short. The limit is 5,000 characters, but a contract a reviewer can hold in mind at once is usually well under half that.

A Worked Example

Here is a contract for a support agent that can look up orders, issue refunds and hand cases to people. Every line maps to something visible in its sessions.

# support-agent

## Purpose
Resolve order problems for customers of the web store in one conversation where
possible, and hand everything else to the support team with the context they need.

## Outputs
- A reply to the customer that states what was done, not what will be done.
- At most one refund per order, through issue_refund, after lookup_order.
- A handoff through escalate_to_human, with the order ID and a one-line summary,
  for every case the agent does not resolve.

## Done when
- The customer's request is resolved, or a handoff has been created.
- Every promise in the final reply matches a tool call in the session.

## Cadence
- On demand, per customer conversation. No scheduled runs.

## Must not
- Refund more than the order total. Offer a handoff instead.
- Refund an order it has not looked up. Look it up first.
- Issue any refund over $200. Hand the case to a human.
- Keep a conversation that mentions a chargeback, a lawyer or legal action.
  Escalate immediately.
- Promise a refund, credit or delivery date it has not actioned.

Notice what is absent: tone guidance, product knowledge, the refund procedure step by step. Those belong in the prompt and the tools. The contract holds only what a reviewer needs to decide whether a session was a success, a failure or a violation.

Mistakes That Make a Contract Useless

• Restating the prompt. A contract that copies the system prompt adds nothing a reviewer did not already have, and it changes every time someone tunes wording. Write the outcome, not the instructions.
• Aspirations instead of criteria. "Provide excellent service" and "be accurate" cannot be confirmed or refuted from a trace. If you cannot picture the session that violates a line, the line will never produce a finding.
• Must not without an instead. "Must not modify ledger entries" tells a reviewer what is wrong; "Report discrepancies instead" tells it what right looked like, which is how it separates a violation from a reasonable judgment call.
• No upper bound on cadence. "Runs nightly" does not make a second run in the same night a failure. "Never more than once" does.
• One contract for many agents. A contract describes one agent. Two agents that share a prompt but do different jobs - a refund agent and a returns agent - need two contracts, or each will be audited against the other's rules.
• Writing it once. The agent changes; the contract has to follow. A new tool, a new output or a new limit is a contract change, reviewed in the same pull request as the code.

A quick test before you register it: give the contract and three real sessions to a colleague who has never seen the agent, and ask them to mark each session as success, failure or violation. If they cannot, or if their answers surprise you, the contract is missing something the agent's owners take for granted, and that gap is exactly what an audit would have stumbled over.

Registering It with Failproof AI

In Failproof AI, a contract belongs to an agent, not to an audit: one contract per agent_id, which the audits covering that agent can use. Create it with the audits API, using a Failproof AI key that has the audits:read and audits:write scopes. The body is the contract text; jq keeps the markdown safely escaped inside the JSON.

curl -s -X POST https://app.befailproof.ai/v1/audits/contracts \
  -H "Authorization: Bearer $FAILPROOFAI_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg body "$(cat contracts/support-agent.md)" \
        '{agent_id: "support-agent", enabled: true, body: $body}')"

# list what is registered
curl -s https://app.befailproof.ai/v1/audits/contracts \
  -H "Authorization: Bearer $FAILPROOFAI_KEY"

Keep contracts in the repository next to the agent's code: the file is the source of truth, a pull request changes it, and the change reaches Failproof AI through the same API or the dashboard. Contracts are not managed through the fp CLI, so a small script around the call above is the usual way to keep the registered text and the file in step. Review a contract change the way you would review the code change that caused it.

How a Contract Changes What an Audit Finds

An audit reviews a population of sessions for a stated failure goal and combines trace evidence, evaluation results and policy hits into findings. Each finding carries an analysis, a recommendation, a severity, the affected sessions and evidence queries. The contract is part of what the audit is configured with, and it sharpens the recommendation as much as the finding: a fix proposed against a stated Must not line is concrete in a way a fix for "unusual behaviour" cannot be. The difference it makes is easiest to see side by side; the findings below are illustrations of the kind of wording to expect, not real output.

An audit also takes a brief and up to five reference URLs. Put material that changes often there - a runbook, a refund policy page - and keep the contract for the stable definition of the agent's job, so a policy page update does not require rewriting what the agent is for.

When a Contract Rule Should Become a Runtime Policy

Do not turn every Must not line into enforcement. First use evaluations and failure analysis to learn whether the rule is being broken, how often and in what context. A narrow rule such as "no refund over $200" can be checked from one tool call and may belong inside the tool or in a pre-tool policy. A rule such as "do not promise what you have not actioned" needs the whole session and remains an evaluation concern. The policy compliance guide explains that split, while turning eval results into runtime guardrails covers backtesting and observing a policy before enforcement.

A contract does not enforce behavior by itself. It gives evaluations and automated failure analysis a stable standard, so violations can be found, grouped and fixed. Only a runtime control in the action path can prevent a matching action before it happens.

When You Do Not Need a Contract

If the agent does one narrow thing and code checks already cover every failure you care about, a contract adds little to your automated checks - though it still helps the next person who reads the agent. If you do not run audits over populations of sessions at all, write the five sections anyway and use them as the rubric for whatever review you do run. The half hour it takes is mostly spent discovering which rules nobody had written down.