Security review checklist for agent observability vendors

Why Agent Traces Need a Stricter Review

An LLM observability review used to be about prompts and completions. An agent trace carries far more. Every tool call and its result is an event: the shell command the agent ran, the file it read, the rows a query returned, the environment variable it printed while debugging. A coding agent's session can contain a production connection string it found in a config file. A support agent's session contains customer records.

So the vendor is not storing "LLM logs". It is storing a record of everything your agents touched. Review it the way you would review a log-management vendor that receives application logs with secrets in them, and ask who else sees the data, because some products send transcripts to a model to judge or summarize them.

The Checklist

Data Flow

• What is collected by default, and what is opt-in: tool inputs, tool outputs, file contents, environment?
• When does data leave the machine - continuously, at session end, or only after an account is connected?
• Does the vendor send transcripts to a model provider for hosted judges, summaries or an assistant? Which provider, on what terms?

Redaction

• Is PII or secret redaction applied before data leaves your machine, at ingestion, or only at display?
• Is it pattern-based, model-based or both, and which SDKs and languages does it cover?

Retention and Deletion

• How long is data kept on each tier, and can you shorten it?
• Can you delete one session, one end user's data, or everything - and how long until deletion reaches backups?

Identity and Access

• Which tier includes SSO/SAML? SCIM provisioning?
• Can roles stop most engineers from reading production transcripts?
• Are there audit logs of who viewed or exported what?

Deployment and Assurance

• Self-host or on-prem: which tier, and what runs where? Which region does the hosted service use?
• Which compliance reports exist - type, period, scope - and under what NDA?
• Is there a subprocessor list, with notice of changes? What are the breach-notification terms?

Questions Only Agent Tools Raise

• What runs on the developer's machine, with what permissions? A hook or SDK inside the agent's harness runs with the agent's access. Ask for the source, or at least a precise description of what it does.
• If it can block, does it fail open or closed? An enforcement point that fails open lets actions through while it is down; one that fails closed stops work. Ask which, for each component.
• Can the agent switch it off? An agent with a shell can edit configuration files. In Failproof AI, block-failproofai-commands is always on; ask any vendor how its hook resists being disabled by the agent it watches.
• Who wrote the rules? Community policy packs are third-party code. Failproof AI's are GitHub releases, checksum-pinned, each "published by its author, not audited by Failproof AI" - read a pack's manifest before enabling it.
• Does the vendor's own model read your data? Built-in judges, assistants and automatic-fix features often do, which makes their model provider a subprocessor.

How Failproof AI Handles Security and Deployment

Failproof AI supports two deployment patterns: local policy enforcement without a Cloud account, and a connected setup for tracing, evaluations, failure analysis and alerts. The table separates documented capabilities from details that still need confirmation during a security review.

Local policy enforcement does not require a Cloud account. Once Cloud is connected, the review should cover trace collection, model access and the controls around stored production data. Failproof documents organization-level isolation, scoped API keys and code-based evaluations that do not call a model. SSO/SAML and RBAC start on Scale. Confirm model providers, data regions, subprocessors, deletion, audit logs and breach terms in writing.

What Other Vendors Publish

Running the Review Without Stalling the Trial

Security reviews stall trials when they begin too late or treat every proof of concept like a production deployment. Use a staged review:

1. Before the trial: decide what data the trial may use. Synthetic sessions, or an internal agent with no customer data, need little review.
2. During the trial: ask for the vendor's compliance report, subprocessor list and data processing agreement. These take the longest to arrive.
3. Before production data flows: run the checklist above, in writing, and test the answers you can - where data goes, what redaction removes, whether roles actually hide transcripts.
4. At renewal: re-ask the questions whose answers drift - subprocessors, retention defaults, new features backed by a model.

The test step matters most. Send a session containing a fake API key and a fake email address, then look at what the vendor stored and who can see it. That one experiment answers the redaction and access questions better than any questionnaire.

When a Light Review Is Enough

Not every trial needs the full checklist. If you run the Failproof AI CLI for local enforcement only, with no Cloud account, the review covers MIT code on your machines and the policy packs you enable - review it as you would any open-source developer tool. A trial on synthetic sessions, or on an agent that touches no customer data, can go ahead on a lighter review too. Save the full checklist for the moment production transcripts start flowing to a vendor, and start it early: compliance reports and data processing agreements are the slowest documents in any deal.