Self-hosted agent evaluation platforms

Three Things "Self-Hosted" Can Mean

Vendors use the word for different arrangements, and a security review cares which one you are buying.

• Open-source platform. You run the UI, API, workers and databases from public images without a hosting contract. Langfuse, Latitude and the Apache-2.0 parts of Future AGI fit this model. Some enterprise features still require a licence key.
• Source-available platform. Phoenix can be deployed on your infrastructure under the Elastic License 2.0. The licence permits internal use but restricts offering it to third parties as a managed service.
• Enterprise self-host. The same product, deployed in your cloud or data center under a commercial agreement. LangSmith, Galileo and Failproof AI offer this.
• Hybrid. Braintrust's model: the data plane that stores traces, logs and datasets runs in your cloud; the web UI, authentication and metadata stay on Braintrust's control plane.

An open-source SDK is not self-hosting. judgeval, Judgment Labs' SDK, is Apache-2.0, but the platform it reports to is hosted.

What You Can Run Yourself, Platform by Platform

Two details change the math. Langfuse's free self-hosted edition includes every core feature - tracing, LLM-as-a-judge evaluators, prompt management, datasets - but project-level RBAC, audit logs, data retention management and server-side masking need its Enterprise license. Future AGI's self-hosted install sends telemetry by default; FUTURE_AGI_TELEMETRY_DISABLED=1 turns it off.

What You Are Signing up to Operate

Look at the last column again. Langfuse, LangSmith, Future AGI and Failproof AI all list ClickHouse, and all pair it with PostgreSQL. Agent traces are wide and numerous - every tool call, every result - which is the workload a column store is built for. That is a sound choice, and it is also a database your team now patches, backs up, sizes and gets paged for.

• Upgrades. Each vendor release may carry schema migrations. Someone reads the changelog before running them.
• Capacity. Retention is now your disk, not a line on a pricing page.
• Backups and restore. Tested, not just configured.
• Access control. SSO, RBAC and audit logs may be features you still pay for, even on your own hardware.

None of this is a reason not to self-host. It is a reason to ask who will own it before the pilot, not after the first disk alert.

Failproof AI Deployment Options

Failproof AI has two deployment models. The MIT-licensed failproofai CLI runs policies locally across supported agent harnesses and does not require a Cloud account. Failproof AI Cloud handles trace storage and querying, evaluations, automated failure analysis, findings and alerts. Self-hosting the Cloud platform is an Enterprise deployment on Kubernetes 1.27+ with Helm 3, ClickHouse and PostgreSQL, with Redis optional and SMTP required for production.

If the whole evaluation platform must run on your infrastructure without a commercial agreement, choose an open-source option such as Langfuse or Latitude. Failproof AI suits a different requirement: local policies can run without Cloud, while teams that need tracing, evaluations and failure analysis inside their own environment can use the Enterprise deployment.

Choosing Between Them

• No budget, must self-host this quarter: Langfuse or Latitude. Both are MIT and ship the same images as their clouds.
• Want the widest feature set and can run a big stack: Future AGI, with the telemetry flag set.
• Only the data has to stay with you: Braintrust's hybrid model keeps traces and datasets in your cloud and leaves the UI hosted.
• Need an agreement, support and an SLA: Enterprise from LangSmith, Galileo, Braintrust or Failproof AI.
• Evaluating Judgment Labs or Raindrop and need data on your infrastructure now: neither offers general self-hosting yet; ask each vendor for a date in writing.

What to Test in a Self-Host Pilot

A pilot that only proves the containers start has tested the easy part. Before you commit, run these against whichever platform you shortlist:

1. An upgrade. Deploy one version behind the latest, send real traces, then upgrade. Note what the migration touched and how long ingestion paused.
2. A restore. Back up PostgreSQL and the trace store, wipe a test environment and restore it. A backup you have never restored is a hope.
3. Your identity provider. Confirm SSO works on the self-hosted edition you would actually run, and whether it needs a paid license there too.
4. Outbound traffic. Watch what the install calls out to: telemetry, license checks, CDNs. Future AGI documents its telemetry flag; ask the others.
5. Judge traffic. If the platform runs LLM judges, find out whose model endpoint the transcripts go to. Self-hosting the platform does not keep data in-house if the judge calls a public API.
6. Export. Pull a week of traces and scores out in a format you could load somewhere else.

Each takes an afternoon or so. Together they tell you what the on-call rotation will look like before you have one.

When the Hosted Tier Is Fine

Self-hosting answers a data-residency or contractual requirement. If you do not have one, a hosted tier is usually the cheaper choice once you count the on-call time. Check first whether redaction before ingest would satisfy your reviewers, and whether a vendor's region already does: Latitude's cloud runs in AWS eu-central-1, for example. Keep the self-hosted option as your exit, not your starting point. The security review checklist lists what to ask a hosted vendor instead.