Can you self-host Judgment Labs?

the short answer

You cannot self-host the Judgment Labs platform today. The Apache-2.0 judgeval SDK runs in your code, but hosted tracing, evaluation, monitoring and alerts use the Judgment platform. Its docs list platform self-hosting as coming soon, with early access through contact@judgmentlabs.ai. Failproof AI offers its complete Cloud platform as an Enterprise deployment in your environment. Latitude and Langfuse can be self-hosted under MIT, while Future AGI publishes an Apache-2.0 core with separately licensed enterprise code.

Platform self-host: Coming soon, per the docs; early access via contact@judgmentlabs.ai.
SDK: judgeval, Apache-2.0: runs in your process and reports to the hosted platform.
Free self-host today: Latitude (MIT), Langfuse (MIT outside ee folders), and Future AGI's Apache-2.0 core.
Enterprise self-host today: Failproof AI (self-hosted Cloud), Galileo (VPC or on-prem), Braintrust (data plane).

What Runs Where Today

Judgment Labs has two halves. The judgeval SDK is open source under Apache-2.0 and runs inside your application, tracing it over OpenTelemetry. What it reports to is Judgment Labs' hosted platform: the README's quickstart exports a JUDGMENT_API_KEY and a JUDGMENT_ORG_ID, and its online monitoring scores production traffic "server-side". Trace storage, judges on live traffic, dashboards and alerts all run there.

The self-hosting page says: "Self-hosting capabilities for the Judgment Platform are coming soon." It directs inquiries and early-access requests to contact@judgmentlabs.ai. The public site and docs do not describe VPC deployment, an on-premises installation or SOC 2. A private option may be available under contract, so ask directly if one of those requirements decides the purchase.

If You Can Wait for It

If Judgment Labs' judges are why you are buying - Agent Judge over long trajectories, AutoRubrics, Behavior Discovery - waiting may be the right call. Ask, in writing:

When, and whether that date can go in the contract.
What ships: the whole platform, including judges on live traffic and Behavior Discovery, or a subset.
Who operates it: fully self-managed, or a vendor-managed deployment in your cloud account.
What it runs on: the databases and services you will operate, and the upgrade path.
How it is priced against the hosted version, which is itself not published.

Meanwhile, the question is whether a hosted pilot is acceptable. For some teams a pilot on non-production traffic, with sensitive fields redacted before they leave the process, passes review while the self-hosted version is built. For regulated data it usually does not.

Options That Run on Your Infrastructure Today

Tool	Self-host on	What you operate
Failproof AI	CLI free (MIT); self-hosted Cloud on Enterprise	CLI: nothing beyond the developer machine. Cloud: Kubernetes 1.27+ with Helm 3, ClickHouse, PostgreSQL, optional Redis, SMTP for production
Latitude	Free, MIT	The same images as its cloud, via Docker Compose or Swarm, Helm, or Railway
Langfuse	Free, MIT outside `ee` folders; Enterprise adds audit logs and project-level RBAC	Postgres, ClickHouse, Redis and blob storage; Docker Compose for testing, Kubernetes (Helm) for production
Future AGI	Free, Apache-2.0; code under `ee/` needs an Enterprise key or Cloud plan in production	Requirements page: PostgreSQL, ClickHouse, Redis, Temporal, MinIO and a code executor; Docker Compose; no Helm yet
Galileo (now Splunk Agent Observability)	Enterprise	Hosted, VPC or on-prem, by contract
Braintrust	Enterprise	The data plane (API, PostgreSQL, Redis, object storage, Brainstore) in AWS, GCP or Azure via Terraform; Braintrust runs the control plane
Raindrop	Partner beta	Self-hosting and VPC for "a select group of initial partners"

Based on each vendor's docs and pricing pages.

Failproof AI offers self-hosted Cloud on Enterprise. The Kubernetes deployment covers agent tracing, code-based and LLM-based evaluations, automated recurring-failure analysis, findings with recommended fixes, alerts, queries, dashboards and policy management. This is the option to compare when the requirement is to keep the complete observability and evaluation workflow inside your environment. The MIT CLI is separate and runs local hook policies on developer machines.

Latitude ships the same platform images used by its cloud and documents Compose, Helm and Railway deployment paths. It covers tracing, prompt management, datasets and post-session evaluations with rules, an LLM judge or JavaScript.

Langfuse self-hosts for free with the core platform - tracing, prompt management, datasets and LLM-as-a-judge evaluators - and, per its self-host pricing, enterprise SSO. It is general LLM observability rather than a product built around agent behaviors.

Future AGI combines evaluations, simulation, a gateway and guardrails, and it is the heaviest stack in this table: its Docker Compose guide starts "all 21 services". The repository has an Apache-2.0 core and separately licensed code under ee/. Self-host telemetry is on by default per its README; set FUTURE_AGI_TELEMETRY_DISABLED=1 to turn it off.

Galileo and Braintrust put self-hosting behind an enterprise contract. Braintrust's is a hybrid: your cloud holds the data plane, and Braintrust runs the web UI, authentication and metadata, which meets some residency requirements with less to operate.

What Self-Hosting Costs You

The license is the small part of the bill. Three of the stacks above store traces in ClickHouse, and a trace store for agents grows fast, because every tool call and result is recorded. Plan for people-time, not just machines:

Stateful services to run. Each database or queue means backups, upgrades, capacity planning and an on-call page. Count them before you choose.
Upgrades. A self-hosted instance only gets new features when you upgrade it, so budget time for upgrades or accept falling behind the cloud version.
Judge tokens. Self-hosting the platform does not self-host the judge. If your evaluators call a hosted model, transcripts still leave your network unless you also run the model.
Retention and deletion. On your own infrastructure, retention limits and deletion requests are yours to implement and prove.

When Hosted Is Fine

If the requirement behind "self-hosted" is a specific control, such as a data processing agreement, retention limit or data region, a hosted or vendor-managed VPC deployment may satisfy it with less operational work. Ask the security team to name the required control before committing to databases, backups and upgrades. If you need agent observability, code-based and LLM-based evaluations, recurring-failure analysis and alerts inside your own VPC sooner, talk to the Failproof AI team about deploying the platform in your environment.

FAQ

Is judgeval self-hostable?

judgeval is a client library, so it runs in your own process. It does not store or judge traces on its own: the quickstart requires a Judgment API key and organization ID, and monitoring runs on the hosted platform. The docs list self-hosting for that platform as coming soon.

When will Judgment Labs support self-hosting?

Not published. The docs say coming soon and direct inquiries and early access to contact@judgmentlabs.ai. Treat any date you hear on a call as a roadmap statement, and if your purchase depends on it, ask for the date and the terms in the contract.

Does Judgment Labs offer VPC or on-prem deployment?

Its public site and docs do not describe VPC or on-premises deployment, while the self-hosting page says platform self-hosting is coming soon. Ask Judgment Labs directly because private deployment options are sometimes negotiated rather than listed publicly.

Which self-hosted option is lightest to run?

By component count, Langfuse (Postgres, ClickHouse, Redis, blob storage) and self-hosted Failproof AI Cloud (ClickHouse, PostgreSQL, optional Redis, on Kubernetes with Helm 3) are smaller than Future AGI, whose Compose guide starts 21 services. Latitude ships the same images as its cloud with Compose, Helm and Railway recipes. Braintrust's hybrid mode leaves the control plane to Braintrust.

Get Started

Failproof AI is free to start. It finds recurring failure modes across agent sessions using code-based and LLM-based evaluations, groups the evidence into findings, and recommends fixes. Bring the eval suite you already have, alert the right owner when behavior drifts, and turn a tested fix into a policy that prevents the failure from recurring. See pricing for the tiers.

start free get a demo →

Sources

Checked against each vendor's own site and docs on 2026-09-14. Products change; if a detail here is out of date, tell us at support@befailproof.ai.