Judgment Labs vs Langfuse (2026)

the short answer

Pick Judgment Labs if judging agent behavior is the whole job: Agent Judge scores long trajectories against a rubric, and Behavior Discovery looks for failures you did not know to test for. Pick Langfuse if you want an open-source observability base you can self-host for free today, with prompt management, managed LLM-as-a-judge evaluators and published prices from $0. Neither blocks an agent action at runtime.

Judgment Labs: Rubric and code judges, behavior discovery, sampled monitoring. Hosted platform, Apache-2.0 SDK. Pricing by demo.
Langfuse: Tracing, prompt management, evals, datasets, playground. MIT core. Cloud from $0 to $2,499 a month.
Self-host: Judgment Labs: listed as coming soon. Langfuse: free MIT edition today; Enterprise self-host at custom pricing.
Blocks an action: Neither. Both score runs after they happen.

What Each Product Actually Is

Judgment Labs has three layers. The open-source judgeval SDK (Apache-2.0) traces agent runs over OpenTelemetry with @Tracer.observe() and auto-instruments OpenAI, Anthropic, Google GenAI and Together. Judges score those traces: Agent Judge follows a natural-language rubric, Code Judge runs your Python, and both run online, offline or on demand. Agent Behavior Monitoring (ABM) runs judges continuously on a sample of production traces and turns detections into alerts, webhooks, automations and dataset actions. The platform all of this reports to is hosted and closed.

Langfuse covers tracing (its Python SDK is a thin layer over the OpenTelemetry client, and the server accepts OTLP at /api/public/otel), prompt management, evaluation (LLM-as-a-judge, code-based evaluators, user feedback, manual labels and custom pipelines over the API), datasets for experiments, and a playground. The core is MIT-licensed, with enterprise features in a separately licensed ee folder. The repository has about 34,600 GitHub stars, and its README says Langfuse has been part of ClickHouse since January 2026.

So both trace and both run LLM judges, but they are built for different problems. Judgment Labs is a narrow, deep product about judging agent behavior and finding the behaviors worth judging. Langfuse is the everyday engineering platform for an LLM application, where evaluation is one tab among several. Which fits depends on whether your hard problem is "what is my agent doing wrong" or "where do I keep traces, prompts and scores for everything".

Side by Side

As of September 2026, from each vendor's public site, docs and README.
capability	Judgment Labs	Langfuse
Agent tracing	OpenTelemetry via judgeval	OpenTelemetry-based SDKs, OTLP endpoint
LLM-as-a-judge	Agent Judge (rubric), Code Judge	Managed evaluators, prebuilt templates
Finding unknown failure patterns	Behaviors, Behavior Discovery	You define what to score
Prompt management	Not in its docs	Versioned, labelled, linked to traces
Alerts	ABM alerts, webhooks, Slack	Slack, webhook, GitHub Actions
Open source	Apache-2.0 SDK; platform closed	MIT core; ee features licensed
Self-host	Coming soon	Free: Docker Compose, Helm, Terraform
Public pricing	Demo only	$0 / $29 / $199 / $2,499 a month
Blocks an action at runtime	Monitoring runs after execution	Evaluators do not block

The last row is the one both vendors state plainly. Judgment Labs' docs describe monitoring on the "completed traces" an instrumented agent sends, with measures that cover "traces the judge evaluated, not necessarily every trace sent to the project", and automations that "notify a team or trigger an action" afterwards. Langfuse's security docs pair "LLM Security libraries for run-time security measures" with Langfuse "for the ex-post evaluation of the effectiveness of these measures". Both are monitors by design, which keeps judging off the request path and means neither stops a bad action while it happens.

Where Judgment Labs Is Stronger

On the judging itself, Judgment Labs has gone further. Langfuse's managed evaluators are general-purpose: prebuilt templates, filters on observation and trace fields, sampling, scores attached to the observation. Judgment Labs is built around the agent trajectory as the unit of judgment.

Rubric judges for long trajectories. Agent Judge reasons over a whole trace against a natural-language rubric; the company published its approach to long-context agent evaluation in May 2026.
Finding what to judge. Behaviors, Behavior Discovery and AutoRubrics go after failures you have not written an evaluator for yet. Langfuse assumes you already know what to score.
Monitoring as a workflow. ABM tracks detection rates and routes detections to alerts, webhooks, automations and datasets, and offline tests run production-derived cases against a candidate agent.
Coding agents in the loop. An MCP server gives coding agents access to Judgment project data, and a judgment-plugin sends Claude Code and Codex sessions for judging.

The team raised $32M across seed and Series A in May 2026, both led by Lightspeed, and publishes on evaluation methods. That lead on agent judges is what the money is for.

Where Langfuse Is Stronger

Langfuse is stronger on everything around the judge, and on how you get hold of it.

Open source you can run today. The self-hosted edition is MIT with "all core platform features and APIs" and unlimited usage; LLM-as-a-judge evaluators, the playground, annotation queues and prompt experiments are not held back for a paid licence. Judgment Labs lists platform self-hosting as coming soon.
Prompt management. Versioned prompts with labels, cached client-side by the SDK and linked to traces so you can compare scores by prompt version. Prompt management does not appear among the sections of Judgment Labs' documentation.
A price list. Every cloud plan, from the free Hobby tier to $2,499 a month, has a published allowance and overage rate.
Alerts on every plan. Threshold alerts on observations and on numeric, categorical or boolean scores, with separate warning and alert levels, delivered to Slack, a webhook or GitHub Actions. Hobby allows 2 per organisation; self-hosted v4 and later has no limit.
Reach. About 34,600 GitHub stars against roughly 1,100 for judgeval as of September 2026, and integrations with LangChain, LlamaIndex, the OpenAI SDK, the Vercel AI SDK and more.

The price of that openness is operations. Self-hosted Langfuse is Postgres, ClickHouse, Redis or Valkey, S3-compatible object storage, and web and worker containers. Docker Compose on one VM is the testing path; Kubernetes with Helm, or Terraform on AWS, Azure or GCP, is the production one.

One Looping Agent, Both Tools

Take a LangGraph research agent that sometimes calls the same search tool with near-identical queries until it hits its step limit. No exception fires; the run just costs ten times what it should and returns a thin answer.

Judgment Labs: rubric, sample, alert
Instrument with judgeval's LangGraph integration. Write an Agent Judge rubric - "the agent does not repeat an equivalent search more than twice" - and attach it to monitoring. Sampled traces are scored after they complete; when the detection rate crosses your threshold, an alert or webhook fires and the flagged traces can go into a dataset. If nobody had named the loop yet, Behavior Discovery is the feature meant to surface it.
Langfuse: evaluator, score alert, experiment
Instrument with the LangChain integration or OpenTelemetry. Create an LLM-as-a-judge evaluator from a template or your own prompt, filtered to this agent's traces and sampled. Add an alert on the average of that score over a one-hour window, sent to Slack. Put failing traces in a dataset, change the prompt, run an experiment against the dataset, and promote the winning prompt version by label.
The next looping run
In both products it runs to its step limit, is scored afterwards and, if it falls in the sample, feeds the alert. Judgment Labs is better placed to tell you the loop is a recurring behavior across agents; Langfuse is better placed to ship the prompt fix and compare it with the old version.

Neither does step zero for you. Someone has to decide that a repeated search is a failure and write it down precisely enough for a judge to agree with them.

Pricing and Buying

The buying experience is where the two differ most. Judgment Labs has no pricing page and its homepage call to action is "Book demo". Langfuse publishes every cloud tier and lets you start on the free one.

	Judgment Labs	Langfuse
Start without talking to sales	Demo first	Yes - Hobby cloud tier, or self-host
Free tier	Not published	Hobby: 50k units a month, 30 days of data, 2 users
Entry paid tier	Not published	Core: $29/month, 100k units, 90 days, unlimited users
What is metered	Not published	Units - traces, observations and scores; $8 per 100k over the allowance, lower with volume
SSO	Not published	Pro ($199) plus the $300/month Teams add-on, or Enterprise ($2,499)
Self-host	Listed as coming soon	Free MIT edition; Enterprise self-host at custom pricing

As of September 2026, from judgmentlabs.ai and langfuse.com/pricing.

A worked Langfuse estimate, with every assumption stated: 10,000 agent sessions a month, each producing 1 trace, 20 observations (LLM calls and tool spans) and 2 scores, is 230,000 units. On Core that is $29 plus 130,000 units over the allowance at $8 per 100,000, or $39.40 a month, assuming overage is billed pro rata. Count your own observations per session before trusting it; an agent that makes 200 tool calls a session is a different bill.

Retention is the other lever in the Langfuse price list: 30 days of data on Hobby, 90 on Core, three years on Pro and Enterprise. If you need to compare this quarter's agent with last year's, retention, not the unit count, is what moves you up a tier.

There is no Judgment Labs number to set beside it. On the demo, ask about trace volume, judge runs, sampling rate, retention, SSO and the self-hosting timeline, and get the answers in writing.

Running Both, or Moving Between Them

Both instrument with OpenTelemetry, so tracing is the portable part of either choice. The judges, rubrics, prompts, alert rules and score history are not: a rubric tuned in Agent Judge and an evaluator prompt tuned in Langfuse are separate assets, calibrated separately.

Some teams will want both: Langfuse as the system of record for traces and prompts, Judgment Labs where its judges and behavior discovery pay for themselves. That is two bills and two places to look, so it suits a team with an owner for evaluation rather than one where evals are everybody's side job.

Which to Choose

Choose Judgment Labs when judging agent behavior is the job - rubric judges over long trajectories, rubric generation, behavior discovery - and a demo-led engagement suits how you buy.
Choose Langfuse when you want an open-source, self-hostable observability base with prompt management, general-purpose LLM-as-a-judge evaluators and a price you can read before you sign up.
Choose Failproof AI if you also need the failing action stopped before it runs: it enforces policies at the agent hook layer (PreToolUse) in Claude Code, Codex, Cursor and other harnesses, scores finished sessions with evaluations you write, and starts on a free tier.

FAQ

Is Langfuse free to self-host?

Yes. The open-source edition is MIT-licensed, with all core platform features and APIs and unlimited usage, including LLM-as-a-judge evaluators, the playground, annotation queues and prompt experiments, per Langfuse's self-host pricing page in September 2026. Self-hosted Enterprise, at custom pricing, adds audit logs, data retention policies, project-level RBAC and SCIM. You operate Postgres, ClickHouse, Redis and object storage yourself.

Does Judgment Labs have prompt management?

Its documentation, as of September 2026, covers tracing, judges, datasets, monitoring, offline tests, integrations, an MCP server and SDK reference; prompt management is not among the documented sections. Langfuse ships it: versioned prompts with labels, cached client-side by the SDK and linked to traces, so scores can be compared by prompt version.

Do Judgment Labs and Langfuse judge every trace?

Both are usually run on a sample. Judgment Labs' monitoring scores traces the judge evaluated, not necessarily every trace sent. Langfuse's LLM-as-a-judge evaluators run asynchronously on observations matched by filters, with sampling options. Judging everything is a cost decision: every judged trace is a model call, and in Langfuse every score is also a billable unit.

Can Judgment Labs or Langfuse block an agent action?

No. Judgment Labs' monitoring judges completed traces, and its automations notify a team or trigger an action - a webhook, a dataset action, a further evaluation - after the run. Langfuse's own security docs leave run-time protection to separate security libraries and position Langfuse for ex-post evaluation of how well they work. Blocking needs a check that runs before the tool call, inside the agent or its harness.

Get Started

Failproof AI is free to start. It finds recurring failure modes across agent sessions using code-based and LLM-based evaluations, groups the evidence into findings, and recommends fixes. Bring the eval suite you already have, alert the right owner when behavior drifts, and turn a tested fix into a policy that prevents the failure from recurring. See pricing for the tiers.

start free get a demo →

Sources

Checked against each vendor's own site and docs on 2026-09-14. Products change; if a detail here is out of date, tell us at support@befailproof.ai.