judgment labs vs latitude

An Open SDK on a Hosted Platform, and an Open Platform

Judgment Labs is an evaluation company. Its judgeval SDK, Apache-2.0, traces agent runs over OpenTelemetry and sends them to the hosted Judgment platform, where judges score them: an Agent Judge that follows a natural-language rubric, and a Code Judge that runs your Python. Agent Behavior Monitoring runs judges continuously and alerts on detection rates; Behavior Discovery surfaces failure modes from unlabeled production trajectories. The platform itself is not open source.

Latitude, from Barcelona, traces sessions, tools, memory and cost over OpenTelemetry with about 40 integrations; evaluations score completed sessions; humans annotate; and since July 2026 Agent Dispatch sends a coding agent to investigate a failure and open a fix PR. The platform repository, latitude-llm, moved from LGPL-3.0 to MIT on May 20, 2026.

Latitude has changed shape more than once, and it is fair to state that as history rather than a knock: data apps, then a prompt playground, then Latitude Agents, then an agent-builder, and in June 2026 a V2 focused only on agent observability. V2 removed the prompt gateway, triggers and hosted tools; V1 keeps running for existing customers, but new features ship only to V2. If you are evaluating Latitude today, you are evaluating V2.

Side by Side

Both products sample and both work after the fact. Judgment Labs' monitoring scores the traces a judge evaluated, "not necessarily every trace sent"; Latitude checks each completed session against every active evaluation's scope and sampling. Neither blocks or changes a response. The differences are in what is open, where it runs, and what happens once a failure is found.

What "Open Source" Gets You in Each

With Judgment Labs, open source means the instrumentation. judgeval is on PyPI under Apache-2.0, with about a thousand GitHub stars, and there are small JavaScript, Go and Java siblings. You can read exactly what the SDK captures and sends, which matters to a security reviewer. You cannot run the judges, the monitoring or the dashboards yourself; those live in the hosted platform, and the self-hosting page says it is "coming soon".

With Latitude, open source means everything. Its deployment docs say every self-host option "deploys the same images with the same configurable values" as the cloud. There are two routes: a single host with Docker Compose, or a highly available cluster on Kubernetes with a Helm chart. You run five application services (web, api, ingest, workers, workflows) on top of Postgres with pgvector, ClickHouse, Redis, Temporal and an object store. That is a real system to operate, and it is free.

One Data-Residency Review, Both Vendors

Take a 120-person European legal-tech company. Its contract agents read client documents, and its DPO has one question before any tool is trialled: where do traces live, and can we keep them inside the EU or on our own servers?

1. Latitude cloud

   Latitude's data-protection docs put the cloud in AWS eu-central-1, Frankfurt. The Starter tier is free with unlimited seats and 30-day retention, so the DPO can review a live account the same day. Pro, at $99 a month, includes SOC 2 and ISO 27001 reports.

2. Latitude self-hosted

   If the answer has to be "our servers", the MIT platform runs on your own infrastructure with Compose or Helm, on the same images as the cloud, at no licence cost. Custom on-premises deployment with SAML SSO, RBAC and an SLA is the Enterprise tier.

3. Judgment Labs

   The platform is hosted, and its hosting region is not stated in the public sources we checked. Self-hosting is listed as coming soon, with an email address for early access. The judgeval SDK can be audited today, but the traces it sends go to the hosted service.

For this company, Latitude is the only one of the two that can pass the review this month. That says nothing about which judges better; it says the question was asked before judging came up. If your data can go to a US-hosted service, or if the Judgment Labs demo produces a region and a date you can live with, the comparison moves back to the product.

After the Alert: A Dataset or a Pull Request

The biggest product difference is what each tool does with a failure once it has one. Judgment Labs turns it into evaluation material. Monitoring alerts in Slack or through a webhook, flagged traces go into datasets, and offline tests run your fixed agent against those production cases before you ship. The loop ends with evidence that the fix works.

Latitude turns it into a pull request. Agent Dispatch, launched July 10, 2026, triggers on new signals, escalations or monitor incidents and hands the failure to Cursor, Claude Code, Linear or any webhook; the coding agent "investigates through Latitude's MCP, traces the failure back to the cause, and opens a PR on your side". You set per-project guardrails on what it may do, mute signals you do not want acted on, and review every PR before it merges. The loop ends with a diff.

The two are complementary more than competing: a PR still needs a test, and a test still needs someone to write the fix. But they signal what each company thinks the bottleneck is. Judgment Labs thinks it is knowing whether you are right; Latitude thinks it is the engineering hours between an alert and a patch.

Python Judges or JavaScript Scripts

A smaller difference that decides some teams: the language of custom checks. Judgment Labs' Code Judge runs Python; Latitude's custom evaluations are JavaScript, alongside rule conditions its docs describe as "deterministic checks, free and instant" and LLM-as-judge evaluations. Latitude also keeps evaluations it generated aligned to human annotations over time, though a detector you write yourself "runs exactly as written". If your ML team lives in Python and your product team in TypeScript, that line tends to fall along the org chart.

Where Each One Is Stronger

Judgment Labs

• Judging long trajectories. Agent Judge is built to reason over a whole trace against a rubric, and the team published its long-context evaluation work in May 2026. When the question is whether a forty-step run actually did the job, that focus shows.
• Finding what to judge. Behavior Discovery surfaces failure modes and usage patterns from unlabeled production trajectories, and Behaviors tag sessions by what users asked, what the agent did and what its environment returned. Latitude also generates evaluations from signals; Judgment Labs has made discovery its research focus.
• Richer judge output. Judges return binary, categorical or numeric results with reasoning, so one judge can say which of several failure types occurred rather than only whether one did.
• SDK languages. judgeval has JavaScript, Go and Java siblings next to the Python library.

Latitude

• Ownership. The entire platform is MIT. You can read, fork and run every part of it, and the cloud runs the same images.
• Starting. A free Starter tier with unlimited seats, no demo required, and a $99 Pro tier that includes SOC 2 and ISO 27001 reports.
• Humans in the loop. Annotation is built in, and evaluations Latitude generates are kept aligned to human judgment over time.
• The repair step. Agent Dispatch turns a detected failure into a pull request you review, rather than a ticket someone has to pick up.

Neither list is a ranking. They describe two companies that decided different things were scarce: Judgment Labs thinks it is good judgment, Latitude thinks it is the time between an alert and a fix.

Pricing

Latitude's price is published, but its unit is not fully explained: the pricing page lists "What is a credit and how are they calculated?" as a question without answering it there, so ask before you model a bill. Judgment Labs publishes no plans at all; the site's one call to action is "Book demo". Neither gap tells you which is cheaper for your volume. Latitude's gap is one question in a support ticket; Judgment Labs' is a sales cycle.