judgment labs vs raindrop

Judge-First and Signal-First

Judgment Labs and Raindrop are often shortlisted together, and not only because Lightspeed led rounds in both. Each sells a way to find the agent failures that throw no exception: the answer that is wrong but confident, the tool call that succeeds and does the wrong thing, the agent that gives up. They approach it from opposite ends.

Judgment Labs is judge-first. The open-source judgeval SDK traces runs over OpenTelemetry; an Agent Judge follows a natural-language rubric and reasons over the trace, a Code Judge runs Python you write, and Agent Behavior Monitoring runs judges continuously on production traces and alerts on detection rates. Behavior Discovery looks for failure modes in unlabeled trajectories before you know to write a judge.

Raindrop is signal-first. You describe a behavior on the Signals page; Raindrop's agent reads your traces, writes the rules, turns them into a classifier and scans your events, and you mark samples Match or Not Match until it is right. Failures in single runs are Stumbles; the same stumble recurring across runs and users becomes an Issue, ranked by severity. A Triage Agent investigates root causes, and anything you can do in the UI, a coding agent can do over MCP.

Side by Side

The row that decides most evaluations is the second one. A signal answers "did this happen?" A judge can also answer "how bad was it?" or "which of these five things went wrong?" Whether you need the extra resolution depends on what you do with the answer.

A Rubric and a Signal Are Different Instruments

Raindrop's bet is that most production failures are easy to recognise and tedious to specify. "The agent makes more than five tool calls when the user is reporting a production issue" is the kind of description its docs recommend, and a yes/no classifier drafted from it, refined by you labelling a sample, is quick to get right. You do not have to label every event; Raindrop surfaces a sample. The output is a rate: how often the behavior happens, per event or per user, which is exactly what you chart and alert on.

Judgment Labs' bet is that the valuable failures are the ones that need reading. Whether a forty-step research run actually answered the question, or whether an escalation should have happened, takes interpretation over the whole trajectory, and a rubric is how you encode that interpretation. Its judges can return a category or a number with reasoning attached. One constraint is worth knowing before you design around it: numeric judges can score a live trace when applied on demand, but cannot be enabled as continuously monitored behaviors. For always-on monitoring, you frame the judge as true or false, or a category, much as you would a signal.

In practice the two converge on "is this behavior present?" for monitoring. They differ in who writes the definition (you, or Raindrop from your description), how much of the trajectory it reasons over, and what else you can do with it: Judgment Labs feeds judged traces into datasets and offline tests; Raindrop feeds signal rates into Issues, alerts and Experiments.

One Silent Failure, Both Tools

A research agent starts telling users "I could not find anything on that" even though its search tool returned relevant results. No error, no exception, and the thumbs-down rate creeps up. Here is the loop in each product.

1. Raindrop: describe, label, alert

   You write the behavior on the Signals page: the agent says it found nothing when a search tool returned results. Raindrop drafts the classifier and scans recent events; you mark a sample Match or Not Match and ask in chat for the rule to exclude empty searches. A signal alert fires in Slack when the rate crosses your threshold, and the recurring stumble shows up as an Issue ranked by severity. The Triage Agent looks for a root cause, and a coding agent can pick the issue up over MCP.

2. Judgment Labs: rubric, monitor, dataset

   You write an Agent Judge rubric: if any search result is relevant to the question, the final answer must use it; return true when the agent claimed nothing was found. A binary judge can run as a monitored behavior, so ABM scores production traces after they complete and alerts on the detection rate in Slack or through a webhook. Flagged traces go into a dataset. Behavior Discovery may have surfaced the pattern before anyone wrote the rubric.

3. Shipping the fix

   With Judgment Labs you run the fixed agent offline against the flagged production cases and ship when the judge passes. With Raindrop you ship the prompt change behind a feature flag and let Experiments compare the two cohorts on the signal rate.

Neither loop is wrong. The Judgment Labs loop gives you confidence before the change reaches users. The Raindrop loop gives you evidence from logged production events after it does, which is the evidence Raindrop argues matters most: its own post on evals says "the truth increasingly lives in production."

Experiments Are Raindrop's Clearest Advantage

Raindrop's Experiments read existing traffic; its docs are explicit that they "don't run or replay your agent". They do not split or route traffic: you ship the change yourself, for example behind a feature flag, then define a baseline and an experiment cohort of already-logged events by model, feature flag, property, tool or date range, pick the signals that define success, and get a result per signal with a 95% confidence interval and a p-value, labelled likely better, likely worse or mixed. Creating experiments requires Pro. Nothing in Judgment Labs' published product does this.

The catch is volume. The docs state no traffic minimum, but two small cohorts make a noisy comparison by nature, so the interval stays wide until enough events pile up. A B2B agent that runs 300 times a day will wait a long time for an interval that excludes zero. That team is better served by offline tests on a fixed set of production cases, which is the loop Judgment Labs is built around.

Open Source, Security and Where the Data Lives

Both companies open-source a developer tool and keep the platform closed. Judgment Labs' judgeval is an Apache-2.0 Python SDK with JavaScript, Go and Java siblings; it instruments your agent and sends traces to the hosted Judgment platform, which is not open source. Raindrop's Workshop is an MIT-licensed local agent debugger and eval runner, launched in May 2026, for poking at a run on your laptop; the production platform is hosted.

Neither is something you can run on your own infrastructure today without a special arrangement. Judgment Labs' docs say self-hosting the platform is "coming soon". Raindrop 2.0, in June 2026, opened self-hosting and VPC to a select group of initial partners, pitched as "traces and agent data never leave your infrastructure", and announced SOC 2 Type II. Raindrop also has PII Guard, which redacts at ingestion: that controls what gets stored, not what the agent does.

If a security review needs a SOC 2 report, SSO and audit logs, Raindrop documents where each sits: SOC 2 Type II announced, SSO/SAML and audit logs on Enterprise. For Judgment Labs, VPC, on-prem and SOC 2 terms are not in the public sources we checked, so they belong on the demo question list rather than in anyone's assumptions.

Pricing: One Published, One Not

Raindrop publishes its plans in its docs. Hobby is free for 1,000 events a month, and ingestion stops at the cap for the rest of the calendar month; it includes one custom signal, with Issues, full Stumbles and experiment creation locked. Pro is $299 a month with a 14-day trial and 10 custom signals, and every event is metered on top: $0.003 each up to a million, then $0.002. SSO/SAML, edge PII redaction, audit logs and warehouse export are Enterprise.

Judgment Labs publishes no plans, no free tier and no meter; the one call to action on its site is "Book demo". That is not evidence it costs more. It does mean the table above has one column you can fill in this afternoon and one you cannot. Ask on the demo how price scales with traces and judge runs, what sampling rate monitoring uses, and when self-hosting will ship.