raindrop vs braintrust

The Argument Between Them

In September 2025 Raindrop's CTO, Ben Hylak, published "Thoughts on Evals", a reply to Braintrust's CEO on whether evals are the future. His case: "As agents get more capable, they become more unpredictable and harder to test deterministically," and "the truth increasingly lives in production." He also wrote that "most of our customers, however, haven't found their evals to be as useful as they thought" - a statement about Raindrop's own customers, not a measurement.

Braintrust's product is the other side of that argument. Its docs call experiments "the immutable, comparable record of your eval runs", used to "catch regressions before they reach production." Read both positions as what they are: each vendor describing the half of the problem it sells. The useful question for a buyer is which half your failures come from.

What Each Product Actually Is

Raindrop monitors AI products and agents in production. It traces runs, including over OpenTelemetry, with SDKs for the Claude Agent SDK, Vercel AI SDK, LangChain and CrewAI. Signals turn a plain-language description of a behavior into a classifier you refine by labeling. Issue detection groups "stumbles" into recurring issues ranked by severity, a Triage Agent works through them, and fixes go to coding agents over MCP. Experiments compare a baseline cohort with an experiment cohort of already-logged events, split by model, feature flag, property, tool or date range; in Raindrop's words, "Experiments read existing traffic; they don't run or replay your agent." Alerts go to Slack.

Braintrust is built aroundEval(): data (test cases with inputs, optional expected outputs and metadata), a task (the function under test, up to a multi-step agent) and scorers or classifiers. Each run is an experiment. Around it sit datasets, playgrounds, prompt versioning, tracing, online scoring that "evaluates production traces automatically as they're logged, running asynchronously with no impact on latency", a GitHub Action for CI, the MIT-licensed autoevals scorer library, and Loop, an AI assistant that investigates your data and builds scorers, datasets and dashboards.

Side by Side

Notice that neither product is purely what the argument says. Braintrust scores production logs; Raindrop ships a local eval runner. The difference is which end each product is built around, and which end it charges for.

Where Raindrop Is Stronger

• Comparing changes on real traffic. Ship a model, prompt or tool change - behind a feature flag, say - and Experiments compare the new cohort of logged events with a baseline. Small cohorts give noisy comparisons, so this pays off for agents with plenty of traffic; at that scale it answers questions a dataset cannot.
• Finding what you did not test for. Issue detection groups stumbles into recurring issues ranked by severity without you writing a scorer first, and Signals let you name a new behavior in a sentence.
• Triage as a product. A Triage Agent and hand-off of fixes to coding agents over MCP cover the step after detection.
• Named customers. Homepage logos include Speak, Vercel, Clay, Framer and AngelList, and the Tolan case study reports 27.8% fewer memory issues.

Raindrop is also the more coherent product for anyone who shares its view. The CTO's post is a position, openly argued, and the plans follow it: Hobby gives you tracing, search, alerts and signal presets, and Pro is where Issue Detection and Experiments live. If you agree that production is where the truth is, that coherence is a strength. If you do not, no amount of configuration will make Raindrop a dataset-first eval tool.

Where Braintrust Is Stronger

• Knowing before merge. Immutable experiments compared over time, and the eval-action GitHub Action that runs evals for node, python or go and posts a live summary comment on the pull request.
• A scorer library. autoevals, MIT, in Python and TypeScript: Factuality, ClosedQA and Summarization judges, RAG scorers such as Faithfulness and Context precision, heuristics such as Levenshtein and JSON diff, and embedding similarity.
• Graded scores. Scorers return numbers and classifiers return labels, so you can track "slightly worse" as well as "failed". Raindrop Signals are yes/no.
• An easier start. Starter is $0 with unlimited users and keeps ingesting past its allowance at a metered rate; Raindrop Hobby stops ingesting at 1,000 events and keeps Issues and Experiments preview-only.
• Self-hosting today. On Enterprise you run the data plane - API, PostgreSQL, Redis, object storage, Brainstore - in AWS, GCP or Azure, while Braintrust hosts the UI and authentication. Raindrop's self-hosting is a partner beta.

A Model Swap, Both Ways

You want to move a customer-facing agent to a newer model. The worry is not a crash; it is a quieter agent that refuses more, or a chattier one that gets facts wrong.

1. Braintrust: dataset, experiment, pull request

   Run the frozen dataset through Eval() with the old and the new model as two experiments and compare them; the eval action posts the scores on the pull request that changes the model. Ship if the scores hold, then let online scoring watch a sample of production logs with the same judges.

2. Raindrop: ship to a slice, compare signals

   Ship the new model to part of your users behind your own feature flag. Then create an Experiment with the old model as the baseline cohort and the new one as the experiment cohort, and compare Signals - frustration, refusals, task failure - between them. Raindrop reads the logged events; it does not route traffic itself. With plenty of traffic the comparison settles quickly; with small cohorts, change and noise look alike.

3. What each misses

   The dataset misses what real users do that nobody wrote down. The staged rollout shows some users the worse model before you know it is worse, and cohorts from an agent with a few dozen users are too small to say much.

The honest resolution of the public argument is that your user count settles it. An internal agent with forty users should lean on a dataset; a consumer agent with a million users will learn more from production than any test set can hold. Most teams in between want a small dataset as a gate and production signals as the smoke alarm.

Running Both

The public argument makes them sound exclusive; the products are not. A team can run Braintrust experiments as the pre-merge gate and Raindrop on production for the failures no dataset holds. Raindrop issues become candidate cases for the Braintrust dataset, and a Braintrust experiment tells you whether the fix for a Raindrop issue worked before it ships.

The costs of doing that are real: two bills, two sets of classifiers and scorers describing overlapping failures, and two places to look during an incident. If you run both, name one as the place alerts come from and the other as the place fixes are proven, and write that down before the first incident rather than during it.

If you can only afford one, the user-count rule applies. Under a few thousand daily users, start with Braintrust and a dataset of fifty real failures. Above that, production signals start to see failures a small dataset cannot hold, and Experiments compare a shipped change against real traffic in a way no offline suite can.

Pricing and Buying

A worked example at 100,000 agent runs a month, with every assumption stated. Raindrop, if each run produces one billable event and every event is billed (the plans doc lists no included allowance on Pro): $299 + 100,000 × $0.003 = $599 a month. If each run produces five events, it is $299 + $1,500 = $1,799.

Braintrust, if each run is 40 KB of processed data (4 GB in total) and two scorers run on a 20% sample (40,000 scores): Pro covers it inside the allowance at $249 a month. Starter would be 3 GB over at $4 plus 30,000 scores over at $2.50 per 1,000, so $87 a month, with 14-day retention. The two meters measure different things - events against gigabytes and scores - so check your own event count and trace size before comparing the totals.