raindrop vs langsmith

What Each Product Actually Is

Raindrop, a YC W24 company, monitors AI products and agents in production. It traces runs, including over OpenTelemetry, with SDKs for the Claude Agent SDK, Vercel AI SDK, LangChain and CrewAI. The core is Signals: you describe a behavior in plain language, Raindrop drafts a classifier, runs it on sample events, and you label results to refine it. Issue detection groups "stumbles" into recurring issues ranked by severity, a Triage Agent works through them, and fixes can be handed to coding agents over MCP. Experiments compare a baseline and an experiment cohort of already-logged events, defined by model, feature flag, property, tool or date range; in Raindrop's words, "Experiments read existing traffic; they don't run or replay your agent."

LangSmith is the LangChain team's platform, pitched as "Know what your agents are really doing." It traces with native support for popular agent frameworks and OpenTelemetry, SDKs for Python, TypeScript, Go and Java, and message threading for multi-turn chats. It evaluates two ways: offline, running your application against datasets to produce experiments you compare; and online, running LLM-as-judge and code evaluators on production traces. Human review, pairwise evaluators, cost and P50/P99 latency dashboards, Insights that cluster traces and summarise errors, and webhook and PagerDuty alerts complete it. Its site says it "works with any LLM framework", including the OpenAI SDK, Anthropic SDK, Vercel AI SDK and LlamaIndex, not just LangChain.

Raindrop is opinionated about where the truth lives - its CTO wrote that "the truth increasingly lives in production" - and it builds for teams with a lot of users. LangSmith is a general platform covering the whole loop from dataset to production, with less of an opinion about which end matters more.

Side by Side

The rows that decide most evaluations are the third and fourth. Raindrop judges a shipped change by comparing cohorts of real production events and treats offline evaluation as a local tool; LangSmith treats datasets and experiments as a first-class workflow and tests on production only by scoring what already happened.

Where Raindrop Is Stronger

• Comparing changes on production data. Ship a model, prompt or tool change - behind a feature flag, say - and an Experiment compares the new cohort of logged events with a baseline. Small cohorts give noisy comparisons, so this is a strength for high-traffic agents more than for an internal tool with forty users.
• Signals in plain language. Describe the behavior, get a classifier, label a few results to sharpen it. Yes/no classifiers are easy to count and alert on, at the cost of the nuance a graded score carries.
• Triage built in. Stumbles, severity-ranked issues, a Triage Agent, and fixes handed to coding agents over MCP. Raindrop is built for the moment after detection, not only the detection.
• Proof in its category. Homepage logos include Speak, Vercel, Clay, Framer and AngelList, and a published Tolan case study reports 27.8% fewer memory issues.
• Workshop. An MIT-licensed local agent debugger and eval runner, launched in May 2026, with about 1,070 GitHub stars.

Where LangSmith Is Stronger

• Both halves of evaluation. Offline experiments on datasets before you ship, online evaluators on production traces after, and failing production cases fed back into the dataset. You can test a change before a user sees it.
• Evaluator variety. LLM-as-judge, code, human review and pairwise evaluators, where Raindrop Signals are yes/no classifiers.
• Alerts where on-call lives. Webhook and PagerDuty alerts. Raindrop alerts go to Slack only: threshold alerts, a daily digest and scheduled Agent Briefs.
• A lower entry price. A free developer seat with up to 5,000 base traces a month, and Plus at $39 per seat with 10,000 base traces, against Raindrop Pro at $299 a month before events.
• SDKs and ecosystem. Python, TypeScript, Go and Java SDKs, and first-party support from the team behind LangChain and LangGraph, while still tracing apps that use neither.
• A documented self-host path. Self-hosted LangSmith is an Enterprise add-on on Kubernetes with PostgreSQL, ClickHouse, Redis or Valkey, and blob storage. Raindrop's self-hosting and VPC option is a beta for "a select group of initial partners".

One Silent Failure, Both Tools

A consumer tutoring agent starts declining hard questions after a model upgrade: "I can't help with that" shows up far more often. Nothing errors, latency is fine, and the dashboards look normal.

1. Raindrop: signal, issue, experiment

   A Signal written in plain language - "the assistant declines a question it should be able to answer" - becomes a classifier; you label a few dozen hits to sharpen it. Issue detection surfaces the rise as a recurring issue, and a Slack alert or the daily digest tells the team. An Experiment with events from the old model as the baseline cohort and the new model as the experiment cohort shows whether the refusals track the upgrade.

2. LangSmith: evaluator, alert, dataset

   An online LLM-as-judge evaluator on production traces flags unwarranted refusals; a webhook or PagerDuty alert fires past a threshold, and Insights clusters the affected traces. Failing traces go into a dataset, and an offline experiment compares the old and new model on it, pairwise if you like, before anything changes in production.

3. Where they end up

   Raindrop answers "did the upgrade cause this, for real users" by comparing cohorts of logged production events. LangSmith answers it with a dataset and an experiment. Both find out after users have met the failure; the difference is how you prove the fix.

Scale decides which proof you can afford. With a lot of traffic, a cohort comparison gives a clean answer quickly. With a few hundred users, the cohorts are small and noisy, and a dataset of the failing conversations is the faster route to confidence.

What to Check on a Trial

• Your event and trace counts. Instrument one real agent for a day and count Raindrop events and LangSmith traces per session. Both bills scale with that number, and neither vendor can tell you what it is.
• Where alerts land. If on-call runs on a paging tool, test how a Raindrop Slack alert would reach it before you decide it does not matter.
• How much traffic you really have. Raindrop Experiments compare cohorts of logged events, and small cohorts give noisy comparisons. If your agent has few users, plan on offline comparisons instead, which is LangSmith's home ground.
• Retention you need. LangSmith base traces are kept 14 days. Decide which traces you need for longer and price the extended upgrade before signing.
• Framework fit. Both trace LangChain. Raindrop lists SDKs for the Claude Agent SDK, Vercel AI SDK and CrewAI; LangSmith lists native tracing for popular frameworks plus OpenTelemetry. Check your exact stack and version.
• Self-host and residency. Raindrop self-hosting is a partner beta; LangSmith self-hosting is an Enterprise add-on. If either is a requirement, it is a sales conversation on both sides.

Run each trial long enough to see at least one real failure, and write down beforehand what you expect each product to show you. A Signal or an evaluator tuned during the trial to catch the failure you already know about proves less than one that finds something you did not.

Pricing and Buying

Raindrop, assumptions stated: at 100,000 events a month, with every event billed at $0.003 because the plans doc lists no included allowance on Pro, the bill is $299 + $300 = $599 a month. At 1,000,000 events it is $299 + $3,000 = $3,299. How many events one agent run produces depends on your instrumentation, so count yours before using either figure.

LangSmith for a five-person team on Plus is 5 × $39 = $195 a month, with up to 10,000 base traces. Above that, traces are pay-as-you-go; the pricing page lists storage and compute units (LSU at $1.00, LCU at $1.50) rather than a per-trace rate, so we could not turn a trace volume into a dollar figure from the public page. Base traces are kept 14 days; extended retention costs more, and LangSmith's pricing page and billing docs give different periods for it (400 and 180 days), so ask which applies.