judgeval vs deepeval

What Each Library Actually Is

judgeval is the open-source Python SDK of Judgment Labs. Its README describes four things. OpenTelemetry-based tracing, where @Tracer.observe() "automatically captures inputs, outputs, and LLM token usage". Agent judges: "prompt-based scorers to evaluate agent behaviors at scale" that produce structured behaviors, meaning scored, labelled outputs. Online monitoring that scores "live production traffic server-side with no latency impact". And auto-instrumentation for OpenAI, Anthropic, Google GenAI and Together AI, with framework support for LangGraph, OpenLit and the Claude Agent SDK. You configure a JUDGMENT_API_KEY and JUDGMENT_ORG_ID, and the data goes to the Judgment platform for storage and analysis.

So the library is the client and the judging happens on the platform. Judgment Labs' docs describe two judge types there - Agent Judge, which follows a natural-language rubric, and Code Judge, which runs your Python - usable online, offline against a dataset, or on demand. The platform is hosted and not open source, and its docs list self-hosting as coming soon. The README does not list a catalogue of prebuilt metrics.

DeepEval works like Pytest for LLM apps: you write test cases and assert on metric scores. It ships ready-to-use metrics: agentic ones (Task Completion, Tool Correctness, Goal Accuracy, Step Efficiency, Plan Adherence, Argument Correctness), RAG ones (Answer Relevancy, Faithfulness, Contextual Recall, Precision and Relevancy), multi-turn ones (Knowledge Retention, Conversation Completeness, Role Adherence and more), MCP and multimodal metrics, general checks such as Hallucination, Bias, Toxicity and JSON Correctness, and G-Eval and DAG for custom criteria. Its docs say almost all predefined metrics use LLM-as-a-judge, and every metric returns a score between 0 and 1 with its reasoning. Confident AI, from the same team, is the optional hosted platform for reports and production monitoring.

Side by Side

The shape of the table is the shape of the decision. DeepEval is a complete tool on its own, and a platform is something you can add later. judgeval is half of a product; the other half is a hosted service you buy from Judgment Labs, which publishes no pricing as of September 2026.

Where DeepEval Is Stronger

• It works with nothing else. Per its tracing docs, spans, test cases on spans, metadata and token costs "run entirely locally"; data leaves your machine only if you log in to Confident AI.
• Metrics on day one. Tool Correctness, Task Completion, Faithfulness and the rest are imported, not written. A team without an evaluation specialist gets a first score the day it installs the package.
• Any judge model. The docs say "You can use ANY LLM judge", naming OpenAI, Azure OpenAI, Ollama, Anthropic, Gemini and LiteLLM, or your own model wrapped in DeepEvalBaseLLM. A local model through Ollama keeps judge calls on your own hardware.
• CI by default. deepeval test run is a native pytest integration, so an eval suite fails a build the way a unit test does.
• Component-level evals. Test cases attach to individual spans - an agent, a tool, a retriever - and are scored separately, without restructuring the app.
• Adoption. Roughly seventeen times judgeval's GitHub stars, and listed integrations for LangChain, LangGraph, Pydantic AI, CrewAI, OpenAI Agents, Google ADK, LlamaIndex, Mastra and others.

Where judgeval Is Stronger

judgeval's strengths are the platform's strengths, reached through the SDK. Both libraries need a hosted service for continuous production scoring - Judgment for judgeval, Confident AI for DeepEval - so the fair comparison is between those services' approaches.

• Judging built around agent behavior. Judges produce labelled behaviors rather than only a column of numbers, and the platform tracks detection rates, alerts, webhooks and dataset actions on them.
• Rubric judges for long trajectories. Agent Judge reasons over a whole trace against a natural-language rubric, the problem Judgment Labs wrote about in its May 2026 post on long-context evaluation.
• Discovery. Behavior Discovery and AutoRubrics look for failure patterns you have not named, where a metric library measures the things someone already thought to measure.
• SDKs beyond Python. judgeval-js, judgeval-go and judgeval-java sit beside the Python library, for teams whose agents are not all Python.
• Server-side, no latency. Online monitoring runs on Judgment's side, so production scoring adds no work to your request path.

One RAG Support Agent, Both Libraries

A support agent answers billing questions from a retrieved policy document and can call a lookup_invoice tool. You want to know whether it answers from the policy, calls the tool when it should, and gets worse after changes.

1. DeepEval: tests first

   Write test cases from real tickets - the input, the agent's answer, the retrieved context - and attach Faithfulness, Answer Relevancy and Tool Correctness with thresholds. Run deepeval test run in CI. Decorate the retriever and the tool call with @observe to score them separately. Nothing leaves the machine unless you log in to Confident AI for reports and production traces.

2. judgeval: platform first

   Instrument the agent with @Tracer.observe() and the provider auto-instrumentation, set the API key and org ID, and traces flow to Judgment. Write an Agent Judge rubric for "answers only from the retrieved policy" and a Code Judge for the invoice lookup; run them offline against a dataset, then online on sampled production traffic, alerting on the detection rate.

3. What you have after a week

   With DeepEval: a CI gate on standard metrics, and production monitoring only if you adopt Confident AI. With judgeval: production judging and behavior tracking, and a commercial relationship with Judgment Labs, since the platform is sold by demo.

In both cases the metric or the rubric is the hard part. A Faithfulness score is only as good as the context you hand it, and a rubric is only as good as the people who agreed on what it means.

Cost and Lock-in

Both libraries are free; the costs sit elsewhere. DeepEval's LLM-as-a-judge metrics call a model you choose, so the judge bill is yours and visible on your provider invoice. Confident AI, if you add it, is a separate product with its own pricing, not covered here. judgeval's cost is the Judgment platform, which publishes no prices as of September 2026.

Lock-in follows the same line. DeepEval test cases and metrics are Python files in your repository, and you can keep them if you drop the platform. judgeval tracing is OpenTelemetry-based, so the traces are portable, but the judges, behaviors and monitoring rules live on the Judgment platform and do not travel with the SDK.

Neither README describes stopping an agent action before it executes. judgeval's online monitoring scores traffic server-side after the fact, and DeepEval's metrics score test cases and spans that have already run. Both are measurement tools, and that is the right scope for a library: the thing that blocks an action has to live in the agent or its harness, not in the evaluation code.

Both also leave the hardest decision to you: what to measure. DeepEval's ready-made metrics make it easy to measure something on day one, which is not the same as measuring the right thing. A Tool Correctness score tells you whether the agent called the tools you expected; it does not tell you whether the refund it issued was allowed. That question needs a criterion written for your domain - a G-Eval or DAG metric in DeepEval, an Agent Judge rubric in Judgment - and a few dozen human labels to check the judge agrees with the people who own the policy.

A practical order for a small team: start with two or three ready-made metrics and one custom criterion, run them in CI on fifty real cases, and only then decide whether you need a hosted platform for production. If the answer is yes, the platform choice matters more than the library choice.

A fair middle path exists for teams that like both ideas: DeepEval in CI for the known cases, and a production judge - on Judgment, Confident AI or a platform of your own - for the unknown ones. What matters is that someone owns the rubric in both places.