jev evals in deepeval

Choose the Fields Before the Questions

DeepEval’s JevEval docs turn selected test-case fields into Jev state. You supply typed questions, option credits and weights. The wrapper calculates the metric score. ConversationalJevEval does this for a conversation. DeepEval also describes using Jev inside other metrics in an earlier integration post, which is a separate path.

A question about tool-result faithfulness needs the actual tool result in state. If you only pass the final answer, Jev cannot check its source. The trace-state guide shows how to select that evidence without dumping an entire unbounded run into one request.

A Bounded Tool-Result Faithfulness Metric

The imports and field names follow DeepEval’s documentation. Check them against your installed package. With threshold=None, you can collect scores and labels before choosing a gate. This example leaves claim splitting and tool-result selection to the caller. For a long answer, use the groundedness guide to test individual claim-evidence pairs.

from deepeval.metrics import JevEval
from deepeval.metrics.jev_eval import Noul, Score
from deepeval.test_case import SingleTurnParams

tool_faithfulness = JevEval(
    name="Tool faithfulness",
    evaluation_params=[
        SingleTurnParams.ACTUAL_OUTPUT,
        SingleTurnParams.TOOLS_CALLED,
    ],
    questions=[
        Noul("Every factual claim in actual_output is supported by tools_called."),
        Score("How much of actual_output is grounded in tools_called?", levels=[
            "Fabricated", "Mostly fabricated", "Mostly grounded", "Fully grounded"
        ]),
    ],
    threshold=None,
)

Read the Metric Score Without Changing Its Meaning

For a Choice answer marked with None credit, DeepEval drops the question when those answers take at least half the probability. Look at score_breakdown when the aggregate rises. The metric may have skipped a difficult question rather than seen a better agent run.

DeepEval builds its text reason from the decisions in code. It is a readable record of the metric’s own outputs, with no new citation or Jev-generated explanation. Check the judgments against labels using evaluator validation.

Choose Score-Only, Thresholded or Strict Use Deliberately

The default 0.5 threshold ships with the software. It has not been fitted to your false-pass costs. Choose a threshold on a calibration split, then report false passes, false failures and review load on untouched cases. The threshold guide covers that step.

Where JevEval Fits Among Eval Workflows

JevEval works when you can name the bounded questions and accept a weighted score as the result. A task that needs open-ended critique or evidence citations calls for a generative judge. The platform comparison covers what DeepEval, LangSmith, jevals and Failproof store around those judgments.

To replace a judge, replay the same blind cases through both and save the native outputs. Compare wrong calls at a chosen review rate, failed calls, latency and total cost. Check calibration where each result supplies a meaningful probability.