jev rag groundedness evaluation

Groundedness Is a Relationship, Not Answer Quality

A response is grounded when its claim is supported by the evidence it cites or was allowed to use. A grounded answer can still be irrelevant, incomplete or based on a poor source. An ungrounded answer may accidentally be true from outside knowledge. Keep factual correctness, citation support, retrieval relevance and task completion as separate measurements.

Ragas similarly distinguishes faithfulness from context precision and other RAG metrics. Its taxonomy is useful independent context, but Jev requires a bounded answer design rather than an open-ended critique. The RAG use-case page maps retrieval and generation stages.

Decompose the Answer Before Judging Support

Jev cannot generate an arbitrary list of claims, so claim extraction is a separate stage. If one sentence contains two propositions, split them when one could be supported and the other unsupported. Preserve offsets into the generated answer so a UI can highlight the affected text.

Use a Bounded Support Taxonomy

A binary “grounded?” question often forces missing or partial evidence into the same bucket as contradiction. A four-way Choice supports better diagnosis, but it still chooses among supplied options; evaluate an explicit no-fit path if the taxonomy is incomplete. Store probabilities for every relation.

{
  "state": {
    "claim": "The plan includes unlimited API calls.",
    "cited_passages": [
      {"id": "pricing-4", "text": "The plan includes 10,000 API calls per month."}
    ]
  },
  "questions": {
    "support": {
      "type": "choice",
      "instructions": "How does the cited evidence relate to the claim?",
      "criteria": {
        "supported": "The evidence establishes the full claim",
        "contradicted": "The evidence conflicts with the claim",
        "insufficient": "The evidence neither establishes nor conflicts with the full claim",
        "not_applicable": "The text is not a factual claim requiring this evidence"
      }
    }
  }
}

Do Not Leak Retrieval or Answer-Generation Mistakes into the Label

Each stage can fail independently. Feeding all retrieved context rather than the cited evidence can let a judge rescue a bad citation from an uncited passage. Decide whether the metric asks “supported anywhere in context” or “supported by this citation” and encode that difference in state.

Validate Against Adjudicated Claim-Evidence Pairs

Jev provides no rationale. For high-stakes review, link the verdict directly to the claim and passage and use a human or explanatory model when reasons are required. A rationale must itself be checked against evidence; fluent explanation is not proof.

• Include explicit support, contradiction, partial support, missing evidence and non-factual language.
• Label source authority separately from textual entailment.
• Measure per-class errors, calibration, coverage and aggregation behavior.
• Slice by claim type, passage length, retrieval rank, language and adversarial text.
• Compare with a deterministic citation check and an explanatory LLM judge on identical pairs.

Aggregate Without Hiding One Severe Unsupported Claim

A mean groundedness score can let many trivial supported claims cancel one consequential unsupported claim. Preserve claim-level results, weight or veto by impact explicitly, and report both claim coverage and answer-level failure. Missing citations should not be assigned a neutral score.

Use composite scoring only when tradeoffs are deliberate. For production session analysis, attach every result to its trace and citation evidence as described in agent-trajectory evaluation.