How to evaluate a RAG agent for groundedness

1. Capture Retrieval Evidence in the Session Trace

Instrument the RAG agent so each Failproof session contains the user question, every retrieval call, the returned chunks or documents, and the final response. Preserve document IDs and useful chunk metadata. Groundedness cannot be evaluated reliably if the answer and its evidence live in separate systems.

Include evidence from every retrieval step in the session, not only the final search. An agent may search several times, read a file and call an API before answering. Its own earlier statements are not evidence, even when the final response repeats them.

2. Create Separate Evaluations for Each Failure

Use a code-based evaluation for exact citation and quote checks. Use an LLM-based evaluation for claim support, relevance and completeness. Tell the groundedness evaluation to use only the retrieved evidence, even when it knows a claim is true from its own training.

Keep these as separate results. One blended quality score cannot tell the team whether to repair retrieval, generation, citation formatting or source coverage. Failproof AI runs the evaluations in the cloud and can also run your existing evaluation suite.

3. Test and Deploy the Evaluations

Test each evaluation against real sessions before deployment. Include a grounded answer, an unsupported claim, a wrong-document retrieval and a question the index cannot answer. Each evaluation should move only for the failure it is intended to measure.

Deploy an immutable evaluation version and keep the key stable. Re-evaluate individual sessions while refining the criterion, then backfill the accepted version across recent RAG sessions to establish the baseline.

fp evals --since 24h --aggregate
fp evals --agent-id support-rag --aggregate
fp --json evals --aggregate --env production

4. Run Audits to Find Recurring RAG Failure Patterns

Create a Failproof audit over the RAG agent with a goal such as "find recurring causes of unsupported answers" or "find topics where retrieval returns no useful evidence". The audit analyzes the failed evaluations together with the underlying retrieval calls and answer traces.

Related failures become findings with affected sessions, severity, evidence queries and a recommended fix. Common patterns include one stale document cited repeatedly, a missing topic in the index, a query rewrite that selects the wrong source, and a prompt that encourages the model to answer when retrieval returns nothing.

fp audits list
fp audits findings --status open --limit 20

5. Query and Dashboard Groundedness Over Time

Use Analyze → Queries to break results down by agent, environment, document, topic or deployment window. Save queries for groundedness failures, empty retrievals, stale-source findings and sessions where relevance is low but the answer still claims completion.

Build a dashboard from those saved queries. Keep groundedness, relevance and completeness side by side. A refusal can be perfectly grounded and useless, while a complete answer can be unsupported. The dashboard should preserve that distinction.

fp agent ask "Which production RAG sessions failed groundedness in the last 7 days? Group the affected sessions by retrieved document and agent version."
fp query run <saved-query-id>

6. Fix the RAG Failure and Verify It

Assign the finding to the owner of the retrieval or answer path. Fix the stale source, missing index coverage, chunking, query rewrite, prompt or model behavior identified by the trace evidence. Keep the evaluation versions unchanged while verifying the product change.

Alert on important groundedness, relevance or completeness changes through email, Slack, webhook or the dashboard. Continue the audit so a new variant does not hide behind a recovered average.

Failproof AI has already built the trace storage, cloud evaluation execution, versioning, re-evaluation, backfill, recurring-failure analysis, native queries, dashboards, alerts and issue workflow. Your team defines what groundedness means for the product and repairs the RAG system. You do not need to build a separate scoring and investigation service around it.