How to build an eval dataset from production traces

What a Production Eval Dataset Is For

An eval dataset is a fixed set of inputs, each with a label or an expected outcome, that you replay against a new prompt, model or tool change before it ships. Its value comes from being fixed: the same cases, scored the same way, give numbers you can compare across versions. Production traces are the right raw material because they carry the real distribution of requests and the real ways your agent fails, which no brainstorm reproduces.

Datasets built from traces go wrong in predictable ways. They fill up with happy paths, because those are most of the traffic, so the agent aces the suite and still fails in production. They leak customer data into a repository that far more people can read than the production database. Their labels drift as different people apply different standards. And they change silently - someone fixes a label or adds ten cases - so this month's 91% and last month's 88% were measured on different tests.

Choosing Sessions: Three Buckets

• Failures. Sessions a judge scored low, sessions with tool errors, sessions that escalated to a human, and sessions behind a support ticket or a thumbs-down. These are the cases that justify the dataset. Each one you add is a regression test for a failure you have already paid for once.
• Edge cases. Low-volume agents, rare tools, unusually long sessions, unusual languages or input formats, and requests near a policy boundary, such as a refund just over the limit. Production has them; your happy-path tests do not.
• A random slice. A plain random sample of ordinary sessions. Without it, a dataset of failures rewards an agent that refuses everything, and you cannot tell whether a change that fixes the hard cases broke the easy ones.

For a first version, 100-300 carefully reviewed cases is often more useful than a much larger weakly labeled set. A roughly even split across the three buckets can be a starting point, but tune the mix to the decisions the suite must support. Report each bucket separately as well as overall. Cluster near-duplicate failures before sampling so one incident does not dominate the suite, while retaining production prevalence separately if you also want an estimate of real-world failure rates.

Capturing a Session Without Leaking Data

Store what a replay needs: the user input, event trajectory, outcome and relevant metadata such as agent, environment, date and model version. Scrub the exported dataset copy before writing it to a repository, shared bucket or labeling system; a later clean-up pass rarely reaches every duplicate. Pattern-based scrubbing catches obvious identifiers and secrets but not every name or free-text detail, so treat it as a floor and have a person review cases before they enter a frozen version.

# capture.py - write a scrubbed copy of one session to the dataset inbox
import hashlib
import json
import os
import re
from pathlib import Path

INBOX = Path(os.environ.get("DATASET_INBOX", "datasets/inbox"))
SCRUB = [                                              # order matters: cards before phones
    (re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"), "<email>"),
    (re.compile(r"\b(?:\d[ -]?){13,19}\b"), "<card-number>"),
    (re.compile(r"\+?\d{1,3}[\s.-]?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}\b"), "<phone>"),
    (re.compile(r"(?i)bearer\s+[a-z0-9._~+/-]+=*"), "Bearer <token>"),
    (re.compile(r"\b(?:sk|pk|rk)[-_][A-Za-z0-9_-]{16,}\b"), "<api-key>"),
]

def scrub(text: str) -> str:
    for pattern, replacement in SCRUB:
        text = pattern.sub(replacement, text)
    return text

def capture(session: dict, reason: str) -> Path:
    """session: session_id, agent_id, environment, events, and any scores you have."""
    record = {**session, "capture_reason": reason}
    text = scrub(json.dumps(record, default=str))
    name = hashlib.sha256(session["session_id"].encode()).hexdigest()[:16]
    INBOX.mkdir(parents=True, exist_ok=True)
    path = INBOX / f"case-{name}.json"
    path.write_text(text)                               # same file on a retry
    return path

The replacements are plain ASCII with no quotes, so the scrubbed JSON stays valid. The file name is a hash of the session id, so capturing the same session twice overwrites rather than duplicates, and the raw id never appears in a path.

Labelling the Cases

A label is only as good as the rubric behind it. Write the criterion down before anyone labels - "pass if the agent completed the task or clearly said it could not" - with one passing and one failing example. Record on each case the label, who gave it, the rubric version and a one-line note on why. The note is what lets a later reader tell a wrong label from a hard case.

Separate two things that are easy to conflate. A label says whether the recorded session was good. A replay needs an expected outcome that a new run can be checked against, because the new agent will produce a different transcript and the old label says nothing about it. "Refund denied, customer offered a human agent" is an expected outcome. "Fail" is only a label on last month's transcript. Record both where you can: the label calibrates your judge on real transcripts, and the expected outcome is what the regression suite scores the next version against. For open-ended tasks with no single right answer, the expected outcome becomes a short list of properties the result must have and must not have.

Have two people label the same 30-50 cases independently and compare. Disagreements may expose rubric ambiguity, missing evidence or a genuine label error; classify the cause before changing the rubric. The same labeled cases can then calibrate an LLM judge against human labels, so one careful labeling pass supports both regression testing and judge validation.

Freezing and Versioning

Freeze reviewed cases into a numbered version with a manifest of content hashes. A version is never edited. A wrong label or a new case means a new version, with a line in a changelog, and a score is always reported with the version it was measured on.

# freeze.py - turn reviewed cases into an immutable, hashed dataset version
import hashlib
import json
import sys
from datetime import date
from pathlib import Path

def freeze(reviewed: Path, version: str) -> Path:
    out = Path("datasets") / version
    if out.exists():
        sys.exit(f"{out} already exists: versions are immutable, pick a new one")
    out.mkdir(parents=True)
    files = []
    for case in sorted(reviewed.glob("case-*.json")):
        if "label" not in json.loads(case.read_text()):
            sys.exit(f"{case.name} has no label; label it or leave it out")
        data = case.read_bytes()
        (out / case.name).write_bytes(data)
        files.append({"file": case.name, "sha256": hashlib.sha256(data).hexdigest()})
    manifest = {"version": version, "frozen_on": date.today().isoformat(),
                "cases": len(files), "files": files}
    (out / "manifest.json").write_text(json.dumps(manifest, indent=2))
    return out

if __name__ == "__main__":
    print(freeze(Path(sys.argv[1]), sys.argv[2]))       # python freeze.py datasets/reviewed v3

Keep a held-out split that you never look at while tuning prompts, and score it only before a release. Otherwise the prompt slowly fits the dataset, and the dataset stops predicting production. Retire cases deliberately, in a new version, when the product changes so much that they no longer describe a real request.

Pitfalls, and How to Check Your Work

• Check the mix. Count cases per agent, bucket and month. A dataset that is 80% one agent measures that agent.
• Check for leaks. Grep every new version for @, long digit runs and key prefixes before it merges. The scrubber missed something the first time; it always does.
• Check the labels age well. When the product changes what "correct" means, relabel in a new version rather than letting old labels fail a correct agent.
• Check replays are possible. A case that depends on yesterday's database state or a live third-party API cannot be replayed faithfully. Record the tool results it saw, or stub the tools in the replay.
• Check the failures are still failures. Once a fix ships, the case that exposed the bug should pass. If it keeps failing, either the fix did not work or the expected outcome is wrong, and both are worth knowing before the next release.
• Check the size is honest. Fifty cases put a 90% pass rate within about ±8 points. That is enough to catch a collapse, not a few points of drift.

Building the Dataset with Failproof AI Traces

Failproof AI stores session traces, evaluation results and failure findings that can help select cases for each bucket. Use failed evaluations and findings to collect known problems, query metadata for edge cases, and add a random production sample so the dataset does not become a catalog of failures only. Export the selected sessions, scrub the dataset copies and place them in the review inbox:

# pull.py - export chosen sessions from Failproof AI Cloud, scrubbed before writing
import hashlib
import os
import sys
import urllib.request
from pathlib import Path

from capture import INBOX, scrub

def pull(session_id: str) -> Path:
    req = urllib.request.Request(
        f"https://app.befailproof.ai/v1/sessions/{session_id}/export",
        headers={"Authorization": f"Bearer {os.environ['FAILPROOFAI_KEY']}"},
    )
    raw = urllib.request.urlopen(req).read().decode("utf-8", errors="replace")
    name = hashlib.sha256(session_id.encode()).hexdigest()[:16]
    INBOX.mkdir(parents=True, exist_ok=True)
    path = INBOX / f"case-{name}.json"
    path.write_text(scrub(raw))                       # same file on a re-run
    return path

if __name__ == "__main__":
    for session_id in sys.argv[1:]:                   # python pull.py <session-id> ...
        print(pull(session_id))

During review, add the human label, expected outcome, rubric version and selection bucket to each exported case. Evaluation results surface known failures; automated failure analysis groups related sessions into findings, which helps avoid filling the dataset with dozens of copies of the same problem. Queries and random sampling provide the ordinary and edge-case traffic needed for balance.

Failproof AI provides the traces and exports but does not currently document a managed dataset and replay workspace. Keep frozen versions in your repository or controlled storage and run replays in your own CI. If managed datasets are the primary requirement, Judgment Labs documents dataset workflows connected to behavior monitoring, while Latitude supports building datasets from traces for replay and regression testing.

When You Do Not Need a Dataset Yet

If the agent is days old and has no real traffic, start with a small set written from the specification and add production cases as evidence arrives. Runtime checks can prevent known actions, but they do not test broader outcomes or reveal new failure modes, so keep regression cases for important behavior. If the agent changes weekly in ways that invalidate expected outcomes, keep the frozen set small and focused rather than building a large suite you cannot maintain.