Migrating agent evals between platforms

What Moves, and What Does Not

Raw data and code are usually portable. Product state is not. A score sits between the two because your evaluator produced it from a transcript rendered by the platform. Recalibrate even when the rubric and judge model stay the same.

OpenTelemetry Is the Portable Layer, with Caveats

The one decision that makes a future migration cheap is instrumenting with OpenTelemetry. Much of the category accepts it: Langfuse receives OTLP at /api/public/otel, LangSmith at https://api.smith.langchain.com/otel, Braintrust at https://api.braintrust.dev/otel, and Judgment Labs, Raindrop, Latitude, Future AGI and Arize Phoenix all support OpenTelemetry for tracing. Moving live traffic is then a change of endpoint and credentials, not a re-instrumentation.

The caveat is the data model. The OpenTelemetry GenAI semantic conventions now live in their own repository and, in Langfuse's words, "are still evolving". Each vendor maps incoming spans to its own model - Braintrust maps gen_ai.* attributes onto its input, output and metrics fields, and LangSmith publishes its own mapping table - so the same span can look different in each. Check that the fields your judges read survive the trip before you cut over.

During the overlap, send to both. An OpenTelemetry Collector pipeline can list more than one exporter, so one instrumented agent feeds the old platform and the new one at once:

# otel-collector.yaml - one pipeline, two backends during the overlap
receivers:
  otlp:
    protocols:
      http:
exporters:
  otlphttp/old:
    endpoint: https://old-vendor.example.com/otel   # plus that vendor's auth headers
  otlphttp/new:
    endpoint: https://new-vendor.example.com/otel   # plus that vendor's auth headers
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/old, otlphttp/new]

Keep Judges as Code You Own

A judge configured in a vendor's UI - a template, a prompt box, a model dropdown - is the hardest thing to move, because it is half prompt and half platform behavior: how the transcript is rendered, where it is truncated, how the reply is parsed. A judge written as a plain function that takes a transcript and returns a verdict moves with a small adapter. Split it that way from the start:

# evals/judges.py - no vendor imports; this file moves between platforms unchanged
import json
import os

import anthropic

JUDGE_MODEL = os.environ["JUDGE_MODEL"]
client = anthropic.Anthropic()                    # reads ANTHROPIC_API_KEY


def judge_task_completion(transcript: list[dict]) -> tuple[bool, str]:
    prompt = (
        "Did the agent complete the task it was given? "
        'Reply with JSON only: {"pass": true or false, "reason": "<one sentence>"}\n\n'
        + "\n".join(json.dumps(e, default=str) for e in transcript)
    )
    msg = client.messages.create(
        model=JUDGE_MODEL, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    text = msg.content[0].text
    try:
        verdict = json.loads(text)
        return verdict.get("pass") is True, str(verdict.get("reason", ""))
    except (json.JSONDecodeError, AttributeError):
        return False, f"unparsable judge output: {text[:200]}"

Where a platform cannot take this file as it is, it gets a few lines of glue: a scorer in that vendor's SDK, or a CI script. Decide per platform whether an unparsable reply counts as a fail or as no score. Keep the human labels next to this file, keyed by your own session ids, so calibration can be recomputed anywhere.

Migrating to or from Failproof AI

Failproof AI ingests sessions through hooks in supported agent harnesses or through its Python SDK and framework adapters. OTLP ingest is not documented, so teams with an existing OpenTelemetry pipeline should confirm compatibility before planning the migration. Recent session history can be backfilled within the allowance of the selected tier.

• Evaluations: keep evaluator logic in code where possible. Existing evaluation suites can run in Failproof AI Cloud without being converted into a proprietary judge format.
• Historical sessions: use backfill for recent local history, then verify the imported events and metadata against the source.
• Exports: the public /v1 API and Cloud CLI return session and evaluation data as JSON. Confirm how findings, comments and audit history will be exported if those records matter to the migration.
• Parallel operation: Failproof policies can run alongside another tracing or evaluation product, so a team can add failure analysis and tested runtime fixes without an immediate full cutover.

The migration test should cover more than score parity. Confirm that automated failure analysis finds the same known patterns, that findings link to complete evidence, that alerts reach the correct owners and that a previously fixed failure is recognized if it returns. Those workflow checks expose migration gaps that a trace-count comparison will miss.

A Migration Plan

1. Inventory

   List every agent, judge, dataset, dashboard and alert on the old platform, with an owner for each. Anything nobody claims is a candidate for not migrating.

2. Export history

   Pull traces, scores, datasets and labels out while the old contract is live, and check which exports your tier includes. LangSmith, for example, limits bulk export to Enterprise for customers who signed up after August 3, 2026; Langfuse offers API, UI and scheduled blob-storage exports.

3. Send to both

   Fan traffic out to both platforms for a few weeks, so the new one builds its own history on the same sessions.

4. Recalibrate

   Run your judges on the new platform over your labeled sessions and recompute agreement. The rubric has not changed; the transcript the judge sees may have.

5. Rebuild alerts

   Set thresholds from the new platform's score series, not the old one's numbers, and check each alert fires on a known-bad session.

6. Cut over, keep read access

   Switch the old exporter off, but keep read access or the export until its retention would have run out, for investigations that reach back.

When Not to Migrate at All

Do not migrate solely because one capability is missing. A team may keep its current tracing and evaluation system while adding failure analysis, a different alerting layer or runtime policies beside it. Migrate the core platform when duplicated instrumentation, fragmented investigations or contractual requirements cost more than the cutover. If you are moving to Latitude, it publishes migration guides from Langfuse, LangSmith and Braintrust.