Agent evaluation infrastructure is the discipline of building a control plane (what to run, whether to ship) and a data plane (running agents and recording what happened) around agentic systems — treating evaluation as durable engineering, not a one-off benchmark score. The core claim: this infrastructure outlives any single model, harness, or runtime rewrite, because it's the layer that lets a team tell whether a change actually helped.
The Concept
Lee Hanchung (Director of Machine Learning, Moody's; author of Evaluation and Alignment) synthesized this framing in a June 13, 2026 post, "The Hidden Technical Debt of AI Systems: Agent Evaluation Infrastructure" — extending the Sculley et al. 2015 "hidden technical debt" argument from classic ML pipelines to agentic systems. His core distinction: single-turn chat evaluation is prompt → response → score; agent evaluation is not, because agents complete tasks by changing the environment, so scoring the final output alone is no longer sufficient.
The unit of work is the episode — a bounded attempt from initial state to terminal condition — and the recorded artifact is a trace, not an input-output pair: messages, tool calls, and their results.
Control Plane vs. Data Plane
| Plane | Contains |
|---|---|
| Control | Task suites, configurations, gates, dashboards, regression tracking — what to measure and whether to ship |
| Data | Model, harness, runtime, tools, memory, environment state, traces — how a run actually happened |
Keeping these separate is the load-bearing idea: if task definitions live inside the harness that executes them, every harness rewrite silently changes what "passing" means, and regressions become impossible to attribute to a specific cause. AI2's olmo-eval is cited as a working example — the same task definition runs as a plain baseline or wrapped in tools and scaffolding without changing what it measures.
Five Evaluation Surfaces
- Output — did the task actually complete (tests pass, spreadsheet reconciles)
- Trace — process-level: tool call sequences, retries, parameter drift, unsupported reasoning
- Memory — scratchpads, conversation context, skill updates; pollution can persist across episodes
- Environment — state deltas: file diffs, DB writes, calendar changes
- Mechanistic interpretability — internal activations (only available for non-API/open-weight systems)
The canonical failure mode the framework is built to catch: an agent calls the right tool, gets an empty result, and fabricates an answer anyway. The final output looks fine. Only the trace shows the reasoning was unsupported.
Practices It Prescribes
- Perturbation tests: hold the task fixed, vary the path — change turn limits, inject tool failures, add conflicting documentation — and check the agent still gets there.
- Ablation tests: remove one component (memory, semantic search, browser access) and measure the delta.
- Checkpoint/resume/replay/branch: resume continues an attempt from a checkpoint; replay re-executes a known trajectory; branch changes exactly one factor (model, tool availability, budget) from a shared checkpoint — this requires real state infrastructure (git worktrees or copy-on-write dirs for filesystems, branchable DB snapshots, browser profile/cookie/DOM recordings), not just prompt logs.
- Score process invariants, not golden paths: "no unsafe writes," "no jumping to conclusions" — properties that hold across many valid trajectories — rather than exact-match against one reference path.
- Report uncertainty, not just deltas: pair runs and report a confidence interval and minimum detectable effect alongside every number, to avoid mistaking noise for a regression or a win.
- Minimum viable trace record: run id, task id, step index, model + harness version, prompt/config hash, tool name, arguments hash, observation hash, latency, cost, permission boundary, state delta pointer, checkpoint id, verifier result, failure labels.
Why This Matters Now
2026 agent evaluation practice is visibly consolidating around this shape. Braintrust, Confident AI, and newer OTel-native tools like Laminar (rollout debuggers for grading multi-step trajectories) are converging on trace-based, production-loop-fed eval suites — one 2026 industry guide recommends ≥500 real traces before trusting an aggregate metric. That's the data-plane half of this framework showing up as product. The control-plane half — decoupling task definitions from harness so regressions are attributable — is less commonly implemented and is where most teams are still accumulating debt.
Why Assess, Not Trial
This is one practitioner's synthesis (June 2026), not a standardized methodology — there's no shared tooling equivalent to a pytest or a CI gate for "did the control/data plane separation hold." The individual pieces (OTel-style tracing, perturbation testing, LLM-as-judge trace scoring) are independently well-established and already show up in tools on this radar (Arize Phoenix, Braintrust, Langfuse, MLflow, AgentOps). What's new is naming the control/data-plane split as the organizing principle and arguing it deserves the same durability investment as CI. Track this to see whether the framing gets adopted broadly or whether it stays a useful mental model without dedicated tooling.
Relationship to Adjacent Entries
- Harness Engineering — Assess. Harness engineering asks what environment the agent runs in; evaluation infrastructure asks how you know whether that environment change helped. The data plane described here is effectively "the harness, instrumented for measurement."
- Arize Phoenix, Braintrust, Langfuse, MLflow, AgentOps — Trial/Assess. These are data-plane implementations: tracing, replay, and eval scoring. None of them enforce the control-plane discipline (decoupled task suites, gated regression tracking) this entry argues is the durable layer.
Key Characteristics
| Property | Value |
|---|---|
| Synthesized by | Lee Hanchung, Director of ML at Moody's; author, Evaluation and Alignment |
| Published | June 13, 2026 |
| Builds on | Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015) |
| Core unit | Episode (bounded attempt) → trace (messages + tool calls), not input-output pair |
| Named example | AI2 olmo-eval — task/harness decoupling |
| Type | Architectural practice / engineering discipline |
Sources
- Lee Hanchung — The Hidden Technical Debt of AI Systems: Agent Evaluation Infrastructure — June 13, 2026
- Lee Hanchung — About — author background, Director of ML at Moody's
- Confident AI — LLM Agent Evaluation Metrics in 2026 — corroborates trace-based eval consolidation industry-wide
- Braintrust — Agent Observability: The Complete Guide for 2026 — corroborates production-trace-to-eval-suite feedback loop