Agent evaluation infrastructure is the discipline of building a control plane (what to run, whether to ship) and a data plane (running agents and recording what happened) around agentic systems — treating evaluation as durable engineering, not a one-off benchmark score. The core claim: this infrastructure outlives any single model, harness, or runtime rewrite, because it's the layer that lets a team tell whether a change actually helped.
The Concept
Lee Hanchung (Director of Machine Learning, Moody's; author of Evaluation and Alignment) synthesized this framing in a June 13, 2026 post, "The Hidden Technical Debt of AI Systems: Agent Evaluation Infrastructure" — extending the Sculley et al. 2015 "hidden technical debt" argument from classic ML pipelines to agentic systems. His core distinction: single-turn chat evaluation is prompt → response → score; agent evaluation is not, because agents complete tasks by changing the environment, so scoring the final output alone is no longer sufficient.
The unit of work is the episode — a bounded attempt from initial state to terminal condition — and the recorded artifact is a trace, not an input-output pair: messages, tool calls, and their results.
Control Plane vs. Data Plane
| Plane | Contains |
|---|---|
| Control | Task suites, configurations, gates, dashboards, regression tracking — what to measure and whether to ship |
| Data | Model, harness, runtime, tools, memory, environment state, traces — how a run actually happened |
Keeping these separate is the load-bearing idea: if task definitions live inside the harness that executes them, every harness rewrite silently changes what "passing" means, and regressions become impossible to attribute to a specific cause. AI2's olmo-eval is cited as a working example — the same task definition runs as a plain baseline or wrapped in tools and scaffolding without changing what it measures.
Five Evaluation Surfaces
- Output — did the task actually complete (tests pass, spreadsheet reconciles)
- Trace — process-level: tool call sequences, retries, parameter drift, unsupported reasoning
- Memory — scratchpads, conversation context, skill updates; pollution can persist across episodes
- Environment — state deltas: file diffs, DB writes, calendar changes
- Mechanistic interpretability — internal activations (only available for non-API/open-weight systems)
The canonical failure mode the framework is built to catch: an agent calls the right tool, gets an empty result, and fabricates an answer anyway. The final output looks fine. Only the trace shows the reasoning was unsupported.
Practices It Prescribes
- Perturbation tests: hold the task fixed, vary the path — change turn limits, inject tool failures, add conflicting documentation — and check the agent still gets there.
- Ablation tests: remove one component (memory, semantic search, browser access) and measure the delta.
- Checkpoint/resume/replay/branch: resume continues an attempt from a checkpoint; replay re-executes a known trajectory; branch changes exactly one factor (model, tool availability, budget) from a shared checkpoint — this requires real state infrastructure (git worktrees or copy-on-write dirs for filesystems, branchable DB snapshots, browser profile/cookie/DOM recordings), not just prompt logs.
- Score process invariants, not golden paths: "no unsafe writes," "no jumping to conclusions" — properties that hold across many valid trajectories — rather than exact-match against one reference path.
- Report uncertainty, not just deltas: pair runs and report a confidence interval and minimum detectable effect alongside every number, to avoid mistaking noise for a regression or a win.
- Minimum viable trace record: run id, task id, step index, model + harness version, prompt/config hash, tool name, arguments hash, observation hash, latency, cost, permission boundary, state delta pointer, checkpoint id, verifier result, failure labels.
August 2026 Implementation Evidence: A2E
A2E (Agent Auditing Engine) is an early public implementation of much of this control/data-plane split. Its Agent Task Protocol decouples benchmark tasks from harness adapters, while an OpenTelemetry/OpenInference monitor records model calls, tool calls, state transitions, errors, latency, token use, and artifacts. A centralized evaluation layer then scores both outcomes and process dimensions. The authors report composing 23 benchmarks with nine harnesses without per-pair integration code, and finding that model–harness combinations occupy different success/efficiency frontiers across tasks rather than producing one universal winner. (paper, repository)
This is useful evidence that harness-native, lifecycle-aligned evaluation is becoming executable rather than merely conceptual. It is still too early to promote the technique: the paper is a new preprint, the repository was created in August 2026, no independent replications were found during this scan, and the GitHub repository does not currently declare a detected license. More importantly, each harness–benchmark cell in the headline matrix contains only five sampled tasks, giving correctness scores a coarse 0.20 resolution and high per-cell variance; the authors explicitly say the table is not intended to rank frameworks. Treat A2E as a reference implementation to inspect or reproduce, not yet as a drop-in organizational standard.
August 2026 Boundary Evidence: QuoteBench
QuoteBench isolates a failure that endpoint scores usually hide: a model can generate a correct shell command while a harness's serialization, interpolation, or re-parsing path corrupts it before execution. The released benchmark freezes 56 incident-derived tasks, validates final program state, and publishes 12,999 rollout records. Replaying identical replies through one extra parser lowered success by 55.4–73.2 points across eight same-window configurations; in a separate provider-native tool campaign, schema adherence stayed at 98–100% while execution accuracy still varied. (paper, benchmark implementation)
The transferable evaluation rule is to score three surfaces separately: the generation contract the model saw, the transport/execution path that transformed the output, and the final-state verifier. A matched aggregate can hide equal and opposite failures at those boundaries, so a harness migration should replay the same outputs through both old and new paths before attributing the delta to model quality.
August 2026 Recovery and Sampling Evidence
AgentRewind makes checkpoint, replay, and branch semantics executable for stateful engineering work: it records aligned agent-context and controlled-environment checkpoints, restores both, and injects bounded memory from the failed attempt. Its public recorder supports local record/replay/fork workflows, while MettleBench supplies 82 long-horizon assignments with deterministic ordered-checklist evaluators. This is stronger evidence than prompt-only replay, but the implementation is alpha and cannot reliably reverse process memory or external side effects; checkpoint coverage must be declared and tested rather than assumed. (paper, recorder, MettleBench)
optstop attacks a different evaluation-infrastructure cost: fixed repetition budgets. The released Python package applies hierarchical Bayesian precision thresholds per item and model-task grouping, integrates with inspect_ai, and includes a conservative rare-success mode. The paper's nine-setting shadow experiment removed 57–97% of planned trials while preserving its aggregate conclusions, but the headline results come from one 200-item, 10-epoch design; MCMC cost, exchangeability, randomized item order, and default-parameter behavior remain acceptance conditions for any real deployment. (paper, implementation)
Why This Matters Now
2026 agent evaluation practice is visibly consolidating around this shape. Braintrust, Confident AI, and newer OTel-native tools like Laminar (rollout debuggers for grading multi-step trajectories) are converging on trace-based, production-loop-fed eval suites — one 2026 industry guide recommends ≥500 real traces before trusting an aggregate metric. That's the data-plane half of this framework showing up as product. The control-plane half — decoupling task definitions from harness so regressions are attributable — is less commonly implemented and is where most teams are still accumulating debt.
Why Assess, Not Trial
The methodology is not standardized, and there is still no shared equivalent to pytest for asserting that the control/data-plane separation holds. A2E is a concrete candidate implementation, but it is new and lacks independent adoption evidence. The individual pieces (OTel-style tracing, perturbation testing, LLM-as-judge trace scoring) are independently well-established and already show up in tools on this radar (Arize Phoenix, Braintrust, Langfuse, MLflow, AgentOps). Keep this in Assess until reproducible cross-harness results and durable CI integrations show that the organizing principle survives beyond one research artifact.
Relationship to Adjacent Entries
- Harness Engineering — Assess. Harness engineering asks what environment the agent runs in; evaluation infrastructure asks how you know whether that environment change helped. The data plane described here is effectively "the harness, instrumented for measurement."
- Arize Phoenix, Braintrust, Langfuse, MLflow, AgentOps — Trial/Assess. These are data-plane implementations: tracing, replay, and eval scoring. None of them enforce the control-plane discipline (decoupled task suites, gated regression tracking) this entry argues is the durable layer.
Key Characteristics
| Property | Value |
|---|---|
| Synthesized by | Lee Hanchung, Director of ML at Moody's; author, Evaluation and Alignment |
| Published | June 13, 2026 |
| Builds on | Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015) |
| Core unit | Episode (bounded attempt) → trace (messages + tool calls), not input-output pair |
| Named example | AI2 olmo-eval — task/harness decoupling |
| Type | Architectural practice / engineering discipline |
Sources
- Lee Hanchung — The Hidden Technical Debt of AI Systems: Agent Evaluation Infrastructure — June 13, 2026
- Lee Hanchung — About — author background, Director of ML at Moody's
- Confident AI — LLM Agent Evaluation Metrics in 2026 — corroborates trace-based eval consolidation industry-wide
- Braintrust — Agent Observability: The Complete Guide for 2026 — corroborates production-trace-to-eval-suite feedback loop