Technology RadarTechnology Radar

Harness Engineering

agentworkflowobservabilitymulti-agentinference
Assess

Harness engineering is the discipline of designing the systems, constraints, and feedback loops that wrap around AI agents to make them reliable in production. Where context engineering asks "what information should the agent have?", harness engineering asks "what environment should the agent operate in?" — and treats the answer as an engineering problem, not a prompting problem.

The Concept

The term was crystallised by Mitchell Hashimoto (HashiCorp/Terraform co-founder) in his February 5, 2026 blog post "My AI Adoption Journey," where he described Stage 5 of AI adoption as "Engineer the Harness":

"Every time you discover an agent has made a mistake, you take the time to engineer a solution so that it can never make that mistake again."

On February 13, 2026, OpenAI published "Harness engineering: leveraging Codex in an agent-first world" — a detailed account of building a one-million-line codebase using only AI agents over five months, with zero lines of manual code. The finding: the team spent most of its engineering effort not writing code, but engineering the harness that made the agents reliable.

Birgitta Böckeler (Thoughtworks) formalized the structure in her February 17, 2026 analysis on martinfowler.com, breaking a harness into three components:

Component What It Does
Context engineering Designs what information the agent sees and when
Architectural constraints Enforces codebase structure, conventions, and guardrails
Garbage collection Removes dead code, stale docs, and accumulated AI noise

The Core Insight

The model isn't the variable. The harness is.

LangChain's coding agent moved from 52.8% to 66.5% on Terminal-Bench 2.0 — jumping from Top 30 to Top 5 — by changing nothing about the model. Same model, different harness, dramatically better results.

OpenAI's internal experiment found: when an agent made a mistake, the instinct was always to "try a better model." The actual fix was always the same: "what capability is missing, and how do we make it legible and enforceable for the agent?"

The Harness in Practice

A harness typically includes:

  • AGENTS.md / structured docs: Treat it as a table of contents, not an encyclopedia. 100 lines pointing to deeper docs, not 10,000 lines of everything the agent might ever need.
  • Enforced architecture: Dependency layers and structural linting rules the agent cannot violate. At OpenAI: Types → Config → Repo → Service → Runtime → UI, with structural tests that fail if agents violate layering.
  • Per-task isolated environments: Each agent run gets a fresh, isolated instance (worktree, container, sandbox). Prevents environment contamination between concurrent runs.
  • Verification loops: Every change is validated against tests, linters, and type checkers before it lands. Self-correction happens in the loop, not after.
  • Observability: Agents use logs, metrics, and spans to reproduce bugs and validate fixes. If the agent can't observe its own output, it can't self-correct.
  • Entropy management: Background cleanup tasks remove AI-generated slop on a schedule, keeping the codebase legible for future agent runs.

The Edit Format Problem

One underappreciated dimension of the harness is edit format — how the model expresses the changes it wants to make to a file. This includes tool schemas, diff formats, search/replace patterns, and whole-file rewrites. These choices are engineering decisions, not model decisions, and they have outsized impact on reliability.

Can Öztürk (February 2026) benchmarked 15 LLMs across different harness configurations and found:

  • Format choice alone swung GPT-4 Turbo from 26% to 59% on Aider's coding benchmark — a 2.3x improvement with zero model change.
  • No single edit format dominates across all models and use cases (confirmed independently by JetBrains' Diff-XYZ benchmark).
  • An 8% improvement in Gemini through harness optimization cost roughly $300 in benchmarking — more impact than most model upgrades, at a fraction of the cost.

Cursor's response: they trained a separate fine-tuned 70B model whose sole job is to take a draft edit and merge it into the file correctly — a harness component dedicated entirely to solving the edit format problem.

The insight: "Often the model isn't flaky at understanding the task — it's flaky at expressing itself. You're blaming the pilot for the landing gear."

A related academic thread: Meta-Harness (Lee et al., Stanford/MIT, March 2026, arxiv:2603.28052) shows that harnesses can be optimized end-to-end automatically — an agentic proposer searches over harness configurations using full access to prior source code, execution traces, and scores. +7.7 points on classification tasks while using 4× fewer tokens.

Lilian Weng's July 2026 synthesis, "Harness Engineering for Self-Improvement", extends that idea into a practical progression: prompts → structured context → workflows → harness code → optimizer code. The near-term lesson is not uncontrolled self-modification. It is a bounded propose → evaluate → accept loop backed by durable filesystem artifacts, verifier-grounded failure analysis, and held-out regression tests. Weng also highlights an important distinction: a model's ability to write a plausible harness update is not the same as its ability to benefit from that update over a long horizon.

AutoDesign turns that pattern into a runnable, domain-specific meta-harness: a fixed model/harness pair proposes one bounded harness change per rollout, evaluates it on an independent development set, and accepts only measured improvements before generating editable posters, slides, web pages, and narrated video. The authors report a 12.4-point aggregate improvement across seven model-agent configurations, and the public repository includes the optimizer, evaluator, CLI, and artifact pipeline. A merged DeepSeek Harness adapter later passed 41 focused tests, the project's 53-section smoke suite, and a real coding-agent integration smoke, showing that the invocation boundary transfers to another harness; the promised artifact end-to-end run and independent quality replication are still absent. This is promising implementation evidence, not general production proof. (paper, repository, DeepSeek adapter)

Production Evidence: Shopify Dispatch

Shopify's July 2026 account of its internal Dispatch security harness adds a concrete production pattern: partition a large codebase into model-sized domains, run parallel category-specific Hunters, verify candidates sequentially with a different model and real application tests, then leave credentials, Git operations, storage, deduplication, and scoring to deterministic code. Shared architecture artifacts and diff-only follow-up scans reduce repeated discovery work and token spend.

Shopify reports complete scans across more than 80 applications, thousands of runs over roughly six weeks, and more than 300 findings. Full scans cost $50–$300 and incremental scans $5–$50 using public frontier models. These are Shopify's self-reported operational results, not an independent benchmark, but they are stronger evidence than a lab-only evaluation because the harness operates against Shopify's production code and test infrastructure.

The tooling is also becoming easier to adopt. Microsoft released the core Agent Framework Harness for Python and .NET with planning, per-model-call persistence, compaction, file memory, approvals, web search, and OpenTelemetry enabled by default. Background agents, file access, looping, and shell tooling remain opt-in and still carry prerelease warnings, so this is evidence of ecosystem convergence rather than a complete standard playbook.

Why Assess (Not Trial Yet)

The concept is new — coined February 2026 — and the evidence base is strong but still concentrated at a small number of advanced teams (OpenAI, Stripe, Anthropic, and Shopify). Most organisations are still at the earlier stages of AI adoption (getting single agents to work reliably) and not yet building harnesses in the structured sense.

The term itself is entering mainstream vocabulary rapidly, but the practices are not yet standardised. There's no "harness engineering playbook" with agreed tooling and patterns the way there is for CI/CD or testing.

Assess means: understand the concept, identify whether your team is ready to invest in harness-building, and track the emerging tooling (AGENTS.md, Roast, Goose hooks, Claude Code hooks).

Relationship to Adjacent Entries

  • Context Engineering — Assess. Context engineering is one component of a harness. If you're doing context engineering well, you're already building part of a harness.
  • Background Coding Agents — Trial. Stripe's Minions and Spotify's Honk are mature harness implementations, even if they didn't use the term. The patterns documented there are harness engineering in practice.
  • AGENTS.md — Trial. AGENTS.md is the primary context artifact in any harness.
  • Agent Evaluation Infrastructure — Assess. If harness engineering is the discipline of building the environment an agent runs in, evaluation infrastructure is the discipline of instrumenting that environment well enough to know whether a harness change actually helped.

Named Org Proof Points

Organisation What They Built Evidence
OpenAI 1M-line codebase, 1,500 PRs, 5 months, zero manual code Harness engineering blog post
LangChain +13.7 ppts on Terminal-Bench 2.0 from harness changes alone OpenAI harness engineering paper
Stripe 1,300+ PRs/week via Minions + Blueprint harness Minions blog
Anthropic 100K-line C compiler built by 16 parallel Claude Opus 4.6 agents OpenAI harness engineering paper
Shopify Dispatch AppSec harness across 80+ applications; 300+ findings Shopify Engineering

Key Characteristics

Property Value
Coined by Mitchell Hashimoto (Feb 5, 2026)
Popularized by OpenAI blog post (Feb 2026), Birgitta Böckeler/Thoughtworks (Feb 17, 2026)
Related concept Context Engineering (subset), Harness ⊃ Context
Type Architectural practice / engineering discipline

Sources