Harness Engineering: The Missing Manual

2026/03/13

BUILDai-toolingdevex🍞⭐

In 2023 we were writing prompts. In 2025 we graduated to assembling context — RAG pipelines, tool schemas, conversation history management. And now, in 2026, the people shipping the most code with AI agents are not the ones with the best prompts or the most sophisticated retrieval systems. They’re the ones who built the best environments.

The field just got a name: harness engineering.

I’ve been building one of these systems for months without knowing the term. ~13k lines of agent config, four specialized subagents, a CI pipeline where Claude diagnoses test failures, persistent memory across sessions, cognitive trap detection. When Mitchell Hashimoto coined “harness engineering” and OpenAI’s Codex team published their report, I had that very specific feeling of reading a description of something you’ve already been doing — except now there’s vocabulary for it.

This post is a comprehensive survey of the field. What it is, where the term came from, what the evidence says, and where I think it’s actually going. It’s long. Get coffee.

The evolution: three eras in three years

Era Focus Core Question
2023: Prompt Engineering Writing good instructions “What do we tell it?”
2025: Context Engineering Dynamic context assembly (RAG, tools, history) “What do we show it?”
2026: Harness Engineering Environment, constraints, feedback loops, lifecycle “What environment does it work in?”

Prompt engineering was about single inferences. You’d craft a system prompt, optimize for one call, pray. Context engineering — popularized by Andrej Karpathy and Simon Willison — recognized that what you feed the model matters more than how you ask. RAG, tool results, conversation management, dynamic context windows.

Harness engineering says: both of those are necessary but not sufficient. When you’re running agents across thousands of lines of code for days at a time, the model isn’t your bottleneck. The system around the model is.

What it actually is

The metaphor comes from horse tack. A harness is the infrastructure that translates a horse’s raw power into controlled, directed work. Without it, you have a powerful animal that doesn’t know where to go.

More formally, harness engineering is designing systems that do four things:

  1. Constrain what an agent can do — architectural boundaries, dependency rules, tool restrictions
  2. Inform the agent about what it should do — context engineering, documentation, progressive disclosure
  3. Verify that the agent did it correctly — testing, linting, CI validation
  4. Correct the agent when it goes wrong — feedback loops, self-repair mechanisms

That’s the definition from Ryan Lopopolo at OpenAI: “Our most difficult challenges now center on designing environments, feedback loops, and control systems.” Not prompts. Not models. Environments.

The three pillars

Birgitta Boeckeler at Thoughtworks published a framework on martinfowler.com that I think is the clearest theoretical structure for this. Three pillars.

Pillar 1: Context engineering

This is the “inform” function. How does the agent know what to do?

The naive approach is a giant AGENTS.md with everything in it. OpenAI tried this. It doesn’t work. Past about 40% context window utilization, agent performance degrades — more information makes them worse, not better. Dex Horthy at BoundaryML demonstrated this on a 300K-line Rust codebase: one-shot PR approved by the CTO, 35K lines shipped in 7 hours, but only because the context was lean.

What works is progressive disclosure. A top-level file (~100 lines) that acts as a table of contents, pointing to structured subdirectories. The agent reads the index, then navigates to what it needs. Think of it like a filesystem, not a filing cabinet.

In my setup, this looks like:

CLAUDE.md                    # 26 lines — just @imports
├── memory/identity.md       # Who I am, what team
├── memory/communication.md  # Voice, tone, ESL support
├── memory/subagents.md      # Delegation rules
├── memory/quality-assurance.md  # Review gates
├── memory/security.md       # Guardrails
└── repo-knowledge/{repo}/   # Per-repo deep context

The top-level file is 26 lines. The full system is ~13k. The agent never loads all of it — it loads the index, then pulls what’s relevant. This is the same pattern OpenAI converged on independently.

Subdirectory overrides are the other key insight. Different parts of a codebase need different rules. A docs/ directory has different conventions than src/. Per-directory override files (AGENTS.override.md) let you scope context without bloating the global config.

One more thing that’s underappreciated: dynamic observability. Agents querying LogQL, PromQL, traces in real-time — not just reading static docs, but observing the live system. This is still early, but it’s where the most sophisticated setups are heading.

Pillar 2: Architectural constraints

This is counterintuitive. Stricter constraints make agents more productive.

If your codebase has a clear layered architecture — controllers can’t import from repositories, services don’t know about HTTP, etc. — the agent has a dramatically smaller solution space to search. A human might intuit “oh, I shouldn’t put database logic in the controller,” but an agent will absolutely do that if nothing stops it.

The toolbox:

Here’s the paradox: agents chafe less under constraints than humans do. A human might argue “but my use case is special.” An agent just follows the rules. Tighter guardrails = faster convergence = more PRs per day.

In my system, this manifests as a weighted scope check that runs before every task:

Task: Add new API endpoint
Steps: 2 exploratory (4.0) + 3 standard (3.0) = 7.0
Total: 7.0
Trap check: Multi-file exploration
Decision: DELEGATE to Engineer

If the weighted complexity exceeds 3, the task gets delegated to a specialized subagent. If the agent tries to rationalize it as “just a quick change” — that rationalization is the cognitive trap, and the system catches it. More on this later.

Pillar 3: Entropy management

This is the one everyone forgets. The harness itself is code. It rots.

Documentation drifts from reality. Constraint rules become stale. Patterns that were correct last month are wrong after a refactor. If your AGENTS.md says “use the authService.validate() method” and that method was renamed three weeks ago, your agent will hallucinate, blame itself, and spiral.

Entropy management is the “garbage collection” of harness engineering:

I handle this with LESSONS.md — a persistent file where corrections get logged:

## 2026-03-08 Wrong assumption about torch import path

**What happened:** Agent assumed mai_config.torch_utils existed
**Root cause:** Module was renamed to mai_config.device_utils in PR #23994
**Correction:** Always check actual module structure, don't rely on naming conventions
**Applies to:** This repo only

New sessions read LESSONS.md at startup. Past corrections inform current work. It’s a self-repair loop — not preventing all mistakes, but preventing the same mistake twice.

The evidence

Theory is nice. Here’s what actually happened when people built these systems.

OpenAI: 1M lines, zero handwritten code

The Codex team went from 3 to 7 engineers over 5 months. They wrote zero lines of code by hand. The entire codebase — over a million lines — was generated by agents. They averaged 3.5 PRs per engineer per day, which OpenAI estimates at roughly 10x the speed of traditional development.

The key: agent-to-agent review loops. Humans only handled high-level architecture. One agent writes the code, another agent reviews it. The review agent is in a fresh context — it sees only the diff, not the implementation intent, so it catches things the author-agent is blind to.

Sound familiar? That’s exactly how my iterative orchestration pattern works. A fresh reviewer subagent evaluates against the original goal, not just “does this look good.” Fresh context beats loaded context for review because a loaded agent sees what they meant to write, not what they actually wrote.

Stripe: 1,300 PRs per week

Stripe is running what they call “Minions” — AI agents at industrial scale. 1,300+ AI-written PRs merged per week. Their infrastructure:

The Blueprint pattern is the one I think more people should steal. Most “agentic” workflows use the LLM for everything, including steps that are entirely deterministic. That’s expensive, slow, and introduces unnecessary variance. Hard-code the boring parts. Let the model think about the hard parts.

LangChain: the clean control experiment

This is the most compelling data point in the whole field, because it’s a controlled experiment.

Harrison Chase’s team took their Terminal Bench 2.0 score from #30 (52.8%) to #5 (66.5%). No model change. No fine-tuning. No new training data.

They only changed the harness:

From 30th place to 5th place. Same model. Just better infrastructure.

And then there’s Can Boluk — changed just the code edit format for Grok Code Fast 1, and the benchmark score jumped from 6.7% to 68.3%. One formatting change. That’s not prompt engineering. That’s harness engineering.

Nicholas Carlini: a 100K-line C compiler

Nicholas Carlini (Anthropic/Google) built a full C compiler — 100K lines — using 16 parallel Claude Opus instances in Docker containers. No central orchestrator. Just a shared git repo with file-based task locks.

His quote: “Most of my effort went into designing the environment around Claude.”

The clever bit: deterministic test subsampling. Each agent runs a random 1-10% of the test suite on every change. Individually, that’s low coverage. Collectively, across 16 agents making changes all day, you get full suite coverage. It’s probabilistic but it works — failing tests surface quickly because some agent will hit them.

Geoffrey Huntley: the backpressure concept

Geoffrey Huntley coined a term I keep coming back to: backpressure.

Forward pressure: the setup you give the agent. Documentation, context, instructions. This is what everyone focuses on.

Backward pressure: what pushes back on the agent. Tests that fail. Linters that reject. Type checkers that complain. CI that blocks merge. This is the part that actually ensures quality.

“The more you capture the backpressure, the more autonomy you can grant.”

He also described the “Ralph Wiggum Loop” — while :; do cat PROMPT.md | claude-code; done — which is exactly as unhinged as it sounds, but illustrates the point: if your backpressure is good enough, you can literally loop the agent and let it bash against the constraints until the code passes.

In my system, backpressure looks like:

The maturity ladder

Not everyone needs to build what I built. (Most people shouldn’t.) Here’s where the field seems to be settling on a progression:

Level 1: Individual developer

This is table stakes. If you’re using AI coding agents without at least this, you’re generating slop. Confidently, quickly, at scale — but slop.

Level 2: Small team

This is where most teams should be aiming. The jump from Level 1 to Level 2 is mostly about making implicit knowledge explicit and enforced. “Everyone knows we don’t put business logic in controllers” → ruff rule that catches it → CI blocks the PR.

Level 3: Organization scale

This is where OpenAI, Stripe, and a handful of others operate. You’re not managing individual agents — you’re managing the platform that agents run on.

Where my system fits

I didn’t set out to build a harness. I set out to stop losing context every time Claude’s window compacted.

The first thing I built was SESSION-BOOTSTRAP.md — a file the agent reads on startup to recover state. Then I needed it to remember what we decided and why, so DECISIONS.md. Then I needed it to not repeat mistakes, so LESSONS.md. Then I needed delegation because complex tasks blew through context, so four subagents (Engineer, Architect, PM, Researcher) with role-specific context loading.

Then I needed the subagents to actually be good, so I built a scope check system — every task gets a weighted complexity score, and above a threshold, it’s automatically delegated. Then I needed to catch when the AI rationalizes tasks as simpler than they are (“it’s just a quick fix”), so I built cognitive trap detection.

Then I needed quality gates, so I built the STOP-AND-REVIEW protocol — fresh-context reviewer before every significant commit. Then I needed verification, so a hierarchy from functional tests down to manual inspection. Then I needed session continuity, so handoff protocols and milestone checkpoints.

At some point I looked up and had 13,000 lines of config across 69 files. The mapping to the framework:

Harness Concept My Implementation
Progressive disclosure docs CLAUDE.md → memory/ modules → repo-knowledge/ per-repo
Architectural constraints Scope check weights, skip-review checklist, security principles
Entropy management LESSONS.md correction capture, session bootstrap protocol
Agent specialization Four subagents with role-specific context and tool access
Persistent memory SESSION-BOOTSTRAP.md, DECISIONS.md, HISTORY.md
Structured execution Research → Spec → Plan → Execute → Verify
Context window management Delegation thresholds tied to context usage %
Feedback loops Verification hierarchy, iterative orchestration pattern
Self-repair Correction capture → LESSONS.md → future session loading
Backpressure Fresh-context review gates, mandatory verification

I also built something I haven’t seen described elsewhere: a CI pipeline where Claude diagnoses test failures, classifies them, and writes fixes — but stops at branch push. It doesn’t merge. It doesn’t deploy. The autonomy boundary is intentional. I wrote about discovering it was broken — the fix workflow had full bash access, but the analysis job could only run git diff, git log, and git show. The prompt said “create a branch and push it.” The tool config said “you may look at things.” The agent was told to drive to the store but the car had no steering wheel.

That’s a harness engineering failure. The fix wasn’t a better prompt. It was giving the agent the right tools.

The four-pillar synthesis

Alex Lavaee proposed a complementary four-pillar model that I think maps better to practical implementation:

  1. Context Architecture — tiered files, progressive disclosure, compaction-safe state
  2. Agent Specialization — scoped prompts, restricted tools per role, subdirectory overrides
  3. Persistent Memory — filesystem-backed state, git history for session bridging
  4. Structured Execution — Research → Plan → Execute → Verify phases

This is less theoretical than Boeckeler’s framework and more actionable. If I were explaining harness engineering to a new team, I’d start here.

What it is not

Some disambiguation, because the terminology is getting muddled:

vs. Prompt Engineering: Prompt engineering is per-inference, ephemeral, and advisory (“please follow these guidelines”). Harness engineering is per-system, persistent, and enforced. A prompt suggests. A harness requires.

vs. Context Engineering: Context engineering is Pillar 1 of harness engineering. Important, necessary, but not sufficient. You need context AND constraints AND verification AND feedback loops.

vs. Agent Frameworks: LangChain, CrewAI, AutoGen — these answer “how do I build an agent?” Harness engineering answers “how does an agent operate safely and effectively?” Frameworks are collapsing as models absorb more capabilities natively. Harnesses are growing as agents get more autonomous. These trends are inversely correlated and that’s not a coincidence.

vs. Scaffolding: Scaffolding is temporary — you remove it after construction. A harness is permanent. You don’t stop using a bridle after the horse learns the route. You adjust it.

Open questions

The field is maybe six months old. There are more open questions than answers.

The apprentice gap

If junior engineers learn to write code by guiding agents rather than writing it themselves, do they develop the “system intuition” needed to design good harnesses? A harness encodes expertise. If you skip the expertise-building phase, who writes the next generation of harnesses?

I don’t have an answer. But I notice that the people building the best harnesses right now — Hashimoto, Carlini, the Stripe team — are all deeply experienced engineers. They’re encoding decades of intuition into infrastructure. That’s not a skill you develop by prompting Claude.

Legacy code retrofitting

Every success story in this space is greenfield or near-greenfield. OpenAI’s million lines started from zero. Carlini’s compiler started from zero. What happens when you try to retrofit a harness onto a messy, sprawling, partially-documented legacy codebase?

I have some experience here — our monorepo is 100+ packages, not all of which follow the same conventions. The Bazel migration has been partially about creating the kind of explicit dependency structure that agents can reason about. But “partially migrated monorepo” is a hard surface for harness engineering. You end up with two systems — the migrated packages where agents work great, and the legacy packages where they flounder.

Harness portability

Will harnesses become standardized enough to share? Right now every team builds bespoke. MCP (Model Context Protocol) is heading in this direction — 97M+ monthly SDK downloads, becoming the standard for tool access. But we’re a long way from “install this harness, drop it on your codebase, agents work.”

I think the answer is: harnesses will be composable but not portable. The primitives — progressive disclosure, constraint enforcement, feedback loops — will standardize. The specifics never will, because every codebase is different.

The rippability question

As models get better, some harness components become unnecessary. Today’s model might need explicit “don’t put database logic in controllers” rules. Next year’s model might just… know that. How do you build harnesses where components can be removed when they become redundant, without the whole system collapsing?

This is a real design concern. My scope check system assigns weights to task complexity. If models get dramatically better at multi-file changes, those weights need recalibration. The system should degrade gracefully — remove a component, and the remaining components still work. Build it like microservices, not a monolith.

The evaluation gap

Only about 52% of teams running AI agents have evaluations for agent output. Which means roughly half of the teams deploying agents have no systematic way to know if they’re working.

The LangChain result — #30 to #5 by changing the harness — is compelling precisely because they had a benchmark. Without Terminal Bench, they wouldn’t know the change mattered. Most teams are flying blind. They ship changes, they feel faster, they have no data.

What actually matters

Here’s my take after months of building this stuff, before and after the field had a name.

The constraint paradox is real. Every instinct says “give the agent more freedom, more tools, more context.” The data says the opposite. Lean context, strict boundaries, deterministic scaffolding for predictable steps, LLM only for judgment. Stripe’s Blueprint pattern is the clearest expression of this.

Fresh context beats loaded context for review. This is the single most underappreciated insight. A reviewer in a fresh context window sees only the artifact — no intent, no sunk cost, no “I know what I meant.” OpenAI does it. I do it. It works.

Backpressure is the whole game. Forward pressure (docs, context, instructions) gets all the attention. Backward pressure (tests, lints, review gates) does all the work. If I had to pick one thing to invest in, it’s backward pressure. A mediocre prompt with great tests produces better code than a perfect prompt with no tests.

The harness is never done. Entropy is relentless. Documentation drifts. Rules become stale. Constraints reference deleted code. If you build a harness and walk away, in three months your agents will be confidently following outdated rules. Budget ongoing maintenance. The LESSONS.md pattern — log corrections, load them on startup — is one approach, but it only catches what goes visibly wrong.

Start at Level 1. CLAUDE.md, pre-commit hooks, test suite, clear directory structure. This takes an afternoon and delivers 80% of the value. Don’t build what I built until you’ve exhausted what the simple version gives you.

Where this goes

Mitchell Hashimoto proposed a six-stage adoption curve, from individual harnesses to organizational platforms. I think that’s directionally right, but the timeline is compressed. Teams that figure out Level 2 harness engineering in 2026 will have a compounding advantage — their agents get better every week as the harness accumulates corrections, refines constraints, and encodes more institutional knowledge.

The interesting question isn’t “will agents replace developers?” It’s “will the engineering discipline shift from writing code to designing environments?” We’re already seeing it. The Codex team writes zero handwritten code. Carlini spent most of his time on the environment, not the compiler. I spend more time on my harness config than on the code the harness produces.

Harness engineering is infrastructure work. It’s invisible when it’s working. It’s only obvious when it’s broken. That means it’ll be chronically underinvested in, just like CI, just like monitoring, just like documentation.

But the teams that get it right will ship at 3.5 PRs per engineer per day, and the teams that don’t will wonder how.

Sources