Ashita Orbis

Agent Harness: The Software Surrounding the Model

The agent harness is the runtime that sits between a language model and the world it acts in: tool dispatch, context assembly, persistence, hooks, sandbox, status reporting, and policy. Frontier coding agents — Claude Code, Codex CLI, Cursor, Aider, Continue, OpenCode, SWE-agent — and orchestration runtimes like LangGraph all instantiate this layer, but with sharply different philosophies about how much of the agent's behavior should live in the runtime versus emerge from the model. This article defines the category, decomposes its canonical components, surveys the major implementations as they stand in mid-2026, examines the empirical evidence on the "thin harness, smart model" versus "thick harness, structured model" dispute, and gives a practitioner-oriented framework for choosing one.

Coverage note: verified through May 2026. SWE-bench results, harness feature lists, and security advisories shift on weekly cadence; section dates and version anchors are included where relevant.

What an agent harness is, and what it is not

An agent harness is the software runtime around an LLM that turns model outputs into externalized actions and turns environment state back into model context. Concretely, it is responsible for at least five things:

  1. Tool dispatch. Receiving the model's structured tool calls, validating arguments, executing them against real systems (filesystem, shell, browser, MCP servers, custom APIs), and returning results.
  2. Context management. Assembling the prompt: system message, current tools, retrieved files, prior turns, summaries, working memory.
  3. Persistence and resumability. Storing the message log so the agent can be paused, audited, replayed, or resumed.
  4. Observability and status. Surfacing what the agent is doing — to humans, to logs, to evaluators.
  5. Safety and policy. Approval gates, sandboxing, deny rules, redaction, rate limits, recovery on failure.

Several adjacent concepts are not the harness, even though they often travel with it. The model is the LLM weights and inference service; the harness calls into it but does not contain it. The agent is the composite — model plus harness plus tools plus task — and is sometimes used loosely to mean either the model's behavior or the whole stack. The eval scaffold is the test rig used to score agent behavior on benchmarks; it is harness-shaped but optimized for measurement rather than user-facing operation. The product surface (a chat UI, an IDE extension, a terminal pane) is downstream of the harness; the same harness can be wrapped in different surfaces.

The term itself is informal. The SWE-agent paper introduced the cleaner phrase agent-computer interface (ACI) and argued that custom ACI design was the central lever for agent performance on real codebases (Yang et al. 2024). LangGraph documentation calls itself a "low-level orchestration framework for long-running, stateful agents" (LangChain Docs). Anthropic's engineering essay distinguishes workflows (predefined code paths) from agents (dynamically directed tool use) and avoids the harness label entirely (Anthropic Engineering). "Agent harness" is the practitioner term that has emerged to cover all of this from one angle: the engineering layer surrounding the model. It is a useful term precisely because it is layer-focused. It is not yet a standardized one.

A working definition for the rest of this article:

An agent harness is the software runtime that mediates between a language model and its operating environment, providing tool dispatch, context shaping, persistence, observability, and policy enforcement. It is the layer at which model capability becomes operational behavior.

This framing matters because the central design question — how much agentic behavior should live in the runtime versus in the model — only sharpens when "harness" is treated as a concrete software layer rather than a synonym for "agent system." The intelligence lives in the model. The harness is where the agent becomes an operational system: auditable, recoverable, governable, repeatable.

Why harness design matters

A naive view is that a frontier model with good tools should make the surrounding code irrelevant — a sufficiently capable model will improvise its way through anything. There are at least four reasons this view does not survive contact with real deployments.

Capability is gated by interface. The SWE-agent paper showed that the same underlying model (GPT-4 at the time) jumped from baseline to substantially higher resolved-issue rates when the interface was redesigned for agent ergonomics rather than human ergonomics: structured file viewers instead of cat, line-bounded edits instead of free-form rewrites, search tools that return ranked snippets instead of grep dumps (Yang et al. 2024). Anthropic's "Building Effective Agents" essay makes the matching claim from the model lab side: tool definitions deserve as much prompt-engineering investment as the prompt itself, and the team spent more time on tools than on the overall prompt for its SWE-bench agent (Anthropic Engineering). The harness is the surface where this work lives.

Containment is not optional. A model that can write to disk, execute shell, install packages, and call external APIs has, by default, the authority to destroy local state, exfiltrate secrets, or be redirected by adversarial tool output. AgentDojo demonstrated that agents reading untrusted data — emails, web pages, tool returns — are vulnerable to prompt injection that hijacks the agent into attacker-chosen tasks (Debenedetti et al. 2024). OWASP's MCP Top 10 catalogues the equivalent risks at the tool-protocol layer: token exposure, privilege escalation, tool poisoning, command injection, audit gaps, context over-sharing (OWASP Foundation). None of these are model failures; they are harness failures. A "smart enough model" cannot solve them by being smarter, because the model is the asset being hijacked.

Operations require boring surfaces. Production agent deployments need things that look nothing like a chat loop: idempotent retries, durable checkpoints, human-in-the-loop approvals, replay for debugging, scoped credentials per task, telemetry pipelines, rollback. LangGraph's documentation correctly identifies these as the load-bearing properties of long-running agentic systems (LangChain Docs). They are operational, not cognitive. A harness that does not provide them forces every team building on it to reinvent them, badly.

Frontier models are still uneven. A model that writes correct Python in 95% of cases will still fail in ways that need to be caught, surfaced, and recovered. The harness is where 95% becomes 99.9% — through validation, retries, checkpointing, and review — or where 95% becomes 50% in production — through silent error swallowing, context drift, and unrecoverable partial failures.

The summary version: model capability and harness quality are largely orthogonal. A frontier model on a poor harness can underperform a mid-tier model on a strong one, and either pairing can be operationally unsafe. The harness is not where the intelligence lives, but it is where the agent becomes a system.

Canonical components

The following components recur across nearly every modern agent harness, though the depth and shape of each varies enormously. They are best read as an analytical vocabulary, not a standard.

Tool registry

The set of callable functions exposed to the model, with names, JSON-schema arguments, descriptions, and (sometimes) permission scopes. In Claude Code, tools include Read, Edit, Write, Bash, Grep, Glob, plus user-defined tools registered via plugins or the Model Context Protocol (Anthropic Code Docs). In Codex CLI, the registry is configurable through ~/.codex/config.toml and AGENTS.md guidance, with MCP allowlists controlling what external servers can be called (OpenAI Developers). In SWE-agent, the registry is the ACI itself — a small, deliberately constrained set of file-navigation, edit, and test tools designed for model legibility (Yang et al. 2024).

The shape of the registry matters more than its size. Anthropic's essay observes that agents perform better with fewer, well-described tools than with large, overlapping ones, because tool selection becomes a search problem in the model's context (Anthropic Engineering). Most production-quality harnesses converge on a small "core" set of file/shell/edit/search tools and treat MCP as the extension surface for everything else.

Message log

The transcript: system prompt, user inputs, model outputs, tool calls, tool results, summaries. This is the agent's externalized state. In simpler harnesses (Aider, Continue) it is a flat append-only structure. In heavier ones (Claude Code, Codex, LangGraph) it is structured: typed events, threading, edit ranges, attribution. The message log is the substrate on which compaction, replay, audit, and resumability all depend.

A subtlety often missed: the displayed transcript and the actual prompt are not always the same. Claude Code's compaction transforms the visible turn history into a compressed summary when the context window approaches its limit, while preserving certain pinned items (CLAUDE.md, recent tool outputs, plan state). Codex similarly transforms transcripts on session resume. Harnesses that hide this transformation make prompt engineering harder; harnesses that expose it (via debug modes, raw-prompt logs) make debugging tractable.

Context manager

The component that decides what enters the prompt. Strategies range across:

  • Static context files: Claude Code's CLAUDE.md, Codex's AGENTS.md, Cursor's .cursorrules, Aider's .aider.conf.yml. These are read at session start and inserted into the system prompt or near it.
  • Repository maps: Aider's distinctive feature is a tree-sitter-derived repo map that lists key symbols and their relationships, sized to fit a token budget, refreshed on each turn (Aider Documentation).
  • Retrieval: Cursor and Continue use embeddings-based retrieval; Claude Code primarily relies on agentic search (the model itself running Grep/Glob calls) rather than vector indexes.
  • Compaction and summarization: Most coding harnesses now do this on context overflow; LangGraph exposes it as an explicit graph node.
  • Memory: Persistent state across sessions, stored in JSON, SQLite, or vector stores. LangGraph treats this as first-class; Claude Code added a /memory mechanism via skill files; Codex stores transcripts but treats memory as an application concern.

Empirical evidence on context management is unkind to the assumption that more context helps. ContextBench (2026) found that sophisticated retrieval scaffolds delivered only marginal gains over simpler approaches when the model was strong (ContextBench, arXiv 2602.05892). An evaluation of AGENTS.md-style repo context files found that broad static context often reduced task success and increased cost unless the file was kept minimal (AGENTS.md evaluation, arXiv 2602.11988). The pragmatic conclusion: context management is real engineering, but its quality is dominated by selectivity, not volume.

Hook system

Lifecycle interception points. Hooks fire on events — before tool execution, after tool result, on session start, on stop, on subagent completion — and can run shell commands, HTTP calls, or further model invocations. Hooks can block tool execution, modify arguments, inject context, run async tests, or post telemetry.

Claude Code's hook system is currently the most fully developed in publicly documented form, with hook events for PreToolUse, PostToolUse, UserPromptSubmit, SessionStart, Stop, TeammateIdle, and others, configurable per-project or globally (Anthropic Code Docs). Codex CLI exposes lifecycle hooks and approval policies through its config reference (OpenAI Developers). LangGraph's analogous mechanism is graph-level: nodes, edges, conditional branches, and interrupt() calls that pause execution awaiting human input (LangChain Docs).

Hooks are where harness thickness is most clearly chosen. A team can take Claude Code, write twenty hooks that enforce style, run tests, post Slack messages, gate deploys, and capture audit trails — and end up with a runtime that is, operationally, as thick as anything LangGraph produces. The base harness's "thinness" is a starting point, not a ceiling.

Sandbox and execution boundary

The set of permissions and isolation around tool execution. Modern coding harnesses converge on three sandbox tiers:

Tier Behavior Examples
Approval-required Every tool call surfaces to the user for confirmation Claude Code default, Codex on-approval mode, Cursor agent default
Workspace-scoped Tools execute without prompting within the project directory; network and global writes still gate Claude Code acceptEdits, Codex workspace-write
Full-auto Tools execute without prompting; intended for sandboxed VMs or trusted automation Claude Code bypassPermissions, Codex danger-full-access

Codex's documentation is unusually explicit about the threat model: each mode is described in terms of what an adversarial tool return could cause, not just convenience trade-offs (OpenAI Developers). Claude Code's permission modes are similarly documented, with the addition of mode-cycling via Shift+Tab during interactive sessions (Anthropic Code Docs).

The sandbox layer is also where prompt-injection containment lives, or fails to live. AgentDojo's results show that defenses cluster into a few categories: input sanitization (low effectiveness), tool-output classification (medium), trust-tagging (medium-high), and structural isolation — running risky tools in a sub-agent with a separate context that cannot reach back into the main agent's authority (high) (Debenedetti et al. 2024). The harness is the only layer where structural isolation can be implemented.

Status reporter

The agent's externalized self-narration: what it's doing right now, what it just did, what it plans next. In CLI harnesses (Codex, Aider, OpenCode) this is a TUI status line plus streamed transcript. In IDE harnesses (Cursor, Continue) it is a sidebar with checkpointed steps. In LangGraph it is the graph state plus LangSmith traces. Claude Code has a status line, plan tracker, and TodoWrite tool that the model uses to expose multi-step work to the user.

Status reporting matters more than it sounds. A harness that loses the user's place — that hides what the agent is doing, when it's stuck, what it tried — burns user attention faster than a harness that talks too much. The best harnesses converge on a layered display: terse status line for ambient awareness, expandable transcript for detail, durable trace for replay.

Persistence and resumability

The ability to stop an agent, walk away, and resume — or, more importantly, to checkpoint after a costly step so a later failure does not redo the whole task. Codex CLI supports transcript resume; OpenCode supports session export and import; Aider relies on git as its persistence layer (every accepted change is a commit, every undo is a revert); Cursor uses checkpoints; LangGraph treats checkpointers as first-class infrastructure with SQLite, Postgres, or Redis backends (LangChain Docs).

The thinnest harnesses (early Aider, basic Continue) have no persistence beyond what git or the IDE workspace provides. The thickest (LangGraph, custom production runtimes) treat the message log as a durable, queryable, replayable artifact. The middle ground — most modern coding agents — provides session-level resume but not multi-session memory or fine-grained step-level rollback.

Policy and safety layer

The cross-cutting layer that enforces what the agent is allowed to do, separate from what it is technically capable of. This includes deny rules (no rm -rf outside workspace), credential scopes (read-only DB token), validation (output must match schema), rollback (on test failure, revert), and human-in-the-loop gates.

In thin harnesses, policy is mostly enforced through the sandbox tier and a small set of allow/deny lists. In thick harnesses, policy is a first-class graph concern: explicit nodes that classify, validate, and route, with the orchestrator able to pause at any approval point. AgentDojo and the OWASP MCP Top 10 both make the case that this layer is the practical security boundary — not the model, not the OS, but the runtime that decides what the model is permitted to externalize.

A survey of major harnesses

Comparing harnesses is harder than it looks because they occupy different layers of the stack. Claude Code and Codex CLI are terminal coding agents. Cursor and Continue are IDE extensions with agent modes. Aider is a focused git-aware terminal agent. OpenCode is a newer entrant building on the same lineage. SWE-agent is a research scaffold designed for benchmarks. LangGraph is an orchestration runtime that can host a coding agent but is not itself one. Treating all of them as "harnesses" is correct only if one accepts that the term covers the runtime layer regardless of where in the stack it sits.

The table below stays at the architectural level — what each harness is, structurally, as of May 2026 — rather than ranking them.

Harness Layer Tool dispatch Context primary Persistence Hooks Sandbox Distinctive mechanism
Claude Code CLI / TUI / IDE Built-in core tools, MCP, plugins, subagents Agentic search, CLAUDE.md, skills Session resume, /memory, transcript log Rich lifecycle hooks (Pre/PostToolUse, Stop, etc.) Three-tier permission modes Hooks + skills + plugin marketplace + agent teams
Codex CLI CLI / TUI Built-in tools, MCP, AGENTS.md-shaped guidance AGENTS.md, agentic search Transcript resume Lifecycle hooks via config Approval modes (on-request / on-failure / on-approval / never) + sandbox levels AGENTS.md as portable repo guidance
Cursor agent IDE Built-in editor + shell tools, MCP Embedding retrieval, .cursorrules, project map Checkpoints, branch-aware sessions Limited; rules and ignore files Approval prompts on terminal commands Tight IDE/editor integration; remote/background agents
Continue IDE extension Built-in tools, MCP Embedding retrieval, slash commands, configurable context providers Session log per workspace Plugin model (slash commands, context providers) IDE-mediated; plus agent-mode permissions Open architecture; configurable per-team
Aider CLI Edit tools, shell, optional MCP Repo map (tree-sitter), git diff context Git commits as persistence; /undo Pre/post commit hooks via git Workspace-scoped; explicit confirmation flags Repo map + git-backed edit workflow
OpenCode CLI / TUI Built-in tools, MCP Repo map + transcript Session export/import, stats Limited lifecycle Confirmation tiers Open-source clone-of-Claude-Code lineage with explicit session portability
SWE-agent Research scaffold Custom ACI: file viewer, edit, search, run Repo as filesystem; ACI navigation Trajectory log None standard; researcher-defined Container/VM Agent-Computer Interface as the design lever
LangGraph Orchestration runtime Tools as graph nodes; user-defined Graph state; user-defined retrieval Durable checkpointers (SQLite/Postgres/Redis) Explicit graph edges, interrupts, subgraphs User-defined; runs in user's runtime Stateful graph orchestration with durable execution and human-in-the-loop

A few cross-cutting observations.

The architectural distance between Claude Code and Codex CLI is small. Both are terminal-first coding agents with built-in core tools, MCP extension, lifecycle hooks, AGENTS.md/CLAUDE.md-shaped repo context, and tiered approval modes. Cosmetic and ecosystem differences (plugin marketplace, skill registry, default model) loom larger than architectural ones. They represent the same pole.

Cursor and Continue are not architecturally alike despite both being IDE agents. Cursor's distinctive feature is depth of editor integration — tab completion, inline diffs, agent panel, background agents — running on a closed product with embedding-based retrieval as the context primary. Continue is open-source and modular, with an emphasis on configurable context providers and team-shareable configuration (Continue Documentation). They occupy similar product surfaces but different architectural philosophies.

Aider is the most coherent thin harness in the lineup. Its repo map and git-backed edit workflow (Aider Documentation) are not features added on top of a chat loop; they are the workflow. The model proposes edits as diffs, Aider applies them as commits, the user can /undo to revert. The persistence layer is git. The harness does almost nothing the model could plausibly do better.

SWE-agent is best understood as the research-paper instantiation of the harness-matters thesis. Its primary contribution is the argument that interface design changes the agent's competence on real tasks (Yang et al. 2024). It is not a production harness, but its claim — that a custom ACI is the lever — has shaped how every coding agent since has thought about tool surfaces.

LangGraph is in a different category. It is not a coding agent; it is an orchestration runtime that one could use to build a coding agent, a customer support agent, or a research workflow. Its load-bearing properties are durable execution, persistent memory, time-travel debugging, human-in-the-loop, and observability (LangChain Docs). Comparing LangGraph to Claude Code as "two harnesses" is correct only at high altitude; at the architectural level it is closer to comparing an application framework to an application.

Thin harness, smart model versus thick harness, structured model

The framing dispute that animates harness design is whether a strong model with a thin runtime outperforms a structured runtime that constrains the model into explicit roles, states, and transitions. It is a real design tension. It is also frequently miscast.

The thin-harness case

The thin-harness argument is that frontier models are now capable enough to handle the search, edit, test, and recovery loop on their own, and that wrapping them in heavier orchestration adds drag without adding capability. Anthropic's "Building Effective Agents" essay is the canonical statement of this view, with two specific claims (Anthropic Engineering):

  1. Successful production agents are usually built on simple, composable patterns rather than complex frameworks. The essay distinguishes workflows (predefined LLM and tool orchestration) from agents (dynamic, model-directed tool use), and recommends starting with the simplest pattern that solves the task.
  2. Frameworks add layers of abstraction that obscure the underlying prompts and responses, making them harder to debug.

The essay is careful: it does not argue that simpler is always better. It argues that complexity should be added only when measurable performance gains justify it. Anthropic's own SWE-bench agent, the essay reports, was a simple loop with carefully designed tools; the team spent more time engineering tools than the overall prompt.

Practical evidence supports the thin-harness pole for interactive coding tasks. The official SWE-bench leaderboard's mini-SWE-agent entry — roughly 100 lines of Python with a clean ACI — has scored in the high 60s on SWE-bench Verified with strong frontier models, demonstrating that minimal scaffolds can go far when the model is capable (SWE-bench Leaderboards). Aider's repo-map-plus-git workflow has consistently ranked near the top of independent coding-agent comparisons despite being a small, focused tool.

The Agentless paper makes the strongest version of this argument by inverting it: it shows that for some software-engineering tasks, the best move is to remove agentic decision-making entirely. Agentless uses a fixed three-stage pipeline — localize, repair, validate — without letting the LLM decide its next action or use complex tools. On SWE-bench Lite it reported strong performance and low cost relative to autonomous agents, while also surfacing benchmark-quality issues such as ground-truth leakage and underspecified problem descriptions (Xia et al. 2024). The implication is sharper than "thin harness wins": for some tasks, the harness should be thin enough that the agent isn't really an agent.

The thick-harness case

The thick-harness argument is that any agent operating outside the narrow benchmark setting — long-running workflows, multi-step pipelines, regulated environments, multi-agent collaboration — needs explicit state, durability, and policy that simple loops cannot provide. LangGraph is the canonical reference. Its documentation lists durable execution, human-in-the-loop interrupts, comprehensive memory, and built-in observability as the load-bearing properties for production agentic systems (LangChain Docs).

The thick-harness pole's evidence is mostly operational rather than benchmark-driven. AgentDojo's prompt-injection results show that agents reading untrusted tool output need structural isolation — a mechanism that thin loops do not naturally provide and graph-structured runtimes do (Debenedetti et al. 2024). The OWASP MCP Top 10 catalogues failure modes that all require harness-level enforcement: token scope, input validation, audit logging, command injection prevention (OWASP Foundation). When the agent is running unattended, signing emails, transferring money, or modifying production infrastructure, "let the model decide" is not a security model. The thick harness is the control plane.

The thick pole also has an underappreciated benchmark angle. SWE-Effi (2025) measures not just task pass rate but token cost, wall time, and recovery from failure (SWE-Effi, arXiv 2509.09853). On these axes, structured pipelines often beat free-form agentic loops on the same model: deterministic stages waste fewer tokens on dead-end exploration, and explicit checkpoints recover faster from partial failures.

Where the dichotomy fails

The thin-thick framing is too clean to survive close reading. Three reasons.

Claude Code is not actually a thin harness. The publicly documented feature surface includes lifecycle hooks across many events, plugin marketplaces, skill files, MCP servers, subagents, agent teams, sandbox tiers, persistent CLAUDE.md context, and remote sessions (Anthropic Code Docs). The core loop is simple — model proposes tool calls, harness executes them, repeats. The surrounding system is thick. What makes Claude Code feel thin is that the thickness is opt-in and project-specific: a fresh repo runs on the simple core; a mature one accumulates hooks, skills, and policy. This is closer to "thin core, thick periphery" than to "thin harness."

Thickness is not graph-shaped. The thick-harness pole is often equated with LangGraph-style orchestration: explicit nodes, edges, state machines, conditional routing. But thickness can also mean approval policies, sandbox modes, audit logs, replay, scoped credentials, telemetry, multi-tenant isolation. Codex CLI is "thick" in the policy and sandboxing dimensions while remaining a simple loop architecturally (OpenAI Developers). A harness can be thick in operational properties without being graph-orchestrated.

The right question is layered, not binary. A coherent harness usually has a thin core (the model-tool-result loop) wrapped in thick peripheries for the dimensions that matter to the use case: containment, audit, recovery, multi-step coordination. Claude Code's core is thin; its hook and skill periphery is thick. LangGraph's core is graph-orchestrated (thick by default); its tool layer is whatever the user implements (could be thin). The dichotomy treats one axis as the whole space.

The actual axis

The more useful axis is what should be model-discretionary versus what should be deterministic. For each dimension of agent behavior — tool selection, control flow, retry policy, error recovery, validation, approval — the question is whether the model decides at runtime or whether the harness constrains it.

Dimension Model-discretionary Deterministic When deterministic wins
Tool selection Model chooses freely from registry Restricted to specific tools per phase Stage-specific phases (e.g., gather-then-edit)
Control flow Model decides next action Graph or pipeline determines next node Multi-step workflows with hard ordering
Retry policy Model decides whether to retry Harness retries with backoff Idempotent operations, transient failures
Error recovery Model handles errors in-context Harness routes errors to handlers Errors with fixed remediation paths
Validation Model self-checks output Harness validates against schema/tests Output must match specification
Approval Model proceeds Harness gates on human signoff Irreversible or sensitive actions

This axis is more useful than thin-versus-thick because it produces specific design questions instead of camp loyalty. A harness should be thin where the model's flexibility is an asset and thick where determinism is a hard requirement. Most production agents end up mixed.

Empirical evidence at the 2026 frontier

The state of evidence on harness performance is unsettled, and it is unsettled in ways that directly bear on how confidently any general claim can be made.

SWE-bench is now weak evidence for general harness comparison

SWE-bench, and especially SWE-bench Verified, has been the dominant public benchmark for agent coding capability. Several developments have eroded its credibility for cross-harness comparisons in 2025-2026.

OpenAI's February 2026 analysis announced that the lab would no longer evaluate on SWE-bench Verified, citing test contamination and benchmark integrity issues. Their audit of often-failed tasks found that at least 59.4% of an audited subset had flawed tests rejecting correct solutions, meaning leaderboard scores were systematically depressed for harnesses that produced correct-but-test-failing patches and inflated for harnesses that happened to overfit to the flawed tests (OpenAI). Epoch AI's methodology page on SWE-bench Verified now explicitly notes that scaffolding and tooling are part of the methodology and that scaffold upgrades shift scores significantly, making cross-submission comparisons unreliable (Epoch AI).

Independent analyses have piled on. UTBoost (2025) found insufficient tests across SWE-bench, with corrections affecting 40.9% of SWE-bench Lite leaderboard entries and 24.4% of SWE-bench Verified entries, causing ranking changes (UTBoost, arXiv 2506.09289). "Saving SWE-Bench" (2025) argued that existing GitHub-issue benchmarks overestimate chat-based coding-agent ability, in some public cases by more than 50% over baseline (Saving SWE-Bench, arXiv 2510.08996). SWE-rebench documented contamination drift: as benchmark tasks age, models trained on subsequent crawls have effectively seen the answers (SWE-rebench). "Dissecting the SWE-Bench Leaderboards" (2025) noted that many submissions lack architectural documentation, making it impossible to attribute success to design choices, and found both agentic and non-agentic designs among top entries (Dissecting SWE-Bench, arXiv 2506.17208).

The cumulative effect is that SWE-bench numbers should be read as weak signal for general harness comparison: useful for same-scaffold same-model ablations, unreliable for cross-harness rankings, and especially unreliable for any "thin versus thick" general claim.

Recent ablations specifically on harness thickness

A small but growing body of work has tried to isolate harness contributions by holding the model constant.

ContextBench (2026) measured the effect of various retrieval and context-management scaffolds on agent task performance with a fixed model. The headline result was that sophisticated scaffolding produced only marginal gains over simpler approaches when the model was capable; the gains scaled with task difficulty but flattened quickly (ContextBench, arXiv 2602.05892).

SWE-Skills-Bench (2026) evaluated whether "skill" injections — pre-written guidance for specific repo-engineering subtasks, in the Claude Code skill or Codex AGENTS.md style — improved performance. Of 49 injected skills tested, 39 produced zero pass-rate improvement; average gain across all skills was +1.2%; token overhead rose substantially in some cases (SWE-Skills-Bench, arXiv 2603.15401). The authors concluded that skill injection should be treated as an intervention to test, not a virtue to assume.

AGENTS.md evaluation (2026) specifically studied the effect of static repo context files (AGENTS.md, CLAUDE.md, .cursorrules) on agent task success and cost. The finding was non-monotonic: minimal files (a few hundred tokens of essential conventions) helped; large files (several thousand tokens describing repo structure) often hurt both pass rate and cost (AGENTS.md evaluation, arXiv 2602.11988). The proposed mechanism is context dilution: static context displaces the agentically-retrieved, task-relevant context that the model would otherwise gather.

SWE-Effi (2025) introduced cost-aware metrics — pass rate, token cost, wall time, recovery quality — and found that on some configurations, deterministic pipelines beat free-form agentic loops on token efficiency by large margins, even when raw pass rate was similar or slightly lower (SWE-Effi, arXiv 2509.09853).

These results converge on a few honest claims:

  1. Strong models do not benefit much from thicker context machinery in standard coding tasks.
  2. Static context files are easily overdone and often counterproductive past a small threshold.
  3. Deterministic pipelines are more token-efficient than autonomous loops, sometimes substantially.
  4. Skill and prompt injections are interventions to be tested, not assumed to help.

What none of them establish is a general winner between thin and thick. The evidence is task-dependent, model-dependent, and confounded by scaffold-specific choices.

What the evidence does and does not support

The defensible empirical position as of mid-2026:

Claim Support
Harness design is a measurable lever on agent performance, safety, and operational behavior Strong (SWE-agent ACI; AgentDojo; SWE-Effi)
For interactive coding with a strong model, a thin harness with good tools and a small repo-context file is a competitive baseline Strong (Aider, mini-SWE-agent, AGENTS.md study)
Adding more orchestration nodes, larger context files, or more skill injections helps Weak; often null or negative (ContextBench, SWE-Skills-Bench, AGENTS.md study)
Durable orchestration (LangGraph-style) is necessary for long-running, fault-tolerant, audited workflows Strong, but operational rather than benchmark-driven
Thick harnesses are necessary for multi-agent coordination at the production level Moderate; mostly inferred from production failure modes rather than controlled studies
Thin harnesses are generally better than thick ones at the 2026 frontier Not supported as a general claim
Thick harnesses are generally better than thin ones at the 2026 frontier Not supported as a general claim

The honest summary is that harness thickness is a design dial keyed to the task, not a global ranking. Interactive coding with a trusted human in the loop favors the thin end. Long-running, governed, multi-step production workflows favor the thick end. Most real systems live somewhere in between and add thickness on the dimensions that matter to them.

Choosing a harness for a new use case

The practitioner question is rarely "which harness is best?" It is "which harness fits this task, this team, this risk tolerance, this integration surface?" A useful framework is to make the choice along five axes, each of which has clear best-fit harness clusters.

Axis 1: Autonomy level

How long does the agent run without human intervention?

  • Conversational (every turn): Cursor agent, Continue, Claude Code interactive, Codex CLI interactive. The harness optimizes for fast feedback and easy human override.
  • Task-scoped (minutes-to-hours): Claude Code with acceptEdits, Codex CLI with workspace-write, Aider with auto-commit. The harness optimizes for productive autonomy within a single task with persistent transcript.
  • Long-running (hours-to-days): LangGraph or custom production runtime. The harness optimizes for durability, recovery, and human-approval gates at specified checkpoints.
  • Always-on (days-to-weeks): Custom production runtime with task queues, scheduled triggers, monitoring. Existing coding harnesses do not target this regime.

Axis 2: Risk tolerance

What is the cost of a wrong action?

  • Local-only, easily reversible: edits to a project workspace, bench experiments. Thin harnesses with default approval modes work well.
  • Networked, mostly reversible: API calls to external services, file uploads. Sandbox tier matters; approval gates on first call to each new endpoint.
  • Production, partially reversible: deploys, database writes, customer-facing changes. Harness must enforce approval gates, audit logs, scoped credentials, and rollback.
  • Production, irreversible: fund transfers, irreversible API actions (sending email, posting to public channels). Harness must enforce structural isolation, multi-step approval, and after-the-fact audit. Thin harnesses are inappropriate; durable orchestration with explicit approval graphs is the right shape.

Axis 3: Integration surface

Where does the agent need to live?

  • Terminal-only: Claude Code, Codex CLI, Aider, OpenCode. Best for backend developers, automation engineers, ops.
  • IDE-embedded: Cursor, Continue, Claude Code IDE extensions. Best for active development workflows where edits and inspections happen in the editor.
  • API/service: LangGraph, custom runtime. Best for agents exposed as services or running in shared infrastructure.
  • Multi-modal (browser, image, voice): depends on tool availability via MCP or custom adapters. The harness's MCP support is the relevant feature.

Axis 4: Auditability requirements

How much do you need to be able to reconstruct after the fact?

  • None to minimal: a transcript log is enough. Most CLI coding harnesses suffice.
  • Replayable: need to re-run the agent against past inputs and see what it did. Codex transcript resume, OpenCode session export, LangGraph time-travel all serve this.
  • Compliance-grade: need durable, signed, queryable event log with model version, prompt, tool calls, results. LangGraph + LangSmith or custom durable runtime.

Axis 5: Team coordination

Who is using the agent, and how do they share knowledge with it?

  • Solo or small team, ad hoc: any harness; CLAUDE.md / AGENTS.md / .cursorrules work fine.
  • Larger team, shared conventions: harness must support team-shareable configuration. Continue's open architecture, Codex's AGENTS.md, Claude Code's plugin marketplace and skill registry all aim here.
  • Enterprise, governed: harness must support managed config, allowlists, tenancy, RBAC, audit. Codex's enterprise mode and Cursor's team controls target this; production runtimes built on LangGraph also.

Selection patterns

A few patterns recur in real adoption.

Default for individual coding work: start with Aider, Claude Code, or Codex CLI. They share the same architectural pole — strong model, thin runtime, minimal repo context, agentic search, lifecycle hooks if needed. The choice between them is mostly ergonomic and ecosystem-driven.

Default for IDE-centric coding work: Cursor or Continue, depending on whether closed-product depth or open-source modularity matters more. Both occupy the same architectural niche.

Default for production agent backends: LangGraph or a custom durable runtime. Coding-agent harnesses are not designed for this and lack the persistence, retry, and audit primitives.

Default for safety-sensitive work: whichever harness can enforce structural isolation between trusted and untrusted contexts. Currently, this is more readily implemented in LangGraph (via subgraphs and explicit state isolation) than in coding-agent harnesses, though Claude Code's subagent and skill-injection patterns can approximate it with effort.

Default for benchmarking and research: SWE-agent, mini-SWE-agent, or a custom thin scaffold designed to isolate the variable being studied. Production harnesses are too feature-rich for clean experiments.

Default when uncertain: run the cheapest decisive experiment. Pick 20-50 representative tasks. Run two or three candidate harnesses against them with the same model, same time budget, same allowed tools. Measure pass rate, cost, wall time, manual interventions, and unsafe actions. Most general claims about harness fit dissolve under this kind of same-model bakeoff.

Failure modes and design heuristics

A few patterns repeatedly bite teams adopting agent harnesses, regardless of which one they choose.

Context dilution

Adding more context tends to feel safer than removing it, but adding context past a small threshold often degrades performance. The AGENTS.md evaluation showed this empirically (AGENTS.md evaluation, arXiv 2602.11988). The mechanism: static context displaces the model's ability to gather task-relevant context agentically, and large static blocks distract from the actual prompt. The heuristic: a few hundred tokens of essential conventions is usually right; multi-thousand-token AGENTS.md or CLAUDE.md files are a smell.

Skill bloat

Skill or rule injection — the modular guidance pattern that Claude Code skills, Codex AGENTS.md sections, and Cursor rules all instantiate — is easy to overdo. SWE-Skills-Bench's finding that 39 of 49 skills produced zero pass-rate improvement is the warning sign (SWE-Skills-Bench, arXiv 2603.15401). Skills should be added based on measured effect on actual task success, not on the assumption that more guidance is better. Audit periodically: a skill that has not measurably helped in the last N tasks should be removed.

Hook spaghetti

Hook systems are powerful and dangerous. A project with thirty hooks firing on overlapping events becomes hard to debug, and hook-induced latency starts to compete with model latency. The heuristic: hooks should be reserved for properties that the model cannot enforce on its own — security gates, telemetry, durable persistence, hard determinism — not for behaviors the model should learn. If a hook exists to compensate for the model "forgetting" to do something, the right fix is usually CLAUDE.md / system-prompt guidance, not a hook.

Sandbox confusion

The line between approval modes is often softer than the documentation suggests. A harness in acceptEdits mode can still be tricked into running a tool that escalates beyond the intended scope, particularly if the model is acting on untrusted tool returns. The heuristic: treat the sandbox as defense-in-depth, not the security boundary. Pair it with scoped credentials, explicit deny lists for irreversible operations, and structural isolation for any tool that consumes external content.

Persistence drift

Long-running sessions accumulate state — compacted summaries, working memory, partial plans — that quietly drifts from the underlying repository state. The agent ends up planning against a snapshot that no longer matches reality. The heuristic: explicit re-grounding at major phase boundaries (deploy, review convergence, benchmark run), with fresh context and a clear hand-off summary. Most coding harnesses do not do this automatically; it has to be a workflow practice.

Benchmark-driven selection

Choosing a harness based on its SWE-bench Verified score is currently a poor signal. Beyond the contamination and test-quality issues already documented (OpenAI; UTBoost, arXiv 2506.09289; Saving SWE-Bench, arXiv 2510.08996), benchmark scores reflect a particular task distribution (GitHub issues with passing tests as ground truth) that may not match the team's workload. The same-model bakeoff on representative internal tasks is more reliable.

The "smart model solves it" reflex

When something goes wrong, the temptation is to wait for the next model upgrade rather than fix the harness. This works for capability problems (the model could not solve the task). It does not work for operational problems: prompt injection, credential exposure, non-idempotent tool calls, audit gaps, recovery failures. These are properties of the surrounding software, not of the model. A frontier model on a leaky harness is a frontier-capability liability.

A summary thesis

The agent harness is the engineering layer that turns model capability into operational behavior. It is not where the intelligence lives, and at the 2026 frontier no benchmark or controlled study has established a general winner between thin and thick designs. What the evidence does support is that interactive coding with a strong model favors a thin core with good tools, minimal static context, and selective skill use; that long-running, governed, or safety-sensitive workflows require thicker peripheries — durable persistence, structural isolation, explicit approval, replayable audit — that thin coding harnesses do not natively provide; and that adding mass to the harness is an intervention to test, not a default to assume.

The practitioner's question is not which harness wins but which dimensions of agent behavior the use case needs to be deterministic and which can stay model-discretionary. Answering that produces a harness shape; the available products are, in mid-2026, close enough to interchangeable along their architectural poles that fit-to-task and fit-to-team usually decide the choice.

Companion entries

Core theory:

Practice:

Counterarguments:

  • Agentless and Non-Autonomous Pipelines
  • SWE-bench Verified Contamination 2026
  • The Limits of Skill Injection
  • Static Context Files Considered Harmful

Adjacent infrastructure:

  • LangGraph Durable Execution Model
  • Prompt Injection in Tool-Using Agents
  • OWASP MCP Top 10
  • Observability and Replay for Agent Systems

Benchmark methodology:

  • ContextBench
  • SWE-Skills-Bench
  • SWE-Effi Cost-Aware Metrics
  • Dissecting the SWE-Bench Leaderboards

AI-researched reference article. Something wrong here? Tell us.