PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems

Paul Barbaste, Tristan Darrigol, Germain Vu, Tom Wiltberger

arXiv:2609.00006v1cs.SEcs.MA

TL;DR

Existing literature lacks a production-source account of coding-harness architecture and evolution. This paper develops that reference through a source-level study of eleven systems, finding recurring design patterns, persistent architectural absences, and a transition from tool to platform.

  • Problem

    Existing work reports benchmark scores or describes conceptual patterns without examining the engineering decisions that make harnesses reliable, extensible, or deployable.

  • Method

    The paper defines a harness and its seven canonical subsystems, then systematically compares eleven coding harnesses at source level alongside a meta-harness contrast point and a longitudinal source diff.

  • Results

    The audit yields 13 cross-cutting observations and 29 recurring design patterns, while the longitudinal analysis documents convergence becoming imitation and behavioral policy migrating from prompt prose to configuration.

  • Takeaways & Limitations

    The study’s central conclusion is that the coding harness has stopped being a tool and become a platform.

  • Takeaways & Limitations

    The Claude Code analysis relies on a publicly circulated March 2026 source snapshot rather than an official release, creating an asymmetry with the other ten systems.

Abstract

from arXiv · show

An agent is a model plus a harness -- the runtime that couples an LLM to the world through a loop, tools, context management, safety controls, orchestration, and extension surfaces. Harness engineering, named as a discipline in early 2026, is the design and evolution of that runtime. This paper gives the young discipline its most comprehensive empirical foundation to date: a source-code anatomy of eleven production coding harnesses (Claude Code, Codex CLI, Gemini CLI, Mistral Vibe, OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, OpenClaw), plus Omnigent, the first meta-harness, analyzed as a contrast point. We define what a harness is, map its seven canonical subsystems with the minimal and maximal implementation of each, and dissect all eleven systems along those subsystems. The audit yields 13 cross-cutting observations and a catalog of 29 recurring design patterns. Two absences survive a threefold corpus expansion: across roughly four million lines of Python, TypeScript, and Rust, no agent runtime imports a general-purpose agentic framework, and none retrieves code with vector embeddings; the field runs on hand-rolled async loops and deterministic retrieval. SKILL.md skills lead MCP in adoption (9/11 vs. 8/11), and ACP ships in six systems with a new third role: harness hosting. Because the original eight systems were re-pinned rather than replaced, the study also contains a controlled longitudinal sample -- the same harnesses source-diffed across one quarter -- showing convergence becoming imitation and behavioral policy migrating from prompt prose to configuration. These threads converge on the paper's thesis: in the first half of 2026 the coding harness completed a turn from tool to platform. The paper closes with 18 design recommendations and a 90-line minimum-viable-harness scaffold.

1 Introduction

The paper establishes harness engineering as a source-code-grounded discipline and argues that coding harnesses have become platforms rather than mere tools. It compares eleven production systems, identifies recurring architectural patterns and absences, and tracks evolution across one quarter.

  • Existing literature reports benchmark scores or conceptual patterns but does not examine the engineering decisions governing harness reliability, extensibility, safety, and context management.
  • The study analyzes eleven production harnesses spanning four provider-native systems and seven open-source projects.
  • The paper defines a reference anatomy, compares the eleven systems systematically, and catalogs 13 cross-cutting observations and 29 recurring design patterns.
  • The retained April snapshots create a controlled longitudinal sample for measuring ninety days of harness evolution in source code.
  • The longitudinal analysis reports convergence becoming imitation, diffusion of patterns, and migration of behavioral policy from prompt prose to configuration.
  • The paper extends its analysis into a platform-turn thesis, practitioner recommendations, and a 90-line minimum-viable-harness scaffold.

2 What Is a Harness?

The paper defines a harness as the runtime surrounding an LLM and organizes it into seven canonical subsystems plus shared interface and session surfaces. The corpus shows that these subsystems range from a roughly 100-line floor to highly engineered production implementations.

  • Harness engineering designs and evolves the runtime that couples an LLM to the world through loops, tools, context, safety, orchestration, and extensions.
  • The paper distinguishes a shipped harness from a scaffold, an imported agentic framework, an evaluation harness, and an orchestrator.
  • All corpus systems implement the seven subsystems, including deliberate absence where a subsystem is unnecessary or omitted.
  • Table 1 maps each subsystem to its function, minimal implementation, maximal observed form, and corresponding paper section.
  • The interface layer and session substrate sit alongside the seven subsystems, supporting human or program control, persistence, resume, and fork operations.
  • Mini-SWE-Agent implements all seven subsystems in roughly 100 lines, while reporting SWE-bench Verified results in the same range as systems three orders of magnitude larger.

3 Background and Related Work

Prior work covers coding-agent benchmarks, conceptual architectures, tool use, and multi-agent coordination, but source-level implementation evidence remains limited. This paper expands that evidence against a rapidly growing July 2026 landscape and related engineering guidance.

  • SWE-Bench and SWE-Agent established benchmark and tool-interface foundations, while later systems added AST-aware retrieval, modular sub-agents, role-based coordination, and trained coding agents.
  • Architectural surveys identify components such as tools, memory, and planning but generally describe what exists rather than how production systems implement it.
  • The closest source-code taxonomy studies thirteen open-source scaffolds across twelve dimensions, whereas this paper includes provider-native systems and extends coverage to safety, orchestration, and extensibility.
  • Multi-agent research provides role-based, conversational, and orchestrator frameworks that contextualize the paper’s analysis of coordination mechanisms.
  • MCP supplies a standard tool-and-data interface, and the paper examines its emergence across coding harnesses.
  • Vendor guidance offers a prescriptive baseline contrasting predefined workflows with LLM-directed agents and enumerating composable agent patterns.
  • The July 2026 corpus is a deliberate sample from a field that has expanded beyond two dozen actively maintained harnesses.
  • The landscape includes provider consolidation, prominent open-source centers of gravity, and a meta-layer that treats whole harnesses as interchangeable components.

4 Methodology

The methodology selects eleven harnesses plus Omnigent as a contrast point across design philosophy, maturity, and market position, then analyzes each system along seven dimensions. Exact July 2026 pins and retained April snapshots support both cross-sectional comparison and longitudinal source diffs.

  • The corpus contains eleven harnesses plus Omnigent, selected to span design philosophy, maturity, and market position.
  • The sample includes one harness per major commercial LLM provider and seven open-source projects occupying distinct research and production niches.
  • OpenClaw is included as an external contrast point because it is a multi-channel assistant gateway that delegates coding through plugins rather than implementing native code editing.
  • Omnigent is analyzed above the corpus as a meta-harness because it wraps harnesses behind a common API without implementing its own editing loop.
  • Each system is analyzed across agent loops, model integration, tools, memory and context, safety, multi-agent orchestration, and extensibility.
  • Exact July 2026 release pins are compared with retained April snapshots for the eight carried-over systems, enabling source-level analysis of three months of evolution.

5 System Overview

The eleven-system corpus spans three orders of magnitude in code size, three programming languages, and fundamentally different design philosophies. Table 4 compares each system at a high level and identifies its distinctive signature capability, while noting source-snapshot and benchmark limitations.

  • The corpus spans three orders of magnitude in code size, three programming languages, and fundamentally different design philosophies.
  • The study removed SWE-Bench Verified scores because the available figures are self-reported, use different model generations and configurations, and sometimes predate current defaults.
  • Table 4 compares eleven systems and names each system’s signature capability—the capability no other corpus system implements in the same form.
  • Claude Code’s table entries use the March-2026 source snapshot, so the July 2026 shipping binary may differ.

6 Agent Loop Design

All eleven systems implement ReAct variations, organized into iterative action-observation, reflection-augmented, and coordinator-worker overlay paradigms. Their loops range from minimal linear control flow to event-sourced, streaming, middleware, budgeted, and concurrent implementations.

  • Taxonomy of Agent Loop Paradigms: All eleven systems implement ReAct variations, but they differ radically in complexity, concurrency, and state management.
  • Taxonomy of Agent Loop Paradigms: Nine systems use iterative action-observation loops, Aider uses reflection augmentation, and coordinator-worker overlays appear in Claude Code, Codex, and configuration-gated Hermes.
  • Iterative Action-Observation Loop: OpenHands persists events in a locked, redacted event log, supports replay and branching, and can execute tool batches in parallel under resource-aware limits.
  • Iterative Action-Observation Loop: Claude Code batches tools by concurrency safety, defaulting tools to unsafe unless they explicitly opt into parallel execution.
  • Iterative Action-Observation Loop: Mistral Vibe factors turn-level policies into middleware profiles, while plan mode restricts writes and gates re-entry through a plan-review event.
  • Reflection-Augmented Loop: Aider feeds lint, compilation, test, and file-mention problems back to the model for up to three corrective reflections, whereas Pi documents the absence of planners, caps, and stuck detection.

7 LLM Integration and Model–Agent Co-design

The eleven harnesses span tight provider coupling to deliberate multi-provider abstractions, while model-specific optimizations depend on centrally paying the conditional-code cost. Prompt construction and behavioral rhetoric vary widely, but independently developed systems converge on controls for tool use, verbosity, surface forms, and scope discipline.

  • Provider abstraction spectrum: The corpus spans five LLM-integration strategies, from single-provider tight coupling to provider-agnostic implementations.The systems differ in how readily they switch providers and in which provider-specific optimizations they can exploit.
  • Provider abstraction spectrum: Mistral Vibe privileges its home provider while offering generic fallback backends for Anthropic, Vertex, OpenAI-compatible, and OpenAI Responses endpoints.Its Mistral backend exposes reasoning_effort and ThinkChunk features that the generic backend does not specialize for.
  • Provider abstraction spectrum: Provider-native optimizations depend on deliberately centralized conditional code, not necessarily on tight coupling.Hermes, Pi, and OpenCode implement cache and reasoning-related behavior across multiple providers through deliberate provider-specific handling.
  • Prompt engineering architectures: Pi’s approximately 170-line prompt builder emits 30–40 lines that co-vary with the active toolset, while delegating most behavioral policy to context files and extensions.Its generated prompt includes tool guidance, project context from AGENTS.md, and an available-skills index.
  • Prompt content and rhetorical style: Nine of eleven prompts contain explicit verbosity directives, whereas OpenHands largely leaves response length to the model.OpenCode uses a fewer-than-four-lines rule, while Mistral Vibe retains a 150-word budget.
  • Prompt content and rhetorical style: The prompts contain operational-risk guidance but no explicit harmful-request refusal language.Refusal-shaped instructions address destructive git operations, secrets hygiene, and dangerous shell patterns rather than policy-level harms.
  • Prompt content and rhetorical style: Prompt rhetoric converges around anti-gold-plating, operational safety, verbosity, and formatting controls, but these rules evolve across systems and snapshots.The corpus reports shared anti-gold-plating language, divergent commit policies, model-dependent banned phrases, and differing verbosity directives.

8 Tool and Action Systems

Tool and action systems range from a single shell call to large, typed, platform-scoped tool surfaces, with deferred loading increasingly used to manage context costs. File editing has reorganized around strict contracts, fuzzy cascades, and model-aware tool selection, while sandboxing spans multiple operating systems.

  • Tool count spectrum: Tool surfaces range from a single bash tool to 109+ tools, with Pi emphasizing sparse defaults and Hermes exposing 69 built-ins through platform-specific scoping.Pi builds seven tools but exposes four by default; Hermes registers 69 and uses a toolset layer to control visibility.
  • Tool interfaces: Tool interfaces range from Mini-SWE-Agent’s single shell call to rich typed contracts separating validation, permissions, concurrency, and rendering.The corpus therefore covers both minimal execution wrappers and mature registries with multiple interface concerns.
  • Deferred tool loading: Deferred loading increasingly reduces prompt exposure by searching or bridging large tool catalogs instead of presenting every schema initially.Codex uses BM25 search, Hermes collapses oversized MCP and plugin schemas into bridge tools, and OpenCode defers skills and experimental MCP catalogs.
  • File editing strategies: File-editing strategies reorganized across snapshots, with Mistral Vibe adopting strict unique-substring replacement and Gemini CLI adding model-assisted repair after cascading matches fail.The change is presented as evidence that stronger models can favor strict contracts over tool-side drift tolerance.
  • File editing strategies: OpenCode and Hermes use elaborate fuzzy replacement cascades, while Aider’s RelativeIndenter makes indentation-relative edits across nesting levels.OpenCode’s cascade includes multiple normalization stages and a disproportionate-match guard; Aider reconstructs absolute indentation after applying relative changes.
  • File editing strategies: Model-aware polymorphism now spans Aider and OpenCode, which select editing formats or toolsets according to the target model.OpenCode gives GPT-family models a Codex-style patch DSL and removes string-replacement tools, while Aider uses a prompt-class factory.
  • Execution sandboxing: Codex and Gemini CLI implement cross-platform sandboxing across Linux, macOS, and Windows using OS-specific isolation mechanisms.Codex uses Bubblewrap, Seatbelt, and restricted-token processes; Gemini CLI combines Docker or Bubblewrap, Seatbelt, and restricted-token processes with additional policy layers.

9 Memory and Context Management

The eleven systems use four practical strategies to manage context growth, with threshold-triggered LLM compaction becoming the dominant production pattern. Persistent memory has become the main frontier, distinguished by who writes and reviews it rather than by embedding retrieval.

  • Compaction strategies: Seven of eleven systems use threshold-triggered LLM compaction, establishing a de facto production standard for context management.Implementations increasingly hybridize compaction with iterative merging, pluggable engines, or session-tree substrates.
  • Minimal strategy: Mini-SWE-Agent deliberately leaves context unmanaged, allowing the message list to grow within the model’s native context window.This choice supports reproducibility and straightforward trajectory analysis as a research baseline.
  • Compaction implementations: Aider, OpenHands, Claude Code, Codex, Gemini CLI, and Mistral Vibe implement progressively richer compression pipelines with configurable triggers, summaries, preservation rules, or pluggable condensers.Mistral Vibe also supports reactive mid-flight compaction and filesystem rewind; OpenHands event-sources condensation events.
  • Compaction implementations: Hermes preserves history through lineage compaction, ending a SQLite session and creating a child session linked by parent_session_id instead of rewriting the transcript.Lineage traversal and deduplicated session search unify context management with session persistence.
  • Compaction implementations: Pi combines iterative summary merging and extension hooks with an append-only session-tree substrate for compaction.Extensions can veto or replace the compaction result through session_before_compact.
  • Persistent memory: Codex delegates long-term memory maintenance to an agent, while Gemini CLI routes extracted memories through a user-reviewed patch inbox.Hermes instead uses bounded Markdown snapshots and deterministic SQLite FTS5 recall without LLM calls.
  • Persistent memory: Persistent memory now differentiates systems through four governance models: agent-maintained, human-gated, model-direct-but-bounded, and pre-turn agentic recall.None uses embedding-based retrieval as its primary memory substrate; deterministic retrieval such as SQLite full-text search remains the production ceiling.
  • Repository context: OpenHands ingests AGENTS.md, .cursorrules, and nested AGENTS.md files as scoped repository rules.This makes its skills subsystem a cross-ecosystem context-ingestion layer.

10 Safety and Permission Models

Safety architectures range from resource limits and interactive confirmation to layered policy, LLM review, sandboxing, and content-threat analysis. The corpus shows that OS isolation is an optional design choice rather than a consequence of system size, while several systems invest in alternative controls.

  • Safety spectrum: Safety mechanisms become more important as agent autonomy increases, spanning indirect-prompt-injection defenses, risk evaluation, and tool-use controls.The corpus therefore compares safety as a subsystem across very different autonomy levels.
  • Layered permission models: Claude Code uses a three-layer permission system, while background and forked sub-agents receive only the first two layers and unresolved cases escalate to a parent coordinator.Per-agent denial state prevents cross-agent pollution, and repeated refusals eventually trigger prompting.
  • Layered permission models: Codex expanded from policy rules plus OS sandboxing to four layers, adding lifecycle hooks and an LLM Guardian reviewer that fails closed on timeout or malformed output.Its Starlark execution policies support inline match and not_match examples validated at parse time.
  • Sandboxing choices: OS-level sandboxing is a choice rather than a consequence of scale; Hermes and OpenCode are large systems without native isolation primitives.Gemini CLI and Codex implement cross-platform sandbox stacks, whereas Mistral Vibe uses branch-level worktree isolation instead.
  • Alternative controls: OpenHands combines LLM and adversarial analyzers with deterministic pattern and policy-rail analyzers, while OpenClaw adds scoped authorization, rate limits, SSRF enforcement, pairing, and prompt-size limits.These systems distribute safety across analyzers, authorization boundaries, and abuse-prevention controls.
  • Alternative controls: Mistral Vibe classifies profiles from SAFE to YOLO and applies per-pattern tool overrides, but has neither an LLM risk classifier nor OS-level isolation.Its safety model relies on explicit matching, hooks, and user prompts.
  • Alternative controls: Hermes replaces OS isolation with policy-as-code, hardline command floors, deobfuscated pattern matching, content scanning, and an optional auxiliary-LLM gate.Its YOLO setting cannot be changed at runtime by prompt-injected skills, and its threat model emphasizes content-borne attacks.
  • Minimal safety: Aider provides interactive confirmation without automated risk assessment, while its reflection loop catches syntax, compile, and flake8 failures before they propagate.Mini-SWE-Agent limits steps, cost, wall-clock time, and consecutive format errors, a configuration sufficient for benchmarks but inadequate for production.

11 Multi-Agent Orchestration

Nine of eleven systems support multi-agent execution, spanning six orchestration patterns from single-agent designs to recursive and protocol-based coordination. Implementations differ in context inheritance, capability control, communication, and optimization priorities.

  • Taxonomy: Nine of eleven systems support multi-agent execution, making this the corpus’s most architecturally diverse dimension.The architectures diverge substantially across the nine systems.
  • Taxonomy: Six patterns range from single-agent and sequential delegation to parallel sessions, hierarchical fan-out, recursive composition, and cross-process registries.The six patterns are represented by different combinations of systems across the corpus.
  • Codex: Codex uses a hierarchical thread tree with fork-controlled inheritance, typed roles, map-reduce fan-out, and persisted parent-child topology.Its orchestration includes CSV-based fan-out under a shared result contract.
  • Claude Code: Claude Code supports ad-hoc sub-agents, typed definitions, and coordinator mode with recursive composition and prompt-cache-sharing forks.Coordinator mode structures work as Research → Synthesis → Implementation → Verification and restricts workers to 16 tools.
  • Cross-cutting comparison: Coordinator-worker emerges across provider-native and open-source systems, but implementations optimize different goals including cost, isolation, and coordination substrates.Pi deliberately keeps sub-agents in extension space, while Hermes uses a SQLite blackboard and Codex uses typed thread structures.

12 Extensibility Mechanisms

All eleven systems provide extension points, with skills and MCP forming the dominant capability-package and tool-integration layers. Skills have overtaken MCP in adoption and increasingly use deferred loading, conditional activation, registries, and security controls.

  • Skills and MCP: Skills package instructions, scripts, metadata, activation conditions, and tool constraints in discoverable directories, while MCP connects agents to separately running tool processes.The two mechanisms can compose, as shown by Claude Code and Mistral Vibe.
  • Skills: Eight of nine skill adopters defer body loading, fetching full skill content only when invoked or otherwise needed.Implementations include dedicated skill tools, mention detection, disclosure chains, and ordinary file reads.
  • Skills: Conditional activation filters skills by runtime requirements or touched paths, reducing irrelevant context.Examples include OpenClaw requirements, Claude Code paths, and OpenHands PathTrigger rules.
  • Supply chain: Skill distribution now includes hosted registries, trust tiers, quarantine, provenance verification, and agent-authored skills.Remote registries appear in four systems, while Hermes and Gemini CLI provide examples of agent-generated skills.
  • Skills and MCP: Nine of eleven systems implement SKILL.md bundles, compared with eight of eleven supporting MCP.Pi is the explicit skills-positive, MCP-negative holdout; Aider and Mini-SWE-Agent abstain from skills.

13 Cross-Cutting Observations and Implications

The audit identifies recurring architectural patterns, two durable absences, and a changing protocol landscape. Production runtimes hand-roll their loops and deterministic code retrieval, while ACP expands from editor integration toward harness hosting.

  • Pattern catalog: 29 recurring architectural patterns are cataloged across the eleven systems, combining seventeen earlier patterns with twelve added or crystallized in July.Tables 11 and 12 organize the catalog across the expanded corpus.
  • Twin absences: Across roughly 4 M lines, no production agent runtime imports a general-purpose agentic framework.The result survived a threefold corpus expansion and a repeated manifest and import sweep.
  • Twin absences: All eleven systems use hand-rolled host-language loops and custom tool registries rather than general-purpose agentic frameworks.Loops use native asyncio, synchronous Python, JavaScript async primitives, or Tokio, with schemas built in-house.
  • Twin absences: Zero of eleven systems use vector-based retrieval for code, relying instead on deterministic mechanisms summarized in Table 13.The search covered vector stores, embedding libraries, and code-retrieval indicators; exceptions concerned conversation memory.
  • Inter-agent protocols: ACP ships in six of eleven systems and now serves editor integration, harness hosting, and—through A2A—cross-vendor mesh communication.OpenHands and Hermes exemplify inward harness hosting, while Gemini CLI uniquely provides the A2A mesh role in the corpus.
  • Trade-offs: Benchmark figures across Mini-SWE-Agent and Codex are not directly comparable because models, runs, configurations, and sandboxing differ.The study leaves matched-condition performance as an open question.

14 The Platform Turn

The paper argues that coding harnesses completed a turn from tools to platforms by mid-2026. Extension ecosystems, marketplaces, governance, service interfaces, switching costs, and meta-harnesses now surround the agent loop.

  • Platform thesis: The platform turn is visible in converged extension substrates, capability marketplaces, governance layers, importable SDKs, service endpoints, and a meta-harness.The competitive unit becomes the ecosystem surface around the agent loop.
  • Platform signals: Skills function as declarative programs that specify behavior, activation conditions, and permitted tools for an LLM runtime.Nine systems implement the skill layer, with activation examples in Claude Code, OpenHands, and OpenClaw.
  • Platform signals: Hooks and event buses became the dominant extension substrate, reaching nine of eleven systems.Codex, OpenHands, and Mistral Vibe illustrate the spread across lifecycle and tool events.
  • Platform signals: Harnesses now expose platform service surfaces, including HTTP APIs, OpenAI-compatible gateways, hosted runtimes, and client-facing interfaces.OpenCode makes multiple UIs clients of its server, while OpenHands exposes an agent through a compatible gateway.
  • Harness–framework merger: Frameworks and harnesses converge on a shared artifact shape: loop, tools, skills, sub-agents, hooks, MCP, and sessions.The merger proceeds from both terminal products and orchestration libraries.
  • Meta-harness: The meta-harness adds cross-harness composition and policy while deliberately leaving editing loops, repository context, and edit application below its boundary.Its common API preserves per-harness capability differences rather than treating harnesses as equivalent.
  • Evolution: In one quarter, convergence became imitation and distinctive patterns diffused rapidly across systems.Examples include copied hook vocabularies, plugin formats, skills directories, deferred loading, plan modes, and checkpointing.

15 Discussion

The discussion argues that production readiness depends on architectural investments beyond benchmark performance, while safety, orchestration, extensibility, and model coupling remain varied. It also states that the analysis is source-based and includes reproducibility and measurement limits.

  • Architectural convergence: 0 of 11 systems use general-purpose agentic frameworks or vector embeddings over code; all rely on hand-rolled loops and deterministic structural retrieval.The framework and retrieval absences were re-verified across the expanded corpus.
  • Architectural convergence: ∼15× higher token use in multi-agent systems motivates prompt-cache sharing, cache-sharing forks, and thought-stripping optimizations.The comparison is against a single chat baseline, not a single-agent pipeline.
  • Orchestration: Seven of eleven systems implement coordinator-worker patterns, mainly for breadth-first exploration rather than parallel implementation.This observed use is consistent with the cited caution that coding tasks contain fewer truly parallelizable subtasks.
  • Safety: Safety investment varies independently of system size: Codex and Gemini CLI sandbox across platforms, while Hermes and OpenCode lack OS-level isolation.The expanded corpus rejects a structural size-to-sandbox correlation, although native sandboxing remains code-expensive.
  • Threats to validity: The study reads source code rather than measuring runtime speed, and benchmark figures come from systems’ own documentation.The authors state that structural knowledge, not precise performance measurement, is the claim.
  • Threats to validity: Claude Code has weaker reproducibility because its analysis uses a circulated March 2026 snapshot rather than an official release.The other ten systems’ source trees are re-derivable from public Git history.

16 Design Recommendations: A Practitioner’s Guide

The practitioner guide recommends starting with minimal, linear harnesses and adding complexity only for observed needs. Its recommendations prioritize deterministic retrieval, explicit safety policy, selective orchestration, interoperable hosting, and distinct roles for skills and MCP.

  • Foundations: 74%+ on SWE-Bench Verified is reported by Mini-SWE-Agent’s minimal scaffold, supporting a linear while loop before middleware complexity.The upgrade path is a middleware pipeline when three or more independent turn-level policies emerge.
  • Foundations: Start with one bash tool and add tools only when observed failure modes justify them.The guide cites evidence that more tools do not always improve outcomes.
  • Context and tools: ∼40% prompt reduction supports deferred tool loading once tool counts exceed ∼15.Claude Code, Codex, and Hermes use deferred or searched tool exposure to control prompt bloat.
  • Context and tools: Use threshold compaction with a preserved recent tail, incremental summary merging, and reactive overflow handling.Examples include Claude Code’s 13 K-token buffer and Gemini CLI’s 50% trigger preserving the last 30%.
  • Context and tools: Use deterministic code retrieval—ripgrep, glob, tree-sitter, and filesystem traversal—instead of vector-embedding RAG.The evidence reports 0/11 systems using vector embeddings for code retrieval.
  • Safety: For semi-trusted developer contexts, use PLAN/DEFAULT/YOLO approval modes with permission-scope patterns.The recommendation combines Gemini CLI’s modes with Mistral Vibe’s layered permission hierarchy.
  • Safety: For enterprise or automated contexts, use OS-level sandboxing, policy-as-code, and per-agent audit trails.Codex and Gemini CLI provide cross-platform sandbox examples, while Claude Code uses an opt-in reusable sandbox runtime.
  • Safety: Keep a safety floor beneath YOLO by expressing rules as data or dedicated policy files rather than imperative code.Examples include Starlark, TOML, hooks, and last-match-wins rulesets.

17 Conclusion and Future Work

The conclusion presents a source-code anatomy of eleven coding harnesses plus a meta-harness, longitudinal evolution, and a platform-turn thesis. It identifies convergence in architecture and standards, while framing evaluation, causation, safety, interoperability, and economics as future research directions.

  • Conclusion: The study combines eleven harnesses, a meta-harness contrast point, and a one-quarter longitudinal diff to document the field’s turn into platforms.The source-level corpus covers provider-native and open-source designs.
  • Conclusion: Benchmark performance does not predict loop sophistication, but architectural sophistication predicts production readiness dimensions such as safety, user experience, and extensibility.Mini-SWE-Agent’s roughly 50-line loop reports frontier-range benchmark results.
  • Model–agent co-evolution: Provider-native optimizations and multi-provider compatibility are both viable, but only vendors can retune scaffold behavior server-side on model-release day.The update loop, rather than capability alone, determines coupling costs.
  • Protocols and orchestration: Seven systems use hierarchical coordinator-worker patterns, while ACP has expanded from editor integration to harness hosting.Sub-agent coordination remains in-process in eight of nine multi-agent systems.
  • Extensibility: Skills lead MCP at 9/11 versus 8/11, with registries, trust tiers, provenance verification, and cross-vendor discovery.The skills layer also includes agent authors and self-improving or extraction workflows.
  • Twin absences: Across roughly four million lines, no runtime uses a general-purpose agentic framework or RAG over code; systems use hand-rolled loops and deterministic retrieval.The absences survive a threefold corpus expansion and a three-month re-audit.
  • Practical contribution: The analysis yields 18 evidence-anchored recommendations and a 90-line scaffold implementing ten of them directly.The authors describe this as an evidence-based how-to rather than a single-vendor whitepaper.
  • Future work: Future work includes unified evaluation of safety, user experience, cost efficiency, and extensibility alongside correctness.This addresses the stated gap between benchmark performance and production readiness.

A Detailed Comparison Tables

The appendix provides comparison tables and implementation-level snapshots spanning prompts, APIs, tool dispatch, permissions, sandboxing, compaction, and context management. These records expose concrete differences in how harnesses allocate context, concurrency, and safety controls.

  • Comparison tables: Table 16 records the rhetorical content of the eleven system prompts, while Table 17 records advanced API features used by each system.Both tables are dated to July 2026.
  • Comparison tables: Table 18 compares Claude Code and Codex pipelines head-to-head from user request to completed task.The paper presents the comparison as highlighting their divergent orchestration positions.
  • Prompt and tool architecture: Claude Code separates static cached prompt content from dynamic per-turn context and defers undisclosed tools through keyword search.Deferred loading reduces prompt tokens by ∼40%.
  • Prompt and tool architecture: Codex converts 25–30 built-in tools to OpenAI Function format and defers MCP tools with BM25 search.The implementation exposes only a subset initially.
  • API protocols: Claude Code streams through Anthropic’s Messages API, while Codex uses the Responses API with WebSocket primary and SSE fallback.The records also distinguish Claude extended-thinking budgets from Codex reasoning-effort levels.
  • Tool execution: Tool dispatch batches concurrency-safe calls but serializes unsafe calls, with Claude Code allowing up to 10 concurrent executions.Codex uses an execution lock gated by each tool’s supports_parallel flag.
  • Safety architecture: Permission architectures range from Claude Code’s hooks, classifier, and dialog to Codex’s four-layer policy, reviewer, and sandbox stack.Sub-agents skip Claude Code’s interactive dialog layer.
  • Safety architecture: Codex uses Bubblewrap, Seatbelt, or restricted Windows tokens; Claude Code uses an opt-in sandbox runtime with network restrictions and worktrees.The comparison records platform-specific enforcement and branch isolation.
Loading 2609.00006v1…