PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of Harness Design for Coding Agents

Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani, Xiaoyang Wang

arXiv:2609.20804v1cs.AIcs.CLcs.LGcs.SE

TL;DR

Existing coding-harness evaluations often treat systems as monoliths, obscuring the conditional effects of their components. This study varies planning, action space, and context management while holding the execution loop fixed. It finds that component benefits depend on context budget, model capability, and task type.

  • Problem

    Many harness studies evaluate complete systems, leaving unclear which components account for performance differences or when component effects transfer across models, tasks, and budgets.

    Source: §1 · p. 2

  • Method

    The study holds a lightweight harness's execution loop fixed while varying planning, action space, and context-management implementations across models and coding benchmarks.

    Source: §1 · p. 2, §6 · p. 15

  • Results

    Harness-component effects vary by context budget, model capability, and task: context management helps most under tight windows, T4 is most efficient, planning shifts from success support to cost reduction, and tools suit bash proficiency.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    Harness design is a conditional systems problem: select components for the target model, task type, and resource budget rather than adopting them by default.

    Source: §6 · p. 16

    The results estimate effects of the specific implementations studied, not a universally optimal harness; the action-space change bundles multiple interface properties.

    Source: Limitations · p. 16

Abstract

from arXiv · show

Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.

1 Introduction

The paper addresses the lack of component-level evidence about when coding-harness choices help across models, tasks, and resource budgets. It varies three harness components in a controlled study and finds their effects depend on model capability and context budget.

1 Introduction

  • Existing harness evaluations often compare complete systems, confounding whether performance differences arise from planning, tools, context management, or their interaction with the model.Prior evidence had not jointly characterized these implementation-level choices across context-window budgets and model scales on long-horizon coding tasks.Source: §1 · p. 2
  • The study holds the execution loop fixed while varying planning, action space, and context management across four models and two long-horizon coding benchmarks.It tests five context strategies at four context-window budgets and separately ablates planning and action space.Source: §1 · p. 2
  • Context management matters most under tight context-window budgets, preventing overflow from prematurely ending execution and allowing progress to editing and verification.Its accuracy benefit diminishes as the context window expands.Source: §1 · pp. 2–3
  • T4, which stages elision before LLM summarization, provides the strongest context-management efficiency while maintaining similar mean success to other managed strategies.Recoverable elision is rarely used and does not improve accuracy over elision alone.Source: §1 · p. 3
  • Planning raises success at additional cost for weaker models, but for stronger models mainly lowers cost by removing redundant post-edit verification, with small accuracy changes.Source: §1 · p. 3
  • Predefined tools improve performance for models with weaker bash proficiency, while bash-capable models can use bash-only interfaces at lower cost, with trade-offs varying by task type.Predefined tools reduce shell-command reliance; bash-capable models can combine multiple operations per call.Source: §1 · p. 3

2 Harness Design

The harness uses a fixed ReAct execution loop to support controlled comparisons of planning, action interfaces, and context management. Supporting workspace safeguards, diagnostics, and stuck detection remain fixed across ablations.

2 Harness Design

  • The modular harness follows a ReAct loop, while workspace access controls, post-edit diagnostics, and stuck detection stay fixed across ablations.This provides a common execution substrate for varying the three components.Source: §2 · p. 3

2.1 Planning

  • Planning is a persistent, model-maintained task plan injected into turns and updated through a tool; disabling it removes its instructions, reminders, injections, and tool.The comparison estimates the effect of this scaffold, not planning as a general reasoning strategy.Source: §2.1 · p. 3

2.2 Action space

  • The action-space comparison is between predefined workspace tools and bash-only, and includes interface instructions, workspace-state tracking, and validation support.Predefined file tools enforce read-before-write checks and trigger diagnostics after supported edits; bash-only retains bash and auxiliary tools required by other enabled components.Source: §2.2 · p. 3, Table 1 · p. 4

2.3 Context management

  • Context management compacts only the history's middle region, preserving the preamble and a recent window; T4 elides bulky observations above B1 and summarizes older events if history exceeds B2.The five tiers isolate combinations of no management, elision, recall, and summarization; T1–T3 act at B2, while T4 elides at B1 and summarizes at B2.Source: §2.3 · p. 5, Table 2 · p. 5

2.4 Other components

  • Fixed supporting components include workspace safety gates, recoverable tool errors, immediate Python diagnostics, and detection of repeated failing actions.The safety checks constrain workspace paths and edits; diagnostics surface common Python issues, while stuck detection can prompt a changed approach or end a run early.Source: §2.4 · pp. 5–6

3 Experiment

Across four models and two benchmarks, the experiments isolate planning, action-space, and context-management effects while holding the harness execution loop fixed. Context management chiefly helps when windows are tight; the best strategy and component trade-offs vary with model capability and task type.

3.1 Setup

  • The study evaluates three Nemotron-3 sizes and Mistral-Medium-3.5-128B on SWE-Bench Verified and Terminal-Bench 2.1, reporting success rate and mean cost per task.The benchmarks contain 500 verified GitHub issues and 89 command-line tasks, respectively.Source: §3.1 · p. 6

Implementation Details

  • The design evaluates 176 settings: five context-management tiers at four window budgets, plus matched planning and action-space ablations at T4/128k.Planning and action-space comparisons hold the other components fixed; T4/128k is their baseline.Source: Implementation Details · p. 6

3.2 Main Results

  • Across 32 matched comparisons, recall-enabled T2 beats T1 in 15, loses in 14, and ties in three; recall is rarely invoked and does not improve accuracy over elision alone.Among 64 T2 and T4 settings, 36 never invoke recall_event, and the median invocation rate is zero.Source: §3.2 · pp. 8–9
  • At 32k, managed–T0 success gaps reach 35.7 percentage points on SWE-Bench and 9.5 on Terminal-Bench, shrinking to 2.7 and 2.8 points at 128k.T0 overflow falls as budgets grow, while managed tiers have zero overflow failures at every budget.Source: §3.2 · p. 7, Table 3 · p. 7, Table 4 · p. 8
  • T4 maintains success rates comparable to T1–T3 and has the lowest cost in seven of eight model–benchmark panels, with early elision reducing reliance on summarization.Across the four budgets, T4 also has the lowest mean cost per task and lowest average peak-context ratio.Source: §3.2 · pp. 7–8, Figure 4 · p. 10, Figure 5 · p. 10
  • Planning raises success for Nemotron-3 30B but costs more; for Nemotron-3 550B and Mistral, it lowers SWE-Bench cost by about 30% and 32%, with small success-rate decreases.The success-rate decreases are 2.0 and 0.4 percentage points, respectively; the trajectory analysis attributes removed turns largely to post-edit verification.Source: §3.2 · p. 9, Figure 6 · p. 11
  • Predefined tools help weaker models, while bash-only can improve success and reduce cost for bash-capable models; the preferred interface also depends on task type.Mistral’s full tool set improves SWE-Bench success, while bash-only adds success on the more shell-centric Terminal-Bench.Source: §3.2 · pp. 10, 12, Figure 7 · p. 11

4 Analysis

Trajectory analysis connects harness-component effects to distinct changes in execution: context management extends runs, planning changes where they stop, and bash-only changes action granularity. These patterns differ by model and, for context management, by window budget.

Context management

  • At 32k, context management extended median trajectories from 20–30 turns to approximately 50–180, while largely preserving phase ordering and proportions.At 128k, differences across tiers largely diminished; the analysis therefore fixes context management at T4/128k for subsequent comparisons.Source: §4 · p. 13

Planning

  • For Nemotron-3 30B, disabling planning reduced median SWE-Bench trajectories from 40 to 5 turns; 68.6% terminated without an edit, versus 27.8% with planning.Planning helped sustain execution through an initial edit attempt, with the extra turns contributing to higher cost.Source: §4 · pp. 13–14
  • Planning reduced median SWE-Bench trajectories from 108 to 74 turns for Nemotron-3 550B and from 68 to 53 for Mistral-Medium-3.5-128B, mainly by reducing post-edit verification.These models showed little change in runs ending without an edit or failing at localization; planning primarily reduced redundant verification.Source: §4 · p. 14
  • On Terminal-Bench, planning reduced cost by roughly 26% for Nemotron-3 120B and about 40% for Mistral-Medium-3.5-128B, but increased it by 75% for Nemotron-3 30B.The direction depended on whether planning's truncation of long runs outweighed its prevention of premature termination.Source: §4 · p. 14

Action space

  • For Nemotron-3 550B on Terminal-Bench, bash-only shortened median trajectories from 47 to 31 actions while increasing Write-code actions from 16% to 27%.The pattern suggests capable models bundle operations into fewer shell commands, while predefined tools distribute work across more interactions and incremental repair cycles.Source: Figure 9 · p. 13, §4 · p. 14
  • Across models, bash-only reduced repeated patching and shifted Terminal-Bench file-writing toward coarser create-or-replace actions.Mean re-patches fell from 3.3 to 0.4 for Nemotron-3 30B and from 4.6 to 1.5 for 550B; create-or-replace shares rose from 28% to 64% and 51% to 76%, respectively.Source: §4 · p. 14

5 Related Work

Prior work studies coding agents, harnesses, and context management, but bundled systems and narrow evaluations leave component effects across models and context budgets incompletely characterized. This study frames controlled comparisons as a way to examine those conditional effects.

5.1 Coding Agents

  • Execution-based benchmarks score complete coding-agent systems, conflating model capability with the control loop, action interface, and context-management policy.This motivates controlled comparisons that hold the agent substrate fixed while varying one component.Source: §5.1 · p. 14

5.2 Coding Harness

  • Coding harnesses combine the control loop, tool interface, and context management, while most systems optimize bundled designs and few compare implementation-level interventions across model capability and context-window budgets.Prior work includes local ablations and systems that add long-term memory, task graphs, or modular sub-agents.Source: §5.2 · p. 15
  • Prior harness studies report model-specific architecture preferences, but Liu (2026) examine prompt-level scaffolding on short-horizon tasks without explicitly examining context-window pressure.This study instead tests implementation-level planning, workspace action interfaces, and context-management policies across long coding trajectories and manipulated context budgets.Source: §5.2 · p. 15

5.3 Context Management

  • Context-management methods include static elision, retrieval, and summarization, as well as dynamic approaches often trained through reinforcement learning.These mechanisms are typically evaluated on a single model, leaving their dependence on available window size and model capability uncharacterized; the study sweeps 32k–128k tokens across two model families and three sizes.Source: §5.3 · p. 15

6 Conclusion

The study estimates conditional effects of planning, action space, and context management across four models and two long-horizon coding benchmarks. Its results link component value to model capability, task type, and resource budget.

Conclusion

  • The study presents a controlled empirical framework for examining harness components independently across models and long-horizon coding benchmarks.The harness varies context management, planning, and action space.Source: §6 · p. 15
  • Across four models and two long-horizon benchmarks, context management matters most under tight context budgets, while planning and action-space effects vary with model capability.T4 has the lowest aggregate cost at broadly similar success rates; predefined tools help models with weak bash control, whereas bash-only benefits bash-capable models, especially on shell-centric tasks.Source: §6 · p. 15
  • Trajectory analysis associates context management with longer execution, planning with sustained or shortened trajectories, and bash-only with combining multiple code modifications in one tool call.The study concludes that harness components should be selected for the target model, task type, and resource budget rather than adopted by default.Source: §6 · pp. 15–16

Limitations

The paper bounds its conclusions to the specific implementations, configurations, models, and tasks evaluated. Its appendices document the prompt assembly, component interfaces, and tool behavior used in those experiments.

Limitations

  • The study estimates conditional effects of its specific harness implementations rather than identifying a universally optimal harness.Planning and action space were ablated only under T4/128k; single runs per task and 89 Terminal-Bench tasks limit statistical power, and the evaluated models and benchmarks constrain external validity.Source: Limitations · p. 16

A Harness Prompts

  • Each model invocation combines an action-interface-specific base prompt with instructions and reminders from enabled components; runtime-dependent fields are instantiated separately.These fields include workspace path, session information, current plan, elision statistics, stored event identifiers, and running summary.Source: A Harness Prompts · p. 19

A.1 System prompts

  • The action-space ablation changes both available tools and the instructions presenting them, while planning uses three prompt blocks and external plan state maintained through update_plan.Planning-off removes all three blocks and the update_plan tool.Source: §A.1 · p. 19, §A.2 · p. 19

A.3 Context-management prompts

  • Context management uses a separate LLM call for M3 summarization and inserts elision stubs or a running summary into the task-solving model's input.T1 receives an unrecoverable elision stub; T2 and T4 also receive the stored event identifier needed for recall_event.Source: §A.3 · pp. 20–21

A.4 Stuck-detection prompts

  • Stuck detection is fixed across conditions and triggers reminders for repeated tool calls; a continuing identical failing streak terminates the run before the step budget is exhausted.A streak uses the same tool name and byte-identical arguments; permission denials do not count as failures.Source: §A.4 · pp. 21–22

B Tool Descriptions

  • The predefined-tool condition exposes file, search, execution, and web interfaces, while bash-only removes predefined workspace tools and retains bash.update_plan and recall_event are auxiliary interfaces controlled independently by planning and context-management settings.Source: B Tool Descriptions · p. 22

B.1 File input and output

  • Predefined file operations provide bounded reads, full-file creation or authorized replacement, and byte-exact targeted edits; discovery tools list directories, match paths, or search contents by regex.Mutation tools require an eligible recorded read before modifying an existing file, unlike bash modifications, which do not use the same state-tracking and automatic-diagnostic path.Source: §B.1 · p. 23, §B.2 · p. 24

B.3 Execution and web access

  • Both action-space conditions execute shell commands through /bin/sh -c, but only the predefined-tool condition offers concrete-URL retrieval and directs common file operations toward structured tools.update_plan replaces the in-memory plan, while recall_event retrieves stored historical content by event identifier; both remain available under either action-space condition when enabled.Source: §B.3 · pp. 24–25, §B.4 · p. 26

C Trajectory Analysis

The trajectory analysis uses complementary annotations to examine how agents behave, where unresolved repairs fail, and how their work is distributed across each benchmark’s problem-solving process.

C Trajectory Analysis

  • Three complementary schemes analyze trajectory survival, behavioral composition, action granularity, and the stages at which unresolved runs terminate.The appendix applies failure-stage attribution to unresolved SWE-Bench repairs and behavior encodings to SWE-Bench and Terminal-Bench actions.Source: C Trajectory Analysis · p. 26, §C.1 · p. 27

C.1.1 SWE-Bench Failure-Stage Attribution

  • Failure-stage attribution assigns each unresolved SWE-Bench repair its earliest failed stage: file localization, line localization, patch implementation, or verification.The earliest-failure convention prevents downstream symptoms, such as failed tests, from obscuring earlier errors.Source: §C.1.1 · p. 27

C.1.2 SWE-Bench Behavior Encoding

  • SWE-Bench behavior encoding labels each turn by its primary workflow purpose across localization, reproduction, fixing, verification, and auxiliary behavior.The five phases characterize how agents distribute effort across debugging.Source: §C.1.2 · p. 28, Table 8 · p. 28

C.1.3 Terminal-Bench Action-Level Encoding

  • Terminal-Bench action-level encoding classifies individual shell tool calls by semantic intent, using 10 action types grouped into four phases.The phases are understanding, code writing, verification, and auxiliary operations.Source: §C.1.3 · p. 28

C.2 Judge Setup and Human Validation

LLM judges label trajectory actions and failure stages, and human annotations are used to validate their agreement across the two benchmarks.

Judge Setup

  • The judge annotations use GPT-5.5 with high reasoning effort and temperature 0.6 for SWE-Bench turn-purpose and failure-stage labels and Terminal-Bench action-purpose labels.The corresponding judge prompts are reproduced in Figures 40–42.Source: §C.2 · p. 28

Human Validation

  • Three annotators labeled 200 trajectories in non-overlapping splits, covering 15,610 units across Terminal-Bench actions, SWE-Bench actions, and SWE-Bench failure stages.The sample included 5,306 Terminal-Bench action labels, 10,254 SWE-Bench action labels, and 50 failure-stage diagnoses.Source: §C.2 · p. 29
  • Judge–human agreement was approximately 94.2% overall, with a weighted mean Cohen’s κ of 0.929 across the three validation splits.Across splits, overall agreement ranged from 88.7% to 98.4%, with κ from 0.858 to 0.980.Source: §C.2 · p. 29
  • The reported agreement supports using judge-produced annotations to scale trajectory analysis to the full set of experimental trajectories.Terminal-Bench action-label agreement ranged from 79.8% to 97.3% raw agreement, while failure-stage agreement ranged from 87.5% to 100.0%.Source: §C.2 · p. 29

C.3.1 Failure-Stage Results

Failure-stage patterns vary with model capability and targeted ablations, while context-management tiers show little change in the stage distribution; behavior profiles describe active trajectories over time.

C.3.1 Failure-Stage Results

  • Removing planning or predefined tools raises Nemotron-3 30B’s file-localization failure share from 56.3% to 73.8% or 76.6%, respectively.For Mistral-Medium-3.5-128B, bash-only raises file-localization failures from 16.0% to 41.4%; context-management tiers change stage distributions little.Source: §C.3.1 · p. 29
  • At 128k, unresolved Nemotron-3 30B runs most often fail at file localization, while Nemotron-3 550B and Mistral-Medium-3.5-128B more often reach the correct file and lines but fail at patch implementation.Table 10 reports the distribution among unresolved SWE-Bench runs by earliest failed stage.Source: §C.3.1 · p. 29, Table 10 · p. 30

C.3.2 Behavior Profiles

  • Figures 32–35 profile active SWE-Bench trajectories by turn, partitioning them into five behaviors and marking median trajectory length, success rate, and mean cost per task.The profiles hold planning and predefined tools fixed while varying context-management tier and context-window budget.Source: §C.3.2 · pp. 30–31, Figure 32 · p. 31

C.4 Terminal-Bench Trajectory Results

Terminal-Bench trajectory profiles show the distribution of active trajectories across turns and behavioral phases.

C.4 Terminal-Bench Trajectory Results

  • Figures 36–39 partition active Terminal-Bench trajectories by turn into Understand, Write Code, Verify, and Other behavior, with planning and predefined tools fixed.Columns correspond to models and rows to context-management tiers T0–T4.Source: §C.4 · p. 31

C.5 Trajectory-Level Statistics at 128k

Trajectory analyses show that context-management tiers leave behavior largely stable at 128k, while planning and action-space changes alter how trajectories unfold. The appendix also reports recall-call statistics and describes the judge prompts used to encode trajectories and failures.

C.5 Trajectory-Level Statistics at 128k

  • At 128k, context-management tiers leave trajectory lengths, tool calls, re-patching, edit sizes, and most without-edit rates largely stable.For Nemotron-3 30B, median SWE-Bench trajectory length stays between 39 and 42 turns across tiers; for Nemotron-3 550B, it stays between 70 and 74.Source: §C.5 · p. 32, Table 11 · p. 39, Table 12 · p. 40
  • Removing planning shortens Nemotron-3 30B SWE-Bench trajectories and raises no-edit termination, but lengthens Nemotron-3 550B and Mistral trajectories while their no-edit rates stay below 3%.The 30B median falls to 5 turns and its no-edit rate rises from 27.8% to 68.6%; the 550B and Mistral medians rise from 74 to 108 and 53 to 68 turns, respectively.Source: §C.5 · p. 32, Table 12 · p. 40
  • Bash-only reduces re-patching for all four models, but its effect on edit size varies by model and Nemotron-3 30B has the highest Terminal-Bench without-code rate.That rate is 22.5%, consistent with the out-of-interface tool emissions described in §3.2.Source: §C.5 · pp. 32–33, Table 11 · p. 39, Table 12 · p. 40

C.6 Recall-Event Usage

  • Recall usage is reported as mean recall_event invocations per task, not the fraction of trajectories using recall, because a trajectory can invoke the tool repeatedly.Table 13 gives per-configuration invocation rates for recall-enabled strategies T2 and T4 by model and context-window budget.Source: §C.6 · p. 33, Table 13 · p. 40

C.7 Judge Prompts

  • The judge prompts encode SWE-Bench turns into five phases, attribute unresolved failures to the earliest repair stage, and classify Terminal-Bench shell actions into 10 fine-grained types.The prompts reproduce the procedures for turn-purpose classification, failure-stage attribution, and action-purpose classification.Source: §C.7 · p. 33, Figure 40 · p. 41, Figure 41 · p. 42
Loading 2609.20804v1…