PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Model: Demystifying Harness Effects in Software Engineering Agents

Haichuan Hu, Quanjun Zhang, Shengcheng Yu, Zhifei Chen, Tianyu Luo, Chunrong Fang, Zhenyu Chen, Liang Xiao

arXiv:2609.32459v1cs.SE

TL;DR

Harness design in SE agents has not been systematically isolated across components, models, and tasks. The paper evaluates two harnesses across ten open-weight models and three benchmarks, then tests five components in NanoHarness. Harness effects vary by task and model capability, while selected components offer stable gains.

  • Problem

    Harness choices are usually evaluated as part of end-to-end agents, leaving their individual contributions and interactions with model capability and task type insufficiently understood.

    Source: §1 · p. 1

  • Method

    The study compares mini-SWE-agent and OpenCode across ten Qwen and DeepSeek models on three benchmarks, then uses NanoHarness to test five representative components individually and together.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    Harness effects vary by model and task, with diminishing gains on issue repair as capability rises; NanoHarness gains 7.37 percentage points over mini-SWE-agent on Qwen3.7-Max.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the studied tasks, structured tool use and task-specific subagents improve performance, while context compression and general subagents can hurt repository-generation performance.

    Source: §8 · p. 12

    The study tests five representative components and only Qwen and DeepSeek models, and its ProgramBench-centered results may not generalize directly to other datasets.

    Source: §7 · p. 12

Abstract

from arXiv · show

Large Language Model (LLM)-based agents are increasingly used for software engineering tasks, yet their performance is not determined by the base model alone. The agent harness substantially shapes how SE agents interact with repositories, execute actions, and validate solutions. However, the role of harness design remains insufficiently understood, especially across different models, tasks, and harness components. In this paper, we present a systematic empirical study of harness effects in SE agents. We first evaluate two representative harnesses, mini-SWE-agent and OpenCode, with ten models from two prominent open-weight model families, Qwen and DeepSeek, on three benchmarks: SWE-bench Pro, ProgramBench, and GitTaskBench. We then construct NanoHarness, a lightweight modular harness built on top of mini-SWE-agent, and use it to analyze five representative harness components: tool registry, context compression, explicit planning, subagents, and lazy skills. Experimental results show that harness effectiveness depends jointly on model capability and task type. Complex harnesses provide diminishing marginal gains on SWE-style issue repair as model capability improves, but can benefit stronger models on more complex and open-ended repository-level tasks. Component-level analysis on ProgramBench further shows that structured tool use and task-specific subagents provide the most stable improvements, while context compression and general subagents can hurt repository-generation performance. When combined, NanoHarness improves over mini-SWE-agent by 7.37 and 6.21 percentage points on Qwen3.7-Max and DeepSeek-V4-Pro, respectively, recovering most of the gains of product-level harnesses. These findings highlight harness design as a first-class factor in SE-agent performance and provide insights for building more effective and efficient coding agents.

1 Introduction

Harness design can affect SE-agent performance as much as model choice, yet its components and interactions with models and tasks have not been systematically isolated. This study compares harnesses across models and benchmarks, then analyzes individual and combined harness components.

1 Introduction

  • The paper frames harness design as infrastructure around the model, including context retrieval, tool use, execution environments, patch generation, and validation.Public SWE-bench Verified leaderboard comparisons motivate controlled analysis of which design choices drive gains and whether they transfer across models and tasks.Source: §1 · p. 1
  • Harness design remains insufficiently understood because prior SE agents bundle choices such as context retrieval, tools, planning, testing, and patch selection into end-to-end systems.The paper notes that this leaves unclear which components drive improvements and how effects vary by model and task.Source: §1 · p. 1
  • The study evaluates mini-SWE-agent and OpenCode with ten Qwen and DeepSeek models across SWE-bench Pro, ProgramBench, and GitTaskBench, then tests five harness components in NanoHarness.The component study evaluates isolated and combined additions, alongside performance and execution behavior.Source: §1 · p. 2
  • Harness effects vary with task and model capability: complex harnesses offer diminishing gains on issue repair but can benefit capable models on open-ended repository tasks.The study also reports that complex harnesses mainly compensate for weaker models on SWE-style issue repair.Source: §1 · p. 2
  • NanoHarness combines representative harness components; its component and trajectory analyses identify structured tools and task-specific subagents as stable improvements, while compression and general subagents can hurt repository generation.The paper reports gains over mini-SWE-agent of 7.37 percentage points for Qwen3.7-Max and 6.21 for DeepSeek-V4-Pro.Source: §1 · p. 2
  • The contributions comprise a three-benchmark, ten-model empirical study, a modular NanoHarness for component analysis, and behavioral findings about structured tool use and regulated exploration.The study reports more than 10B Qwen and 20B DeepSeek prompt tokens.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe a modern agent harness as modules organized around the model. They introduce tools, context and memory, planning, subagents, and reusable skills as core modules.

2.1 Architecture

  • The architecture decomposes the harness into 12 modules around the model, distinguishing core capabilities from supporting components such as environment, permissions, logging, and evaluation.The paper focuses on five core modules and leaves supporting modules’ internal mechanisms aside.Source: §2.1 · p. 3

2.2.1 Tools

  • Tools expose repository actions such as inspection, editing, search, shell execution, and testing; their interface shapes what state the model can observe and how it acts.The paper emphasizes the interface’s role in safe repository interaction.Source: §2.2.1 · p. 3

2.2.2 Context and Memory

  • Context and memory govern which task information, observations, outputs, and intermediate reasoning remain available during the agent loop.Effective management preserves useful state while limiting irrelevant or redundant history.Source: §2.2.2 · p. 3

2.2.3 Planning

  • Planning helps agents decompose tasks, track intermediate goals, and choose subsequent actions through explicit plans, replanning, or lightweight action selection.Source: §2.2.3 · p. 3

2.2.4 Subagents

  • Subagents delegate task parts to specialized roles or execution flows, potentially improving decomposition while adding coordination and communication overhead.Source: §2.2.4 · p. 3

2.2.5 Skills

  • Skills encode reusable workflows such as debugging, validation, and recovery from failed attempts, reducing repeated decisions when relevant.Source: §2.2.5 · p. 3

3 Study on SE Agent Harness

The study examines joint model, harness, and task effects, then isolates representative harness mechanisms in a modular system. Its questions cover performance, component contributions, and execution behavior and efficiency.

3.1 Overall Study Design

  • The study combines a joint-effects evaluation across models, harnesses, and SE tasks with component-level analysis of individual mechanisms and their combinations.Figure 3 summarizes these two complementary parts.Source: §3.1 · p. 3, Figure 3 · p. 4
  • The joint-effects design compares weaker-to-stronger models within families, lightweight and product-level harnesses, and benchmarks spanning software lifecycle stages.The unified matrix evaluates each model under each harness across each task.Source: §3.1 · p. 4
  • NanoHarness adds five independently switchable mechanisms to mini-SWE-agent—tool registry, context compression, explicit planning, subagents, and lazy skills—for individual and combined analysis.The component study also examines performance, behavior, and efficiency.Source: §3.1 · p. 4

3.2 Research Questions

  • The research questions ask how model, harness, and task effects relate; how components act individually and together; and how mechanisms change tool use, context, token use, and long-horizon failure modes.Source: §3.2 · p. 5

4 Experimental Setup

The experiments use three complementary SE benchmarks, two contrasting harnesses, and ten open-weight models from Qwen and DeepSeek. A shared protocol sets task limits and evaluation environments.

4.1 Tasks and Datasets

  • The benchmarks cover repository-level code generation, real-world issue repair, and repository-centric task solving across SE lifecycle stages.These are ProgramBench, SWE-bench Pro, and GitTaskBench, respectively.Source: §4.1 · p. 5

4.2 Harness Selection

  • mini-SWE-agent provides a lightweight command-line baseline, while OpenCode represents a more complex product-level harness with integrated repository, tool, context, and task support.Source: §4.2 · pp. 5–6

4.3 Model Selection

  • The study selects five models each from the open-weight Qwen and DeepSeek families to support reproducible comparisons across capability levels within similar lineages.Models are ranked using official Coding Index scores, and all are evaluated in non-thinking mode.Source: §4.3 · p. 6

4.4 Implementations

  • All models use temperature 0.0; task runs have step and time limits, with RQ1 using random benchmark subsets and component studies using all 200 ProgramBench instances.ProgramBench’s step cap is reduced to 300 for cost control; SWE-bench Pro and GitTaskBench use 250 steps.Source: §4.4 · p. 6

5.1 RQ1: Relationship Among Models, Harnesses, and Tasks

Harness effects vary with both model capability and task type: gains generally shrink on SWE-style repair as models strengthen, while complex harnesses can benefit stronger models on more open-ended repository tasks. The scaling pattern also differs between Qwen and DeepSeek.

5.1.1 RQ1.1-Scaling of Harness Performance with Model Capability

  • On SWE-bench Verified, OpenHands’ gain over mini-SWE-agent falls from +13.60 pp to +5.47 pp, and SWE-agent’s from +9.60 pp to +4.07 pp, as Claude models advance.The table reports matched-model harness improvements over mini-SWE-agent.Source: §5.1.1 · p. 6, Table 3 · p. 6
  • For Qwen on SWE-bench Pro, OpenCode’s advantage over mini-SWE-agent narrows from about 6 pp with Qwen3-Max to 2.67 pp with Qwen3.7-Max.OpenCode keeps agent steps relatively stable while its tool calls rise from 39.87 to 72.23; mini-SWE-agent remains near one call per step.Source: §5.1.1 · p. 7, Figure 5 · p. 7, Table 4 · p. 7
  • DeepSeek scaling is non-monotonic: OpenCode gains more on intermediate models, then nearly converges with mini-SWE-agent on DeepSeek-V4-Pro at 34.3% versus 33.7%.For DeepSeek-V4-Pro, OpenCode uses fewer steps but more tool calls per step than mini-SWE-agent, while the performance gain is very small.Source: §5.1.1 · p. 7, Figure 6 · p. 7, Table 5 · p. 7

5.1.2 RQ1.2-Harness Performance on Different SE Tasks

  • On ProgramBench, weaker models do not gain more from OpenCode, while the latest models leverage it consistently; on GitTaskBench, the turning points are Qwen3.6-Max and DeepSeek-V3.1.ProgramBench results use a random sample of 70 instances; GitTaskBench includes all 54 instances.Source: §5.1.2 · p. 8, Figure 9 · p. 8, Figure 11 · p. 9
  • SWE-bench Pro has relatively fixed workflows and localized edits, whereas ProgramBench requires repository generation and GitTaskBench demands flexible, independently designed workflows.The paper notes sustained industry attention to SWE-style tasks may also make models more adaptable to that task type and less reliant on harness support.Source: §5.1.2 · p. 8

5.1.2 RQ1.2-Harness Performance on Different SE Tasks.

  • Complex harnesses bring diminishing gains on fixed-workflow SWE-bench Pro as models improve, but can benefit stronger models on ProgramBench and GitTaskBench.The reported task comparison identifies model capability, task complexity, and task type as joint factors in harness effectiveness.Source: Figure 7 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

5.2 RQ2: Component-Level Effect Analysis

On ProgramBench, NanoHarness’s combined configuration outperforms mini-SWE-agent on both tested models and comes close to product-level harnesses. Individual components vary: structured tools and task-specific delegation are strongest, while compression and general subagents can hurt.

5.2.1 RQ2.1-Individual Effect

  • Tool registry and task-specific subagents provide the most consistent individual gains, including 4.57 and 5.91 points for Qwen3.7-Max and 3.54 and 4.46 points for DeepSeek-V4-Pro.The authors connect these gains to structured repository operations and task-oriented delegation during multi-module implementation and validation.Source: §5.2.1 · p. 9
  • Planning yields small gains for both models, while lazy skills help DeepSeek-V4-Pro more than Qwen3.7-Max.The authors suggest procedural guidance is more helpful when a model can effectively decide when and how to invoke it.Source: §5.2.1 · p. 9
  • Context compression lowers scores by 4.86 points for Qwen3.7-Max and 3.85 points for DeepSeek-V4-Pro, and general subagents also degrade performance.The paper attributes compression risk to losing detailed requirements, interface constraints, or implementation state needed later in coding and debugging.Source: §5.2.1 · p. 9

5.2.2 RQ2.2-Combined Effect

  • NanoHarness improves over mini-SWE-agent by 7.37 points on Qwen3.7-Max and 6.21 points on DeepSeek-V4-Pro, exceeding every single-component variant.The full configuration scores 42.88 to 50.25 for Qwen3.7-Max and 45.14 to 51.35 for DeepSeek-V4-Pro.Source: §5.2.2 · pp. 9–10
  • The combined gains are not the sum of isolated effects: component interactions can add overhead, yet positive mechanisms can dominate when coordinated.NanoHarness remains slightly behind OpenCode and Claude Code while closing most of their performance gap with a lightweight modular design.Source: §5.2.2 · p. 10

5.3 RQ3: Effects of Harness Mechanisms on Agent Behavior and Efficiency

Harness components change how agents explore repositories and use context. NanoHarness improves performance with controlled prompt growth, while task-specific subagents help regularize both excessive probing and shallow validation on ProgramBench.

5.3.1 RQ3.1-Effects on Tool Use, Context Usage, and Token Usage

  • The full NanoHarness raises scores by 7.37 and 6.21 points while tool calls grow 131.18% and 83.58%, but prompt tokens rise only 16.37% and 7.26%.The authors characterize this pattern as improved performance through more structured interaction rather than simply more prompt budget.Source: §5.3.1 · p. 10, Table 7 · p. 11
  • Task-specific subagents increase tool calls by 115.76% for Qwen3.7-Max and 73.10% for DeepSeek-V4-Pro, while context compression cuts prompt tokens by 54.29% and 77.02% but accompanies score drops of 4.86 and 3.85 points.The paper suggests added tool use helps when directed toward the task, whereas compressed histories may lose requirements or debugging state.Source: §5.3.1 · p. 10, Table 7 · p. 11

5.3.2 RQ3.2-Effects on Failure Modes

  • ProgramBench failures include both excessive low-value probing followed by rushed completion and insufficient probing followed by shallow smoke-test validation.The figure compares Qwen3.7-Max probing-time distributions across harness configurations; the paper reports that task-specific subagents reduce both extremes.Source: §5.3.2 · p. 11, Figure 13 · p. 11
  • In two cases, NanoHarness reduces 133 probes to 43 while pass rate rises from 24.68% to 68.17%, and increases 3 probes to 45 while pass rate rises from 61.74% to 75.18%.The examples illustrate reduced unproductive exploration in one case and expanded behavioral exploration in the other.Source: §5.3.2 · p. 11

6 Related Work

Prior SE-agent work often evaluates complete systems or task-specific pipelines, leaving model, harness, and task effects entangled. This paper instead isolates representative harness components and studies their performance, behavioral, and efficiency effects.

6.1 LLM-based Agents for Software Engineering

  • Prior coding-agent research spans code generation, repository problem solving, multi-role workflows, and repair systems, but commonly evaluates complete systems, benchmarks, or task-specific pipelines.The paper names ChatDev, MetaGPT, SWE-agent, AutoCodeRover, RepairAgent, and Agentless among the related systems.Source: §6.1 · p. 12
  • Because those evaluations entangle model, harness, and task effects, this work treats harness design as the research object and measures it across models and tasks..Source: §6.1 · p. 12

6.2 Agent Harness

  • SE harnesses support repository navigation, editing, command execution, testing, and recovery, but existing mechanisms are usually evaluated within complete agents rather than isolated.The paper's controlled analysis examines representative components' performance, behavioral, and efficiency effects.Source: §6.2 · p. 12

7 Threats to Validity

The validity of the study is bounded by its selection of five harness components, two model families, and ProgramBench as the primary dataset. The authors note that these choices may limit coverage and generalization.

7 Threats to Validity

  • The five selected harness components may omit other relevant mechanisms, which the authors plan to evaluate in future work.Source: §7 · p. 12
  • Model-family coverage is limited to Qwen and DeepSeek across 10 representative variants, with prompt-side usage exceeding 10B and 20B tokens, respectively, at an estimated cost above $7,000 USD.Source: §7 · p. 12
  • Because primary experiments use ProgramBench, the conclusions may not directly generalize to other datasets.The authors characterize ProgramBench as a repository-generation benchmark spanning requirement understanding through artifact generation.Source: §7 · p. 12

8 Conclusion

The paper treats harness design as a factor in LLM-based software-engineering agent performance and finds its effects depend on model capability and task type. Component and trajectory analyses identify promising and potentially harmful design choices.

8 Conclusion

  • Harness gains depend on model capability and task type across the three benchmarks.Source: §8 · p. 12
  • Structured tools and task-specific subagents improve performance, while context compression and general subagents can hurt.Source: §8 · p. 12
  • Trajectory analysis indicates that effective harnesses regularize exploration and tool use, motivating joint consideration of models, harnesses, tasks, and execution behavior in future agent design.Source: §8 · p. 12

Related papers

Loading 2609.32459v1…