PaperlayerView exact revision on arXiv

Source-linked AI summary

The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

Naman Vats, Oleg Golev

arXiv:2607.22585v1cs.AI

TL;DR

Model-only coding-agent rankings leave harness effects on cost, latency, and oversight under-specified. The paper compares two models across three harnesses on 50 Terminal-Bench Pro tasks and finds a 40× token-cost gap alongside smaller paired pass-rate differences.

  • Problem

    Model-name leaderboards often under-specify the harness, although developers care about cost per resolved task, completion time, and oversight burden.

    Source: §1 · p. 1

  • Method

    The study compares two coding models across three harnesses on 50 Terminal-Bench Pro tasks, measuring pass rate, token use, interaction, latency, and failure modes.

    Source: §1 · p. 1, §3.5 · p. 3, §5.1 · p. 5

  • Results

    Harness choice shifts tokens-per-solved-task by 40×, while paired pass-rate differences are 0–8 pp and mostly within bootstrap noise.

    Source: §1 · p. 1, §4.7 · p. 5

  • Takeaways & Limitations

    Human-centered coding-agent evaluation should compare harness–model pairs and report cost, idle-turn, and failure-mode signals alongside pass rate.

    Source: §7 · p. 6

    With n = 50 tasks, several pass-rate effects fall within bootstrap noise and are descriptive rather than statistically definitive.

    Source: §6 · p. 5

Abstract

from arXiv · show

Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMax M2.5 across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task subset of Terminal-Bench Pro. Harness choice induces up to a 40x difference in tokens per solved task, while paired within-model pass-rate differences remain 0-8 percentage points (95% paired-task bootstrap CIs include zero except for the largest gap). Failure fingerprints replicate across models (REASON for Goose, VERIFY/MAX_TURNS for OpenHands-SDK, idle-loop/TIME for OpenCode), indicating harness-level biases that are largely model-independent. For human-centered coding-agent evaluation, model name alone is an incomplete comparison unit: harness-model pairs determine real-world cost, latency, and oversight burden; no-action turns are a per-task wait tax, not just a token tax. We therefore recommend selecting harness-model pairs by pass rate under token/latency budgets, and reporting token usage, latency, and full harness specifications alongside any model comparison. We release anonymized configs, raw trial logs, aggregated snapshots, and analysis scripts.

1. Introduction

Model-name leaderboards omit the harness and therefore do not capture cost, completion time, or oversight burden. In a controlled comparison, harness choice produced a large token-cost gap, modest paired pass-rate differences, and failure fingerprints that replicated across models.

1. Introduction

  • Model-name rankings leave out the cost per resolved task, time-to-completion, and oversight needed to keep an agent productive.The introduction notes that these practical measures are not recovered from a model-only score.Source: §1 · p. 1
  • A prior Terminal-Bench 2.0 comparison reported a 5.7-point accuracy gain alongside a 65× input-token difference between harnesses.The paper uses that observation to motivate a controlled study of harness effects with the model held fixed.Source: §1 · p. 1
  • Across 300 trials, harnesses differed by 40× in tokens per solved task, while paired pass-rate differences were 0–8 pp and CIs included zero except for the largest gap.The evaluation crossed three harnesses, two models, and 50 Terminal-Bench Pro tasks.Source: §1 · p. 1
  • Harness-specific failure fingerprints replicated across both models, indicating that these patterns were scaffold properties rather than model properties.The study frames no-action turns—neither editing a file nor issuing a new command—as a proxy for oversight burden.Source: §1 · p. 1
  • The authors argue that human-centered coding-agent evaluation should compare harness–model pairs and provide anonymized configurations and raw trial data.They also describe releasing aggregated snapshots and analysis scripts as supplementary material.Source: §1 · p. 1

2. Related Work

Prior coding-agent work recognizes scaffold choice and interaction quality, but the cited literature does not quantify harness efficiency across a fixed task set and model. This study measures tokens, turns, idle time, and failure modes in a controlled setting.

Coding-agent benchmarks

  • Terminal-Bench Pro extends command-line coding evaluation to 400 tasks across eight domains, while this study samples 50 tasks across eight archetypes.The related-work discussion contrasts this with SWE-bench and successors, which treat scaffold choice as a confound to control.Source: §2 · p. 2

Agent scaffolding and evaluation reliability

  • Earlier work frames agents as systems, but does not quantify harness efficiency across a fixed task set with a fixed model.The discussion also identifies OpenHands-SDK, Goose, OpenCode, and Harbor as relevant platforms and execution infrastructure.Source: §2 · p. 2

Cost, oversight, and human-centered evaluation

  • Token efficiency had been studied for long-context benchmarks, while interaction-aware evaluation calls for measures of oversight burden and verifiability.The paper measures tokens, turns, idle time, and failure-mode mix.Source: §2 · p. 2

3. Experimental Setup

The study compares two coding models across three open-source harnesses on a stratified Terminal-Bench Pro subset, standardizing shared task and environment conditions while retaining native harness behavior. It measures outcomes, token use, interaction, and failures.

3.1. Tasks

  • The evaluation sampled 50 tasks from the public Terminal-Bench Pro set by stratified random sampling across eight domain categories.Tasks were drawn randomly within categories, whose counts followed the source-set distribution; evaluation used deterministic pytest suites.Source: §3.1 · p. 2, Table 1 · p. 2

3.2. Models

  • The study evaluated Qwen 3.6 Plus and MiniMax M2.5, two recent agentic coding models accessed through OpenRouter.The setup passage reports their respective published benchmark results on Terminal-Bench 2.0 and SWE-bench Verified.Source: §3.2 · p. 2

3.3. Harnesses

  • Goose, OpenCode, and OpenHands-SDK were run through Harbor with identical task manifests despite their distinct scaffolding approaches.The harnesses are described respectively as a heavyweight IDE agent, a persistent tool-loop agent, and a micro-agent architecture with delegation, retries, and verification.Source: §3.3 · p. 2

3.4. Standardization Protocol

  • The protocol held task instructions, tests, sandbox, wall-time cap, model gateway, and most prompt content constant while retaining native harness controls and behavior.Goose and OpenHands-SDK had 40-turn caps; OpenCode had only the 900-second wall-time cap, and native APIs, context preloading, retries, and sub-agent logic remained variable.Source: §3.4 · pp. 2–3

3.5. Metrics

  • Metrics included solve status, turns, tokens, no-action turns, budget hits, wall time, and post-hoc failure category, with paired-task bootstrap intervals.Tokens per solved task amortizes cellwide billed token totals over solved tasks; failed and infrastructure-error trials count in the numerator, while pre-LLM HANG and ERROR trials contribute zero.Source: §3.5 · p. 3

4. Results

Harness choice shifts tokens per solved task by 40×, while pass-rate effects are smaller and often indistinguishable at this sample size. The cost ordering and failure patterns vary by harness, and failure fingerprints replicate across both models.

4.1. Pass Rate

  • At n = 50, within-model harness pass-rate ranges are 2–8 pp, and most pairwise differences are not statistically distinguishable from zero.The reported Goose–OpenCode difference on MiniMax is at most −8.0 pp (95% CI [−18.0, 0.0]); on Qwen it is −2.0 pp ([−12.0, +8.0]).Source: §4.1 · p. 3

4.2. Token Cost per Solved Task

  • OpenCode consumes approximately 40× more tokens per solved task than Goose, while harness pass-rate differences remain within 0–8 pp.Average turn counts are 21–27 for OpenCode versus 18–25 for Goose; the token gap is not primarily attributable to turn-count differences. A 40-turn cap would truncate approximately 12–29% of OpenCode trials, mostly already in the failure tail.Source: §4.2 · pp. 3–4, Table 3 · p. 4

4.3. Pareto Frontier: Pass Rate vs. Token Cost

  • Goose lies on the Pareto frontier for both models, while OpenCode is dominated by Goose on cost and by OpenHands-SDK at the same pass rate.Figure 1 plots pass rate against tokens per solved task on a log scale; the pass-rate spread is 38–50% while the cost spread is two orders of magnitude.Source: §4.3 · p. 4, Figure 1 · p. 4

4.4. No-Action Turns

  • OpenCode averages 2.0–2.16 no-action turns per task versus 0.2–0.3 for Goose, a 10× ratio across both models.Each no-action turn carries a full context window of input tokens due to message-history accumulation, creating dollar cost and wall-clock latency for interactive users.Source: §4.4 · p. 4, Figure 2 · p. 4

4.5. Failure-Mode Fingerprints

  • Failure fingerprints replicate across models: Goose is REASON-dominated, OpenHands-SDK shows VERIFY and MAX TURNS failures, and OpenCode shows TIME and idle spinning.OpenCode has zero MAX TURNS classifications because it lacks a turn-budget flag through Harbor; runs that might be classified as MAX TURNS elsewhere surface as TIME or HANG.Source: §4.5 · p. 4

4.6. Pass Rate by Task Category

  • On IMPL tasks, OpenHands-SDK reaches 69% pass rate versus 38% for Goose and 38% for OpenCode; all harnesses score 8–17% on SEC and 0% on SYS.Harness rank order is unstable across the remaining categories.Source: §4.6 · pp. 4–5, Table 5 · p. 5

4.7. Summary of Harness vs. Model Effects

  • Harness choice shifts tokens per solved task by 40×, while upgrading the model changes that metric by only 1.0–1.3×.Pass-rate effects of harness and model upgrade are of comparable magnitude and within bootstrap noise at n = 50.Source: §4.7 · p. 5

5. Discussion

The discussion connects harness choice to developers’ costs, waiting time, and oversight, then recommends evaluating harness–model pairs with deployment-relevant metrics.

5.1. What the Cost Gap Means for a Human User

  • ∼40× more tokens per task means ∼40× more API spend, all else equal, outweighing the 1–2× price spread between leading models.OpenCode’s no-action turns also require round-trip API calls, creating a per-task wait tax; users must read or skim idle turns to check they were not destructive.Source: §5.1 · p. 5

5.2. What the Failure Fingerprints Mean

  • Goose stops cleanly when stuck, while OpenHands-SDK may accept plausible-but-incorrect solutions or exhaust iterations, requiring users to verify outputs more carefully.Goose’s failures are REASON-dominated with zero VERIFY; OpenHands-SDK’s cited failure categories are VERIFY and MAX TURNS.Source: §5.2 · p. 5
  • OpenCode more often exhausts the wall-time cap with TIME or HANG failures, but commits only to solutions that pass tests.Users pay in time and tokens, while its positive verdicts can be trusted; the harnesses’ distinct failure fingerprints imply different oversight disciplines.Source: §5.2 · p. 5

5.3. Implications for Benchmark Reporting

  • The authors recommend evaluating harness–model pairs and reporting tokens per solved task, average no-action turns, and failure-category vectors alongside pass rate.They argue model-name-only pass-rate reporting conflates model and harness variance and omits cost and oversight signals relevant to deployability.Source: §5.3 · p. 5

6. Limitations •

The study’s findings are bounded by a 50-task sample, asymmetric token and turn accounting, three evaluated harnesses, and provider settings that may change.

6. Limitations •

  • At n = 50 tasks, several pass-rate effects fall within bootstrap noise and are treated as descriptive, not statistically definitive.Each task corresponds to 2 percentage points.Source: §6 · p. 5
  • Goose reports total tokens without an input/output breakdown, although the 40× ratio is anchored in totals; per-direction comparisons are unavailable for Goose.The total-token ratio is robust to the missing split.Source: §6 · pp. 5–6
  • Goose and OpenHands-SDK enforce a 40-turn cap, while OpenCode has only a 900-second wall-time cap; a turn may represent different numbers of internal LLM calls.OpenCode does not expose a turn-budget flag through Harbor.Source: §6 · p. 6
  • Findings from three harnesses should not be assumed to generalize to other scaffold architectures or unevaluated commercial closed-source agents.Source: §6 · p. 6
  • Provider defaults can change across runs, so exact replication requires pinning the route and sampling settings used at run time.Run metadata records the settings available during execution.Source: §6 · p. 6

7. Conclusion

The conclusion reports a large harness-associated token-cost difference alongside smaller paired pass-rate shifts, and recommends evaluating harness–model pairs with deployment-relevant signals.

7. Conclusion

  • Harness choice introduces a 40× token-cost difference and harness-specific failure fingerprints, while paired pass-rate shifts are at most 8 percentage points.Both findings replicate independently across two recent models.Source: §7 · p. 6
  • For human-centered evaluation, the paper argues for comparing harness–model pairs and reporting cost, idle-turn, and failure-mode signals alongside pass rate.Source: §7 · p. 6
  • The supplementary materials include anonymized configs, raw trial logs, aggregated snapshots, and analysis scripts.Source: §7 · p. 6

Impact Statement

The impact statement frames coding-agent evaluation as a human-centered problem, emphasizing deployment costs and oversight while noting no specific ethical concern beyond those generally associated with releasing evaluation materials.

Impact Statement

  • The work aims to advance coding-agent evaluation by making scaffold choice and its deployment implications visible.Source: Impact Statement · p. 6
  • The stated human-facing concerns are the dollar cost, latency, and supervisory effort a developer absorbs when using these systems.Source: Impact Statement · p. 6
  • The authors identify no specific ethical concern beyond those generally associated with releasing coding-agent evaluation tooling and trial logs.Source: Impact Statement · p. 6

A. Failure Classification Decision Tree

The exception-first decision tree assigns each trial a failure category using exception type first, then solve status, turn-budget status, and completion behavior; TIME is tied to the 900-second timeout.

A. Failure Classification Decision Tree

B. Selected 50 Task IDs

This section lists the 50 selected evaluation task IDs in alphabetical order.

B. Selected 50 Task IDs

  • The task list begins with advanced-json-to-rfc4180-csv-converter and ends with train-sarsa-taxi-agent.The passage identifies the list as alphabetical.Source: B. Selected 50 Task IDs · p. 7
  • The listed tasks include software building, migration, debugging, and dependency-repair tasks.Examples include build-nginx-1-24-production-server, migrate-make-to-cmake-build, debug-bst-segfault-with-gdb, and restore-broken-pip-installation.Source: B. Selected 50 Task IDs · p. 7
  • The list also includes analysis, machine-learning, chess, and puzzle-solving tasks.Examples include apache-log-security-analyzer, train-fraud-detection-model, solve-chess-mate-in-two, and solve-escape-room-puzzle-server.Source: B. Selected 50 Task IDs · p. 7

C. Failure Examples (One per Category) •

The examples illustrate distinct failure categories, from voluntary stopping and failed tests to timeouts, turn limits, setup exits, and infrastructure errors.

C. Failure Examples (One per Category)

Related papers

Loading 2607.22585v1…