PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Model, Different Harness: Different Coding-Agent Results

Sydney Lewis

arXiv:2608.26218v1cs.AIcs.SE

TL;DR

The paper asks whether changing the harness changes coding-agent results when model weights and tasks stay fixed. It compares a full chronological transcript with a treatment that shortens older tool results and responds to stalled work. Under context pressure, treatment improves fail-to-pass performance across all three benchmarks and increases complete solutions on Verified and Pro, supporting evaluation of the model and harness as one solver.

  • Problem

    The paper asks whether changing a coding-agent harness changes outcomes when the model and task remain fixed.

  • Method

    The study pairs control and treatment configurations of the same harness across three coding benchmarks, preserving the run record while mechanically managing the model’s working view and responding to stalled work.

  • Results

    Treatment repaired more required behavior on all three benchmarks under context pressure and completed more tasks on Verified and Pro; outcomes were close on Verified and Pro when context was effectively unconstrained.

  • Takeaways & Limitations

    Coding-agent evaluations should report and test the model, harness, tool interface, context policy, and run controls as one solver.

  • Takeaways & Limitations

    The study compares the mechanical treatment with a full chronological-transcript control, not model-written summaries or retrieval-based views, and each task contributes one greedy trajectory per arm.

Abstract

from arXiv · show

A coding agent combines a model with a harness, which decides what the model sees, which tools it can use, and how the work continues. We ask whether changing the harness changes the result when the model and task stay fixed. We compare two configurations of the same harness on three coding benchmarks. The control supplies the full conversation in time order, while the treatment keeps the same record but mechanically shortens older tool results as the context fills and responds to repeated or stalled work. Under tight context, the treatment raises mean per-task fail-to-pass fraction (F2PF) in all three pressure comparisons and increases complete solutions on SWE-bench Verified and SWE-bench Pro. The tight-window Verified comparison uses 169 tasks, a 20,480-token window, and a fixed 480-second attempt endpoint; on this cohort, treatment raises mean per-task F2PF from 28 percent to 49 percent and complete solutions from 43 to 72. Without model-specific retuning, the same frozen treatment also raises both endpoints on the same cohort for three additional models with different designs. In the wide-window Qwen3.6 comparisons, observed arm outcomes are close on Verified and Pro, while FeatureBench retains a higher mean per-task F2PF under treatment. On the wide-window Verified cohort, treatment also serves fewer prompt tokens per turn. Because changing the harness changed what unchanged model weights could accomplish, coding-agent evaluations should treat the model and harness together as the tested solver.

1 Introduction

The paper asks whether changing a coding-agent harness changes outcomes when the model and task remain fixed. It introduces and evaluates a closed-loop treatment that changes the model’s working view and responds to stalled work while preserving the run record.

  • Long coding tasks make accumulated files, searches, errors, and test logs compete for finite context space.
  • The control presents the full chronological transcript, while the treatment can manage the model’s working view and respond to repeated or stalled work.
  • The paired evaluation holds model weights, tasks, context capacity, and run protocol constant while measuring complete solutions and required failing tests repaired.
  • The contribution is paired evidence that changing the harness changes what the same model accomplishes on the same tasks.

2 From a growing transcript to a closed loop

The treatment keeps a durable execution record while rebuilding a smaller model-visible working view and reacting to detected repetition or command problems. Its fixed rules shorten older tool results, preserve newer results, and intervene without additional model calls.

  • 2. From a growing transcript to a closed loop: The control exposes the full chronological conversation until it no longer fits, whereas treatment shortens older tool-result bodies as the context fills.
  • 2. From a growing transcript to a closed loop: The harness retains the complete in-memory conversation and rebuilds the model’s current view at each step, with separate files preserving the full run record.
  • 2.2 Keep the record, change the view: Treatment shortening leaves the task, model replies, and other conversation messages unchanged while deterministically displaying abbreviated tool results without model summarization.
  • 2.2 Keep the record, change the view: Shortening begins after the estimated full prompt reaches half the configured window; the newest four tool results stay full, while older results retain their beginnings and ends.
  • 2.3 Detect repetition: The detector identifies repeated failures or repeated reading without edits from fixed execution-record facts, then maps those patterns to predetermined responses.
  • 2.4 Act, then look again: The treatment combines the working-view rule, detector interventions, and command safeguards.

3 Comparing treatment with control

The study compares paired control and treatment runs that differ only in a checked-in harness configuration, while holding models, tasks, environments, and protocols fixed. It measures both complete resolution and partial repair, using pressure comparisons across three benchmarks while documenting package and cohort boundaries.

  • Paired design: Both arms use the same task, model weights, context capacity, tool interface, serving setup, evaluator, harness code, and scoring protocol; only the treatment configuration package differs.Pairing estimates the total effect of assigning the treatment condition, not the separate effect of any individual rule.
  • Paired design: The treatment package was frozen before campaigns, but its operating point was not selected through a task-disjoint hyperparameter study.The comparisons therefore evaluate the package as developed rather than a held-out tuning procedure.
  • Models, benchmarks, and task sets: Primary comparisons use Qwen3.6-35B-A3B across SWE-bench Verified, SWE-bench Pro, and FeatureBench, with each benchmark retaining the same task set in both arms.Verified uses 169 tasks at every context size and model; FeatureBench uses 183 canonical tasks, while Pro uses a 316-task demand-screened cohort.
  • Treatment operation: Tight-window runs shortened early and often, whereas the milder Verified comparison shortened later and in fewer runs.Table 1 reports active runs, first shortening turn, and typical reduction across treatment populations.
  • Outcome measures: F2PF measures the share of fail-to-pass tests repaired, complementing binary resolution by recording partial progress on unresolved tasks.Each task contributes one operational F2PF score between 0 and 1, and task-level scores are averaged with equal task weight.
  • Statistical comparison: For resolution, the study uses the exact two-sided McNemar test; for F2PF, it subtracts control from treatment per task and applies the exact two-sided sign test to non-tied directions.The sign-test p-value tests direction, while arm means describe magnitude.
  • Context pressure: The reading boundary estimates when runs fill the context window before making a source change, while resolution and F2PF measure whether they completed or repaired the task.A higher treatment boundary indicates more reading before space runs out.

4 Changing the harness changed the results

Under tight context pressure, treatment improves partial repair across all three benchmarks and increases complete solutions on the two SWE-bench cohorts. With effectively unconstrained context, Verified and Pro outcomes become close between arms, while FeatureBench retains a treatment advantage.

  • Progress and complete solutions under context pressure: All three pressured cohorts show higher mean per-task F2PF under treatment, with exact paired sign tests yielding p < 0.0001.After Holm correction, all three F2PF increases remain significant at 0.05.
  • Progress and complete solutions under context pressure: 10.5% to 19.6%: FeatureBench mean per-task F2PF rises under treatment, while complete solutions increase from 2 to 3.The 183-task pressure cohort shows sparse complete solutions despite the larger partial-repair result.
  • Results across context regimes: At 20,480 tokens, the Verified treatment-control gap is largest across the three observed context windows.The common cohort contains 169 tasks and uses a fixed 480-second budget; cross-window ordering does not isolate window size from run-era change.
  • Results across context regimes: At 262,144 tokens, Verified and Pro outcomes are close between arms, with fewer than 1% of tasks hitting the allotted context.On Verified, the treatment-minus-control F2PF difference is -0.3 percentage points, with a 95% interval of [-4.5, +3.9].
  • Results across context regimes: On wide-window FeatureBench, mean per-task F2PF rises from 23.9% to 30.7%, while complete solutions remain 5 in both arms.The primary task-paired sign test gives p = 0.00022, whereas the repository-level direction sign test gives p = 0.0963.
  • F2PF movement across both regimes: Under pressure, treatment lowers operational-zero assignments across all three comparisons, shifting Verified mainly toward complete repair and FeatureBench toward partial repair.Pro shows both partial and complete-repair movement; at 262,144 tokens, Verified and Pro distributions remain close while FeatureBench retains a visible shift.
  • The treatment moved the reading boundary: Treatment moves the reading boundary to more than twice the control location on every benchmark, with every repository-bootstrap interval excluding one.The reading boundary measures where half the fitted runs stop while reading before the first source change; it does not itself measure task resolution.

5 The same treatment across model designs

The frozen treatment package improves both partial repair and complete solutions across additional model designs on the same tight-window Verified cohort. This transfer comparison changes model design while keeping the treatment unretuned.

  • Transfer across model designs: The treatment package remains fixed without harness-code forks or treatment retuning, although transfer models use model-specific profiles and serving settings.Both arms within each model use the same declared four-bit weight file and compatibility setup.
  • Transfer across model designs: Treatment raises mean per-task F2PF and complete solutions for every tested additional model on the 169-task Verified cohort.The cohort uses a 20,480-token window and a fixed 480-second attempt endpoint.
  • Transfer across model designs: Devstral reaches 2.1× the control mean per-task F2PF and 2.4× as many complete solutions.These are the largest proportional changes reported in the model-design comparison.

6 What these results change

The paired results show that harness configuration changes what fixed model weights accomplish, especially under context pressure. Evaluations should therefore report and test the complete model–harness solver, while recognizing the study’s scope and comparison boundaries.

  • Changing the harness changed completed work while model weights and tasks stayed fixed, so performance belongs to the model and harness together.
  • Treatment produced large gains across all three pressure cohorts, while Verified and Pro outcomes were close when context was effectively unconstrained.
  • The working-view rule shortens older results only after context pressure develops, preserving the full record while managing current working space.
  • Builders should version and test view construction, tool behavior, and run control as solver-design choices, and evaluations should report these settings.
  • The comparison estimates the treatment-package effect as run rather than a compute-matched efficiency effect, and one trajectory per arm leaves run-to-run variation unmeasured.
  • Evidence covers four locally served open-weight checkpoints and repository-level coding tasks, excluding hosted frontier models, human patch-quality judgment, and other task types.

7 What to test next

The next studies should separate the fixed treatment package’s components and test how time, context, and adaptive harness behavior interact across models and tasks. Additional experiments should also measure variation and prevent redundant context growth.

  • Future experiments should first separate the fixed treatment package’s limits before testing complete-harness adaptation across tasks and model designs.
  • A prespecified component study should vary detector responses while holding half-life context and command safeguards fixed.
  • Repeated runs at several time limits could distinguish tasks needing more time from those needing more context.
  • Adaptive-harness experiments should respond to context-filling speed and whether the model has begun editing, using a fixed rule on new tasks.
  • Repeated control and treatment runs would reveal both average change and run-to-run variation.
  • Caching repeated read-only results could mechanically prevent duplicate output when the request and source hash remain unchanged.
  • A direct severe-to-unconstrained window study could show how quickly each model approaches its ceiling and how much pressure-gap recovery treatment provides.

8 Conclusion

The same model produced different coding-agent results under different harness configurations. Treatment improved pressure-cohort outcomes across benchmarks and models, supporting evaluation of the model and harness as one solver.

  • Under context pressure, treatment repaired more required behavior on all three benchmarks and completed more tasks on Verified and Pro.
  • When context was effectively unconstrained, outcomes were close on Verified and Pro, while FeatureBench retained a treatment F2PF gain.
  • On the 169-task Verified cohort with a 20,480-token window and 480-second endpoint, treatment raised both endpoints for every tested model.
  • The paired results support treating the model and harness as one solver and reporting the harness, tools, context policy, and run controls.

A.1 Outcome selection

The outcome is the best finalized task result produced under each arm’s fixed attempt budget and scoring protocol.

  • Each task contributes one finalized result per arm under the fixed attempt budget and scoring protocol.

A.2 Cohorts and evaluation setup

The study uses paired, frozen control and treatment comparisons across three benchmarks, with task-level outcomes recorded under benchmark-specific cohorts and fixed execution protocols. The comparisons include pressure and wider-window regimes, plus sensitivity analyses for denominators, repositories, and provenance.

  • Evaluation design: Each paired comparison used one task-level trajectory per arm under greedy decoding, estimating paired differences rather than pass@k or run-to-run variation.Tasks, not resampled trajectories, were the units of inference.
  • Cohorts and regimes: The reported tables cover complete resolution and mean per-task F2PF across primary pressure and larger-window Qwen3.6 comparisons, with a post-hoc common-cohort Verified projection.Repository-cluster sensitivity analyses repeat the primary comparisons.
  • Cohorts and regimes: The three Verified comparisons shared a fixed treatment package, while Pro and FeatureBench pressure rows used an earlier complete version; historical 262,144-token rows were not exact final-package replications.The FeatureBench wide-window row also paired an earlier treatment with a later corrected control.
  • Sensitivity checks: 239 of 871 finalized SWE-bench Pro opportunities lacked independent code stamps, limiting the code-identity record without showing that different code ran.The public provenance record identifies all seven Qwen3.6 rows and provides resolved configuration hashes for all four primary pressure cells.
  • Sensitivity checks: 10.53% versus 19.38% was the FeatureBench pressure comparison’s mean per-task F2PF for control and treatment, with 47:9 non-tied directions favoring treatment.The direction count indicates the observed treatment direction was not explained by unequal denominator size.

A.3 Resource use under pressure

Under pressure, the treatment changed the model’s working view and continued runs beyond the control’s context limit, while consuming more total model work. In the wide-window Verified comparison, it served fewer prompt tokens per turn but took slightly more turns, and fitted treatment reading boundaries exceeded control boundaries across benchmarks.

  • Model work: Treatment used more model work in every primary Qwen3.6 pressure comparison and more often remained active after control reached its context limit.The efficacy results include this difference in realized work.
  • Wide-window context use: 7.2% fewer prompt tokens per turn was the treatment geometric-mean ratio in the 262,144-token Verified comparison.This estimate used 153 paired tasks with defined prompt-token-per-turn ratios.
  • Wide-window context use: 1.4% higher was treatment’s mean number of turns among 154 paired tasks with both arm metrics in that wide-window Verified comparison.Prompt tokens per task therefore also reflected differing numbers of turns.
  • Reading boundaries: Treatment fitted reading boundaries above control in all three benchmarks, with treatment-to-control intervals excluding one.Because control locations differed across benchmarks, these window-scale results are descriptive.
  • Harness activity: Mechanical decay shortened older tool results while new tool results entered at full size, allowing assembled prompt length to drop and rise again.Figure 9 records assembled prompt tokens at each model call against the configured window.
  • Observed outcomes: Every Qwen3.6 variant was lower than its frozen baseline on both complete solutions and mean per-task F2PF in the rerun comparison.No altered setting improved resolution beyond the observed rerun difference.
Loading 2608.26218v1…