Source-linked AI summary
Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
Mohsen Arjmandi
TL;DR
Benchmarks often vary models or bundle each vendor’s model and harness, leaving limited evidence for teams choosing a harness for a model they already have. This paper uses paired same-model contrasts on a private task pool; neither contrast resolves an average harness advantage, while the authors’ practical guidance is to consider workload mix, completion behavior, and billed cost.
Problem
Existing coding-agent benchmarks often vary the model or report a vendor’s model and harness as one system, leaving limited evidence for choosing harness software when the model is fixed.
Source: §1 · p. 1Method
The study compares vendor-native and neutral harnesses in paired same-model contrasts on a private, contamination-controlled pool of 80 tasks.
Source: §1 · p. 2Results
Neither contrast resolved an average harness advantage: Opus 4.8 was −1.25 pp (task-bootstrap 95% CI [−10.0, +7.5]) and GPT-5.5 was +1.25 pp (CI [−4.4, +6.9]).
Source: §9 · p. 17Takeaways & Limitations
Harness choice should reflect workload mix, desired completion behavior, and billed cost under the user’s own pricing terms, rather than aggregate solve rate alone.
Source: §6 · p. 15The frozen pool oversamples screening-discordant tasks, so its averages are conditional on that selection rather than a random sample of the suite.
Source: §7 · p. 16
Abstract
from arXiv · showhide
An agentic coding system couples a language model to a harness: the tools, prompts and control flow that turn a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners assume the vendor-native pairing solves more tasks. We measure that assumption with paired same-model contrasts on a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks. The same 80 tasks ran under claude-agent-sdk and under deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5, with gemini-3.5-flash and deepseek-v3.2 as side cells. 792 of 800 planned runs were graded by an isolated oracle. Neither contrast resolves an average advantage for either harness: -1.25 pp for Opus 4.8 (48.8% vs 50.0%, task-bootstrap 95% CI [-10.0, +7.5]) and +1.25 pp for GPT-5.5 (55.6% vs 54.4%, CI [-4.4, +6.9]). The Opus average combines opposite strata: the native harness trails by 9.0 pp on the 61 repository tasks and leads by 23.7 pp on the 19 contest tasks (label-permutation p = 0.003). The partition was chosen after seeing the data and needs a designed replication. Correctness and completion also separate: 22 of 81 runs cancelled at the wall-clock ceiling had produced a passing patch. Re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times on GPT-5.5. These are observed-usage estimates. On the Anthropic account 58 runs left no usage record, and allocating that spend to either cell would move the Opus ratio between 0.7 and 2.3, so the billed ordering is unresolved. This revision corrects an August 2026 manuscript whose cost figures rested on a usage-semantics defect in our own telemetry (Section 5.1). We release the orchestrator, grading oracle, reanalysis code and derived aggregates. The tasks stay private.
1 Introduction
The paper asks whether a vendor-native harness outperforms a neutral harness when both drive the same model, and tests this with paired contrasts on a private, contamination-controlled task pool. It describes the task suite, contamination safeguards, and execution plan, while noting a correction to its cost telemetry and protocol record.
1 Introduction
- A post-study review found that the cost normalizer applied one SDK’s cache-token convention across harnesses, prompting this revision to re-derive cost figures from raw events.The paper also releases the orchestrator, grading oracle, reanalysis code, and derived aggregates.Source: §1 · pp. 1–2
- Paired same-model harness contrasts on a private, contamination-controlled pool of 80 tasks address whether vendor-native harnesses solve more tasks.The paper frames the choice as relevant to teams that already have a frontier model and must choose its driving software.Source: §1 · pp. 1–2
2.1 Task suite
- The 256-task suite combines 179 repository tasks from four private production codebases with 77 post-cutoff contest tasks.Repository tasks include hidden tests and gold patches; contest problems also have hidden test suites.Source: §2.1 · p. 2, §1 · p. 2
2.2 Contamination protocol
- The suite’s contamination controls include a frozen cutoff registry and runtime model-identity checks, but cannot rule out exposure through undocumented channels.The authors did not run the canary-emission probe on the study models, so the claim is bounded by documented cutoffs and private provenance.Source: §2.2 · p. 3
2.3 Experimental matrix, protocol deposit, and amendments
- The 80-task pool oversamples tasks where the two screening models disagreed, so main-run averages are conditional on that selected pool.The plan was committed before scored runs, but no OSF registration was created; the deposited PREREGISTRATION.md described a four-model benchmark rather than harness contrasts.Source: §2.3 · p. 4
3 System and Execution
The evaluation runs coding agents in isolated microVM sessions and grades extracted patches offline after execution. The orchestration records run events and usage in a ledger, while a separate oracle grades patches and AI tools assist research tasks under human direction.
3.1 The platform as instrument
- Each evaluation run is a chat session in a fresh-clone KVM microVM, with vendor approval and sandboxing layers disabled so harnesses face identical constraints.Runner messages are persisted to an append-only event log to make runs replayable and resumable.Source: §3.1 · p. 5
3.2 Orchestration, ledger, and telemetry path
- Terminal runs append tokens, accounted cost, end reason, and attempt to a ledger, while a 1,200-second ceiling cancels sessions and preserves the patch for grading.The system halts if a ledger append cannot be recorded, and operational stop-loss controls use accounted euros.Source: §3.2 · pp. 5–6
3.3 The grading oracle
- A separate Docker-isolated oracle grades extracted patches against hidden tests, with polarity and determinism invariants gating each environment.After purging and regrading suspect keys, the oracle produced 792 verdicts with zero grader errors.Source: §3.3 · p. 6
3.4 Use of AI tools in this research
- AI tools assisted implementation, task preparation, and analysis under human direction, but the evaluated agents were measurement subjects and did not grade patches.The released reanalysis script reads the checksummed archive, and figures and tables are regenerated from its output with assertions against the canonical results sheet.Source: §3.4 · pp. 6–7
4 Capability Results
Neither paired contrast resolves an average harness advantage; the Opus average combines opposite repository and contest patterns. Some runs cancelled at the wall-clock ceiling nevertheless produced passing patches.
4.1 What was graded, under which conventions
- 792 oracle verdicts were graded; published solve rates average pass/fail verdicts over tasks, excluding timeout and apply_error verdicts from each task’s denominator.Interim stop-loss rates differed from final rates; only the full-power figures are citable.Source: §4.1 · p. 7
4.2 The paired contrasts
- −1.25 pp (Opus 4.8) and +0.6 pp (GPT-5.5) under the all-graded convention; neither contrast resolves an average harness advantage.The task-bootstrap 95% intervals are −10.0 to +7.5 pp for Opus and −4.4 to +6.9 pp for GPT-5.5.Source: Figure 2 · p. 8, §4.2 · p. 8
4.3 The workload split
- On Opus 4.8, native trails neutral by 9.0 pp on 61 repository tasks but leads by 23.7 pp on 19 contest tasks; the post-hoc split yields p = 0.003.The split was chosen after seeing the panel and is one of several possible partitions, so the pattern is not a confirmed population interaction.Source: Figure 3 · p. 9, §4.3 · pp. 9–10
4.4 Correct output and autonomous completion
- 22 of 81 runs cancelled at the 1,200-second ceiling had passing patches, separating correctness from autonomous completion in these data.Neutral-harness ceiling hits were more frequent, a latency and cost fact rather than a capability result.Source: §4.4 · p. 10
4.5 Side cells, screening agreement, and per-family structure
- The four frontier cells range from 48.8% to 55.6%; screening native bundles agreed on 91% of 256 tasks, comparing model-harness bundles rather than harnesses alone.The Opus pair’s per-family rates also varied, including 84% versus 61% on contest tasks.Source: §4.5 · p. 10
5 Cost Results, Telemetry Correction, and Calibration
Repricing raw per-turn usage corrected the study’s cost estimates after a cache-token double count and a pricing error in the original ledger. The corrected observed-usage estimates favor the native harnesses on cost per solved task, but missing Anthropic usage records leave the billed Opus ordering unresolved.
5.1 A usage-semantics defect in the telemetry path
- Deepagents double-counted cache tokens for inclusive LangChain usage, while the ledger also charged cache writes at both input and cache-write rates.For one example trajectory, 1,000 inclusive input tokens became 1,950 ledger tokens; cache-read share was reported as 46% rather than 90%.Source: §5.1 · p. 11
- Raw-event reconciliation was exact for 693 of 703 rows with usage, but C6 cost was not estimable because only 16 of 131 rows had usage records.Ten rows differed by −2.6% to +0.2% from the double-added total; runs without a result_done record had no tokens in either source.Source: §5.1 · p. 11, Table 7 · p. 11
5.2 Cost per solved task from observed usage
- 1.63 at list prices for Opus and 1.18 for GPT-5.5 are the observed-usage C2/C1 cost-per-solved-task ratios; the study reports these as 1.3–1.6 and 1.2, respectively.The Opus range reflects alternative pricing bases; neither ratio bounds billed cost.Source: §5.2 · p. 12, Figure 4 · p. 12, Table 8 · p. 12
- On Opus, the $164 cost gap comprised $124 in cache reads, $17 in cache writes, and $23 in output; GPT-5.5’s $41 gap comprised $17 in uncached input, $19 in reads, and $5 in output.The neutral Opus cell had the same cost-component structure as the native cell, but at larger volume.Source: §5.2 · p. 12, Table 9 · p. 13
5.3 What billed spend does and does not confirm
- 58 Anthropic runs had no usage record, and allocating the $288 residual across the Opus cells moves the C2/C1 ratio from 0.68 to 2.29, leaving billed ordering unresolved.These allocations are illustrations, not estimates, and assume the balance and ledger windows coincide to the day.Source: §5.3 · p. 13, Table 12 · p. 14
5.4 The truncation probe
- The truncation probe was inconclusive: it could not establish whether the 30 KB cap applied, and its cost intervals could not distinguish moderate savings from increases.The study recommends retaining raw tool-output lengths and truncation events to demonstrate exposure to a cap.Source: §5.4 · p. 14, Table 13 · p. 14
6 Discussion
The average Opus harness contrast obscures opposite repository and contest patterns, while correctness and autonomous completion also diverge. The paper therefore argues for workload-specific comparisons and separate reporting of solve rate, terminal states, completion time, and billed cost.
Workload composition
- −1.25 pp overall for Opus combines a −9.0 pp repository effect and a +23.7 pp contest effect; the post-hoc split needs a designed replication.The repository and contest strata comprise 61 and 19 tasks, respectively; the partition was chosen after seeing the data.Source: §6 · p. 14
Correctness and completion
- 22 of 81 runs cancelled at the wall-clock ceiling had produced a passing patch, separating correct output from autonomous completion.The neutral harness hit the ceiling more often and issued about twice the tool invocations at similar solve rates.Source: §6 · pp. 14–15
Cost evidence
- Measured token cost and billed operational cost require different evidence because billed spend includes unrecorded turns and actual provider rates.On Anthropic, 58 missing usage records leave the Opus ratio between 0.7 and 2.3 under alternative residual allocations.Source: §6 · p. 15
Experimental design
- A truncation experiment needs demonstrated exposure to its cap; retaining raw tool-output lengths and whether the policy fired makes that comparison interpretable.The probe could not establish whether its 30 KB cap applied because traces retained only previews.Source: §6 · p. 15
Related work
- Related studies report harness effects conditional on workload, context regime, and endpoint, but none uses this study’s vendor-native-versus-neutral comparison.The paper cautions that HarnessRouter’s single-task application-credit results are not comparable to its observed-usage estimates.Source: §6 · p. 15
Practitioner implications
- The neutral harness cost 1.2–1.6 times as much per solved task on observed usage, but the Anthropic billed ordering remains unresolved.The paper recommends choosing based on the user's workload mix, needed completion behavior, and billed cost under the user's pricing terms.Source: §6 · p. 15
7 Threats to Validity and Limitations
The study’s estimates are bounded by task selection, configuration differences, reduced power, post-hoc analyses, incomplete cost records, ceiling-censored runs, and limits on rerunning the benchmark.
Task provenance and selection
- All 179 repository tasks came from one organization, and the pool’s family mix reflects mining yield rather than the population of software work.The pool oversampled screening-discordant tasks; the per-task labels needed to assess white-box coupling were not retained.Source: §7 · p. 16
Configuration differences beyond the harness
- The codex cell used 4 GiB VMs while other cells used 8 GiB, and inference budgets and tool surfaces also differed.The contrasts therefore compare deployed harness configurations on the same model, not identical environments.Source: §7 · p. 16
Power and inference; Post-hoc analyses
- The 80-task pool and two repeats for C2/C4 left task-bootstrap intervals 11–17 pp wide; the planned detectable effect is not an observed bound.The repository-versus-contest workload split and outcome-flow reanalysis were post hoc, and the number of partitions considered cannot be recovered.Source: §7 · p. 16
Cost measurement
- The corrected estimator matched OpenAI billed spend within 2–4%, but a 25–29% Anthropic residual and uncertain cache-write pricing leave billed Opus ordering unresolved.Anthropic’s Opus ratios are observed-usage estimates; C6 cost is not estimable.Source: §7 · p. 16
Termination and ceilings
- The 1,200-second ceiling censored time rather than correctness: 22 of 81 ceilinged runs passed, and a longer ceiling could change neutral-harness solve rates and costs.Interim results at the stop-loss halt also differed from the final results by several points in both directions.Source: §7 · p. 16
Reproducibility after teardown
- The checksummed archive supports the reported numbers but not rerunning agents against the tasks, because per-task mirrors were deleted and contest tests remain private.Track A tasks are reconstructable from private source repositories at recorded commits; the sampled commits resolved in August 2026.Source: §7 · p. 16
8 Related Work
Related work covers repository benchmarks, contamination controls, harness comparisons, and usage accounting. The paper distinguishes its private same-model vendor-native-versus-neutral comparison from prior harness studies with different comparators and settings.
Execution-based coding benchmarks
- SWE-bench established hidden-test repository issue resolution, LiveCodeBench a rolling post-cutoff pattern, and task-quality validation motivated the study’s prompt-finalization protocol.The study adopts SWE-bench’s FAIL_TO_PASS and PASS_TO_PASS discipline and uses post-cutoff contest tasks.Source: §8 · p. 17
Contamination
- Prior contamination mitigations include private suites, post-cutoff harvesting, and canary strings; this study combines them with a frozen cutoff registry and runtime drift gate.The drift gate addresses the possibility that providers swap dated snapshots behind stable aliases.Source: §8 · p. 17
Harness comparisons
- Harness-Bench, Scaffold Effect, and Same Model, Different Harness use different harness comparators or interventions; this study compares vendor production harnesses with a neutral harness on the same model.The cited studies report variation in completion and process measures, pass-rate differences with intervals often including zero, and context-dependent effects, respectively.Source: §8 · p. 17
Inference economics and usage accounting
- Prior work argues for reporting cost alongside accuracy and describes provider-specific cache usage; the paper says multi-SDK double counting and reconciliation against provider balances had not been documented.The paper cites prompt caching as making effective input price depend on trajectory shape.Source: §8 · p. 17
9 Conclusion
Neither harness shows a resolved average advantage, while Opus results differ by task stratum; observed-usage costs favor the native harness, but billed ordering on Anthropic is unresolved. The replication package releases evaluation infrastructure and derived aggregates, while keeping the task suite private.
9 Conclusion
- −1.25 pp for Opus 4.8 and +1.25 pp for GPT-5.5, with task-bootstrap intervals spanning zero, leave neither contrast with a resolved average harness advantage.On Opus, the average combines a repository stratum favoring neutral and a contest stratum favoring native; this post-hoc pattern had permutation p = 0.003 and needs designed replication. The neutral harness cost 1.2–1.6 times as much per solved task on observed usage, while Anthropic billed ordering remains unresolved.Source: §9 · pp. 17–18
A Execution chronology and incidents (condensed)
- 792 of 800 planned runs were graded after the main run stopped at 502 rows and a finish leg followed.The timeline records stop-loss and provider-resume incidents, followed by reanalysis after the September review found the telemetry defect; incidents became regression guards and did not retroactively alter graded data.Source: §9 · p. 18, A Execution chronology and incidents… · p. 18
Data Availability
- The replication package provides the orchestrator, grading oracle, reanalysis script, derived aggregates, and claim-to-evidence records, but not the 256 tasks or hidden tests.The authors offer suite access to an artifact-evaluation committee under a non-disclosure arrangement on request.Source: Data Availability · p. 18
B Configuration summary
The supplement documents configuration and statistical checks, including limits on image pinning, post-hoc workload analysis, and the interpretive scope of secondary tests.
B Configuration summary
- The sandbox image digest was unpinned on all 891 ledger rows, although a build fingerprint was recorded for 889 sessions.The effective configuration was recorded per row; the scored phases used a 1,200-second ceiling and concurrency 12.Source: B Configuration summary · p. 19
C Statistical supplement
- The task-family partition was chosen after seeing the results, so the Opus workload interaction remains post hoc despite its revised q = 0.013.The primary test family changed after reviewer request, moving the interaction from q = 0.10 to q = 0.013.Source: C Statistical supplement · p. 19
- The label-permutation test targets whether workload labels carry information about task-level harness effects, whereas the stratified sign-flip test tests no effect in either stratum.A common nonzero effect across strata can also reject the sign-flip null; the GEE interaction coefficient uses a different, log-odds estimand.Source: C Statistical supplement · p. 20
- Cluster-adjusted McNemar tests gave p = 0.78 for Opus and p = 0.65 for GPT-5.5, consistent with the task-bootstrap intervals.The adjustment accounts for correlation between the two repeats of each task.Source: C Statistical supplement · p. 20
- The pass@1, pass@2, and both-repeats-solved measures use matched subsets of 78 Opus and 77 GPT-5.5 tasks, with a planned-key sensitivity over all 80.Table 15 also reports the planned-key sensitivity, counting any missing or non-pass repeat as failure.Source: C Statistical supplement · p. 20, Table 15 · p. 20
- The OpenAI GEE contest coefficient is not estimable because both cells solve every contest task, and its reported value is a separation artefact.Table 16 provides the coefficient tables for three logistic GEE models with exchangeable within-task correlation and robust standard errors.Source: C Statistical supplement · p. 20, Table 16 · p. 20
- Table 17 identifies the contest family (lcb) as the stratum where the Opus pair reverses direction.It reports per-cell solve rates by task family on the frozen pool.Source: C Statistical supplement · p. 20, Table 17 · p. 21
- Under each SDK’s own usage semantics, all six cells had 87–98% cache-read shares; the ledger’s apparent 48% regime double-counted neutral-harness cache tokens.The panels show the same runs, with the recorded panel’s denominator counting cache reads twice.Source: Figure 5 · p. 21
D Cost supplement
The corrected cost analysis uses SDK-specific usage semantics and frozen list prices, while missing Anthropic usage records keep the billed comparison uncertain.
D Cost supplement
- 1.14 on C1 and 4.8–7.7 on deepagents are the ledger cost overstatement ratios from the combined telemetry defects.C1 reflects cache-write double charging alone; deepagents also had cache tokens double-counted.Source: D Cost supplement · p. 21
- The August manuscript’s 48% cache-read regime came from a ledger denominator that counted cache reads twice.The figure compares the recorded runs with the same runs interpreted using each SDK’s own semantics.Source: D Cost supplement · pp. 21–22
- On Opus, the C2 minus C1 cost gap of $164 comprises $124 in cache reads, $17 in cache writes, and $23 in output.On GPT-5.5, the C4 minus C3 gap of $41 comprises $17 in uncached input, $19 in cache reads, and $5 in output.Source: D Cost supplement · p. 22
- The Anthropic missing-spend allocations are illustrative ranges, not estimates, and depend on account and ledger windows coinciding and the residual being unrecorded model usage.The rules show possible billed-ratio ranges under different allocations of the residual.Source: D Cost supplement · p. 22
E Probe supplement
The truncation probe cannot test its intended hypothesis because the archived tool-result previews do not reveal whether the 30,000-byte cap ever bound.
E Probe supplement
- The probe changed the deepagents tool-output truncation cap to 30,000 bytes on 12 repository tasks, but also differed in concurrency, repeats, and run date.It ran after the finish leg and included one repeat rather than two, with concurrency 8 rather than 12.Source: E Probe supplement · p. 22
- No archived tool-result preview exceeded 2,000 characters, so the archive cannot establish whether any output exceeded 30,000 bytes and the probe’s null is uninterpretable as a truncation test.The stored previews covered 24 main-run sessions and 26 probe sessions.Source: E Probe supplement · p. 22
F Claims of the August 2026 manuscript and their status
- The August manuscript’s claims and their revision status are listed in the paper, with the full claim-to-evidence table accompanying the replication package.The package points into the derived data.Source: F Claims of the August 2026 manuscript… · p. 22
G Incident history
The revision reframes the harness contrasts as unresolved average advantages and corrects the earlier cost analysis after finding a telemetry defect. It also distinguishes graded correctness from completion and records limits in the interim probe and contamination controls.
G Incident history
- The revised analysis leaves point estimates unchanged but treats the −1.2 and +1.2 pp harness contrasts as unresolved average advantages.Timeout and apply_error conventions shift cell means by less than 0.5 pp.Source: G Incident history · p. 23
- 22 of 81 wall-clock-ceilinged runs passed, and the wall-clock ceiling was not a graded failure by design; §4.4 also reports a 48% cache-hit share across deepagents cells.The 48% share is described as a telemetry double-counting artefact; source-semantics shares are 87–98%.Source: G Incident history · p. 23
- A telemetry double count produced about 4.3× self-report inflation in neutral-harness cells, but its mechanism remains unpinned.The passage identifies the double count as an artefact and reports source-semantics shares of 87–98%.Source: G Incident history · p. 23
- Observed-usage repricing puts deepagents at 1.3–1.6 times the cost per solved task on Opus and 1.2 times on GPT-5.5, while Anthropic’s billed ordering remains unresolved.The earlier cost figures were withdrawn after a telemetry double count and cache-write pricing error.Source: G Incident history · p. 23
- The OSF record comprised a repository-dated plan and a different private design deposit, rather than a single matching registration.The passage states that no registration matched the plan as described.Source: G Incident history · p. 23
- The truncation hypothesis was falsified, while the probe remained inconclusive and its manipulation check was impossible from the archive.The contamination claim is bounded to documented cutoffs and private provenance; the canary emission probe was not run.Source: G Incident history · p. 23
- Interim stop-loss solve rates were about 5 pp higher for frontier cells and lower for C6, but only full-power figures are citable.The pattern is consistent with the halt censoring slow runs, though two snapshots cannot establish the mechanism.Source: G Incident history · p. 23
Appendix B
Appendix B documents operational incidents and their disposition, including a grading re-run that yielded the reported verdicts and a telemetry defect that affected cost figures but not graded outcomes. The reproduction pointers emphasize ledger integrity, usage-semantics checks, and logging exposure to policies under test.
Appendix B
- The stop-loss overshot its €100 cap to €275 when in-flight runs continued spending; a predictive dispatch brake limited the main run’s overshoot to €3,303.34 against €3,300.Source: Appendix B · p. 23
- Credit exhaustion and silent runner deaths were scrubbed as retryable infrastructure failures and re-run, recovering screening rows rather than grading them as agent failures.The scrub recovered 72 C3 and 24 C1 screening rows, plus rows after the main halt and in the finish leg.Source: Appendix B · p. 23
- The idempotent purge-and-regrade of 114 suspect keys yielded zero grader errors and the 792 reported verdicts.The initial grading pass had 85 grader errors and 29 timeouts from disk exhaustion on the grading host.Source: Appendix B · p. 24
- The telemetry double count affected the stop-loss currency accounting and every August-manuscript cost figure, but no graded verdict.Source: Appendix B · p. 24
H Reproduction pointers
- The reproduction checklist calls for deterministic session IDs, terminal graded outcomes, bounded infrastructure retries, and halting on an unrecordable ledger append.It also recommends reconciling raw per-turn usage against ledger totals and billed spend before publishing prices.Source: H Reproduction pointers · p. 24
- The released materials exclude tasks, golds, hidden tests, and canaries, so external reproduction requires re-mining private code under the published protocol.The checklist also calls for per-tool-call output-length logging and recording whether truncation, summarization, or eviction fired.Source: H Reproduction pointers · p. 24