Source-linked AI summary
Stop Comparing LLM Agents Without Disclosing the Harness
Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, Chandan K. Reddy
TL;DR
Current long-horizon agent benchmarks often attribute joint model–harness outcomes to models while leaving the execution harness undisclosed. The paper formalizes the harness as a closed-loop controller, reports controlled evidence that harness variance can dominate model variance, and proposes disclosure and variance-aware evaluation. Its conclusion is limited to long-horizon tasks involving comparable frontier models and remains empirically falsifiable.
Problem
Agent benchmarks report scores as model properties even though context construction, tool mediation, verification, recovery, and stopping jointly affect each outcome.
Method
The paper combines a closed-loop control formalization, benchmark and deployment evidence, controlled variance decomposition, and a harness-aware disclosure framework.
Results
Average harness variance is 18.48 pp^2 versus 2.37 pp^2 for model variance, a 7.80× ratio, with six ranking reversals in the controlled grid.
Takeaways & Limitations
Long-horizon leaderboard comparisons should disclose the harness and use locked-harness or factorial protocols before attributing performance differences to models.
Takeaways & Limitations
The thesis is restricted to long-horizon tasks with comparable frontier models and can be falsified if model variance exceeds harness variance in a controlled experiment.
Abstract
from arXiv · showhide
This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps. We formalize and defend the Binding Constraint Thesis: in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements. We support this thesis along three lines. First, a control-theoretic formalization treats the harness as the controller of a closed-loop dynamical system and the LLM as the stochastic policy it governs, which explains why small harness changes can produce performance shifts that exceed those obtained by substituting one model for another. Second, published benchmarks, industry deployments, and a controlled variance decomposition show that harness-induced variance can substantially exceed model-induced variance, including cases of model ranking reversal. Third, we propose a harness-aware evaluation framework with a disclosure standard and a variance decomposition protocol. Until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.
1 Introduction
Current agent benchmarks report scores as if they were properties of models, although execution harnesses jointly produce those scores and are rarely disclosed. The paper argues that valid comparisons require controlling or varying the harness explicitly.
- The attribution problem: Benchmarks commonly attribute one reported score per model–benchmark pair to the model alone.This convention directs attention toward model scaling, tool-use fine-tuning, and prompting.
- Harness effects: 77.0% versus 69.7%: changing only the harness raises Terminal-Bench 2 pass@1 with the model held fixed.Independent monitoring also reports up to 15 percentage points of scaffold-only variation on SWE-bench Verified.
- What the harness does: The harness constructs context, mediates tools, validates outputs, and decides when to retry, escalate, or stop.These functions form the infrastructure layer between the model and task.
- Proposed evaluation standard: Valid cross-model comparisons require either one disclosed locked harness or a factorial protocol that varies harness choice and reports variance components.Disclosure covers context construction, tool-call mediation, and output verification.
- Consequences: Undisclosed harnesses make leaderboard conclusions systematically attribute joint model–harness outcomes to the model.The paper also argues that harness changes receive too little attention despite their importance for long-horizon performance.
2 Why Current Evaluation Conflates Model and Harness
Agent evaluation conflates model and harness because every step from context construction through recovery and stopping is a harness decision, yet published scores usually record only the model. Evidence across benchmarks and proposed responses indicates that harness substitutions can change rankings and that disclosure, rather than model scaling or more benchmarks alone, addresses the attribution problem.
- How the harness enters evaluation: Agent evaluation constructs context, parses outputs, routes tools, handles errors, compresses trajectories, and stops through harness decisions.The published score records the model while eliding these execution choices.
- Observed conflation: 55.4% versus 45.9%: Claude Opus 4.5 scores differently on SWE-bench Pro under Claude Code versus the standardized SEAL scaffold.Other reports include up to nearly 48 percentage points of single-model scaffold variation and 11–15 points of scaffold-only variation.
- Observed conflation: Harness substitutions can reverse model rankings, including a SWE-bench Pro ordering change after adding the WarpGrep search subagent.These shifts routinely exceed the 2 to 4 percentage point changes reported as meaningful model advances.
- Why attribution matters: The same composite score is read as a model property, benchmark progress measure, and deployment agent property, but only the deployment reading is well-defined without disentanglement.Current reporting does not separate the model and harness contributions needed for the first two readings.
- Structural explanation: The closed-loop account assigns stability, context drift, and control lag to the harness controller rather than the open-loop model policy.The harness controls how state is filtered, validated, exposed, and updated across steps.
- Responses and solution: Locking one harness enables cleaner model comparisons, but the paper argues that standardization alone embeds design choices and hides harness-side gains.The proposed solution is disclosure with either locked-harness or factorial evaluation.
- Responses and solution: More capable models, more benchmarks, and harness optimization each address part of the problem without resolving attribution unless the harness becomes an experimental condition.The text reports execution and orchestration failures as predominant over knowledge failures in one cited analysis.
3 The Binding Constraint Thesis
The Binding Constraint Thesis states that, for comparable frontier models on long-horizon tasks, harness-induced variance can match or exceed model-induced variance. It formalizes this claim through closed-loop reliability channels and tests it with variance decomposition, while restricting the thesis to an empirical regime that can be falsified.
- Thesis: HV is often comparable to or larger than MV in the stated regime, while single-harness benchmark protocols leave HV unmeasured.The paper therefore treats undisclosed leaderboard comparisons as incomplete and potentially misleading.
- Variance decomposition: Total performance variance is decomposed across model effects, harness effects, and their interaction over the model–harness grid.The interaction matters because the same harness change can produce different gains for different models.
- Reliability channels: Harness controller variance operates through stability, context drift, and control lag, which govern long-horizon reliability channels.These quantities describe progress stability, preservation of task-relevant context, and delay between anomaly detection and corrective feedback.
- Scope: The thesis applies to long-horizon tasks requiring multiple steps of tool use, error recovery, and context management among comparable frontier models.It excludes settings where model capability gaps are so large that model variance mechanically dominates.
- Falsifiability: The thesis is falsifiable by holding the model fixed while varying harnesses, holding the harness fixed while varying models, and comparing the resulting variance components.The stated test is whether model variance exceeds harness variance on a comparable long-horizon task distribution.
4 Evidence
Public benchmark evidence and a controlled factorial experiment show that harness changes can produce larger performance differences than model substitutions, including ranking reversals. The evidence supports the Binding Constraint Thesis while noting that observational public data do not isolate the responsible harness components.
- The evidence combines public leaderboard observations with a controlled factorial experiment under matched conditions.
- 4.9 points separate the six leading frontier models under the standardized SEAL scaffold, while varying the harness around Claude Opus 4.5 widens the range to 9.5 points.The within-model harness range is roughly twice the within-harness model range in this slice.
- 2.1 to 2.2 points from adding the WarpGrep search subagent can reverse the MiniMax 2.5 versus Claude Opus 4.6 ordering.The added tool sits on otherwise identical infrastructure, making the model–harness interaction visible.
- 34 points separate Claude Sonnet 4.5 and GPT-5 Medium across SWE-bench Verified Mini scaffolds, while o4-mini shows a gap of nearly 48 points.
- 11 to 15 percentage points of scaffold-only variation are reported on SWE-bench Verified, with scaffold choice described as having the single biggest impact on overall performance.
- Public evidence remains observational because public harnesses vary across multiple dimensions and were built by teams with different engineering budgets.
- 7.80× is the aggregate HV/MV ratio in the controlled grid, with six ranking reversals across nine model-pair and harness-pair comparisons.Changing the harness moves GLM-5.1 by 13.0 points and GPT-5.4 and Kimi K2.6 by 8.5 points each; model changes within fixed harnesses move scores by 3.0, 2.5, and 5.0 points for H1, H2, and H3.
- Trajectory analysis attributes the H1 → H2 shift primarily to reduced control noise and the H2 → H3 gain primarily to closing the verification and recovery loop.
5 A Harness-Aware Evaluation Framework
The proposed harness-aware framework makes harness configuration explicit, measures its contribution through controlled model-by-harness comparisons, and links aggregate variance to trajectory-level mechanisms. It is designed to make harness variance reportable, interpretable, and attributable.
- The framework comprises a structured Harness Card, a variance decomposition protocol, and trajectory-level metrics.
- The Harness Card discloses execution, tool, context, scheduling, observability, and verification layers under the seven-layer ETCSOVG taxonomy.
- A minimal valid variance protocol uses a 2 × 2 model-by-harness grid with task order, execution environment, evaluation script, API parameters, and stopping rules held constant.
- η2_p is a fixed-effects quantity on the chosen grid, can be positively biased in small grids, and should be reported alongside ω2 or a bootstrap interval.
- η2_p is neither necessary nor sufficient for ranking flips, so the protocol pairs it with explicit reversal counts.
- Recovery Rate, Context Retention, and Control Lag connect harness-induced variance to stability, context drift, and corrective-signal timing.Recovery Rate is reported over k ∈ {1, 3, 5, 10}; Context Retention tracks task-relevant content; Control Lag counts steps to a corrective signal.
6 Alternative Views and Counterarguments
The paper addresses four counterarguments by arguing that harness effects persist with stronger models, standardization is only one valid evaluation regime, interactions can reverse rankings, and deployment composition does not eliminate the need for attribution.
- Model capability will dissolve the harness problem: Harness complexity has grown alongside model capability, while managed-agent designs and AHE gains indicate that stronger models do not remove harness effects.The paper attributes long-horizon reliability to controller feedback structure.
- Standardization is enough: Locked-harness evaluation is valid, but disclosure remains necessary because standardized configurations interact with models and may suppress harness-side gains.The paper presents locked-harness and factorial protocols as complementary valid regimes.
- Interaction effects are small enough to ignore: Six ranking reversals across nine model-pair / harness-pair comparisons show that interaction effects can invalidate stable model orderings.SWE-bench Pro and Terminal-Bench examples provide additional evidence of model–harness coupling.
- The harness is part of the deployed agent: Although deployment optimizes model–harness compositions, model-comparison evaluation must disclose the harness to isolate model contributions.The proposed disclosure and variance-decomposition protocols distinguish deployment optimization from model attribution.
7 Conclusion and Discussion
The paper recommends treating harness configuration as part of agent evaluation and directs researchers, benchmark designers, and practitioners toward disclosure, controlled variation, and broader optimization targets.
- Researchers: Researchers should report the harness alongside hyperparameters and decoding settings because model scores without harness specifications omit part of the experimental condition.The paper identifies the harness as a necessary component of reproducible model evaluation.
- Benchmark designers: Benchmarks should offer locked-harness tracks for model comparison or factorial tracks that report harness and model variance components.The paper connects these tracks to ongoing scaffold and standardization programs.
- Practitioners: At a fixed model, moving from H1 to H3 shifts pass@1 by 8.5 to 13.0 percentage points, versus 2.5 to 5.0 points when changing models at a fixed harness.The paper uses this controlled-grid contrast to frame the engineering object as the full operating surface around the model.
- Open questions: The paper leaves open how to define comparable frontier models, measure harness distance, balance disclosure with locking, and extend trajectory-level diagnostics.These unresolved questions limit current cross-paper variance comparisons and motivate additional metrics and perturbation tests.
- Conclusion: The conclusion identifies the execution harness as the binding constraint for long-horizon agents and calls for Harness Cards, variance decompositions, and trajectory-level metrics.It characterizes the model as an open-loop policy and the harness as the controller that closes the loop.
A Example ETCSOVG Disclosure Card
The disclosure framework uses ETCSOVG fields to make harness configurations explicit and illustrates the required information with a compact H3 example.
- ETCSOVG disclosure fields: Table 3 defines the ETCSOVG disclosure fields needed to separate model effects from harness effects.The table is the full field set expected for benchmark submissions.
- H3 example: The compact Table 4 example instantiates the ETCSOVG fields for the H3 configuration used in the controlled experiment.The example demonstrates how the disclosure standard can be applied to one experimental harness.
B Harness Configuration Specifications
The controlled study fixes model-side conditions, constructs a stratified 100-task SWE-bench subset, checks run-level stability, and compares harness costs with pass rates.
- Model selection: Kimi K2.6, GLM-5.1, and GPT-5.4 were selected as comparable models with coding scores of 45.4, 45.3, and 44.6 on the April 23, 2026 snapshot.The snapshot justified comparability at selection time, while live leaderboard values may differ.
- Subset construction: The experiment sampled subset100 with seed 42 and difficulty stratification, then ran every model-harness cell on all 100 tasks in two final runs.The subset preserved the approximate difficulty mix of the 500-task split.
- Model-side controls: All three models used official API services in default configurations across H1, H2, and H3, with no harness-specific API tuning.This keeps the experiment focused on harness variation rather than decoding variation.
- Run-level checks: The first and second runs produced HV/MV ratios of 8.72× and 6.76×, respectively, indicating that harness dominance was not an averaging artifact.Run-level grids recomputed the same variance measures used in the main table.
- Configuration specification: Table 8 enumerates shared controls and varied harness mechanisms at the detail required to reproduce the variance decomposition.The specification distinguishes whether future comparisons change the model, the harness, or both.
- Resource accounting: H2 is the cheapest controller in aggregate, while H3 spends more on verification and recovery to obtain the best pass rate.Sequential time is an accounting quantity rather than parallel-campaign wall-clock time, and per-model costs vary with pricing and tokenization.
C Perturbation Stress-Test Protocol Details
The proposed stress test examines whether small perturbations to controller inputs produce disproportionate output instability, as predicted by the closed-loop dynamical-systems framing. It is presented as future work rather than evidence for the thesis.
- Scope: The proposed control-theoretic test has not been run and is recorded as future work rather than evidence for the thesis.The paper distinguishes this diagnostic proposal from the completed variance decomposition.
- Setup: The planned experiment fixes GPT-5.4 and H2, then perturbs context ordering and tool-output values.Context chunks would be shuffled without changing their contents, while numerical tool outputs would receive noise.
- Metrics: Output instability is the fraction of task trajectories whose final outputs differ from clean-baseline outputs, normalized by perturbation magnitude where applicable.Robust agents should remain near the clean-baseline variance floor, whereas sensitive agents should scale disproportionately with perturbation magnitude.
- Variance context: Table 7 reports run-level pass@1 scores and HV/MV variance decomposition before averaging across the two final runs.The table defines HV and MV as population variances over percentage-point scores.
- Expected diagnostic value: The stress test would measure whether the controller absorbs small changes in context ordering and tool observations.Lower instability for H3 than H1 under matched perturbations would support the role of verification, drift checks, and rollback in closed-loop robustness.
D Industry Evidence Summary
Industry and applied benchmarks provide evidence that agent outcomes depend substantially on execution and orchestration infrastructure, not only on the underlying model. Reported systems therefore treat runtime architecture and harness quality as material design variables.
- Implication: Industry competition is presented as increasingly dependent on harness quality rather than model choice.This observation motivates treating execution infrastructure as a disclosed component of agent evaluation.
- Managed-agent architecture: Decoupling the Brain, Hands, and Session layers reduced time-to-first-token by 60% at p50 and over 90% at p95.The reported gains are infrastructure improvements rather than model improvements, and the architecture notes that harness assumptions go stale as models improve.
- Benchmark evidence: Agent failures in APEX-Agents were predominantly execution and orchestration problems rather than knowledge failures.Across 480 professional tasks, zero-score rates ranged from 40% to 62% across configurations using the same underlying model.
- Public agent systems: Public agent systems commonly wrap frontier models in product-specific execution layers for repository navigation, software development, or runtime interaction.Examples include SWE-agent, OpenHands, Codex CLI, Claude Code, OpenCode, Terminus, and Hermes Agent.
E Mechanism Analysis from Trajectory Logs
Trajectory logs explain harness variance through distinct control-loop mechanisms across H1, H2, and H3. H2 constrains actions and preserves relevant state, while H3 adds verification and recovery signals that expose ineffective progress.
- Variance and rankings: Cross-model variance is smallest under H2 at 1.17 pp2 and largest under H3 at 4.22 pp2, with H1 at 1.72 pp2.The comparison indicates that harness mechanisms can either suppress or expose model differences, depending on their design.
- Mechanism analysis: The H1→H2 improvement reflects reduced control noise rather than increased model knowledge.Stricter formatting, smaller tool schemas, compressed recent history, and structured error feedback reduce malformed outputs, repeated exploration, and broad patches.
- Mechanism analysis: H2 keeps the model on a narrower action path by retaining relevant evidence, retrieving earlier steps, and clarifying tool boundaries.This produces smaller patches that are less likely to disturb retained tests.
- Verification and recovery: H3 adds per-step verification, anomaly checks, and recovery signals that turn ambiguous execution traces into corrective observations.Its verifier flags ineffective commands, failed edits, incomplete fixes, and regression risks before submission.
- Verification and recovery: H3’s additional gains primarily come from closing the loop around verification and regression avoidance, although it frequently spends more steps and tokens.The harness enables the same base model to act on feedback about execution failures and regression risks.