PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Wenxuan Ding, Qin Lu, Changlong Yu, Sha Li, Shuowei Jin, Xin Liu, Zhenyu Shi, Greg Durrett

arXiv:2610.22466v1cs.AI

TL;DR

Harness-evolution search gains may reflect benchmark-specific overfitting or additional inference-time compute, leaving their generalizability uncertain. HDA attributes gains using controlled harness comparisons and held-out evaluation. Across significant-gain settings, overfitting and test-time scaling account for a large share, while held-out gains often fall short of search-set gains.

  • Problem

    Search-set scores do not identify why harness evolution improves performance, and held-out evaluations may share benchmark artifacts that limit evidence of transfer to new tasks or practical distribution shifts.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    HDA compares baseline and evolved harnesses with a compute-matched baseline and an artifact-neutralized evolved harness, then evaluates held-out performance.

    Source: §3 · p. 5

  • Results

    Among 21 settings with statistically significant search-set gains, overfitting and test-time scaling explain 54% and 14% of gains on average, and together exceed 50% in 14 settings.

    Source: §1 · p. 2, §5 · p. 6

  • Takeaways & Limitations

    Held-out evaluation alone does not distinguish test-time scaling from improvement at a fixed inference budget, so HDA complements it by attributing overfitting, scaling, and residual improvement.

    Source: §5 · p. 9, §7 · p. 9

    HDA neutralizes only detected mechanisms that can be disabled without affecting unrelated harness components, so its overfitting estimate is a lower bound.

    Source: §3 · p. 5

Abstract

from arXiv · show

Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizable improvement. We apply HDA to 11 executor models and three harness-evolution frameworks across math, coding, creativity, and agentic tasks. In 21 of these settings that show statistically significant search-set gains, we find that a large share of the improvement is explained by overfitting or test-time scaling. Specifically, among these 21 settings, overfitting and test-time scaling account for 54% and 14% of the gain on average; their combined share exceeds 50% in 14 settings. The residual and held-out transfer vary substantially across tasks, models, and frameworks. Importantly, search-set gains often fail to transfer to held-out tasks, and provide limited evidence that they will persist under practical distribution shifts. Moreover, validation-based signals are insufficient to prevent overfitting. These results motivate attribution-aware evaluation in future harness evolution research.

1 INTRODUCTION

Harness evolution can improve a fixed-weight agent, but search-set gains do not reveal whether improvement transfers beyond the benchmark. Harness-Delta Attribution (HDA) separates gains attributed to overfitting and test-time scaling from a residual upper bound on generalizable improvement.

1 INTRODUCTION

  • Harness evolution iteratively edits non-parametric components around a fixed-weight model using task feedback to optimize performance on the search set.The evaluated components can include harness code, skills, and tool-use or orchestration patterns.Source: §1 · p. 2
  • Search-set gains leave transfer uncertain because held-out instances may share task construction and dataset artifacts with the search split.Such evaluations provide limited evidence that gains persist on new tasks or under practical distribution shifts.Source: §1 · p. 2
  • HDA attributes harness-evolution gains to overfitting, test-time scaling, and residual generalizable improvement.It uses a compute-matched baseline and an artifact-neutralized evolved harness to estimate these sources and evaluates held-out transfer.Source: §1 · p. 2
  • The evaluation covers three frameworks, 11 executor models, and 25 settings spanning math, software engineering, creative generation, and embodied planning.The paper also contributes a reusable HDA skill and evaluation suite.Source: §1 · p. 2
  • Among 21 settings with statistically significant search-set gains, overfitting explains 54% and test-time scaling 14% of gains on average; together they exceed 50% in 14 settings.Different frameworks also repeatedly discover benchmark-specific shortcuts, including a recurring correct answer choice in LiveMath.Source: §1 · p. 2

2 PRELIMINARIES

An agent combines a fixed executor model with a harness of non-weight components. Harness evolution uses a proposer and accumulated evaluation history to generate and assess candidates, with optional validation for gating or selection.

2 PRELIMINARIES

  • An agent is the pairing of a fixed executor model M and harness H, whose components include prompts, skills, tools, memory, and orchestration logic.Source: §2 · p. 2
  • HDA compares baseline and evolved harness scores, a compute-matched baseline, and an evolved harness with detected shortcuts disabled to estimate gain sources.The remaining gap estimates generalizable improvement.Source: Figure 2 · p. 3
  • Starting from baseline H0, a proposer uses the accumulated filesystem-backed history to generate candidate harnesses for evaluation on the search set.The history can contain harness versions, trajectories, evaluation results, and other artifacts.Source: §2 · p. 3
  • Candidate feedback may include scores, per-task rewards, trajectories, and execution logs, which are added to the accumulated history.Source: §2 · p. 3
  • Frameworks expose persisted artifacts directly or insert selected records into a structured prompt or optimizer state.The evaluated harness pool is used to select an evolved harness.Source: §2 · p. 3
  • An optional validation set can gate updates or select harnesses.Source: §2 · p. 3

3 HARNESS-DELTA ATTRIBUTION: DISSECTING GAINS FROM HARNESS EVOLUTION

Harness-Delta Attribution (HDA) separates search-set gains into overfitting, test-time scaling, and residual improvement by comparing evolved harnesses with controlled variants.

Overfitting: Detection and neutralization

  • In one Meta-Harness ALFWorld run, the 0.8B harness grew from 66 to 503 lines (7.6×), reached 100% search-set success, and made 0 LLM calls.The controller hard-coded benchmark priors.Source: §3 · p. 4, Figure 3 · p. 5

Harness-Delta Attribution

  • After paired-bootstrap significance screening, HDA compares the evolved harness with a compute-matched baseline and a neutralized evolved version, with human review at both checkpoints.Insignificant search-set gains receive no attribution.Source: §3 · p. 4

Test-Time Scaling: Compute matching

  • The compute-matched baseline preserves the original harness while scaling its inference budget, estimating gains reproducible through added compute rather than harness evolution.The scaling procedure uses task-appropriate measures such as LLM calls or token usage; its practical value depends on acceptable latency and inference cost.Source: §3 · p. 4

Overfitting: Detection and neutralization

  • HDA defines overfitting as exploiting benchmark-specific patterns and neutralizes detected shortcuts with minimal changes that preserve unrelated evolved components.Audits cover direct shortcuts and benchmark-derived heuristics; benefits may disappear on new instances that retain the task objective but not the exploited pattern.Source: §3 · pp. 4–5
  • O is a lower-bound estimate of overfitting because HDA removes only detected mechanisms that can be disabled without changing unrelated harness components.Implicit or entangled mechanisms may remain.Source: §3 · p. 5

Generalizable Improvements: Residual upper bound

  • The residual G is the gain left after subtracting detected overfitting and test-time scaling, and is an upper bound—not evidence—of generalizable improvement.HDA separately assesses transfer by comparing baseline and evolved harnesses on held-out tasks.Source: §3 · p. 5

4 METHOD AND EXPERIMENTS

The evaluation spans four benchmark domains, three harness-evolution frameworks, and 11 executor models, with validation-based harness selection described for Meta-Harness.

Benchmarks

  • The evaluation covers mathematical reasoning, creative generation, interactive embodied planning, and software coding benchmarks.These are LiveMath, CREATE, ALFWorld, and SWE-bench Verified, respectively.Source: §4 · p. 5

Evolution Frameworks

  • Meta-Harness evolves harness code, SkillOpt edits frozen-agent skills, and AHE evolves prompts, tools, skills, and memory through an observability-driven loop.AHE uses Opus 4.8 as both proposer and executor in the reported self-evolution setting.Source: §4 · p. 5

Evaluation results

  • Table 1 reports HDA results for 25 model–benchmark–framework settings, with O, T, and G shown as capped percentages of search-set gain ΔS.The table marks statistically insignificant gains with † and held-out test regressions with ⋆.Source: Table 1 · p. 6

Validation

  • For Meta-Harness, aggregate validation scores are provided during evolution, and both search-selected and validation-selected harnesses are tested on held-out data.Per-instance validation results and trajectories remain hidden.Source: §4 · p. 6

Models

  • The study evaluates 11 Qwen- and Claude-family executor models, with the exact executor set varying by benchmark.The Qwen models span Qwen3, Qwen3.5, and Qwen3.6 and range from 0.8B to 35B.Source: §4 · p. 6

5 RESULTS

Across significant settings, overfitting and test-time scaling explain much of the observed search-set gain, while the source of gains varies by benchmark. Held-out gains often fall short of search-set gains, and validation selection does not reliably close that gap.

5 RESULTS

  • Overfitting and test-time scaling explain 54% and 14% of gains on average, respectively, and together exceed half the gain in 14 of 21 significant settings.The reported percentages are clipped to [0%, 100%].Source: §5 · p. 6
  • Attribution sources differ by benchmark: ALFWorld and LiveMath are dominated by overfitting, CREATE by test-time scaling, and SWE-bench shows clearer residual generalizable improvement.Across five ALFWorld executors, evolved harnesses reach 100% search-set success, with 98% of their gain attributed to overfitting.Source: §5 · p. 7
  • CREATE’s Creative Utility rewards larger pools of strong, diverse answers, and test-time scaling contributes +5.1 CU, or 75% of observed evolution gain, in AHE.Under Meta-Harness, a compute-matched baseline recovers 49% of Claude Haiku 4.5’s gain and outperforms the evolved Qwen3.6-35B-A3B harness.Source: §5 · p. 7
  • All three frameworks independently rediscover the LiveMath meta-option shortcut, while ALFWorld recipes and CREATE scaling strategies also recur across frameworks.Shortcut-driven gains complicate fair framework comparison, motivating rich, low-artifact benchmarks.Source: §5 · p. 7
  • Excluding ALFWorld, held-out gains are smaller than search-set gains in 13 of 15 settings and below half in 8; four settings have negative held-out gains.The figure compares search-set and held-out gains by setting; red markers flag negative held-out gains.Source: §5 · p. 7, Figure 4 · p. 8
  • Validation selection improves held-out scores in 10 of 16 Meta-Harness settings and reduces them in 6, yet its gain remains below the search-set gain in 14 settings.Mean held-out changes are +8.0 points on ALFWorld, +0.8 on LiveMath, +8.5 on SWE-bench Verified, and −0.4 CU on CREATE.Source: §5 · pp. 7–8, Figure 5 · p. 8
  • ALFWorld can appear to transfer strongly despite 89% of its evolution gain coming from overfitting, as deterministic benchmark recipes can cover held-out instances.Small-model harnesses become rule-based controllers without LLM calls; larger-model harnesses encode related benchmark knowledge through prompts, templates, and deterministic rules.Source: §5 · pp. 7–8
  • On LiveMath, Meta-Harness 0.8B reaches 100% accuracy on the 53 questions containing the meta-option but 31% on the other 71, so held-out gains can reflect shared artifacts.The same meta-option artifact occurs across search, validation, and test splits.Source: §5 · pp. 8–9

6 RELATED WORK

Prior work evaluates harness evolution, individual harness capabilities, and methods to reduce overfitting, but evaluation practices can leave search-set optimization difficult to distinguish from generalizable improvement. This work complements those approaches by attributing observed gains to overfitting, test-time scaling, and a residual upper bound on generalizable improvement.

6 RELATED WORK

  • Harness optimization spans prompts, skills, code, tools, workflows, memory, context, and environments, while this work focuses on where reported gains come from and whether they generalize.Prior work also studies self-evolution and separate proposer or optimizer agents.Source: §6 · p. 9
  • Reporting performance on the in-distribution search set can confound optimization with generalizable improvement; HDA complements work on evaluation suites, in-situ evaluation, and regularization.HDA attributes observed gains to detected overfitting, test-time scaling, and the remainder as an upper bound on generalizable improvement.Source: §6 · p. 9

7 CONCLUSION

The paper presents HDA as a portable, semi-automated protocol for attributing harness-evolution gains and argues that held-out evaluation alone is insufficient. Its conclusion calls for attribution-aware evaluation, explicit accounting for compute, and benchmarks with meaningful shifts and fewer artifacts.

7 CONCLUSION

  • HDA attributes observed harness-evolution gains to overfitting, test-time scaling, and a residual upper bound on generalizable improvement.The conclusion recommends combining HDA with held-out evaluation and accounting for test-time scaling and cost constraints.Source: §7 · p. 9

A HARNESS EVOLUTION DETAILS

B EXPERIMENT SETUP DETAILS

B.1 EXECUTOR MODELS

  • The experiment uses 11 executor models across 25 settings: 16 Meta-Harness, 6 SkillOpt, and 3 AHE.Nine models are open-weight Qwen-family models; the other two are Claude Haiku 4.5 and Claude Opus 4.8.Source: §B.1 · p. 14

B.2 BENCHMARKS AND SPLITS

The benchmarks use different split designs and task-specific performance measures. HDA standardizes attribution while retaining human review for compute matching and overfitting neutralization.

B.2 BENCHMARKS AND SPLITS

  • LiveMath uses a 35/18/124 train/validation/test split, but all splits share the same question construction and use accuracy.Source: §B.2 · p. 14
  • ALFWorld’s 120 test games include fresh instances and held-out task–room combinations, with success measured within 50 environment steps.Source: §B.2 · p. 14
  • CREATE’s 561-item test set includes held-out relation domains, and performance is measured by Creative Utility.Source: §B.2 · p. 14
  • SWE-bench Verified uses 48/24/50 search/validation/test instances, sampled after filtering for tasks resolvable within one hour.Source: §B.2 · p. 14

C THE HDA SKILL

B.3 VALIDATION-BASED SIGNAL FOR META-HARNESS

  • The validation protocol retains the highest-search-score and highest-validation-score harnesses, then evaluates both on held-out test data.Source: §B.3 · p. 15

D ALFWORLD: BENCHMARK-SPECIFIC PROCEDURAL SCAFFOLDS

On ALFWorld, evolved harnesses encode simulator-specific procedures in different forms, while LiveMath frameworks rediscover a shared answer-choice shortcut. Both cases show why held-out performance alone may not establish generalization.

D ALFWORLD: BENCHMARK-SPECIFIC PROCEDURAL SCAFFOLDS

  • ALFWorld’s Meta-Harness and SkillOpt encode similar simulator-specific recipes, appliance mappings, object-location priors, and environment rules.Source: D ALFWORLD: BENCHMARK-SPECIFIC… · p. 15
  • The ALFWorld procedures appear as executable controller logic in Meta-Harness and natural-language instructions in SkillOpt; the zero-LLM controller reaches 93.3% held-out success.Source: D ALFWORLD: BENCHMARK-SPECIFIC… · p. 15

E LIVEMATH: THE SHARED META-OPTION SHORTCUT

F CREATE: METRIC-DRIVEN TEST-TIME SCALING

CREATE’s metric rewards pools of valid, diverse responses, and evolvable orchestration can increase calls and token use. Compute matching recovers substantial gains when orchestration can change, while SkillOpt’s fixed call flow yields no significant gains.

F CREATE: METRIC-DRIVEN TEST-TIME SCALING

G SWE-BENCH VERIFIED: MIXED GAINS AND REUSABLE GUARDRAILS

SWE-bench shows residual harness gains alongside overfitting and test-time scaling, but the source and held-out transfer vary by setting. Its residual attribution is an upper bound, not a guarantee of transfer.

G SWE-BENCH VERIFIED: MIXED GAINS AND REUSABLE GUARDRAILS

  • For Qwen3-4B-Instruct-2507, HDA attributes the SWE-bench gain as O/T/G = 23/23/54; for Qwen3-30B-A3B-Instruct-2507, it is 0/6/94.The first harness combines general edit guardrails with a search-fitted patch-size rule and uses 1.44× as many LLM calls as baseline; the second detects a general parser failure.Source: G SWE-BENCH VERIFIED: MIXED GAINS AND… · p. 17, Figure 12 · p. 20
  • Weaker Meta-Harness executors transfer positively, but Qwen3.6-27B and Qwen3-Coder-30B-A3B regress on held-out tests despite 0/0/100 search-set attributions.Source: G SWE-BENCH VERIFIED: MIXED GAINS AND… · p. 17

Related papers

Loading 2610.22466v1…