PaperlayerView exact revision on arXiv

Source-linked AI summary

Forging More Tasks into Better Harnesses: Agentic Harness Evolution via Successive Scale Racing

Jingzhe Liu, Jiliang Tang, Tong Zhao, Neil Shah, Clark Mingxuan Ju

arXiv:2610.28815v1cs.AI

TL;DR

Existing harness-evolution methods do not reliably benefit from larger task pools, and evidence combinations affect optimization outcomes. SSR searches evidence scales through quality-and-coverage ranking and candidate racing. It achieves top performance across the reported benchmark settings, scales positively with larger task pools, and reduces validation executions.

  • Problem

    Whether harness evolution benefits from larger task pools is underexplored, and evidence selections can produce substantially different outcomes, with well-selected subsets outperforming full-pool evidence.

    Source: §1 · p. 1, How can we leverage experience from… · p. 2

  • Method

    SSR ranks task experience by quality and coverage, generates multiscale candidate harnesses, and races them under a fixed execution budget.

    Source: How can we leverage experience from… · p. 2, §4.1 · p. 4, §4.3 · p. 5

  • Results

    SSR achieves top performance across three benchmarks and two execution setups, leading in all six settings, and its racing mechanism cuts validation executions by 60%.

    Source: §7 · p. 8

  • Takeaways & Limitations

    As the task pool expands from 40 to 200 tasks, SSR accuracy rises from 64.3% to 68.3%, supporting positive task-pool scaling in the reported evaluation.

    Source: §6.3 · p. 7

    The study treats noisy, conflicting, and redundant evidence as a primary driver of scaling degradation, while acknowledging that other systemic factors may contribute.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Agentic harness evolution optimizes an agent's instructions, tools, and control flow by using its task execution experiences as evolving evidence. While more evolution tasks should intuitively improve performance, our preliminary study reveals a paradoxical failure mode: the performance of existing methods degrades as the task pool expands. Our further analysis demonstrates that this degradation stems from suboptimal evidence selection during optimization, as different evidence combinations lead to substantially different evolution outcomes. To address this issue, we propose Successive Scale Racing (SSR), a framework that adaptively searches for optimal evidence combinations in each evolution round. SSR constructs diverse evidence combinations based on the corresponding task quality and coverage, and efficiently races the resulting candidate harnesses under a small compute budget. Empirically, across three distinct benchmarks, SSR achieves state-of-the-art results in all six configurations. Notably, as the task pool grows from 40 to 200, SSR steadily improves the pass rate from 64.3% to 68.3%, while the baselines' performances all drop. Furthermore, its racing mechanism identifies near-optimal candidates while reducing validation executions by 60%.

1 Introduction

Agentic harness evolution uses execution experience to refine harnesses, but larger task pools do not reliably improve existing methods. SSR searches evidence combinations each round, ranking them by quality and coverage and racing candidate harnesses under a fixed execution budget.

1 Introduction

  • Harness evolution uses execution trajectories as evidence: an optimizer distills lessons from them to improve the harness, repeating until a stopping condition or execution-budget limit.The harness governs model interaction through instructions, tools, context management, and control flow.Source: §1 · p. 1
  • AHE’s accuracy drops from 63.0% at 80 tasks to 57.7% at 200 tasks, illustrating that larger task pools do not consistently improve existing harnesses.The motivating question is how to leverage experience from more tasks.Source: §1 · pp. 1–2, How can we leverage experience from… · p. 2

How can we leverage experience from more tasks

  • Evidence selection matters because a well-selected subset can outperform full-pool evidence, while noisy or redundant trajectories can blur optimizer focus.Different trajectory selections produce different harness updates, motivating adaptive evidence search.Source: How can we leverage experience from… · p. 2
  • SSR searches evidence combinations each round using quality-and-coverage ranking, multiscale candidates, and performance racing under a fixed execution budget.It refreshes trajectories and re-ranks evidence as the incumbent harness evolves.Source: How can we leverage experience from… · p. 2
  • The paper evaluates SSR on SWE-Bench Pro Verified, DeepSWE, and GAIA-2 across two execution setups.The supplied introduction identifies these evaluation domains and setups without reporting their results here.Source: How can we leverage experience from… · p. 2

2 Related Work

Related work spans harness design and automated optimization, while self-evolving systems use execution experience to refine agents and harnesses. Approaches range from prompt and module changes to trajectory-based evidence and cross-task comparisons.

2.1 Agent Harness Engineering

  • Harness design includes reasoning–action loops, multi-agent conversation, sandboxed execution, and prescribed repair workflows, while optimization spans instructions, prompts, pipelines, and feedback propagation.Other approaches optimize computational graphs, evolve modules, or update tool functions.Source: §2.1 · p. 2

2.2 Self-Evolving Agents and Harnesses

  • Self-evolving systems reuse experience through feedback, reflection, skills, extracted insights, guidelines, manuals, memory, or playbooks, and later work evolves code or structures trajectory evidence.Retro selects a challenging, diverse coreset, while MGM compares across tasks and lineages.Source: §2.2 · p. 2
  • Existing methods’ performance does not consistently improve as evolution task pools grow.Figure 2 compares performance changes with more evolution tasks.Source: Figure 2 · p. 3

3 Preliminary Study

The preliminary study tests task-pool size and evidence selection on SWE-Pro-V with Codex, reporting final Avg Pass@1 (%) averaged over three runs. Existing methods do not reliably benefit from larger pools, and evidence choice affects outcomes.

3 Preliminary Study

  • The study uses SWE-Pro-V with Codex and reports final Avg Pass@1 (%) averaged over three independent runs.Section 6.4 extends the evidence comparison to DeepSWE.Source: §3 · p. 3

3.1 Effect of Task-Pool Size

  • AHE peaks at 63.0% accuracy with 80 tasks but falls to 57.7% with 200, and none of Retro, AHE, HGM, or MGM consistently benefits from larger pools.The study varies pools from 40 to 200 tasks; expanding a pool can even degrade performance.Source: §3.1 · p. 3

3.2 Effect of Evidence Configuration

  • TOP 40 from the 160-task pool outperforms ALL 160 by 3.4% and BOTTOM 40 by 5.4%, while a random 40-record subset trails ALL 80 by 1.3%.Figure 3 compares harness performance under different task-pool evidence configurations.Source: §3.2 · p. 3, Figure 3 · p. 3

4 Successive Scale Racing

SSR ranks experience, generates candidate harnesses from nested evidence prefixes, races them on shared task batches, and recursively updates an incumbent. The process refreshes evidence after accepted updates and stops at the execution budget or after consecutive rounds without strict improvement.

4 Successive Scale Racing

  • SSR consists of Representative Evidence Ranking, multiscale candidate generation, candidate performance racing, and recursive evolution.Training tasks supply optimization evidence, while disjoint validation tasks contribute to candidate selection.Source: §4 · p. 3

4.1 Representative Evidence Ranking

  • RER orders experience records by intrinsic quality and collective coverage, using importance-weighted similarity and diminishing returns rather than a static sort by individual quality.Record quality is actionability multiplied by confidence; importance separately parameterizes coverage.Source: §4.1 · pp. 3–4

4.2 Multiscale Candidate Generation

  • SSR treats evidence scale as a per-round search variable and generates nested prefixes from the RER ordering, with the shared evidence core supporting validation batching.Nested prefixes isolate scale as the candidate variable and provide near-optimal coverage at their length.Source: §4.2 · pp. 4–5

4.3 Candidate Performance Racing

  • Candidate harnesses and the protected parent race through successive halving on shared cumulative task batches, with weaker children eliminated while the parent remains an active reference.The mixed evaluation set includes validation tasks and parent-success or parent-failure training tasks, so its score is not validation-only.Source: §4.3 · p. 5

4.4 Recursive Evolution

  • After an accepted update, SSR reruns the parent on training tasks and rebuilds experience records and rankings; evolution stops at budget exhaustion or after P rounds without strict improvement.If the parent is retained, existing records and ordering are reused while fresh candidates are generated.Source: §4.4 · p. 5

5 Theoretical Analysis

The analysis gives single-round guarantees for evidence coverage and candidate selection, while distinguishing evidence-coverage guarantees from execution reward and population-level claims.

5 Theoretical Analysis

  • Every length-k prefix of RER’s exact greedy ordering has a (1 − 1/e) approximation guarantee for the evidence-coverage objective F.For a fixed evidence pool with nonnegative weights and similarities, F is normalized, monotone, and submodular.Source: §5 · p. 6
  • The racing guarantee assumes a fresh uniform validation permutation, fixed stage sizes, predictable training-task choices, and fresh rollouts in [0, 1] with stationary conditional means.The mixed-score error radius combines rollout noise, validation-task sampling error, and training–validation discrepancy weighted by the training fraction.Source: §5 · p. 6
  • On the joint error-control event, each elimination can lose at most 2εr, while the protected-parent bound controls regression using final-stage error.These are conditional single-round guarantees on a finite pool; they establish neither population generalization nor monotonic improvement across evolution rounds.Source: §5 · p. 6

6 Experiments

Across six benchmark–execution configurations, SSR leads the strongest baselines; its performance also improves with larger task pools, and racing reduces validation cost while retaining near-best selection quality.

6.1 Experimental Setup

  • Experiments use three benchmarks, disjoint train/validation/test splits, matched optimizer and executor settings, and three-run mean Average Pass@1 for main test results.Ablations and controlled analyses use a single evolution run per condition.Source: §6.1 · pp. 6–7

6.2 Overall Performance

  • SSR achieves state-of-the-art performance in all six benchmark–execution configurations, exceeding the strongest baselines by 1.3%–3.0%.Under Codex, SSR scores 65.7% on SWE-Bench Pro Verified, 68.0% on DeepSWE, and 38.0% on GAIA-2; under Claude Code, it scores 63.3%, 65.3%, and 36.0%, respectively.Source: Table 1 · p. 7, §6.2 · p. 7

6.3 Effects of Task Experience and Search Resources

  • 64.3% to 68.3%: SSR’s accuracy rises as the task pool expands from 40 to 200, while AHE falls from its 63.0% peak at 80 tasks to 57.7% at 200.Execution budgets scale proportionally with the task pool.Source: §6.3 · p. 7, Figure 5 · p. 8
  • 66.3% to 71.0%: with a fixed task pool, SSR improves as execution caps rise from 800 to 4,000 and stays ahead of MGM, which rises from 51.0% to 60.3%.Candidate-generation caps increase from 15 to 35 for SSR and 55 to 145 for MGM.Source: §6.3 · p. 7, Figure 5 · p. 8

6.4 Effects of Evidence Selection Strategies

  • SSR’s multiscale strategy exceeds TOP K RESAMPLING by 1.4% on SWE-Pro-V (65.7% vs. 64.3%) and 1.3% on DeepSWE (68.0% vs. 66.7%).Repeatedly sampling identical evidence or supplying the full evidence set yields suboptimal performance on both benchmarks.Source: §6.4 · p. 7, Figure 6 · p. 8
  • RER’s quality-and-coverage ordering improves over Quality-only by 1.0% on SWE-Pro-V (65.7% vs. 64.7%) and 0.7% on DeepSWE (68.0% vs. 67.3%).Quality-only ordering improves over random ranking on both benchmarks.Source: §6.4 · p. 7, Figure 6 · p. 8

6.5 Racing Cost–Quality Audit

  • At 40% of full validation cost, Racing scores 65.3%, within 0.4% of the 65.7% full-matrix upper bound, while saving 60% of validation executions.The audit compares 12 candidate harnesses on a 160-task matrix; increasing the budget to 60% yields no further performance gains.Source: Figure 7 · p. 8, §6.5 · p. 8

7 Conclusion

The paper concludes that SSR addresses degradation as task pools scale by ranking evidence, generating multiscale candidates, and racing them under a fixed compute budget. It reports top performance across all tested settings and 60% fewer validation executions for near-optimal candidate selection.

7 Conclusion

  • SSR ranks task experience by quality and coverage, generates multiscale candidate harnesses, and races them under a fixed compute budget.Source: §7 · p. 8
  • Across three benchmarks and two execution setups, SSR achieves top performance in all six settings and enables positive task-pool scaling.Source: §7 · p. 8
  • Racing identifies near-optimal candidates while cutting validation executions by 60%.The paper presents evidence-driven candidate diversity and execution-based selection as a paradigm for agent self-improvement.Source: §7 · p. 8

Limitations

The authors identify unresolved causes of task-pool degradation, alternative evidence-selection approaches, and limits on how broadly SSR’s empirical findings generalize.

Limitations

  • The degradation from expanding task pools may have systemic causes beyond noisy, conflicting, and redundant evidence.The study treats these evidence properties as a primary driver, not a complete explanation.Source: Limitations · p. 9
  • Dynamic context pruning, fine-grained trajectory credit assignment, and reinforcement-learning-based optimizer tuning remain unexplored alternatives to searching over evidence scales.SSR is described as an initial step toward mitigating the scaling bottleneck.Source: Limitations · p. 9
  • The empirical evaluation uses specific foundation models as task agents and optimizers, so generalization to other architectures remains unestablished.The authors call for validation with open-weight models of varying scales and domain-specialized language models.Source: Limitations · p. 9

Ethical Considerations

The appendix presents conditional guarantees for evidence coverage and single-round racing, while ethical guidance emphasizes deployment safeguards and review of task-derived artifacts.

Ethical Considerations

  • Benchmark evaluations do not establish safety in unrestricted deployment; consequential tool actions warrant isolation, restricted permissions, and human oversight.The authors also recommend reviewing trajectories and generated artifacts for sensitive information or bias before sharing or deployment.Source: Ethical Considerations · p. 9

A Proofs and Technical Conditions

  • The proof of the confirmed SSR procedure uses ranking-based fixed quotas, mixed equal-task-weight scores, a protected parent, and selection after a completed stage.The appendix adds no confidence threshold to the algorithm.Source: A Proofs and Technical Conditions · p. 12

A.1 Formal Guarantees

  • The RER objective is normalized, monotone, and submodular, giving every greedy-order prefix a coverage guarantee, not an execution-reward guarantee.One ordering supplies nested evidence inputs with a guarantee at every scale.Source: §A.1 · p. 12
  • The single-round theorem bounds the cost of selecting among evidence scales and limits regression against the protected parent under its stated conditions.The guarantees are conditional and finite-pool; they do not establish population generalization or monotonic improvement across rounds.Source: §A.1 · p. 12

A.2 Proof of the RER Prefix Guarantee

  • The RER prefix guarantee applies to its coverage objective and provides no approximation guarantee for execution reward.The proof makes no assumption that judge scores are calibrated to downstream reward.Source: §A.2 · p. 13

A.3 Single-Round Probability Model

  • The probability model assumes bounded rollouts with stationary conditional means; arbitrary correlations that change those means and drifting environments are outside its scope.The race also uses a fresh uniform validation permutation, fixed stage sizes, and training-task choices made before new-batch outcomes.Source: §A.3 · p. 13
  • The parent is protected during elimination, and at the final stage a child replaces it only when its score is strictly higher.The output maximizes the last-stage empirical score among surviving children and the parent, with the parent winning ties involving itself.Source: §A.3 · p. 13

A.4 Uniform Error for Mixed Evaluation

The mixed-evaluation analysis gives a simultaneous, discrepancy-dependent error bound and derives what it guarantees for racing selection. Its guarantees are conditional on a finite validation pool and do not establish population generalization or monotonic performance as the evidence pool grows.

A.4 Uniform Error for Mixed Evaluation

  • Adaptive training-task selection does not make selected-task rewards fresh samples from the validation population, so the discrepancy term cannot be assumed small.The proof cautions that conditioning on the final adaptive task set can invalidate a naive independence argument.Source: §A.4 · pp. 13–14
  • The mixed-evaluation error bound holds for every evaluated harness at each completed stage, with a pathwise term for training-to-validation discrepancy.The bound combines rollout noise, validation-task sampling, and discrepancy; the discrepancy can be random and adaptive, and need not be small.Source: §A.4 · pp. 13–14

A.5 From Stage Errors to Selection Loss

  • At intermediate elimination stages, the best active child may be removed, and the resulting stage-selection loss is bounded using the stage error.If no stage completes, the method returns the parent and provides no nontrivial selection guarantee relative to unevaluated children.Source: §A.5 · p. 15

A.6 Interpretation and Limits

  • If twice the stage error is below the unique best harness’s minimum reward gap at every completed stage, the racing process selects that harness.The paper describes this as a conservative sufficient condition, not an optimal sample-complexity claim.Source: §A.6 · p. 15
  • Training-task and validation rankings can disagree even without rollout noise, so a vanishing validation-target bound requires a discrepancy assumption or a changed selection objective.The paper illustrates this with two children whose expected mixed-score ordering can reverse despite exact expected scores.Source: §A.6 · p. 15
  • The theorem concerns expected reward on a fixed finite validation pool; it makes no multi-round or test-distribution claim and does not establish monotonic SSR performance as the pool grows.A larger eligible evidence set can increase the best attainable reward when old configurations remain available, but this does not prove monotonic SSR performance.Source: §A.6 · pp. 15–16

B SSR Implementation Details

SSR builds shared racing batches from validation tasks and RER-ranked training evidence, then applies explicit budget, stopping, and candidate-selection rules. Validation trajectories are withheld from optimizer evidence, while training trajectories support candidate generation and outcome labels.

Evaluation-task construction

  • Each race batch combines unused validation tasks with unused successful- and failed-task evidence, prioritizing the intersection of surviving candidates’ nested RER prefixes.If an intersection cannot fill a class quota, selection continues outside it in RER order; the parent is excluded from the prefix intersection.Source: B SSR Implementation Details · p. 16
  • Validation trajectories are not exposed to the optimizer, and the test set is never used in racing decisions.Training trajectories provide candidate-generation evidence and outcome labels; parent–child comparisons use executions from the current race.Source: B SSR Implementation Details · p. 16

Integer quotas and ties

  • After each intermediate stage, fixed quotas approximate a constant number of removals, and survivors are ranked by cumulative score, then harness diff size, then candidate index.A child tied with the parent never replaces it; these are empirical comparisons without a confidence threshold.Source: B SSR Implementation Details · p. 16

Budget boundaries and stopping

  • The race starts a stage only when its full execution cost fits the remaining budget; otherwise it selects from the last completed stage or retains the parent if none completed.An incomplete evidence refresh terminates evolution and returns the incumbent; within-race scores do not carry automatically into a new race.Source: B SSR Implementation Details · p. 16
  • Evolution stops after two consecutive rounds without strict improvement, while an unchanged parent reuses collected records and rankings but receives freshly generated candidates.The no-improvement counter resets after an accepted update.Source: B SSR Implementation Details · p. 16

Judge and optimizer interface

  • The judge scores transferability directly, and the optimizer receives selected summaries and suggestions with access to their original trajectories on demand.The optimizer may modify instructions, tool-related code, context management, and execution flow.Source: B SSR Implementation Details · p. 16

C Experimental Configuration

The experiments use two model-and-reasoning settings for the main benchmarks, with analyses using Codex. The configuration table defines benchmark abbreviations and indicates how task-pool and resource lists are ordered.

Experimental configurations

  • The main experiments use Codex with GPT 5.5 high and Claude Code with Opus 4.8 high; all analyses use the Codex setting.Within each setting, the executor, optimizer, and experience judge use the same model and reasoning effort.Source: C Experimental Configuration · p. 16
  • The embedding model is BAAI/bge-large-en-v1.5 with 1,024-dimensional outputs, following Retro.The paper identifies Pan et al. (2026b) for this choice.Source: C Experimental Configuration · p. 16
  • Table 2 uses S, D, and G for SWE-Pro-V, DeepSWE, and GAIA-2, and orders task-pool and resource lists by increasing size.Its scaledynamics column distinguishes parent–scale probes from final-winner scale analysis.Source: Table 2 · p. 17

Evidence and repetitions

  • Evidence scales round up from percentages, while TOP K FIXED, TOP K RESAMPLING, and FIXED N use distinct evidence-selection rules.Training evidence uses one rollout per task and is refreshed after an accepted update; final testing uses three rollouts outside the execution budget.Source: C Experimental Configuration · p. 17
  • The audit replays a fixed 13 × 160 × 2 execution matrix three times, and scale-probe winner frequencies aggregate 20 independent five-round runs.Budgets include training-trajectory collection and parent/child racing executions.Source: C Experimental Configuration · p. 17

D Variability of the Main Results

This section describes SSR’s result variability and evidence-scale selection, as well as the DeepSWE harness’s solve methodology and end-to-end tracing of data and state.

D Variability of the Main Results

  • Standard deviations ranged from 0.27% to 0.98%, with SSR at 0.27% to 0.54%; SSR’s gaps over the strongest baseline ranged from 1.3% to 3.0%.The paper reports this variability descriptively and does not attach significance tests to individual comparisons.Source: D Variability of the Main Results · pp. 17–18

E Evidence Scale across Parents and Evolution Rounds

  • The largest observed single-candidate gains occurred at k = 40 for A, k = 80 or 120 for B, and k = 120 for C.No scale maximized all three curves; these candidates illustrate snapshot variation, not mean optimal scales or an evolution-stage effect.Source: E Evidence Scale across Parents and… · p. 18
  • Across 20 runs of five rounds, small-scale winner frequency fell from 75% to 20%, while large-scale winner frequency rose from 5% to 40%.Later rounds more often selected winners generated from broader evidence; these are winner frequencies, not survival probabilities.Source: E Evidence Scale across Parents and… · p. 18, Figure 8 · p. 18

F More Related Work

  • SSR races candidates generated from nested evidence prefixes of one evolving parent, using fixed-quota elimination with a protected parent and a regenerated candidate pool.Evaluation batches mix held-out validation tasks with training tasks selected from the survivors’ evidence.Source: F More Related Work · p. 18

G.1 SWE-Pro-V Harness

  • The SWE-Pro-V harness identifies recurring prior-run failures in surface coverage, contract tests, verification, toolchain discovery, and safe temporary-file handling.Source: §G.1 · p. 20

G.2 DeepSWE Harness

  • Trace each feature from public input to its final consumer, checking state ownership, identity keys, cleanup, and metadata preservation across persistence and reconstruction.The passage also flags absence versus legal zero or empty values, skipped callbacks or cleanup on early returns, and shared-input mutation as risks to inspect.Source: §G.2 · p. 25

Related papers

Loading 2610.28815v1…