PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness Evolution Hits a Ceiling: When Weight Training Should Begin

Yuan Tian, Bing Hu, Hao Wang, Binghang Lu, Fang Wu

arXiv:2610.11655v1cs.AIcs.CLcs.LG

TL;DR

The paper asks whether gains from evolving a long-horizon agent’s harness still require that runtime or can be retained in trained weights. It compares harness evolution with trajectory-based LoRA, using failure composition to distinguish process from content failures. On DeepPlanning, harness evolution improves held-out scores, and LoRA transfers gains under the seed harness.

  • Problem

    A higher score from a self-evolving harness does not reveal whether the trained model retains the gain or still depends on the runtime.

    Source: §1 · p. 2

  • Method

    The paper labels trajectories by their first failure signal, evolves the harness, then crosses seed and evolved harnesses with base and trajectory-trained weights.

    Source: §1 · p. 2

  • Results

    Harness evolution adds +0.133 held-out for Qwen3.5-4B and +0.119 for Qwen3.5-9B; LoRA adds +0.13 under the seed harness for both sizes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Failure composition supports evolving the harness when delivery is low or F3 dominates, and training weights when delivery is saturated and F4 dominates.

    Source: §7 · p. 9

    The study does not compare the loop with a matched-budget test-time search baseline.

    Source: §7 · p. 10

Abstract

from arXiv · show

Improving a long-horizon LLM agent means evolving the harness around a frozen model or training its weights. We let a self-evolving harness make the system stronger first, then cross seed and evolved harnesses with base and trained weights to learn which gains the trained model keeps and which still need the runtime. We show that the right lever can be read off the agent's failure composition: labelling failed trajectories by the first signal that fires separates process failures (blocked calls, loops, exhausted step budgets) from content failures (a delivered plan that is poor). Harness evolution repairs the former, the behaviour it instils can be trained into the weights, and content failures are what weight training is for. On DeepPlanning, a self-evolving harness loop lifts the held-out score of Qwen3.5-4B from 0.16 to 0.30 and of Qwen3.5-9B from 0.32 to 0.44; for 4B, held-out delivery rises from 55% to 90% while content failures are left for the weights. LoRA adapters trained on evolved-harness trajectories internalise the gain: under the original harness they add +0.13 on held-out tasks for both sizes; on 4B they stack with the harness to more than double the held-out score, and on 9B the adapter alone matches the full evolution line, cutting content failures from a quarter of trajectories to one in twenty. A placebo adapter trained on answer-shuffled trajectories falls below the base model. The loop transfers to WebArena-Lite (+0.09 on 117 unseen tasks), where the gain lives in what the model sees and adapters do not add to it. The result is a diagnose-then-intervene rule applied twice: read the failure composition to choose between harness and weights, then read what the accepted edits changed to decide which gains to train in. Scores are four-rollout means against fresh anchors, same-night except where marked, across eight models from six families and two benchmarks.

1 INTRODUCTION

The paper compares harness evolution with weight training to determine which gains persist in the model and which still require the runtime. It uses failure composition to guide the choice of intervention and reports held-out gains from both.

1 INTRODUCTION

  • The paper tests a failure-based rule: harness evolution targets process failures, learned behavior can transfer to weights, and weight training targets content failures.The taxonomy labels trajectories by the first failure signal, separating blocked calls and loops from a poor delivered plan.Source: §1 · p. 2, Figure 1 · p. 2
  • Harness evolution improves held-out scores over the seed harness by +0.133 for Qwen3.5-4B and +0.119 for Qwen3.5-9B.For 4B, delivery rises from 59% to 88% as loop failures fall from 64% to 37%; for 9B, loops fall from 39% to 17%.Source: §1 · p. 2
  • The harness loop leaves content failures in place, and its evidence package does not expose the scorer’s failure composition to the proposer.The package foregrounds gate and checkpoint metrics rather than the cause of F4.Source: §1 · p. 2
  • LoRA trained on evolved-harness trajectories adds +0.13 under the seed harness for both model sizes; on 4B it stacks with evolution, while on 9B it reaches the evolved-harness level.The placebo adapter trained on answer-shuffled trajectories falls below the base model.Source: §1 · p. 2
  • Across nine evolution lines on eight models from six families, the seed measurements identify five lines with harness gains and three without; the loop also transfers to a second benchmark.The reported gains for the five successful lines range from +0.05 to +0.14.Source: §1 · p. 3

2 BACKGROUND AND RELATED WORK

Related work spans agent-failure taxonomies, harness optimization, trajectory fine-tuning, and evaluation under noise. This paper distinguishes itself by choosing interventions from failure composition and measuring both levers under a calibrated noise model.

2.1 FAILURE ANALYSIS OF LLM AGENTS

  • Earlier failure taxonomies classify error types, often using human or LLM annotation; this paper organizes failures by the lever that removes them and uses deterministic checks.Ecdysis uses recurrence across tasks to decide which failures the harness should absorb; this paper tests the split by applying both levers.Source: §2.1 · p. 3

2.2 SELF-EVOLVING HARNESSES AND AUTOMATED AGENT DESIGN

  • Harness-design work searches agent programs, workflows, or components, while this paper adds an account of which failure classes evolution removes and leaves.AHE introduced the propose-verify loop on which this work builds.Source: §2.2 · p. 3

2.3 TRAINING AGENTS ON TRAJECTORIES, AND MOVING HARNESS GAINS INTO THE WEIGHTS

  • Trajectory fine-tuning and harness-aware training are established approaches; this paper switches interventions based on failure composition and tests size dependence with a shuffled-answer placebo.It uses plain fine-tuning on the model’s own trajectories rather than long-horizon reinforcement-learning credit assignment.Source: §2.3 · p. 3

2.4 EVALUATING AGENTS UNDER NOISE

  • Agent evaluations vary in hold-out quality, noise reporting, and exposure of failure composition; this paper measures both levers under one calibrated noise model and turns the findings into a failure-based rule.DeepPlanning provides programmatic scoring and a clean split, while several broader benchmarks rely on LLM judges or do not expose failure composition.Source: §2.4 · pp. 3–4

3 A DIAGNOSTIC FRAMEWORK

The framework labels failed trajectories by the first signal that fires, then uses delivery and delivered-plan quality to interpret score changes. It also accounts for selection optimism when choosing among noisy candidate readings.

3.1 THE FAILURE TAXONOMY

  • The taxonomy assigns the first fired signal to F1 evidence failures, F2 harness friction, F3 execution-control failures, or F4 planning-capacity failures; a plan scoring at least 0.5 is healthy.F4 generally means a delivered plan scores below 0.5 without an earlier process signal.Source: §3.1 · p. 4
  • Delivery rate and mean quality among delivered plans multiply to give the composite score; F2 and F3 are process classes the harness can directly address, while F4 is not distinguished from healthy trajectories by process signals.The paper tests whether behavior associated with removed process failures can be learned by the weights.Source: §3.1 · p. 4

3.2 TWO IDENTITIES BEHIND THE NUMBERS

  • Keeping the highest of several noisy candidate readings creates selection optimism; with four rollouts, its bound is 0.025–0.030, and a fresh anchor after each acceptance prevents optimism carrying into the next anchor.With one rollout, the bound is 0.05–0.06, above the 0.035 resolvability threshold.Source: §3.2 · p. 4
  • On DeepPlanning, score changes decompose into delivery-rate and delivered-plan-quality contributions, so improving quality is worth more when delivery is already high.The score is the product of delivery rate and mean quality among delivered plans.Source: §3.2 · pp. 4–5

4 EXPERIMENTAL SETUP

Experiments use a held-out travel-planning benchmark and same-night, four-rollout measurements, with a self-evolving harness loop and trajectory-based weight training. Fresh anchors and failure-composition plots track the evolution process.

4.1 BENCHMARK, MODELS AND MEASUREMENT

  • DeepPlanning uses programmatically scored multi-day travel plans; 48 development tasks guide decisions and 72 held-out tasks are reserved for reporting.The study uses Qwen3.5-4B and Qwen3.5-9B as its primary models.Source: §4.1 · p. 5
  • Each configuration is a four-rollout mean, differences below 0.035 are treated as noise, and comparisons are made on the same machine on the same night.Cross-night drift for one configuration reaches 0.05.Source: §4.1 · p. 5

4.2 HARNESS SELF-EVOLUTION PROTOCOL

  • Each evolution round evaluates two proposed harness edits against an anchor, applies fixed rejection rules, retains an accepted candidate, and measures a fresh anchor before the next round.Only harness files, policy parameters, middleware, and tools are editable; the 4B and 9B lines run 13 and 5 rounds, respectively.Source: §4.2 · p. 5

4.3 WEIGHT TRAINING

  • Training uses selected high-scoring trajectories collected under the evolved harness, with separate LoRA and full-parameter fine-tuning configurations.The training sets contain 82 trajectories for 4B and 98 for 9B; the adapters are evaluated on both harnesses and splits.Source: §4.3 · p. 5
  • Per-round plots show fresh anchors and candidate readings; accepted rounds reduce loop failures and increase the healthy share in both lines, while F4 is not reduced.The 9B lessons-file change raises the share of plans with unbacked claims; Figure 2 marks the gap between retained readings and subsequent anchors as selection optimism.Source: Figure 3 · p. 6, Figure 2 · p. 6

5 RESULTS

Harness evolution transferred to held-out tasks on DeepPlanning and WebArena-Lite, while adapters internalized some harness gains. Across DeepPlanning lines, the harness lever’s payoff varied with seed plan quality and failure composition.

5.1 HARNESS SELF-EVOLUTION WORKS AND TRANSFERS

  • 4B gained +0.133 held-out versus +0.143 development, while 9B gained +0.119 held-out versus +0.056 development under the evolved harness.For 4B, delivery rose from 59% to 88% as loop failures fell from 64% to 37%; F4 rose from 10% to 28%. For 9B, delivery was already 97%, loops fell from 39% to 17%, and F4 stayed at 25%.Source: §5.1 · p. 6

5.2 TRAINING INTERNALISES THE HARNESS GAIN

  • On seed harnesses, adapters added +0.126 held-out for 4B qkvo and +0.131 for 9B qkvo+MLP; stacking added +0.082 held-out for 4B, while 9B gains were within noise.The 9B adapter reduced held-out F4 from 28% to 5%; the answer-shuffled placebo scored 0.227 below the base model on held-out tasks.Source: Table 1 · p. 7, §5.2 · p. 7
  • Adapter placement followed the textbook: attention-only worked better for 4B’s behavior-focused data, while adding MLP blocks worked better for 9B’s plan-content data.For 9B, the MLP adapter reduced F4 to 5% on held-out tasks, versus 16% for attention-only.Source: §5.2 · p. 7

5.4 WHERE THE HARNESS LEVER PAYS, ACROSS NINE LINES

  • Harness evolution paid +0.063 to +0.136 on four of five lines whose seed plans scored at least 0.27; it paid nothing on three lines scoring 0.13–0.17.The fifth stronger-plan line’s +0.053 was within its own noise; when delivery was saturated and F4 dominant, Gemma 4 12B’s harness gain was also within noise.Source: Figure 4 · p. 8, §5.4 · pp. 8–9

5.3 THE SAME LOOP ON A SECOND BENCHMARK: WEBARENA-LITE

  • On WebArena-Lite’s 117 test tasks, the evolved harness raised score from 0.256 to 0.344 (+0.088), while accuracy among delivered answers stayed within noise.Delivery rose from 49% to 77%, and repeated-action or unparseable-action failures fell from 165 of 237 undelivered seed trajectories to 13 of 108 under the closing harness.Source: §5.3 · p. 8
  • On WebArena-Lite, adapters did not add to the harness gain: the gain came from showing two full pages instead of one, which the adapters were not shown.Under the two harnesses, test scores were within the 0.064 line of the base model; delivery rose further while accuracy among delivered answers fell.Source: §5.3 · p. 8

6 ANALYSIS

Ablations show that capability-granting components account for most harness gains, while the loop’s evidence package limits which failures it can address.

6.1 WHAT THE HARNESS COMPONENTS BUY

  • Removing tools or lessons caused the largest score losses; most enforcement components were within noise, except the 4B exit gate, which protected held-out delivery.For 9B, removing the ledger and dependent controls cost 0.145 on held-out tasks; for 4B, removing lessons cost 0.120.Source: §6.1 · p. 9

6.2 WHY THE LOOP STOPS AT F4

  • The loop stopped at F4 because its evidence package did not aggregate scorer failure composition, and its accepted edits therefore targeted visible process metrics instead.The proposer made a single tool-free call, and metric definitions sat outside the four editable directories; the authors describe this ceiling as specific to this loop’s evidence package.Source: §6.2 · p. 9

7 DISCUSSION

The discussion proposes diagnosing failure composition before choosing between harness evolution and weight training, then inspecting accepted edits to determine which gains can transfer.

7 DISCUSSION

  • When seed delivery is weak or loops dominate, the paper recommends harness evolution; when delivery is saturated and F4 dominates, it recommends weight training.The authors report no harness gains on three models with delivered-plan scores of 0.13–0.17, while 9B’s adapter alone matched the five-round evolution line on held-out tasks.Source: §7 · p. 9
  • Seed measurements separated five of nine DeepPlanning lines where harness evolution paid from three where it paid nothing; WebArena-Lite’s gain transferred to 117 unseen tasks (+0.088), but training did not add to it.The smallest of the five DeepPlanning gains was within its own noise.Source: §7 · p. 9
  • Applying the taxonomy requires trajectory-level process signals and a task-level scorer; the study also lacks a matched-budget test-time search comparison.The plan-to-JSON converter inflated audited scores by about 0.025, and two comparisons crossed nights.Source: §7 · p. 10

A NOISE FLOOR AND MEASUREMENT PROTOCOL

The measurement protocol uses four-rollout means and fresh anchors to limit noise and selection optimism. Evolution logs and composition tables show that the largest failure shift in both Qwen lines is fewer loops, not fewer content failures.

A NOISE FLOOR AND MEASUREMENT PROTOCOL

B FULL 13-ROUND EVOLUTION LOG (QWEN3.5-4B)

  • The 4B line retained 10 edits across 13 rounds; a plan-completeness checker was its largest single step at +0.039 on the fresh anchor.Rounds 4, 5, and 12 rejected both candidates; after round 11, the context window changed and the harness was re-anchored from 0.305 to 0.269.Source: B FULL 13-ROUND EVOLUTION LOG… · p. 13, Table 3 · p. 13

C FULL 5-ROUND EVOLUTION LOG (QWEN3.5-9B)

  • The 9B line accepted three edits in five rounds, all to the distilled lessons file; every proposed tool was rejected because the agent never called it.An earlier 9B line anchored on a single lucky evaluation and then rejected eight consecutive candidates, prompting the fresh-anchor protocol.Source: C FULL 5-ROUND EVOLUTION LOG… · p. 13, Table 4 · p. 14

D EVOLUTION SUMMARY TABLES AND PER-ROUND FAILURE COMPOSITION

  • F3 falls most in both evolution lines, while F4 is not reduced and remains the largest failure class for 9B and second-largest for 4B.Table 5 reevaluates seed and evolved harnesses on development and held-out tasks; its evolved-harness held-out cells were measured two nights earlier than the other three cells.Source: Table 6 · p. 14, Table 5 · p. 14

E TRAINING DETAILS, RANK ABLATION AND FULL-PARAMETER FINE-TUNING

The supplied training details cover adapter settings, the scope of the rank and full-parameter comparisons, and how adapters shift failure composition. The reported cross-harness comparison shows different combinations of harness and adapter gains for 4B and 9B.

E TRAINING DETAILS, RANK ABLATION AND FULL-PARAMETER FINE-TUNING

  • Adapters use rank 32, α = 64, dropout 0.05, and a 10−4 learning rate; Table 1 adapters were trained for two epochs but taken after one because the second epoch memorises.The placebo permutes assistant outputs across samples while preserving inputs, harness structure, and step count.Source: E TRAINING DETAILS, RANK ABLATION AND… · pp. 14, 16
  • Table 8 compares LoRA with full-parameter fine-tuning and includes a 9B rank ablation, but its bottom-block gains use machine-matched base scores and are not comparable to the rows above.The second textbook uses 198 trajectories over 42 tasks for 4B and 200 over 44 for 9B, with one epoch.Source: Table 8 · p. 15
  • Under the seed harness, the 4B adapter raises delivery and reduces F3, whereas the 9B adapter reduces F4.Table 9 reports failure shares across trajectories, with rows summing to 100% up to rounding.Source: Table 9 · p. 15, E TRAINING DETAILS, RANK ABLATION AND… · p. 16
  • On the seed harness, adapters recover as much as harness evolution; on the evolved harness they add to the 4B score, while the 9B adapter alone reaches the evolved-harness level.The evolved-harness 9B adapter cuts F4, but the freed share reappears as F3 on development and F1 on held-out, leaving mean score unchanged.Source: Figure 5 · p. 16, E TRAINING DETAILS, RANK ABLATION AND… · p. 16

F COMPONENT ABLATIONS

The ablations compare components removed from each line’s closing harness, with different rollout counts and noise thresholds across the two model sizes. The reported paired 4B comparisons show small full-versus-bare gains on tasks both configurations deliver.

F COMPONENT ABLATIONS

  • The 9B cells were run across two identical machines with a cross-machine offset of at most 0.03; 4B cells were run on one machine in one boot.The 4B exit-gate cell was rerun with four rollouts; the 9B comparisons use a 0.035 within-noise threshold.Source: F COMPONENT ABLATIONS · p. 16, Table 11 · p. 17, Table 12 · p. 17
  • The 9B ablation has seven cells with four rollouts each; removing the ledger also removes its dependent exit gate and checkpoint.The 4B ablation has ten cells, mostly with two rollouts per cell, for which differences below 0.05 are within noise.Source: F COMPONENT ABLATIONS · p. 16, Table 11 · p. 17, Table 12 · p. 17
  • Full versus bare is +0.011 over 35 development tasks and +0.022 over 48 held-out tasks, while seed kit versus bare is −0.027 / −0.009.The paired 4B comparisons include only tasks both configurations deliver; multi-draft submissions fall from 8.9% for bare to 1.0% for full.Source: F COMPONENT ABLATIONS · p. 17

G MEASUREMENT PITFALLS AND MITIGATIONS

The measurement pitfalls are selection optimism, ratchet slippage, and accepted edits that never execute. The paper reports fresh-anchor measurements and adds an execution check as mitigations.

G MEASUREMENT PITFALLS AND MITIGATIONS

  • In the 4B line, candidate readings of 0.316, 0.327, and 0.323 corresponded to fresh anchors of 0.282, 0.288, and 0.305.The paper reports anchors rather than candidate readings; Figure 7 compares retained candidates’ readings with fresh anchors measured afterward.Source: G MEASUREMENT PITFALLS AND MITIGATIONS · pp. 17–18, Figure 7 · p. 18
  • A lucky accept can set the next reference too high, making a neutral edit look regressive; the first 9B line then rejected eight consecutive candidates.The mitigation is to re-measure the accepted harness before judging the next candidate.Source: G MEASUREMENT PITFALLS AND MITIGATIONS · p. 18
  • An edit that never executes can pass as neutral; the paper’s mitigation requires evidence that its code path fired during evaluation.One such edit was accepted in the 4B line and three in the WebArena-Lite line.Source: G MEASUREMENT PITFALLS AND MITIGATIONS · p. 18

H FURTHER LINES IN DETAIL

Across the further model lines, results vary with failure composition: several gains appear in delivery or interface behavior, while weak process-failure capacity leaves little for harness edits to repair. The plan-conversion audit also shows why close score comparisons require care.

H FURTHER LINES IN DETAIL

  • Qwen3.6 thinking-off scored +0.063 and hy3 +0.065, while Qwen3.6 thinking-on scored +0.053 within its cell noise; all gains were delivery, not delivered-plan accuracy.An earlier single-rollout Qwen3.6 thinking-on reading of 0.535 fell to 0.44–0.46 four days later, motivating same-night comparisons.Source: H FURTHER LINES IN DETAIL · p. 19
  • All three non-Qwen lines closed within seed noise; their seeds had 0–6% healthy trajectories versus 15–34% for Qwen, and delivered plans scored 0.13–0.17 versus 0.27–0.31.The evidence packages predated failure-composition reporting, so the compositions were computed after the fact.Source: H FURTHER LINES IN DETAIL · p. 19, Table 14 · p. 20
  • MiniCPM delivery rose from 61% to 71% while delivered-plan quality fell from 0.173 to 0.156; Spark’s F1 fell from 21% to 7% as F4 rose from 38% to 60%, leaving its score unchanged.MiniCPM’s product did not move, and Spark’s freed trajectories landed in F4.Source: H FURTHER LINES IN DETAIL · p. 19
  • Ministral began with F2 and F3 totaling 7%; five candidates edited the exit gate, eight of nine rejections followed proposer rollback conditions, and the sole accept was a syntax fix.Source: H FURTHER LINES IN DETAIL · p. 19
  • Gemma 4 12B’s cross-day 0.330-to-0.386 line remeasured at +0.019 development and +0.012 held-out, within noise; 98% delivery and 40% F4 point to the weights branch.Its training recipe produced development gains that did not carry to held-out tasks.Source: H FURTHER LINES IN DETAIL · pp. 19–20
  • Under Gemma’s seed harness, LoRA added +0.034 / +0.018 and full-parameter training +0.038 / −0.003 (dev / held-out); stacking on the evolved harness added no resolvable gain.The reported stacked changes were LoRA −0.042 / −0.013 and full-parameter +0.019 / −0.040 (dev / held-out).Source: H FURTHER LINES IN DETAIL · p. 20
  • On WebArena-Lite, three accepted middleware edits never executed; after the bug fix, the closing re-anchor was within noise, leaving interface edits as the line’s gain.Undelivered trajectories mostly stopped at the 40-call budget (77 of 108); the closing harness read more of each page and walked further.Source: H FURTHER LINES IN DETAIL · p. 20

I PLAN-TO-JSON CONVERSION BIAS

  • The default plan converter silently selected one draft in all 20 multi-draft cases tested, and configuration-dependent multi-draft rates make score differences near 0.025 worth careful reading.The paper reports all scores using that converter; its repeat agreement was 93.5% at default temperature.Source: I PLAN-TO-JSON CONVERSION BIAS · p. 20

J TWO FRONTIER API MODELS UNDER THE EVOLVED HARNESS

The separate-machine check applied evolved harnesses to Kimi K3 and GLM 5.3 as acting agents. Kimi improved on both splits, while GLM’s harness gain cannot be identified without a bare control.

J TWO FRONTIER API MODELS UNDER THE EVOLVED HARNESS

Loading 2610.11655v1…