PaperlayerView exact revision on arXiv

Source-linked AI summary

Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents

Laizhen Li, Jiarui Li, Juanjuan Zhao, Kejiang Ye, Ye Li, Cheng-zhong Xu, Xitong Gao

arXiv:2609.26760v2cs.AIcs.SE

TL;DR

LLM agents serving related tasks often reconstruct recurring control through repeated inference, adding calls and context. Growing Harness learns reusable control code from task feedback, and achieves the highest mean success in five of six benchmark-model settings while reducing calls and cost relative to Tool-Calling.

  • Problem

    Agents repeatedly serving related tasks may reconstruct recurring control through LLM inference, adding calls and growing execution histories.

    Source: §1 · pp. 1–2

  • Method

    Growing Harness grows a shared executable harness from a strategy-free scaffold using failure traces, bounded function-level edits, multi-failure repair, and a success-first held-out gate.

    Source: §1 · p. 3, §5 · p. 10

  • Results

    Across BrowseComp-Plus and WebArena-Verified and three deployment-model scales, Growing Harness achieves the highest mean success in five of six settings while reducing LLM calls by 76.0–91.8% and online cost by 74.4–98.6% relative to Tool-Calling.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Harness growth offers a structural way to move repeatable control into code and reserve model context for task-varying evidence and decisions.

    Source: §5 · p. 10

    Its value depends on reusing the learned harness enough to offset offline optimization.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large language model (LLM) agents often handle streams of related tasks, yet standard harnesses repeatedly ask the model to reconstruct the same control decisions inside each task's context. We study whether task feedback can instead turn recurring control into reusable executable code, while reserving LLM calls for task-specific semantic reasoning. We introduce Growing Harness, a failure-guided training paradigm that learns the agent harness itself from a strategy-free scaffold that exposes fixed model and tool interfaces but encodes no task-solving controller. Function-level execution traces localize each failure to a bounded code surface, an optimizer repairs a window of failures jointly, and a success-first held-out gate rolls back repair sequences that harm prior capability. Accepted edits accumulate in one shared harness, allowing its control structure to emerge from task feedback. Across BrowseComp-Plus and WebArena-Verified with three deployment models from 4B to 120B parameters, Growing Harness achieves the highest mean success in five of six benchmark-model settings and trails the best mean by 0.7 pp. in the sixth. Relative to a Tool-Calling agent, it reduces LLM calls by 76.0-91.8% and deployed-agent inference cost by 74.4-98.6%. On WebArena-Verified, its success remains 44.7-45.3% across model scales, whereas Tool-Calling falls to 6.7% with the 4B model. Ablations show that trace-local edits, joint repair, and gate-based rollback each improve final success. These results show that persistent program growth can move recurring control out of model context and into low-cost code, yielding reusable specialist agents that remain effective with smaller deployment models.

1 Introduction

Repeated deployments on related tasks can leave agents reconstructing recurring control through model inference. Growing Harness learns that control as reusable code from task feedback, while reserving LLM calls for task-specific semantic reasoning.

1 Introduction

  • Harness growth aims to make recurring control reusable across related tasks instead of repeatedly reconstructing it through inference, leaving task-specific semantic reasoning to the LLM.Source: §1 · pp. 1–2
  • The learned harnesses occupy a high-success, low-cost region, with the largest separation from inference-intensive agents on WebArena-Verified and with the 4B deployment model.Source: §1 · p. 3
  • The approach learns the harness from a strategy-free scaffold under fixed LLM and tool interfaces, rather than starting with a predefined task-solving controller.Source: §1 · pp. 2–3
  • Trace-local program growth combines function-level edits, multi-failure training, and held-out rollback to focus repairs and guard against regressions.Source: §1 · p. 3, Figure 2 · p. 3
  • Across two benchmarks and three deployment-model scales, Growing Harness achieves the highest mean success in five of six settings, while reducing LLM calls by 76.0–91.8% and online cost by 74.4–98.6% relative to Tool-Calling.Source: §1 · p. 3

2 Related Work

Prior work studies inference-time control, persistent experience, language-model program optimization, harness optimization, and lower-cost execution. Growing Harness differs by growing a shared harness through local, failure-conditioned updates from a strategy-free scaffold.

Inference-time control in tool-using agents

Persistent experience across tasks

  • Persistent-experience agents store insights, workflows, skills, or tools for reuse, whereas this paper makes the shared harness itself the persistent artifact Wang et al. (2025, 2023); Cai et al. (2023); Yuan et al. (2023).Source: §2 · p. 4

Optimizing language-model programs

Harness optimization

Cost-efficient LLM and agent execution

3 Method

Growing Harness trains the executable harness from a strategy-free scaffold while keeping the model and tools fixed. Traces guide bounded repairs, and held-out gate validation protects previously acquired capability.

3.1 Problem Formulation

  • Training learns only harness code h while keeping model M and tools T fixed, starting from a minimal scaffold without a complete hand-designed strategy.Training, gate-validation, and final-evaluation tasks have separate roles; the optimizer never observes final-evaluation outcomes or traces.Source: §3.1 · p. 4

3.2 Harness Execution and Function-Level Traces

  • For each task, execution traces capture runtime events and measurements, while invoked-function sets identify existing code eligible for repair.The trace records harness-function invocations, model and tool calls, returns, and errors; its function set constrains which existing functions may be modified.Source: §3.2 · p. 5

3.3 Failure-Window Curriculum

  • A bounded window collects failures from unseen tasks for joint repair, removes solved tasks, and retires unresolved tasks after Rmax attempts.If every task exhausts its attempt budget without a repair, the system restores the checkpoint from before optimizing the window.Source: §3.3 · pp. 5–6

3.4 Trace-Guided Function-Level Optimization

  • The optimizer may edit only the entry point and traced functions, add helpers within an edit budget, and preserve interfaces while targeting reusable control.Code handles deterministic operations; LLM calls remain for semantic interpretation, synthesis, fuzzy comparison, and answer generation.Source: §3.4 · p. 6

3.5 Candidate Evaluation and Window Update

  • Candidates are rerun on the entire active window; solved tasks leave and unresolved tasks receive fresh traces and incremented attempt counts.Helpers and control logic accumulate in the harness across rounds rather than remaining task-specific patches.Source: §3.5 · p. 6

3.6 Held-Out Gate Validation and Rollback

  • Held-out gate validation accepts only non-decreasing gate success and rolls back regressing repair sequences; final selection likewise prioritizes success over online cost.The final harness is evaluated once on the separate final-evaluation set.Source: §3.6 · pp. 6–7

4 Experiments

Experiments test Growing Harness on two benchmarks across three deployment-model scales, then examine how its learned harnesses grow and which mechanisms contribute to success. It achieves top mean success in five of six settings while using fewer calls and less online cost than Tool-Calling.

4.1 Experimental Setup

  • Each benchmark uses 200 training, 50 gate, and 50 final tasks, with three independent runs per configuration and shared within-benchmark models, splits, tools, and protocol.BrowseComp-Plus covers open-domain retrieval and evidence synthesis; WebArena-Verified covers multi-step interaction.Source: §4.1 · p. 7

4.2 Main Results

  • 76.0–91.8% fewer LLM calls and 74.4–98.6% lower online cost than Tool-Calling accompany top mean success in five of six settings, with a 0.7 pp. gap in the sixth.Table 1 reports the main results across BrowseComp-Plus and WebArena-Verified.Source: §4.2 · pp. 7–8, Table 1 · p. 8
  • On WebArena-Verified, Growing Harness success stays within 44.7–45.3% across models, while Tool-Calling falls from 30.0% with gptoss-120b to 6.7% with Qwen3.5-4B.The paper reports the largest gain with the smallest deployment model.Source: §4.2 · p. 8

4.3 Harness Growth Convergence Analysis

  • The shared seed grows into a retrieval-and-verification pipeline for BrowseComp-Plus but specialized Shopping, Reddit, and Map handlers for WebArena-Verified.Because training configurations differ, this contrast describes these runs rather than intrinsic benchmark difficulty.Source: §4.3 · pp. 8–9

4.4 Ablation Studies

  • Removing function-level guidance cuts final success from 36% to 18%, gate validation lets gate success fall from 30% to 16%, and single-failure repair lowers final success by 8 pp.Each mechanism removal lowers final-evaluation success; the gate-validation ablation shows a rise followed by a decline.Source: §4.4 · p. 9

5 Conclusion

Growing Harness turns recurring control into persistent code while reserving LLM calls for task-specific semantic decisions. Its value depends on reuse and safe execution.

5 Conclusion

  • Starting from a strategy-free scaffold, Growing Harness uses function-level traces, joint repair of bounded failure windows, and a held-out gate to grow a shared harness.The gate rolls back changes that reduce prior capability.Source: §5 · p. 10
  • Harness growth encodes repeatable control in the harness and reserves model context for evidence and decisions that vary by task.Source: §5 · p. 10
  • The approach’s value depends on enough harness reuse to offset offline optimization; deployment also requires sandboxing, permission boundaries, and generated-code validation.Source: §5 · p. 10

A.1 Datasets

The evaluation uses fixed-size train, gate, and final splits. WebArena-Verified coverage is limited to three websites, with stratified sampling and a fixed seed.

A.1 Datasets

  • Each benchmark is split into 200 train tasks, 50 gate tasks, and 50 final tasks.Train supplies optimization traces, gate supports checkpoint selection, and final is held out for reporting.Source: §A.1 · p. 13
  • WebArena-Verified is restricted to shopping, reddit, and map tasks, stratified across intent templates, task types, and websites.Source: §A.1 · p. 13
  • All dataset splits use fixed random seed 42.Source: §A.1 · p. 13

A.2 Shared Evaluation Configuration

Final-evaluation runs use a shared configuration across methods. The harness, rather than the agent, enforces the call limit.

A.2 Shared Evaluation Configuration

A.3 Baseline Configurations

Baselines share the seed interface, tool APIs, deployment models, and LLM-call budget with Growing Harness. Their benchmark-specific interaction procedures differ.

A.3 Baseline Configurations

  • All baselines use the same seed interface, tool APIs, deployment models, and LLM-call budget as Growing Harness.Source: §A.3 · p. 13
  • On BrowseComp-Plus, methods share the search tool: Tool-Calling alternates reasoning and search, Self-Ask uses follow-up questions, and IRCoT interleaves retrieval with evidence generation.IRCoT then uses a final QA call.Source: §A.3 · p. 13
  • On WebArena-Verified, methods use text-only browser tools; WebDreamer scores candidate actions, while AgentOccam uses compressed observations and branch, prune, and note operations.The alternatives are adapted to the shared seed interface, accessibility-tree observations, LLM budget, and deployment models.Source: §A.3 · p. 13

A.4 Token Prices

Online cost uses provider token prices and excludes judge calls; the appendix also specifies fixed, benchmark-specific harness-training configurations selected without final-evaluation access.

A.4 Token Prices

  • Online cost is computed using the provider token prices in Table 4, excluding judge calls.Table 4 reports token prices in USD per million tokens.Source: §A.4 · p. 13, Table 4 · p. 13

A.5 Hyperparameters

  • Hyperparameters are selected using only the training and gate splits, and Table 5 reports the fixed harness-training configurations.All configurations are selected without access to the final-evaluation sets.Source: §A.5 · p. 13, Table 5 · p. 14
  • Figure 5 compares task-level LLM calls with an efficiency statistic for Growing Harness and Tool-Calling on BrowseComp-Plus and WebArena-Verified using GPT-OSS-20B; cost and token measurements exclude judge usage.Source: Figure 5 · p. 15
  • Figure 6 is a post-hoc abstraction of learned harnesses, not an architecture designed for the optimizer.It depicts a shared retrieval and evidence-verification pipeline for BrowseComp-Plus and a layered web-control program for WebArena-Verified.Source: Figure 6 · p. 15

B Additional Results

Additional results report task-level efficiency measurements and post-hoc abstractions of the learned harnesses, showing distinct executable control paths for the two benchmarks.

B Additional Results

  • Figure 5 reports per-task calls, cost, and token usage for Growing Harness and Tool-Calling under the same 50-call evaluation budget.The measurements correspond to aggregate results in the main paper.Source: B Additional Results · p. 16

B.1 Discovered Harness Structures

  • The learned BrowseComp-Plus harness uses a shared retrieval and evidence-verification pipeline, while WebArena-Verified uses layered web control with resolvers, learned routines, browser fallback, and completion validation.These are post-hoc summaries of discovered code, not hand-designed architectures; both wrap the same fixed deployment LLM and tool interfaces.Source: §B.1 · p. 16

Related papers

Loading 2609.26760v2…