PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness Scaling for Generalizable Agentic Reinforcement Learning

Fei Bai, Huatong Song, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2610.29458v1cs.LGcs.AIcs.CL

TL;DR

Policies trained through a single fixed harness may specialize, limiting transfer across harness configurations. Harness Scaling trains across a structured, diverse set of harnesses, and consistently improves generalization to unseen white-box configurations while transferring effectively to external black-box harnesses.

  • Problem

    Existing approaches typically optimize policies through a single fixed harness, motivating study of transfer across execution configurations.

    Source: §1 · p. 1, §3.1.2 · p. 4

  • Method

    Harness Scaling factorizes harness variation, selects a compact set with balanced factorial coverage, and trains on structurally nearby harnesses paired with the same task.

    Source: §1 · p. 2

  • Results

    Harness Scaling consistently improves generalization to unseen white-box configurations and transfers effectively to Claude Code, where Qwen3-4B-Instruct reaches 37.22 Pass@1 versus 35.08 for black-box SFT and 33.38 for black-box RL.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    Harness diversity is a meaningful scaling dimension for agentic RL, and its benefits depend on how that diversity is selected and incorporated into training.

    Source: §6 · p. 10

Abstract

from arXiv · show

Agent harnesses have become a central component of modern agentic systems, shaping how models use tools, manage context, and coordinate long-horizon interaction. Realizing the benefits of these harnesses depends on how effectively the underlying model can make use of them, motivating reinforcement learning directly through the harness. However, existing approaches typically train through a single fixed harness, which can lead to specialization and limited transfer across harness configurations. To address this, we introduce \textbf{Harness Scaling}, a new scaling dimension for agentic RL that extends training from a single harness to diverse execution configurations. Concretely, we construct a factorized harness space, select a compact training subset with balanced factorial coverage, and organize cross-harness training to combine global coverage with controlled variation for the same task. Across multiple benchmarks, model scales, and RL algorithms, Harness Scaling consistently improves generalization to unseen white-box configurations. Moreover, models trained through structured white-box variation transfer effectively to unseen external black-box harnesses, outperforming models trained through a single complex black-box harness. These results establish harness diversity as a meaningful scaling dimension for agentic RL and highlight the importance of how this diversity is organized for cross-harness generalization.

1. Introduction

Harness Scaling addresses the limits of training through a single fixed harness by varying execution configurations for the same task. It uses structured harness selection and cross-harness training, improving generalization to unseen configurations.

1. Introduction

  • Existing approaches typically optimize through one fixed harness, which may specialize policies and weaken transfer across harnesses.Harness Scaling varies how policies execute and interact with the same underlying task and environment, unlike task or environment scaling.Source: §1 · p. 1
  • The method factorizes harness variation across five components and selects a compact training set balancing component realizations and pairwise combinations.Structurally nearby harnesses are paired with the same task to combine controlled local variation with broad training-space coverage.Source: §1 · pp. 1–2
  • Harness Scaling consistently outperforms single-harness RL on unseen white-box configurations and transfers effectively to external black-box harnesses, outperforming single-harness black-box training.The evaluation spans multiple benchmarks, model scales, and RL algorithms.Source: §1 · p. 2
  • Harness Scaling introduces harness diversity as a distinct agentic RL scaling dimension and finds that how this diversity is organized matters for generalization.The framework factorizes harness variation, selects balanced configurations, and assigns structurally nearby configurations to the same task.Source: §1 · p. 2

2. Related Work

Prior agentic RL scaling expands the tasks agents solve or the environments where they operate, often retaining a fixed execution harness. Harness Scaling instead studies policy optimization across structured harness configurations to generalize to unseen ones.

2.1. Agentic Reinforcement Learning

  • Task scaling broadens problems and environment scaling expands external settings, while these directions often retain a fixed execution harness.Harness Scaling adds diversity in the execution configurations through which policies interact.Source: §2.1 · p. 2

2.2. Harness Design and Optimization

  • Prior harness work improves harnesses or adapts policies to particular harnesses; Harness Scaling optimizes policies across a structured combinatorial space to generalize to unseen configurations.Figure 1 overviews harness-space construction, cross-harness training, and generalization to unseen configurations.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

3. Method

Harness Scaling formalizes harness configurations as combinations of component realizations and aims to generalize across unseen combinations and external harnesses. Its method selects a compact, balanced training set and assigns related configurations to tasks for cross-harness policy optimization.

3.1.1. Factorized Harness Space

  • Harnesses are factorized into five components, and Hamming distance measures how many components differ between configurations.The factorization holds task semantics and the environment fixed while component transformations alter execution settings.Source: §3.1.1 · pp. 3–4, §3.1 · p. 3

3.1.2. Generalization Objective

  • The objective covers both held-out combinations within the constructed harness space and transfer to external black-box harnesses.In-space evaluation uses complete combinations unseen during training; out-of-space evaluation uses real-world harnesses not represented in the constructed space.Source: §3.1.2 · p. 4

3.2. Harness Scaling with Structured Selection

  • Because the factorized space grows combinatorially, the training set is kept compact to preserve broad factorial coverage under a fixed optimization budget.Source: §3.2 · p. 4

3.2.1. Factorial Coverage

  • The selection preserves low-order factorial structure by balancing component realizations and their pairwise combinations.Component balance prevents overrepresentation of particular realizations; pairwise balance gives every realization pair across distinct components equal representation.Source: §3.2.1 · pp. 4–5

3.2.2. Harness Set Construction

  • A strength-2 orthogonal design constructs a compact, exactly balanced subset, leaving many complete harness combinations unseen for evaluation.Its admissible size is constrained by pairwise coverage and the Rao bound.Source: §3.2.2 · p. 5

3.3. Reinforcement Learning across Harnesses

  • Tasks are paired with small groups of structurally related harnesses, combining controlled within-task variation with coverage across the training harness space.These task–harness instances form the basis for rollout collection and policy optimization.Source: §3.3 · p. 5

3.3.1. Cross-Harness Task Assignment

  • For each task, a uniformly sampled anchor is joined by the K−1 training configurations with the smallest Hamming distance, while random anchors distribute tasks across the space.Each task yields K task–harness instances; when K=1, only the anchor is assigned.Source: §3.3.1 · p. 5

3.3.2. Multi-Harness Policy Optimization

  • The K assigned harnesses are used over K successive training steps, while policy updates use the standard clipped objective and retain the underlying RL algorithm's advantage computation.The same cross-harness procedure is applied to PPO and GRPO.Source: §3.3.2 · p. 6

4. Experiments

Experiments evaluate Harness Scaling across models, RL algorithms, benchmarks, and held-out harness configurations. The results show improved in-space generalization, transfer to external black-box harnesses, and benefits from structured harness selection and assignment.

4.1. Experiment Settings

  • The experiments use 18 structured training configurations from a 243-configuration space, two Qwen3 backbones, GRPO and PPO, and three benchmarks with disjoint evaluation tasks.In-space evaluation averages over 18 held-out configurations; external evaluation uses Claude Code and OpenClaw.Source: §4.1 · p. 6

4.2. Main Results

  • In-space gains over Single-Harness RL reach +9.80 and +3.57 points with GRPO and PPO on Qwen3-4B-Instruct, and +5.21 and +4.90 points on Qwen3-30B-A3B-Instruct.The comparison averages results over three benchmarks; Figure 2 shows evaluation curves on ClawGym-Bench.Source: §4.2 · p. 7, Table 1 · p. 7, Figure 2 · p. 7
  • Overall averages improve from 29.81 to 39.04 with GRPO and from 33.91 to 36.93 with PPO for Qwen3-4B-Instruct, and from 37.29 to 42.78 and 36.31 to 41.89, respectively, for Qwen3-30B-A3B-Instruct.The reported gains persist across both backbone sizes and optimization methods.Source: §4.2 · p. 7, Table 1 · p. 7
  • On Claude Code, Harness Scaling achieves average Pass@1 scores of 37.22 and 41.20 for Qwen3-4B-Instruct and Qwen3-30B-A3B-Instruct, exceeding black-box SFT and RL.The respective black-box SFT scores are 35.08 and 38.08, and black-box RL scores are 33.38 and 36.28; both baselines train through OpenClaw.Source: Table 2 · p. 8, §4.2 · p. 8

4.3. Ablation Studies

  • The structured 18-harness set outperforms three first-order-coverage harnesses, Random-18, and the full 243-harness setting under the same training budget; gains do not rise monotonically with configuration count.The full-space setting includes white-box evaluation configurations and is a fixed-budget stress test, not a held-out generalization setting.Source: §4.3 · p. 8, Figure 3 · p. 9
  • Assigning K=1 harness per task already beats Single-Harness RL, K=2 improves further and exceeds Repeat-2, while K=4 adds no improvement.Repeat-2 reuses the same sampled harness; K=2 performs best overall among the evaluated settings.Source: §4.3 · pp. 8–9
  • Random assignment reduces performance by 2.32/2.20 points on ClawGym-Bench and 3.04/1.24 points on PinchBench under in-space/black-box evaluation.The reported pattern supports combining broad coverage from random anchor sampling with controlled local variation from nearest-harness assignment.Source: §4.3 · p. 9

5. Analysis

Harness Scaling is associated with stronger execution capabilities and more robust performance as execution configurations change. It improves behavioral capabilities relevant to varying harnesses, produces more consistent rewards across held-out configurations, and largely preserves base-harness performance.

5.1. Harness-Related Capability Analysis

  • Harness Scaling’s largest capability gains are in state tracking, workflow compliance, and error recovery, with smaller gains in observation reading.Instruction following and interface handling also improve.Source: §5.1 · p. 9

5.2. Cross-Harness Robustness and Specialization

  • Across two held-out harnesses differing in four components, Harness Scaling shifts rewards toward higher performance and closer alignment across configurations.The analysis compares paired rewards for the same tasks; Figure 4c–d shows the reward distributions.Source: §5.2 · p. 10, Figure 4 · p. 10
  • Harness Scaling decreases base-harness performance by 0.19 points while improving ClawGym-Bench in-space performance by 10.48 points and Claude Code performance by 3.60 points.The authors characterize the base-harness change as negligible relative to the cross-harness gains.Source: §5.2 · p. 10

6. Conclusion

The conclusion presents Harness Scaling as a complementary agentic RL scaling dimension focused on how policies execute. The appendices describe five-component harness variation and a compact orthogonal design that leaves higher-order configurations unseen for evaluation.

6. Conclusion

  • Across benchmarks, model scales, and RL algorithms, Harness Scaling improves generalization to unseen white-box configurations and transfers effectively to external black-box harnesses.The conclusion says benefits depend on how harness diversity is selected and incorporated into training, not diversity alone.Source: §6 · p. 10

A. Harness Component Variants

  • The harness space varies system prompt, tool interface, observation, context management, and workflow, with three realizations per component yielding 243 configurations.Each transformation targets one component while preserving the task, environment, and remaining components’ functionality.Source: A. Harness Component Variants · p. 15

B.1. Orthogonal Design and Coverage

  • A strength-2 orthogonal design selects 18 configurations, balancing each component realization and every pairwise combination while leaving higher-order configurations unseen.Each component realization appears six times and each pair across two components appears twice.Source: §B.1 · p. 17

B.2. Training Harness Set

The training harness set is represented by 18 configurations listed in Table 6.

B.2. Training Harness Set

  • The structured training harness set comprises 18 configurations, listed in Table 6.The supplied table caption identifies the configurations as the structured training set Htr.Source: §B.2 · p. 17, Table 6 · p. 17

C.1. Training

The training appendix summarizes the GRPO and PPO setups, explains how context-compacted rollout segments contribute to optimization, and specifies the black-box baseline comparison.

C.1. Training

  • GRPO uses 16 tasks with eight rollouts per task, while PPO uses 128 tasks with one rollout per task; both use a 1 × 10−6 actor learning rate.Training uses veRL with asynchronous vLLM rollouts; PPO also uses GAE with γ = λ = 1.Source: §C.1 · p. 17
  • Context-compacted segments inherit their rollout’s reward, group identity, and normalized advantage, while token-level mean loss weighting prevents extra sample-level weight from segmentation.GRPO reward normalization remains over the eight original rollouts, regardless of segment count.Source: §C.1 · p. 17
  • Black-box SFT and RL train through OpenClaw and are evaluated on unseen Claude Code; the comparison matches backbone, optimization budget, main GRPO hyperparameters, and evaluation protocol.SFT reuses a fixed trajectory set, whereas RL collects trajectories online.Source: §C.1 · pp. 17–18

C.2. Evaluation

The evaluation uses a balanced set of held-out harness configurations for in-space testing and reports performance on an additional external black-box harness. Scores are averaged across runs, with the same judge protocol used for rubric-based evaluation.

C.2. Evaluation

  • The 18 in-space evaluation configurations balance each component’s three realizations equally and span pairwise Hamming distances of 2 to 5.They are excluded from training in the main experiments and ablations, except the full 243-harness setting, where results are a fixed-budget comparison.Source: §C.2 · p. 18, Table 8 · p. 18
  • Pass@1 is averaged across three independent runs, with each task executed once per run under its corresponding harness setting.For rubric-based evaluation, GPT-5.4 judges all methods and harness settings using the same protocol.Source: §C.2 · pp. 18–19
  • On OpenClaw, Harness Scaling consistently improves performance over Single-Harness RL across both model scales and all three benchmarks.OpenClaw is outside the factorized harness space and was not used to train either method.Source: §C.2 · p. 19, Table 9 · p. 19

C.3. Additional Ablation Details

The ablations compare cross-harness variation for the same task with repeated exposure and test sensitivity to the fixed training harness. Behavioral metrics describe six harness-related capabilities beyond task-level performance.

C.3. Additional Ablation Details

  • K=2 outperforms Repeat-2 by 3.02/1.77 points on ClawGym-Bench and 5.90/2.83 points on PinchBench under in-space/Claude Code evaluation.Repeat-2 matches the two-step schedule but reuses the same sampled anchor harness; its changes relative to K=1 are mixed.Source: §C.3 · p. 19
  • Harness Scaling remains stronger than all four evaluated single-harness settings under both In-Space and Claude Code evaluation.The fixed-harness baselines vary in performance, and the selected configurations are structurally separated without using evaluation performance.Source: §C.3 · pp. 19–20, Table 11 · p. 20

D.1. Behavioral Metrics

  • Six trajectory-level dimensions capture instruction following, interface handling, observation reading, state tracking, workflow compliance, and error recovery beyond task-level performance.The metrics assess following harness instructions, using interfaces, extracting information from observations, tracking progress, complying with procedures, and responding to execution failures.Source: §D.1 · p. 20

D.2. Metric Evaluation

The metric evaluation uses GPT-5.4 to score complete trajectories independently across six behavioral dimensions. Scores are aggregated equally across held-out harnesses and normalized for visualization.

D.2. Metric Evaluation

  • For each model scale and each of 18 held-out harnesses, the study collects 200 trajectories per method on the same 200 ClawGym-Bench tasks.The judge receives the task instruction, harness specification, and complete interaction history.Source: §D.2 · p. 20
  • GPT-5.4 scores each of six capabilities independently on a five-point scale, using the same prompt, rubric, and judge model across methods and harness settings.Scores range from largely absent or repeatedly violated behavior at 1 to consistent and effective satisfaction at 5.Source: §D.2 · p. 20
  • Scores are averaged across 200 trajectories within each harness, then equally across all 18 harnesses; visualization divides them by five to yield 0.2–1.0 values.Source: §D.2 · p. 20

Related papers

Loading 2610.29458v1…