PaperlayerView exact revision on arXiv

Source-linked AI summary

HarnessBandit: Joint Learnability-Transferability Scheduling for Multi-Harness Agentic Reinforcement Learning

Hongliang Wei, Xiaobing Tu, Yinggui Wang, Zhengxi Liu, Rongkun Xue, Jinkui Ren, Xiantao Zhang, Debin Zhao, Xiaopeng Fan

arXiv:2609.13739v1cs.LGcs.AIcs.CL

TL;DR

Harness variation can change agent trajectories even for the same task, making the choice of training interface consequential. HarnessBandit schedules one harness per optimizer step using learnability and cross-harness transferability; it improves over mixed schedules on both benchmarks, including a 41.6% ClawEval pass@3 versus 38.5% for the six-harness mixture and 29.8% for the two-harness mixture.

  • Problem

    Different harnesses induce different trajectories for the same task and reward, so a policy trained through one interface need not transfer to another, and multi-harness training must decide how to allocate steps.

    Source: §1 · p. 1

  • Method

    HarnessBandit schedules harness-pure rollout batches as a non-stationary bandit, combining batch learnability and cross-harness gradient alignment while retaining exploration.

    Source: §1 · p. 2, §3 · p. 4, §4.4 · p. 5

  • Results

    HarnessBandit improves over mixed schedules on both benchmarks; on ClawEval, pass@3 is 41.6% versus 38.5% for the six-harness mixture and 29.8% for the two-harness mixture.

    Source: §5.3 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    Learnability and transferability provide distinct, evolving signals for allocating training steps across harnesses, and the resulting schedule is a mild reweighting rather than a collapse to one harness.

    Source: §6 · p. 8

    The evaluation uses disjoint tasks, with PinchBench on an in-training harness and ClawEval on a native evaluation loop outside the training harness set.

    Source: §3 · p. 3

Abstract

from arXiv · show

Language-model agents are increasingly deployed through diverse harnesses that differ in system prompts, tool schemas, control loops, and trajectory formats. The same model can perform unevenly across these interfaces, making robustness to harness variation an important objective. A natural approach is to train a shared policy through multiple harnesses, but doing so introduces a scheduling problem: each training step should favor a harness that currently provides a useful learning signal while also producing an update that benefits the other harnesses. We develop HarnessBandit, an online scheduler that selects one harness per optimizer step. After a group-relative policy optimization (GRPO) update, it observes learnability -- the mean absolute advantage on the batch -- and transferability -- the cosine between a low-dimensional gradient sketch of the current harness and exponential moving averages of the remaining harnesses. The two signals are fused after pooled sliding-window min-max normalization and sampled with a visit-dependent bonus and an explicit exploration floor. We train Qwen3.5-2B across six harnesses on ClawGym and evaluate on PinchBench (held-out tasks, in-distribution OpenClaw) and ClawEval (held-out tasks and harness). HarnessBandit improves over mixed-batch multi-harness training on both benchmarks, while training diagnostics indicate that learnability and transferability provide distinct, evolving signals.

1 INTRODUCTION

Harness variation changes agent trajectories even when tasks and rewards stay fixed, motivating an adaptive choice of training interface. HarnessBandit combines learnability and cross-harness transferability to schedule harness-pure updates, and improves over static mixtures on both benchmarks.

1 INTRODUCTION

  • Different harnesses induce different trajectories for the same task and reward, so training through one interface need not transfer to another.Harnesses combine system prompts, tool schemas, planner or loop policies, and session bookkeeping.Source: §1 · p. 1
  • A useful training schedule must balance non-degenerate learning signals against updates aligned with other harnesses, which static step allocations do not track.Under GRPO, identical rewards within a prompt group yield advantages near zero and almost no policy gradient.Source: §1 · p. 1
  • HarnessBandit treats harnesses as non-stationary bandit arms and adapts which one supplies each pure rollout batch while retaining exploration.The scheduler measures learning signal and cross-harness alignment after each update.Source: §1 · p. 2, Figure 1 · p. 2
  • HarnessBandit reaches 41.6% pass@3 and 0.556 mean score on ClawEval, versus 38.5% / 0.510 for the six-harness mixture and 29.8% / 0.490 for the two-harness mixture.On PinchBench, it records 0.506 versus 0.493 for the two-harness mixture and 0.331 for the six-harness mixture.Source: §1 · p. 2

2 RELATED WORK

Prior work studies tool-using agents, shared-model update interference, curricula and data mixtures, bandit allocation, and gradient similarity. HarnessBandit focuses on choosing which execution harness generates the next on-policy batch for shared tasks.

Agentic reinforcement learning and tool use

  • Agentic RL and tool-use work studies reasoning with environment actions and agent interfaces, while Polar exposes multiple harnesses as rollout backends behind a shared trainer.HarnessBandit uses that interface and asks which harness should generate the next batch.Source: §2 · pp. 2–3

Multi-task learning and conflicting updates

  • Multi-task methods address gradient interference or shared skills, but typically assume fixed task sampling; HarnessBandit instead selects informative, aligned harness rollouts without altering gradients.Here harnesses wrap the same ClawGym instances, so interface mismatch rather than disjoint objectives induces interference.Source: §2 · p. 3

Curriculum learning and data mixing

  • Curriculum and data-mixing methods usually schedule difficulty or tune mixture weights, rather than measuring current on-policy advantage and cross-source gradient alignment.A related two-signal bandit schedules reasoning domains; this work allocates steps among harnesses wrapping the same task collection.Source: §2 · p. 3

Bandit allocation

  • The scheduler combines an exponentially weighted value, visit-dependent bonus, Boltzmann sampling, and a uniform floor; its contribution is harness-specific integration, not a new regret result.The c/√n_h + 1 bonus is described as an implementation choice.Source: §2 · p. 3

Gradient similarity and sketches

  • HarnessBandit uses a fixed, layer-proportional subsample of the last four transformer blocks to compare gradient directions without storing full 190M-dimensional buffers.Linear sketches preserve inner products in the Johnson–Lindenstrauss sense.Source: §2 · p. 3

3 PROBLEM FORMULATION

The formulation treats tasks and harnesses as a product of shared tasks and execution interfaces, then schedules one interface per optimizer step. The scheduler balances current learning signal and alignment with other harnesses as the policy changes.

Tasks and harnesses

  • A task supplies a query, workspace, and checker reward, while a harness wraps it with a prompt, tool schema, and control loop that induces a trajectory distribution.The same task under different harnesses can yield different tokenizations, tool-call syntax, and reward realizations.Source: §3 · p. 3
  • Evaluation uses disjoint tasks; PinchBench runs under a training harness, whereas ClawEval uses a native loop outside the training harness set.The intended generalization is across held-out tasks and, for ClawEval, across interfaces as well.Source: §3 · p. 3

On-policy update

  • GRPO forms advantages against a leave-one-out group baseline, and Dr.GRPO omits division by group standard deviation to avoid exploding updates when reward variance is near zero.The update uses a clipped surrogate with a KL penalty toward a frozen reference; HarnessBandit can use any on-policy method yielding advantages and a batch gradient.Source: §3 · pp. 3–4

Scheduling objective

  • Each optimizer step selects one harness for the entire rollout batch, and arm utilities evolve with the changing shared policy.The scheduler trades off two instantaneous harness properties rather than treating utilities as fixed reward distributions.Source: §3 · p. 4
  • Learnability favors batches with mixed successful and unsuccessful trajectories, while transferability favors updates aligned with other harnesses’ running gradient directions.Optimizing either signal alone can favor updates that are respectively idiosyncratic or currently uninformative; the scheduler combines normalized values subject to exploration.Source: §3 · p. 4
  • The signals are inexpensive training-step instruments intended to allocate a fixed budget toward usable, shareable updates, not the evaluation reward itself.Expected reward on held-out tasks remains the ultimate objective.Source: §3 · p. 4

4 METHOD

HarnessBandit leaves GRPO unchanged and schedules harness-pure batches using learnability and cross-harness transferability. It normalizes and fuses those signals, then samples the next harness while retaining exploration.

4 METHOD

  • HarnessBandit wraps an unchanged GRPO trainer, recording learnability and sketched-gradient transferability after each harness-pure update to select the next harness.Signals are pooled-window normalized before fusion, and the scheduler draws the next harness.Source: §4 · p. 4

4.1 HARNESS-PURE BATCHES

  • Each rollout batch draws all prompts from one harness’s shuffled pool, making its gradient attributable to that harness and allowing filtered groups to be retained per harness.The pools contain the same ClawGym tasks, limiting confounding by task difficulty.Source: §4.1 · p. 4

4.2 LEARNABILITY

  • Learnability averages token-masked absolute advantages by trajectory, prompt group, and harness; equal group weighting avoids long trajectories dominating the signal.Identical rewards within every group yield near-zero learnability, even when the absolute reward is high.Source: §4.2 · pp. 4–5

4.3 TRANSFERABILITY VIA GRADIENT SKETCHES

  • Transferability compares a current gradient sketch with available peer-harness EMAs; sketches sample the last four transformer blocks, excluding embeddings and the output head.The sketch uses proportional coordinate subsampling, and transferability is undefined until another harness EMA is available.Source: §4.3 · p. 5, §4.6 · p. 6

4.4 NORMALIZATION, FUSION, AND VALUE UPDATE

  • Learnability and transferability are min–max normalized over a pooled 32-observation window, fused with α=0.5 when both exist, and used to update per-harness value estimates.An unvisited harness receives the current maximum Q value at selection time.Source: §4.4 · p. 5

4.5 SAMPLING

  • The scheduler ranks harnesses with a value-plus-visit bonus, samples by temperature-scaled softmax with an ε exploration floor, and begins with 30 round-robin steps.The floor ensures a minimum sampling rate; the bonus does not grow with training time, so a harness that stops being drawn cannot restore its own bonus.Source: §4.5 · p. 5

4.6 COMPLEXITY

  • Per step, the add-on gathers d gradient coordinates on K layers, all-reduces a d-vector, and evaluates O(Hd) cosines; it leaves rollout, reward evaluation, and policy loss unchanged.The EMA store occupies a few megabytes, making HarnessBandit a constant-factor add-on rather than a second trainer.Source: §4.6 · pp. 5–6

5 EXPERIMENTS

The experiments compare harness scheduling with mixed-batch training on matched ClawGym training and compute budgets, using held-out tasks with either a familiar or unseen evaluation interface. HarnessBandit leads the mixed schedules on both benchmarks, while diagnostics show distinct learnability and transferability signals and a non-collapsed schedule.

5.1 TRAINING SETUP

  • Training crossed 2,000 ClawGym tasks with six harnesses into 12,000 prompt rows, and compared methods under shared optimizer, data, step-budget, and hardware settings.The policy was Qwen3.5-2B trained with Dr.GRPO for 150 optimizer steps; the comparison included harness-pure HarnessBandit and two mixed loaders.Source: §5.1 · p. 6

5.3 RESULTS

  • PinchBench mean was 0.506 for HarnessBandit, versus 0.493 for the two-harness mixture and 0.331 for the six-harness mixture.The six-harness mixture’s 0.331 is essentially tied with the untuned base model’s 0.330.Source: Table 1 · p. 7, §5.3 · p. 7

5.2 EVALUATION PROTOCOL

  • PinchBench tests 147 held-out tasks through OpenClaw, an in-training harness; ClawEval tests 161 held-out tasks through its native, out-of-training loop with three trials.The table reports PinchBench mean score and ClawEval Pass3, pass@3, and average score under the same 150-step GRPO budget.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 RESULTS

  • On ClawEval, HarnessBandit scored 67/161 (41.6%) pass@3 and 0.556 mean score, exceeding both mixtures on those metrics.It matched the base model on Pass3 at 43/161 (26.7%), while both mixtures were below the base model on Pass3 and mean score.Source: §5.3 · p. 7, Table 1 · p. 7

5.4 BANDIT BEHAVIOR

  • After warmup, harness probabilities are non-uniform but remain distributed: OpenClaw, Pi, and OpenCode end near 0.20, 0.20, and 0.18, with visit counts spanning 21–32.This is a mild reallocation rather than collapse to one harness.Source: §5.4 · p. 7, Figure 2 · p. 8
  • The signals rank harnesses differently: Pi has highest normalized learnability (0.75), while OpenCode is high on both learnability (0.62) and transferability (0.60).Directed pairwise gradient cosines fluctuate around zero with positive and negative intervals, and no source harness is uniformly beneficial.Source: §5.4 · pp. 7–8

6 CONCLUSION

HarnessBandit schedules harness-pure GRPO updates by combining learnability and cross-harness transferability. In the reported evaluation, it outperforms both mixed schedules on PinchBench and exceeds them on ClawEval metrics while matching the base model on Pass3.

6 CONCLUSION

  • HarnessBandit estimates learnability from GRPO advantages and transferability from sketched last-layer gradients, then fuses the signals after shared-window normalization to schedule harness-pure batches.The scheduler uses a UCB-softmax rule on top of the GRPO trainer.Source: §6 · p. 8
  • After 150 GRPO steps, HarnessBandit exceeds both mixed schedules on PinchBench and all three ClawEval metrics, while matching the base model on ClawEval Pass3 at 26.7%.The induced schedule mildly reallocates visits rather than collapsing to one harness.Source: §6 · p. 8

A HYPERPARAMETERS

  • Table A.1 lists GRPO hyperparameters shared across trained methods; the scheduler knobs apply to HarnessBandit, while mixed-batch baselines ignore them.The optimizer, batch size, sample count, step budget, and hardware are matched to isolate harness allocation.Source: A HYPERPARAMETERS · p. 10, Table A.1 · p. 11

B PINCHBENCH CATEGORY MEANS

PinchBench category means vary by method, with HarnessBandit leading in seven categories and the two-harness mixture leading in four. Category comparisons use identical cell counts across methods.

B PINCHBENCH CATEGORY MEANS

  • HarnessBandit leads on analysis, integrations, log analysis, productivity, research, skills, and writing; the two-harness mixture leads on coding, CSV, meeting analysis, and memory.Those four categories align with the OpenClaw and Codex interfaces.Source: B PINCHBENCH CATEGORY MEANS · p. 11
  • The six-harness mixture wins no category, and its writing score is zero, which the authors treat as a recorded score rather than evidence it cannot write.Source: B PINCHBENCH CATEGORY MEANS · p. 11
  • Category cell counts are identical across methods, including coding (n=14) and CSV analysis (n=26).Source: Table B.1 · p. 11, B PINCHBENCH CATEGORY MEANS · p. 11

C BANDIT SCHEDULE

HarnessBandit begins with a round-robin warmup, then reallocates sampling non-uniformly without collapsing to one harness. The final visit counts show only mild reallocation over the 150-step budget.

C BANDIT SCHEDULE

  • The first 30 steps use round-robin sampling, so every harness is observed before softmax sampling begins.Source: C BANDIT SCHEDULE · p. 11
  • After warmup, OpenClaw, Pi, and OpenCode receive more probability, while Qwen Code and Codex remain above the exploration floor.Source: C BANDIT SCHEDULE · p. 11
  • Final visit counts span 21–32 across harnesses, a mild reallocation of the 150-step training budget.Source: C BANDIT SCHEDULE · p. 11

D RAW ONLINE SIGNALS

The raw online signals differ in scale and ranking: learnability stays positive while transferability fluctuates near zero. Their distinct behavior motivates shared sliding-window normalization before fusion.

D RAW ONLINE SIGNALS

  • Learnability is mean absolute advantage and stays clearly positive, while transferability is an average cosine that fluctuates in a narrow band around zero.Source: D RAW ONLINE SIGNALS · pp. 11–12
  • The signals use different units and their harness rankings can differ, so the method applies shared sliding-window min–max normalization before fusion.Source: D RAW ONLINE SIGNALS · p. 12
  • Figure D.1 plots 7-step moving averages of learnability on the left and transferability on the right, before normalization.Source: D RAW ONLINE SIGNALS · p. 11, Figure D.1 · p. 12

E PAIRWISE GRADIENT GEOMETRY

Pairwise gradient cosines vary across source–peer harness pairs and over time, with both positive and negative intervals. This changing geometry makes the scalar transferability state evolve rather than settle on one universally helpful interface.

E PAIRWISE GRADIENT GEOMETRY

F FINAL HARNESSBANDIT STATE

The terminal schedule tilts toward OpenClaw, Pi, and OpenCode, while remaining distributed across all six harnesses.

F FINAL HARNESSBANDIT STATE

  • Terminal sampling probabilities were 0.204 for OpenClaw, 0.197 for Pi, 0.184 for OpenCode, 0.151 for Gemini CLI, 0.137 for Codex, and 0.127 for Qwen Code.Visit counts were OpenCode 32, OpenClaw 26, Qwen Code 24, Codex 24, Gemini CLI 21, and Pi 21.Source: F FINAL HARNESSBANDIT STATE · p. 12
  • Figure E.1 fixes the source harness by panel and uses line color for the peer EMA in directed current-to-peer gradient-cosine plots.The plotted values are 10-step moving averages.Source: Figure E.1 · p. 13
  • Against uniform allocation (about 25 visits per harness), the schedule is a tilt rather than a collapse to one harness.Uniform allocation corresponds to p_h = 1/6 ≈ 0.167.Source: F FINAL HARNESSBANDIT STATE · pp. 12–13

Related papers

Loading 2609.13739v1…