PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents

Chenqian Le, Jiayi Cheng, Qijia He, Runhao Li, Yinghao Li, Xupeng Chen

arXiv:2609.04518v1cs.AI

TL;DR

Multi-harness RL combines exposure to several execution harnesses with a choice about whether their rewards compete for credit, but the contribution of that grouping choice is not isolated by prior recipes. The paper holds experience and update budget fixed while comparing grouping rules, finding that harness choice moves solve rates more than training recipe choice and that cross-harness credit adds no detectable portability.

  • Problem

    Multi-harness RL does not by itself specify whether harness rewards compete in one relative-credit group, leaving the effect of that grouping choice unisolated.

    Source: §1 · p. 3

  • Method

    The study replays the same experience and update budget from a common warm start while changing only whether GRPO groups trajectories per task–harness pair or pools harnesses per task.

    Source: §3 · p. 4

  • Results

    Across 24,000 sealed evaluations, harness choice moves mean solve rate by a factor of 4.3 versus 1.16 across recipes, while Cross minus Within is +0.25 pp on the held-out harness at eight attempts per task.

    Source: §4 · p. 5, §5 · p. 5

  • Takeaways & Limitations

    Multi-harness RL reports should state the grouping boundary and measure deltas under at least one unseen harness to separate source adaptation from portable capability.

    Source: §7 · p. 8

    The study covers one model family, one budget, four source harnesses, one round of on-policy re-collection, and Python-only tasks.

    Source: §9 · p. 9

Abstract

from arXiv · show

Agent reinforcement learning (RL) increasingly runs through full execution harnesses, and a multi-harness recipe mixes two choices: exposing the policy to several harnesses, and comparing their rewards inside one relative-advantage group. We isolate the second choice in repository-level coding. From one Qwen3-8B supervised warm start we replay the same frozen task-harness records from Aider, OpenHands, Qwen Code, and SWE-agent, with the same number of updates, under two rules for group-relative policy optimization (GRPO), Within (one group per task-harness pair) and Cross (harnesses pooled within a task), and score every checkpoint with a sealed SWE-bench Verified oracle on four source harnesses and a minimal harness held out of training. The evaluation harness is the dominant variable: across 24,000 sealed evaluations it moves the mean solve rate from 2.14\% to 9.27\%, a factor of 4.3, where the training recipe moves it by 1.16. The grouping rule is not. On the held-out harness, Cross minus Within is +0.25 pp, 95\% confidence interval [-0.48, +1.02], at eight attempts per task, and +0.16 [-0.41, +0.72] pooled over three training seeds whose individual estimates change sign. Each rule's own seed range, 0.42 to 0.45 pp, exceeds the difference between them. Both rules place their largest gains on the same source harness. The pooled advantage carries the harness: an out-of-fold classifier recovers the generating harness from Cross's advantage +4.48 pp above the shuffled-label baseline and from Within's not at all, and the two rules still reach the same held-out score and action distribution inside each harness. Re-collecting half the training data on-policy does not change this. Cross-harness credit yields configuration adaptation and no more portable capability than within-harness credit. Multi-harness RL reports should state the grouping boundary and test under an unseen harness.

1 Introduction

Multi-harness RL combines exposure to different execution harnesses with a choice about whether their rewards compete for credit. The paper isolates that grouping choice and finds that cross-harness credit does not itself yield more portable capability.

1 Introduction

  • Existing multi-harness systems do not specify whether harness rewards are compared together or normalized separately, leaving the credit-assignment boundary unresolved.The systems discussed also differ in exposure, rollout generation, training regime, and evaluation, so their end-to-end results do not isolate that boundary.Source: §1 · p. 2
  • The experiment replays identical records, tokens, rewards, loss masks, and update budget from one Qwen3-8B warm start, changing only whether GRPO groups are per task–harness pair or pooled per task.Every checkpoint is evaluated with a sealed per-instance oracle on four source harnesses and a minimal harness absent from training.Source: §1 · p. 2
  • Cross-harness credit changes which trajectories carry signed gradient, but not the held-out score or the policy’s action distribution inside each harness.The evaluation harness is the largest effect in the study; seeing several harnesses and comparing them for credit are separate interventions.Source: §1 · p. 3
  • The study contributes a controlled recipe-by-harness matrix, a same-data comparison of GRPO grouping rules, and a measurement of harness identity retained in their advantages.Cross retains harness identity +4.48 pp above the shuffled-label baseline; Within does not, while both rules have the same held-out score and within-harness action distribution.Source: §1 · p. 3

2 Related work

Related work studies harness-dependent evaluation, training through production harnesses, multi-harness transfer, credit assignment, and what RL adds over a starting policy. This paper varies the comparison group while keeping reward resolution fixed.

Harnesses as an evaluation variable

  • Prior work finds agent scores, costs, and failures depend on model–harness pairing, motivating fixed-interface benchmarks and audits of benchmark versus model contributions.This paper measures that heterogeneity on its own checkpoints and uses it to pose a training question.Source: §2 · p. 3

Harness-native reinforcement learning

  • Harness-native RL systems study infrastructure, scaling, and native-harness performance; this work asks how outcomes from several native harnesses should interact in relative-credit estimation.For repository coding, SWE-RL and SWE-Gym establish the outcome-reward recipe replayed here.Source: §2 · p. 3

Multi-harness training and transfer

  • Multi-harness recipes vary exposure and grouping, while prior comparisons also vary other training choices; this study fixes the trajectory manifest and update budget to isolate the grouping boundary.It separates source-configuration effects from held-out portability.Source: §2 · p. 3

Where credit is assigned

  • Prior credit-assignment work refines reward resolution or adjusts group-relative baselines; this paper instead varies which trajectories compete, keeping one binary outcome per episode.The reward resolution is fixed while only the comparison-group boundary moves.Source: §2 · pp. 3–4

What RL adds over the starting policy

  • Whether outcome-reward RL creates capability or reweights existing behavior remains contested, and the answer depends on how capability is measured.Evaluation under an unseen harness separates model–harness-pair gains from gains attributable to the model itself.Source: §2 · p. 4

3 Separating exposure from credit assignment

The experiment holds the training experience and budget fixed while changing where trajectories are grouped for relative reward comparison. Source and held-out profiles distinguish source-configuration gains from portability.

3 Separating exposure from credit assignment

  • The source profile includes both portable model change and source-configuration adaptation, whereas the held-out profile tests portability.Both profiles are evaluated relative to the common SFT warm start.Source: §3 · p. 4
  • The primary arms share a frozen manifest and differ only in whether GRPO compares trajectories within each task–harness pair or pools harnesses for the task.Within and Cross use the same records, tokens, rewards, loss masks, starting checkpoint, and update budget.Source: §3 · p. 4
  • The primary campaign scores checkpoints on four source harnesses and a held-out weak-ReAct interface, using two, four, or eight attempts per task across the stated contrasts.It also retrains both arms under two further seeds and re-collects half of one Cross arm’s training data on-policy.Source: §3 · p. 4
  • Harness means span 2.14% to 9.27%, while recipe means span only 5.55% to 6.46% across the same sealed-oracle evaluations.Figure 2 shades the solve rate in each recipe-by-harness cell and orders harness columns by their mean.Source: Figure 2 · p. 5
  • Infrastructure failures are quarantined rather than scored as model failures.The study’s appendices provide design details, version pins, and completeness gates.Source: §3 · p. 5

4 The evaluation harness dominates the training recipe

Across the same sealed evaluations, solve rates vary much more by evaluation harness than by training recipe. Harness effects also depend on the model, so interface strength is not a model-independent scalar.

4 The evaluation harness dominates the training recipe

  • Harness choice moves mean solve rate by a factor of 4.3, compared with 1.16 across training recipes over 24,000 sealed evaluations.Harness means range from 2.14% to 9.27%, while recipe means range from 5.55% to 6.46%.Source: §4 · p. 5
  • Harnesses differ in success priors, observations, reachable states, tool vocabularies, and control flow, so their score spread is not a fixed interface property.On the same tasks, Qwen3-8B and Seed-Coder-8B score alike under the minimal harness, while Aider and SWE-agent each cost Qwen3-8B about 4 pp versus its minimal-harness score.Source: §4 · p. 5
  • Under Cross, rewards from harness contexts that are not exchangeable in the usual same-prompt GRPO sense compete for credit.The harness strengths observed for one model cannot be treated as a model-independent scalar to subtract from every model’s score.Source: §4 · p. 5

5 Cross-harness credit adds no detectable portability

Cross-harness credit does not separate from Within on held-out weak-ReAct, while both RL arms have positive training-harness estimates over SFT, with their largest single-harness gains on OpenHands.

5 Cross-harness credit adds no detectable portability

  • At k=8, Cross–Within is +0.25 pp [−0.48, +1.02] on held-out weak-ReAct; Figure 3a shows no recipe separates at k=4.The six recipes span 4.01% to 4.73%, narrower than the smallest difference the contrasts can detect.Source: §5 · p. 5, Figure 3 · p. 6
  • Cross is +0.77 pp above SFT on the four training harnesses, while Within is +0.48 pp above SFT, with both arms’ largest gains on OpenHands.Cross’s 95% interval is [+0.03, +1.52]; its 99% interval corrected for five recipe comparisons is [−0.21, +1.75].Source: §5 · p. 6
  • Doubling attempts from four to eight moves Cross–Within from +0.58 to +0.25 pp, while the smallest detectable difference falls from 1.19–1.42 to 0.88–1.14 pp.Both estimates move toward zero as precision improves.Source: §5 · p. 6
  • Across seeds 17, 29, and 41, Cross–Within is +0.62, +0.05, and −0.20 pp; pooled, it is +0.16 pp [−0.41, +0.72].Each arm’s own seed range—0.45 pp for Within and 0.42 for Cross—exceeds the difference between the arms.Source: §5 · p. 6
  • Re-collecting half the data on-policy leaves Cross +0.13 pp [−0.85, +1.10] from offline Cross on the held-out harness.The on-policy Cross arm is +0.75 pp [−0.10, +1.65] from Within there.Source: §5 · pp. 6–7

6 The pooled advantage carries harness identity, without detectable consequence

Cross’s pooled advantage makes generating-harness identity recoverable, unlike Within’s, but the reported held-out score and within-harness action distributions show no detectable consequence of that difference.

6 The pooled advantage carries harness identity, without detectable consequence

  • Cross’s advantage carries +4.48 pp [+3.22, +5.83] recoverable harness identity above its null, while Within is +0.02 pp above its null with an interval spanning zero.The out-of-fold classifier predicts generating harness from advantage alone; chance is 25% across four harnesses.Source: §6 · p. 7
  • The residualized arm retains +4.54 pp [+3.24, +5.90] recoverable harness identity, the same as Cross, after removing 9% of the offset as assembled.Its baseline used 1,008 graded tasks, mostly failed by every harness, while training used 183 tasks where harnesses disagree.Source: §6 · p. 7
  • A baseline estimated on the training population itself and finer-grained credit methods remain next interventions for this design.The proposed channels include per-step value estimates, hierarchical or hindsight assignment, and process supervision.Source: §6 · p. 7
  • Swapping SFT for Cross changes action distributions within a harness by 0.0003–0.0029 bits of Jensen–Shannon divergence, at least 59 times less than moving a checkpoint between harnesses.Across harnesses, the divergence is 0.17 to 0.35 bits; on the held-out harness, some action categories differ while search, inspection, editing, and testing are unchanged.Source: §6 · p. 7, Figure 4 · p. 8

7 Discussion

The study finds configuration adaptation without portable capability, regardless of the grouping boundary. Multi-harness RL reports should specify how credit crosses harnesses and evaluate under an unseen harness.

7 Discussion

  • Both grouping rules show configuration adaptation without portable capability: gains are largest on a source harness, while held-out scores and within-harness action distributions do not change.Cross-harness credit changes which trajectories carry signed gradient and puts the harness’s solve rate into the advantage.Source: §7 · p. 8
  • The study isolates reward comparison across harnesses while holding exposure fixed, unlike recipes that also change task pools, training duration, supervision stages, or harness evolution.Those are separate interventions from the one isolated here.Source: §7 · p. 8
  • Reports should specify experience-generating harnesses and whether advantage normalization crosses boundaries, then report source-harness and unseen-harness deltas.The first two specify the algorithm; the latter two separate source adaptation from portable capability.Source: §7 · p. 8

8 Conclusion

The paper separates evaluation-harness effects from training-recipe effects in coding-agent solve rates. It concludes that cross-harness credit does not improve held-out performance and recommends reporting the grouping boundary and testing an unseen interface.

8 Conclusion

  • The evaluation harness moves solve rate by a factor of 4.3, while the training recipe moves it by 1.16.The gains from replaying multi-harness experience sit on training harnesses, not the held-out one.Source: §8 · p. 8
  • Holding multi-harness exposure fixed, changing the credit-grouping rule alters the learning signal but not held-out score, even with half the data re-collected on-policy.The policy’s action distribution is set by the harness it runs in, not by the credit rule.Source: §8 · p. 8
  • Multi-harness RL studies should report the grouping boundary and test portability under an unseen interface.Source: §8 · p. 8

9 Limitations

The study is bounded by its model, budget, harness, and task setting, and reports reproducibility procedures alongside its AI-use and broader-impact statements.

9 Limitations

  • The study covers one model family, one budget, four source harnesses, one on-policy re-collection round, and Python-only tasks; most estimates also rest on one checkpoint.Source harnesses were evaluated at two attempts per task, and the design resolves differences of about 1 pp and larger.Source: §9 · p. 9

A AI use statement

  • The authors used Claude and ChatGPT for draft critique, analysis support, editing, code, figures, and literature searches, and say they reviewed analysis code and verified suggested citations.They state that generative AI was not used to fabricate, select, or alter experimental data or verdicts.Source: A AI use statement · p. 14

B Broader impacts and ethics

  • The authors identify benchmark-harness underspecification as a risk for ranking, procurement, deployment, and safety evaluation, and report no harm pathway created by publishing this analysis.They recommend specifying the harness, task set, grader, and rollout count; held-out evaluation requires additional rollout and grading compute.Source: B Broader impacts and ethics · p. 14

C Reproducibility

  • Reported numbers are emitted by artifact scripts and recomputed from graded episode pools; rollout generation is not bit-reproducible because of continuous-batching nondeterminism.Statistics are task-clustered and paired, with k ≥4 wherever rollout level matters.Source: C Reproducibility · p. 14

D The crossed study: design and full tables

The crossed study replays matched training records under different advantage-grouping rules and evaluates with a sealed oracle. Its tables organize solve rates by recipe and harness, then report levels on the held-out interface.

D The crossed study: design and full tables

  • Four RL arms branch from one supervised warm start and replay the same 81,216 records, differing in advantage grouping and group contents.Evaluation uses SWE-bench Verified under a sealed per-instance oracle with four-state verdicts.Source: D The crossed study: design and full… · p. 14
  • 4.85% is the re-collected Cross arm’s held-out k=4 level on the 500 tasks gradeable in it and the offline Cross, Within, and SFT arms.The second training half used its own rollouts after re-running the 183 training tasks through four harnesses.Source: D The crossed study: design and full… · p. 14
  • Table 1 reports SWE-bench Verified solve rates by training recipe and evaluation harness, with row means averaging the four harness columns.Its avg@2 metric is the mean over two attempts per task.Source: Table 1 · p. 15
  • Table 2 reports held-out-interface solve levels, including k=4 and k=8 results and additional-seed levels for Within and Cross.The k=4 levels use 499 common tasks; k=8 levels use 500 tasks in the three re-run arms.Source: Table 2 · p. 15

E Training objectives

The training-objective section isolates reward-group construction while holding the primary arms’ replay data and update budget fixed, then specifies the controls and harness/runtime version pins.

E Training objectives

  • Within forms GRPO groups within each task–harness pair, so harness-level reward offsets cancel by construction.Its advantage is the group-standardized reward for that group.Source: E Training objectives · p. 15
  • PlainCross pools harnesses within each task, placing each harness’s solve rate in the group mean.This is the pooled comparison under study; the passage identifies the group mean’s harness offset as the quantity measured in Section 6.Source: E Training objectives · pp. 15, 17
  • Coverage-matched PlainCross keeps the pooled comparison but zeros the advantage on dropped traces, matching Within’s count of traces with non-zero advantage.The remaining traces retain the pooled comparison.Source: E Training objectives · pp. 15–17
  • Resid subtracts a leave-one-task-out, per-harness reward baseline before pooled standardization, using a unit coefficient and no fitted slope.The baseline averages a harness’s rewards over other graded tasks, excluding the task being scored; the out-of-fold classifier is the primary harness-identity diagnostic.Source: E Training objectives · pp. 16–17
  • The second-model-family comparison reports OpenHands and Qwen Code as not measurable for Seed-Coder-8B because all episodes ended in harness errors.Its context limit is 32,768 tokens versus 40,960 for Qwen3-8B; minimal-harness overflows were scored as zero.Source: Table 5 · p. 17
  • SFT is a matched-hyperparameter objective ablation, not a tuned distillation recipe, because it inherits an RL learning rate far below a tuned recipe.Base is the untrained Qwen3-8B anchor for contrasts.Source: E Training objectives · pp. 16–17

F Harness and runtime version pins

E Training objectives

  • All primary arms replay the same 81,216-record manifest and realize 81,200 optimizer steps, isolating changes to the training signal or its source.The controls are Base, which applies no update, and SFT, which uses successful trajectories under supervised loss.Source: E Training objectives · p. 15, Table 7 · p. 18

G Training corpus and regime

This section describes disjoint training and evaluation task sets, then details where pooled reward variance arises and how harness identity, action composition, prior reports, and statistical uncertainty are assessed.

G Training corpus and regime

  • Training uses SWE-Gym tasks and every reported evaluation number uses SWE-bench Verified, with the benchmark families disjoint by construction.The frozen replay regime is one offline epoch; the re-collected arm collects its second-half data on-policy.Source: G Training corpus and regime · p. 18, Table 8 · p. 18
  • Pooled GRPO has non-zero reward variance only on tasks where harnesses disagree; the 183 such tasks span solve rates from 6.9% to 27.0%.The reported rates are Aider 6.9%, SWE-agent 9.8%, Qwen Code 20.9%, and OpenHands 27.0%.Source: G Training corpus and regime · p. 18

H Detected infrastructure failure modes

  • Fail-closed checks detected and corrected infrastructure failures that otherwise could have silently biased results.Documented failures include cross-condition container contamination, missing episodes, grading crashes, repository corruption, and incomplete image caches.Source: H Detected infrastructure failure modes · pp. 18–19

I Harness identity in the advantage

  • Table 9 tests harness identity in advantages with an out-of-fold classifier against a within-task shuffled-label null, with chance accuracy set at 25%.It also reports the advantage’s regression slope on training-time harness solve rate.Source: Table 9 · p. 19, I Harness identity in the advantage · p. 19

J Action composition under each harness

  • Assistant turns are assigned one primary action label, while Table 10 compares within-harness SFT–Cross action distributions and ranges across harness pairs.Aider is excluded from the parser-based action analysis because its transcript has no tool calls or shell commands to label.Source: J Action composition under each harness · p. 19, Table 10 · p. 20

K Reported multi-harness results and what they identify

  • +9.5 and +20.3 pp are OpenForgeRL’s reported held-out gains on ClawEval, but its SFT and RL harness exposure varies together, so the marginal multi-harness RL contribution is not isolated.Orchard’s reported unseen-harness score is bundled with SFT, while Polar evaluates each checkpoint on its own training harness.Source: K Reported multi-harness results and… · p. 20

L Statistical protocol details

  • Paired per-task differences are bootstrapped by resampling tasks, with two-sided 95% intervals; analysis requires at least 95% graded coverage.The sealed oracle uses hidden tests disjoint from harness feedback tests, and a gate experiment found 2.58% inline-versus-oracle disagreement, all optimistic.Source: L Statistical protocol details · pp. 20–21
  • Rerun flips rule out specifically single-run, small-n comparisons, not every difference below six points; estimated detection floors are 1.19–1.42 pp at k=4 and 0.88–1.14 pp at k=8.The pooled three-seed contrast has an approximately 0.8 pp detection floor, and its null does not establish equivalence; each arm’s held-out level shifted 0.42–0.45 pp across seeds.Source: L Statistical protocol details · p. 20
Loading 2609.04518v1…