PaperlayerView exact revision on arXiv

Source-linked AI summary

Training the Whole Agent, Not Just the Model

Lingxiao Tang, Wei Tao, Yiwen Guo, Lingfeng Bao

arXiv:2610.28069v1cs.SE

TL;DR

Training coding agents typically keeps the harness fixed, despite trajectories revealing repair and interaction problems that shift with tasks and policy learning. HarnessAdapt-RL adapts skills, tools, and hooks during reinforcement learning; it improves pass@1 over GRPO across three benchmarks, with deployment retaining evolved tools and hooks but removing skills.

  • Problem

    Coding-agent training typically updates the policy with a fixed harness, although training trajectories expose repair and interaction problems that vary across tasks and change as the policy learns.

    Source: §1 · p. 1

  • Method

    HarnessAdapt-RL generates task-specific mechanism-level skills and evolves tools and hooks from current-policy trajectories during reinforcement learning, training the policy with the adapted harness.

    Source: §1 · pp. 1–2

  • Results

    HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro, respectively, and also outperforms OpenHands-GRPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Mechanism-level skill gains persist after skills are removed, while retaining evolved tools and hooks preserves full performance at deployment.

    Source: §1 · p. 2

    Tool and hook selection relies on finite-rollout observed rates rather than a significance test, so rollout noise and repeated search can affect retained changes.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Training coding agents typically updates the policy while keeping the harness fixed. During training, trajectories reveal flawed repair strategies and interaction problems that vary across tasks and change as the policy learns. We introduce HarnessAdapt-RL, which trains the policy and adapts its harness during reinforcement learning. For flawed repair strategies, a frozen teacher generates task-specific mechanism-level skills. For interaction problems recurring across tasks, a separate teacher model evolves tools and hooks, keeping each change only if it improves the observed pass rate or the observed trainable rate without lowering either. The policy trains with the adapted harness, and its new trajectories guide further adaptation. With a 9B base model, HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro, and outperforms GRPO with the human-designed OpenHands harness. Gains from mechanism-level skills persist after their removal at evaluation. Re-evolving tools and hooks as the policy changes outperforms freezing early versions. Removing them at deployment, in contrast, costs up to 7.7 percentage points. Training therefore produces a model-harness pair: the trained policy and the evolved tools and hooks, with skills removed at deployment.

1 Introduction

Training coding agents typically leaves the harness fixed, even though trajectories expose repair and interaction problems that vary across tasks and change as the policy learns. HarnessAdapt-RL adapts the harness during reinforcement learning and reports higher pass@1 than GRPO across three benchmarks.

1 Introduction

  • Training trajectories expose flawed repair strategies and interaction problems that vary across tasks and change as the policy learns, motivating continued harness adaptation.The paper asks how to design skills, evolve tools and hooks, and choose which components to retain at deployment.Source: §1 · p. 1
  • HarnessAdapt-RL generates task-specific mechanism-level skills and evolves tools and hooks from current-policy trajectories, retaining proposals only when observed pass or trainable rate improves without lowering either.The policy then trains with the adapted harness, and its new trajectories guide later adaptation.Source: §1 · pp. 1–2
  • 9.8, 12.6, and 6.4 percentage points: HarnessAdapt-RL improves pass@1 over GRPO on SWE-bench Verified, Multilingual, and Pro, respectively.It also outperforms GRPO trained with the human-designed OpenHands harness on all three benchmarks.Source: §1 · p. 2
  • Mechanism-level skills perform best among three guidance variants after skills are removed, while gains from those skills persist and evolved tools and hooks remain necessary for full deployment performance.Removing evolved tools and hooks at evaluation reduces pass@1 by up to 7.7 percentage points.Source: §1 · p. 2

2 Problem Formulation

The paper represents a coding agent as a policy paired with a harness that guides actions and supplies tools and runtime feedback. Training adapts this pair, while deployment removes skill guidance and retains the final runtime configuration.

2 Problem Formulation

  • An agent combines a policy that selects actions from interaction history with a harness that executes them and returns observations.The harness is represented as a runtime configuration plus optional skill guidance; its runtime configuration comprises a base configuration, added tools, and hooks.Source: §2 · p. 2
  • Training begins from pretrained policy weights and the base runtime, uses teacher-generated skills and proposed tool or hook changes, and updates the policy from fresh supported rollouts.At deployment, skills are removed and the trained policy is paired with the final runtime configuration.Source: §2 · p. 2
  • The task reward indicates whether the submitted patch passes the task tests, while skill generation and tool and hook selection use training-side evidence.Source: §2 · p. 2

3 Method: Training the Whole Agent

HarnessAdapt-RL adapts skills, tools, and hooks from current-policy trajectories while training the policy with the selected harness. Changes are selected for observed task success and rollout trainability, then used for subsequent policy updates.

3 Method: Training the Whole Agent

  • Separate teachers use current-policy trajectories to generate task-specific repair skills and propose tools and hooks for recurring interaction problems; accepted changes guide subsequent GRPO training and later adaptation.The overview retains a proposal only when it improves observed pass rate or trainable rate without lowering either.Source: §3 · pp. 2–3, Figure 1 · p. 3

3.1 Policy-Adaptive Repair Skills

  • Mechanism-level skills explain when and why a repair strategy applies while leaving code localization and implementation to the policy.Unlike general skill-bank guidance, which does not consistently improve over GRPO, near-answer hints raise training success but lead to worse evaluation than GRPO when removed.Source: §3.1 · p. 3

3.2 Evolving Tools and Hooks for Policy Training

  • A teacher identifies recurring interaction problems across tasks, inspects tool calls and feedback, then proposes tools or hooks to address them as the policy learns.Tools package reusable repair operations, while hooks provide execution-time feedback; updated-policy trajectories guide further adaptation.Source: §3.2 · p. 4

3.3 Training-Aware Harness Selection

  • A harness proposal is accepted only if observed pass rate or trainable rate improves without lowering either, after comparison using the same policy, tasks, skills, and rollout budget.The policy then trains on fresh rollouts, and tools and hooks can evolve again in later epochs using trajectories from the updated policy.Source: §3.3 · p. 4

4 Experiments

HarnessAdapt-RL achieves the highest average pass@1 among the reported configurations, outperforming GRPO and OpenHands-GRPO. Experiments also examine skill guidance, evolving tools and hooks, and retaining the evolved harness at deployment.

4.1 Experimental Setup

  • Training uses a fixed R2E-Gym-Lite subset disjoint from evaluation benchmarks; evaluation reports mean pass@1 across three runs without skills or further harness updates.The base model is Qwen3.5-9B, the base harness is mini-swe-agent, and GRPO receives twice as many policy-training rollouts as other trained configurations.Source: §4.1 · pp. 4–5

4.2 Main Results

  • With a 9B base model, HarnessAdapt-RL exceeds GRPO by 9.8, 12.6, and 6.4 percentage points and OpenHands-GRPO by 6.2, 4.0, and 7.4 on Verified, Multilingual, and Pro.It achieves 44.9% average pass@1; evaluation uses no skills or teacher calls.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Why Mechanism-Level Skills?

  • Mechanism-level skills yield the best performance after removal, while near-answer hints raise training success but underperform GRPO without guidance.Mixed-success-and-failure groups occur in 46.5% of samples with mechanism-level skills versus 39.0% with near-answer hints; concrete-answer rollouts are 40.8% shorter than GRPO, compared with 14.5% for mechanism-level skills, but omit independent-repair steps.Source: §4.3 · pp. 6–7

4.4 How Do Tools and Hooks Evolve During Training?

  • Re-evolving tools and hooks during training improves results, with proposals addressing context management, patching, testing, and revision.A first proposal raises pass rate by 13.7% and trainable rate by 7.4%; later testing changes raise them by 6.3% and 7.4%. A review hook that raised pass rate by 2.7% but lowered trainable rate by 7.0% was rejected, while a later proposal combining review with view_diff was accepted.Source: §4.4 · p. 7

4.5 Why Train and Deploy with the Evolved Harness?

  • Keeping H2 at deployment preserves performance: replacing it with H0 lowers HarnessAdapt-RL pass@1 by 4.8, 7.7, and 4.7 points on Verified, Multilingual, and Pro.Switching H0 to H2 at evaluation raises HarnessAdapt-RL’s Verified score by 4.8 points versus 2.0 for Skill-Mechanism; skills are removed in both evaluations.Source: §4.5 · pp. 7–8, Table 2 · p. 8

4.6 Can the Evolved Harness Be Reused for Training?

  • Frozen-H2+Mechanism scores 56.8%, 38.3%, and 38.9% on Verified, Multilingual, and Pro, within 1.0 point of HarnessAdapt-RL and ahead of Pre-Evolved.Its reuse advantage depends on first discovering H2 through evolution during policy training.Source: §4.6 · p. 8, Figure 4 · p. 8

5 Related Work

Related work covers skill-guided policy learning, harness optimization with fixed model weights, and approaches that adapt policy training alongside harnesses.

Skills and policy learning

  • Skill-based approaches include reusable skills, evolving strategy playbooks, trajectory-derived tasks, and hints or guidance removed at deployment.The paper compares general skill-bank skills, mechanism-level skills, and near-answer hints by trained-model performance after skill removal.Source: §5 · p. 9

Harness design and optimization

  • Harness optimization methods use trajectories, preferences, revision history, or cross-task interactions, while keeping model weights fixed during optimization.HarnessAdapt-RL instead evolves tools and hooks during policy training using trajectories from the updated policy.Source: §5 · p. 9

Policy training and harness adaptation

  • Other work adapts training data, tool-graph memory, complex harnesses, or harnesses alternating with supervised fine-tuning; HarnessAdapt-RL uses GRPO and pass/trainable-rate selection.A proposal is accepted only if at least one observed rate improves and neither decreases; the paper also tests component needs at deployment.Source: §5 · p. 9

6 Conclusion

HarnessAdapt-RL trains coding agents while adapting their harnesses from training trajectories. Its results support deploying the trained policy with evolved tools and hooks, but without task-specific skills.

6 Conclusion

  • HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro.It also outperforms OpenHands-GRPO.Source: §6 · p. 9
  • Mechanism-level skills outperform near-answer hints and general skill-bank procedures after skills are removed at evaluation.The gains from skills persist without deployment-time skills.Source: §6 · p. 9
  • Re-evolving tools and hooks as the policy changes performs better than searching only before training or keeping the first-epoch harness fixed.The evolved harness can also support a new training run without further evolution.Source: §6 · p. 9
  • Removing evolved tools and hooks at deployment lowers pass@1 by up to 7.7 percentage points, so the final model–harness pair retains them.Task-specific skills are removed at deployment.Source: §6 · p. 9

Limitations

The evaluation and adaptation procedure has scope and reproducibility boundaries, including task-domain limits, one base model, single training runs, and noisy candidate selection.

Limitations

  • All tasks are SWE-bench-style software repair, and the skills, tools, and hooks were derived for that setting; usefulness in other agent domains is untested.The paper names web navigation and data analysis as untested domains.Source: Limitations · pp. 9–10
  • The study uses a single 9B base model, leaving uncertain whether larger models or other model families benefit similarly from harness evolution or the same guidance.The authors note larger models may already handle operations packaged into their tools.Source: Limitations · p. 10
  • Each configuration is trained once, so confidence intervals reflect task-level variation for fixed checkpoints rather than variation across training seeds or harness-search runs.Source: Limitations · p. 10
  • Harness candidates are selected using finite-rollout observed rates rather than significance tests, so rollout noise and repeated search can affect retained changes.Final-evaluation statistical analysis does not remove this search uncertainty.Source: Limitations · p. 10

A.1 Task Pool and Assisted Subset

The appendices detail the task pool and skill-generation process, execution limits, training schedule, harness-candidate selection, and rollout accounting.

A.1 Task Pool and Assisted Subset

  • The training pool excludes tasks with 100% pass@4, samples 1,058 remaining tasks, and fixes a uniformly sampled 512-task assisted subset across configurations.The pool is deduplicated against the three evaluation benchmarks, and task order is held constant.Source: §A.1 · p. 14

A.2 Skill Generation

  • Skill teachers see issue and trajectory records, not repositories or hidden tests, and compare two of four preparation rollouts, preferably one success and one failure.When all preparation rollouts succeed or fail, the teacher compares two from that outcome group.Source: §A.2 · p. 14
  • Mechanism skills explain a repair mechanism without repository identifiers or code; near-answer hints provide concrete edits, while skill-bank guidance summarizes recurring procedures.Mechanism skills that fail the leakage screen are regenerated up to three times; if they still fail, the task is trained without a skill.Source: §A.2 · pp. 14–15
  • Second-epoch skills are regenerated, and Skill-Gated requests guidance only when all four preparation attempts fail.For Skill-Gated, the teacher generates a skill from two randomly selected failed trajectories.Source: §A.2 · p. 15

A.3 Execution Limits and Submission

  • Training allows 64K context and 60 turns, versus 128K and 150 turns at evaluation; over-limit trajectories truncate, and submitted patches are tested in fresh containers.Submitting is a separate Bash action; prepare_patch writes changes to patch.txt but does not submit or score them.Source: §A.3 · p. 15

B.1 Hyperparameters and Schedule

  • Configurations train for two epochs over 1,058 tasks, with 23 updates per epoch; GRPO samples eight rollouts per task and other configurations sample four.Configurations share the task pool, order, optimizer, learning rate, and budgets, differing in rollout width, guidance, and harness.Source: §B.1 · pp. 15–16

B.2 Candidate Generation

  • The teacher proposes evidence-backed tool or hook changes, which are compared using fresh rollouts under the same policy; a proposal is retained only if neither observed rate falls and one rises.The rates are pass rate and trainable rate; candidates also undergo format, schema, loading, and smoke-test checks before comparison.Source: §B.2 · pp. 16–17

B.3 Rollout Accounting

  • Candidate comparisons use 184 fresh rollouts per harness on the same 46 tasks, while unknown-reward groups are excluded from policy updates rather than assigned zero.Comparison rollouts are discarded after the decision; rounds with no proposed change need no comparison.Source: §B.3 · p. 17

B.4 Training Runs and Evaluation

The evaluation protocol averages three runs per configuration, while training statistics describe sampled groups and rollouts across 46 updates. Across skill variants, reward-group outcomes differ, and shorter trajectories do not by themselves indicate better learning.

B.4 Training Runs and Evaluation

  • Each configuration was trained once, and the update-45 checkpoint was evaluated three times with one attempt per task per run.Failed or unfinished attempts count as incorrect; training statistics cover all 46 updates and describe sampled groups and rollouts, not unique tasks or independent training replicates.Source: §B.4 · p. 18

C.1 Reward Groups

  • Skill-Mechanism has the largest mixed-outcome share (46.5%), while Skill-Hint has the largest all-pass share (27.5%).The counts are sampled before training filters; mixed-group share alone does not measure how much useful training signal remains.Source: §C.1 · p. 18, Table 7 · p. 18, Figure 5 · p. 18

C.2 Trajectory Length

  • Skill-Mechanism shortens trajectories by 14.5% overall, 15.3% for successful rollouts, and 10.5% for failed ones relative to GRPO.Skill-Hint shortens trajectories by 40.8% overall, 39.6% for successful rollouts, and 36.7% for failed ones, yet performs worse than GRPO without skills.Source: §C.2 · p. 19, Table 8 · p. 19, Figure 6 · p. 19

C.3 Verification Over Training

Test executions per trajectory decline over training, including when the harness is retained or evolved. Paired task-resampling intervals support pooled gains from re-evolution, while describing variation across tasks rather than retraining.

C.3 Verification Over Training

  • Test executions per trajectory fall by 45% for GRPO and 44% for Skill-Mechanism under H0, and by 57% for H1+Mechanism retaining H1.These descriptive statistics come from one training run per configuration and do not establish the cause of the decline.Source: §C.3 · p. 20, Table 9 · p. 20

C.4 Paired Confidence Intervals

  • Confidence intervals resample tasks using the same sampled tasks for both configurations, and do not estimate variation from retraining models.The pooled results cover 1,442 tasks, weighting each benchmark by its number of tasks.Source: §C.4 · p. 20
  • HarnessAdapt-RL exceeds H1+Mechanism by 2.6 percentage points pooled, with a 95% confidence interval of [+0.9, +4.3].Its individual intervals exclude zero on Multilingual but include zero on Verified and Pro; pooled gains over GRPO, Skill-Mechanism, and OpenHands-GRPO also exclude zero.Source: §C.4 · p. 20, Table 10 · p. 21
  • Frozen-H2+Mechanism exceeds H1+Mechanism by 2.2 percentage points pooled, with a 95% confidence interval of [+0.4, +3.9].Frozen-H1+Mechanism versus H1+Mechanism has a pooled difference of 0.6 percentage points, with an interval of [−1.1, +2.3].Source: §C.4 · p. 20, Table 10 · p. 21

C.5 Guidance Coverage and Learning Curves

The guidance comparison contrasts when skills are provided and which trajectories generate them. Skill-Gated finishes below Skill-Mechanism on Verified, but the comparison cannot isolate the contribution of guidance coverage from generation evidence.

C.5 Guidance Coverage and Learning Curves

  • Skill-Gated provides skills only after all prior attempts fail, whereas Skill-Mechanism also guides tasks with successful attempts.The two variants share the mechanism-level prompt, base harness, schedule, and evaluation protocol.Source: §C.5 · p. 22
  • Skill-Gated ends at 47.2% on Verified, close to GRPO’s 47.8% and below Skill-Mechanism’s 52.2%.Figure 7’s training pass rates use rollouts collected after guidance assignment, not the preparation attempts used for gating.Source: §C.5 · p. 22, Figure 7 · p. 22
  • The comparison cannot separate guidance frequency from generation evidence: Skill-Gated uses fewer tasks and generates each skill from two failures.Skill-Mechanism instead prefers a success–failure pair; isolating the factors would require failure-only generation for all assisted tasks.Source: §C.5 · p. 22

C.6 Fixing the First-epoch Harness

A first-epoch harness can be reused from the start of another run, with or without mechanism-level skills. The comparison tests reuse of an existing harness, not whether training would discover it before policy learning.

C.6 Fixing the First-epoch Harness

  • Frozen-H1 exceeds GRPO by 5.4, 6.6, and 2.4 percentage points on Verified, Multilingual, and Pro, respectively.Frozen-H1 reuses H1 from HarnessAdapt-RL’s first epoch and trains without skills, with H1 fixed throughout both epochs.Source: §C.6 · p. 23, Table 11 · p. 23
  • Frozen-H1+Mechanism reaches 55.4%, 33.7%, and 38.6% on Verified, Multilingual, and Pro, almost the same as H1+Mechanism.Figure 8 compares Frozen-H1, Frozen-H1+Mechanism, and H1+Mechanism.Source: §C.6 · p. 23, Figure 8 · p. 23
  • Adding mechanism-level skills raises Frozen-H1 by 2.2, 2.4, and 2.9 percentage points on Verified, Multilingual, and Pro, respectively.This comparison concerns reuse of an existing harness, not whether the same harness would be discovered before policy training.Source: §C.6 · p. 23

C.7 Training and Deployment Costs

The appendix reports rollout and deployment costs, records how harness candidates were evaluated and retained, and documents the teacher prompts and guidance variants.

C.7 Training and Deployment Costs

  • HarnessAdapt-RL uses 18,056 rollouts overall, including 8,464 training rollouts, versus GRPO's 16,928 training rollouts.Rollout totals do not directly measure compute because trajectory lengths differ; teacher token usage is not measured by invocation counts. At deployment, only retained tools and hooks add cost.Source: §C.7 · pp. 24–25, Table 13 · p. 24

C.8 Harness Evolution Record

  • Each harness comparison uses the same policy checkpoint, task batch, skills, and rollout budget for both harnesses; comparison rollouts are not reused for training.Selection compares the candidate with the current harness in that round, accepting it only when neither rate falls and at least one rises.Source: §C.8 · p. 25
  • Six of nine candidates were accepted; first-epoch changes raised pass rates 5.6%–13.7% and trainable rates 7.4%–7.8%, while second-epoch gains were 3.5%–6.3% and 3.2%–7.4%.The retained H2 has eight tools and five hooks; the comparisons evaluate complete bundles, not individual component effects.Source: §C.8 · p. 25, Table 15 · p. 26, Table 14 · p. 26

D Prompts and Teacher Outputs

  • The appendix reproduces the teacher prompts and representative outputs, with runtime-filled placeholders and long instructions split across figures.Figures 9–15 show instructions and input templates; Figure 16 gives a representative output for each guidance type.Source: D Prompts and Teacher Outputs · p. 28, §D.1 · p. 28, §D.2 · p. 28

D.1 Skill Generation Prompts

  • Near-answer hints and mechanism-level skills use evidence from one task, while the skill bank combines evidence across tasks..Source: §D.1 · p. 28

D.2 Representative Teacher Outputs

  • Near-answer hints specify code changes, mechanism-level skills explain repair mechanisms without code identifiers, and skill-bank guidance describes cross-task procedures.Only the Task-specific Skill section of mechanism-level guidance is injected into policy context.Source: §D.2 · p. 28

D.3 Harness Proposal and Implementation Prompt

  • The harness-proposal teacher diagnoses problems, designs a candidate, and implements it in one call.The complete instruction is reproduced across Figures 17–21, with the input template in the final part.Source: §D.3 · p. 28, Figure 17 · p. 36, Figure 18 · p. 37
Loading 2610.28069v1…