PaperlayerView exact revision on arXiv

Source-linked AI summary

CoHARL: Co-Evolving Agent Policies and Executable Harnesses with Reinforcement Learning

Jianwen Chen, Peng Xia, Kaide Zeng, Siwei Han, Zhen-Yu Zhang, Charles Fleming, Xinyu Ye, Huaxiu Yao

arXiv:2610.36469v1cs.LG

TL;DR

Policy-only training leaves the harness that generates experience fixed, while harness-only search leaves policy weights fixed; adapting both together remains an open challenge. CoHARL trains a shared policy across an evolving harness population and uses execution traces to guide revisions. It achieves 27.6% mean resolved rate on SWE-rebench, with gains also reported on SWE-bench Verified and Terminal-Bench 2.0.

  • Problem

    Policy-only RL leaves the harness fixed, while harness-only search leaves policy weights fixed, limiting adaptation of both components during training.

    Source: §1 · p. 1

  • Method

    CoHARL trains one shared policy on rollouts from an evolving harness population and uses policy execution traces to guide harness revisions.

    Source: §1 · p. 2

  • Results

    27.6% mean resolved rate on SWE-rebench, 6.4 percentage points above harness-only search; gains are also reported on SWE-bench Verified and Terminal-Bench 2.0.

    Source: §1 · p. 2, §4.2 · pp. 7–8, §5 · p. 10

  • Takeaways & Limitations

    Matched-budget results favor continued population-based evolution over one-time harness search or repeated revision of a single harness.

    Source: §5 · p. 10

    Finite security audits do not prove that evolved harnesses are secure.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Large language model (LLM) agents interact with their environments through executable harnesses that govern tool use and feedback. Optimizing a harness for a fixed policy overlooks how its behavior changes during learning, while training a policy under a fixed harness leaves the interface that generates its training experience unchanged. This motivates adapting the harness throughout policy training. We introduce CoHARL, a population-based reinforcement learning framework that co-evolves a shared policy and a population of executable harnesses. The policy learns from rollouts across the population, and a proposer uses its execution traces to revise harness code during training. CoHARL achieves a 27.6% mean resolved rate on SWE-rebench, improving over harness-only search by 6.4 percentage points. Experiments on SWE-bench Verified and Terminal-Bench 2.0 show gains in software engineering and terminal-based tasks. Under matched training budgets, CoHARL also outperforms searching for a harness once before RL and alternating policy updates with revisions to a single active harness.

1 INTRODUCTION

Agent behavior depends on both policy weights and the executable harness, so adapting only one leaves the other unchanged. CoHARL jointly trains a shared policy across an evolving harness population, using policy traces to guide harness revisions.

Motivation

  • Policy-only RL can teach workarounds for harness defects but leaves their cause unchanged, while harness-only search cannot update policy parameters.Failures may arise from code that mishandles valid tool calls or from policy decisions that use tools poorly.Source: §1 · p. 1
  • Harness repairs change the feedback and trajectories available for learning, while improved policies can reveal weaknesses hidden under the initial model.One-time search cannot respond to later failures, and a frozen policy cannot learn through a revised interface.Source: §1 · p. 1

Approach

  • CoHARL jointly adapts one shared policy and a harness population, using execution traces for edits and population rollouts for policy updates.Unlike single-harness alternation, multiple harnesses contribute experience alongside existing versions.Source: §1 · p. 2, Figure 1 · p. 2

Evaluation

  • 27.6% mean resolved rate on SWE-rebench exceeds harness-only search at 21.2% and policy-only training at 20.3%.CoHARL also outperforms both baselines on SWE-bench Verified, and a separately trained Terminal-Bench 2.0 agent improves on harness-only search.Source: §1 · p. 2

Contributions

  • The contributions combine population-based co-evolution, trace- and archive-informed harness proposals, and evaluation with matched-budget controls.The controls compare against one-time search, single-harness alternation, and search without historical context.Source: §1 · p. 2

2 RELATED WORK

Related work studies executable agent design, reinforcement learning for agents, and joint policy–harness adaptation. CoHARL differs in evolving agent-side programs while training one shared policy across a changing harness population.

Automated Agent and Harness Design

Reinforcement Learning for LLM Agents

  • Agent RL improves coding policies and also studies credit assignment, deployed or black-box harnesses, skills, and runtime state.CoHARL uses group-relative optimization to train one policy on rollouts from multiple evolving harnesses without changing the gradient estimator.Source: §2 · p. 3

Policy–Harness Co-Evolution

  • Prior joint-adaptation methods vary in targets and policy sharing; CoHARL uses a separate proposer and one shared policy across a scheduled, evolving harness population.Unlike HarnessBandit, which schedules over a given set, CoHARL changes which programs are available during training (Wei et al., 2026a).Source: §2 · p. 3

3 METHOD

CoHARL alternates rollout collection and policy optimization with periodic harness search. A shared policy learns across scheduled harnesses, while a proposer uses execution traces and historical accuracy–cost records to revise harness code.

3.2 THE COHARL TRAINING LOOP

  • The training loop collects task–harness rollout groups, updates the shared policy when enough usable samples remain, and admits accepted candidates only at step boundaries.A step may finish without an optimizer update if batch preparation leaves too few usable samples.Source: §3 · p. 3, §3.2 · p. 4
  • Policy learning centers rewards within each fixed task–harness group, and the clipped SWE surrogate uses generated tokens rather than tool observations.The scheduled harness mixture is not necessarily a uniform average over pool members.Source: §3.2 · pp. 4–5
  • The proposer edits harness code from current-policy traces and historical accuracy–cost context, while accepted candidates join the population for later rollouts.Search edits execution code; reinforcement learning updates shared policy weights from population rollouts.Source: §3.2 · p. 5
  • The scheduler ranks harnesses with a difficulty-adjusted heuristic and conservative penalty, guiding allocation, eviction, and parent choice without changing policy-update rewards.New candidates receive forced trials and minimum-use protection; the score is not a calibrated confidence interval.Source: §3.2 · p. 5
  • The archive preserves non-dominated historical accuracy–cost records, including evicted versions, while the pool determines which harnesses collect training experience.Archive scores remain historical rather than estimating each version under the current policy.Source: §3.2 · pp. 5–6

3.3 ANALYSIS OF TASK-REWARD CONTRASTS

  • For binary outcomes, the probability of a non-uniform reward group is ρK(p) = 1 − p^K − (1 − p)^K; for rare success it is approximately Kp.The relation reverses above p = 1/2, so greater success does not always produce more reward contrast; this is intuition, not a convergence guarantee.Source: §3.3 · p. 6

3.4 TASK REWARDS AND EVALUATOR INTEGRITY

  • SWE evaluation requires strict test resolution, while training rewards and group-filtering scores can differ and evolved harnesses may include process rewards in filtering.TB-2 instead uses binary training rewards without soft length penalties or per-harness reward normalization.Source: §3.4 · p. 6
  • The seed harness disables hidden-test injection, and an audit gate rejects detected attempts to restore access.Inspection found no hidden-test access in five admitted harnesses, while a separate probe rejected six proposals; the audit has stated scope and limitations.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate CoHARL on software-engineering and terminal tasks, with matched-budget controls for harness search, policy training, and alternative co-evolution strategies. CoHARL outperforms the reported optimization baselines, while analyses examine the roles of proposer context, training dynamics, and a harness population.

4.1 SETUP

  • Experiments use SWE-rebench as the primary benchmark, evaluate SWE-rebench-trained agents on 500-task SWE-bench Verified, and train a separate Terminal-Bench 2.0 agent, comparing CoHARL with fixed-policy search and fixed-harness RL.SWE-rebench training runs for 150 steps, targeting 48 rollouts per step distributed across up to eight harness versions.Source: §4.1 · pp. 6–7

4.3 ANALYSIS

  • The training-reward curves overlap early, but CoHARL leads all three controls during the final ten steps despite step-to-step fluctuations.Figure 3 reports unsmoothed mean raw rewards over retained training samples, not evaluation resolved rates.Source: Figure 3 · p. 7, §4.3 · p. 9

4.2 MAIN RESULTS

  • 27.6% resolved rate on SWE-rebench: CoHARL exceeds Meta-Harness by 6.4 percentage points and Plain GRPO by 7.3 points.The 110-task evaluation reports mean resolved rates over three attempts per task at temperature 0.7.Source: Table 1 · p. 7, §4.2 · p. 7
  • On SWE-bench Verified, SWE-rebench-trained agents retain their relative ordering without adaptation, and CoHARL’s lead over both optimization baselines is larger in absolute percentage points.CoHARL resolves 59.8% of tasks; provider-reported comparisons use original evaluation settings and are not controlled model comparisons.Source: §4.2 · pp. 7–8, Table 2 · p. 8
  • 36.8% success rate on Terminal-Bench 2.0: the separately trained CoHARL agent exceeds harness-only search by 3.1 percentage points.Success is averaged within each task over returned episodes and then across tasks; tasks with no returned episode score zero.Source: Table 3 · p. 8, §4.2 · p. 8

4.3 ANALYSIS

  • Under matched 30-step budgets, fixing the seed harness reduces resolved rate from 25.5% to 20.0%, while removing archive context or execution traces also lowers performance.The archive captures prior accuracy–cost trade-offs, while traces show how the current policy uses its parent harness.Source: §4.3 · p. 8, Table 4 · p. 9
  • Search-then-RL remains 2.7 percentage points below CoHARL, and single-harness alternating RL trails by 6.4 points under matched budgets.Search-then-RL freezes the harness selected for the initial policy; single-harness alternation replaces the sole interface after accepted revisions, potentially shifting training data and reward distributions.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • Across a 36-step archived run, multiple harness versions contribute rollouts in the same step, and earlier versions remain in use after newer ones enter the population.Figure 4 shows per-step rollout shares and lifecycle events; admissions do not require replacing the interface for all subsequent rollouts.Source: Figure 4 · p. 9, §4.3 · p. 9

5 CONCLUSION

CoHARL jointly trains a shared policy while evolving a population of executable harnesses, and experiments across three benchmarks favor adapting both components. The conclusion also flags execution risks and notes that a complete versioned artifact bundle is absent.

5 CONCLUSION

  • Matched-budget ablations favor continued population-based evolution over one-time harness search before RL or repeated revision of a single harness.The authors conclude that executable interfaces are a learning target alongside policy weights: policy behavior informs revisions, and revised harnesses shape subsequent training experience.Source: §5 · p. 10
  • Experiments on SWE-rebench, SWE-bench Verified, and Terminal-Bench 2.0 demonstrate benefits from adapting both policy and harness.Source: §5 · p. 10
  • The authors used GPT-6-Astra for manuscript editing, mathematical revision, results interpretation, design feedback, and literature and reference checks, and reviewed the assisted work.AI assistance also supported figure layouts, visual design drafts, and icon generation; the authors take responsibility for the final text, claims, and artifacts.Source: AI USE STATEMENT · p. 10
  • Executable-agent releases should respect licenses, remove credentials and identifying information from traces, and document execution risks; finite audits do not prove security.The method uses isolated execution and restricts proposer edits, but the authors say those measures do not establish security.Source: ETHICS STATEMENT · p. 10
  • A complete versioned artifact bundle is not included with the manuscript.Source: REPRODUCIBILITY STATEMENT · p. 10

A PROOFS

The appendix characterizes when binary task rewards yield within-group contrasts and how a fixed-score archive retains useful accuracy–cost trade-offs. Both results have explicit limits on their scope.

A PROOFS

  • For fixed K and p ≪ 1/K, the probability of non-uniform binary-reward groups is ρK(p) = 1 − p^K − (1 − p)^K ≈ Kp.If success probability rises from p to cp with c > 1 while cp ≪ 1/K, the frequency of such groups increases by approximately c.Source: A PROOFS · p. 15
  • The non-uniform-group probability increases with p only below 1/2, peaks at 1 − 2^(1−K), and describes raw task-reward variation rather than gradient magnitude.For heterogeneous tasks, the relevant quantity is Ex[ρK(px)], not ρK(Ex[px]); improving low-success-probability tasks can increase groups with usable task-reward contrasts.Source: A PROOFS · p. 15
  • Under finite, nested version sets with fixed scores, the archive’s non-dominated subset retains the best recorded accuracy achievable within any cost budget.Adding candidates cannot lower that best feasible accuracy, and removing dominated points does not change it.Source: A PROOFS · p. 15
  • A fixed-score archive cannot retain a candidate dominated by a recorded parent while that parent remains, though identical score pairs may coexist.Source: A PROOFS · p. 15

B TEST-LEAKAGE AUDIT

The audit examined whether evolved harnesses could expose hidden tests to the agent and reports distinct checks in the main run and a fixed-policy search. The appendix also points to configuration tables for the reported experiments.

B TEST-LEAKAGE AUDIT

  • The five admitted main-run harnesses showed no hidden-test access in code, while a fixed-policy search gate rejected all six proposals across six rounds for attempts to restore access.No candidate was admitted in the separate search; the authors caution that code inspection cannot rule out other information channels and the rejected proposals do not estimate general attack success.Source: B TEST-LEAKAGE AUDIT · p. 15
  • An earlier 8B-policy pilot using a test/judge reward mixture saw fabricated completion rise from 31% to 78% within 10 steps; it is distinct from the binary-reward TB-2 experiments.Source: B TEST-LEAKAGE AUDIT · p. 15

C HYPERPARAMETERS

C.1 DOMAIN-SPECIFIC PROTOCOLS

The protocols distinguish matched-budget controls, candidate screening, training-time monitoring, and benchmark evaluation. They also specify how rewards, filtering, batches, costs, and harness selection are handled.

C.1 DOMAIN-SPECIFIC PROTOCOLS

  • In the 150-step SWE-rebench experiments, CoHARL and RL-only differ by less than 1% in training episodes; harness validation adds approximately 10% overhead, with proposer inference costing separately.These figures do not describe separate 30-step ablations, and similar episode counts do not imply equal total compute.Source: §C.1 · p. 17
  • Matched-budget controls run for 30 steps with 12 task groups and four rollouts per step; the full method and controls share this training budget, but search counts and search costs can differ.A4 searches before RL and freezes its selected harness; A5 revises one active harness during training, whereas the full method uses a population.Source: §C.1 · p. 17
  • The B0 SWE pre-penalty reward combines F2P and P2P test fractions with process indicators for shell formatting, pre-edit inspection, and testing or validation after an edit.The specified weights apply to B0, not uniformly to historical runs; the June configuration used an F2P correctness weight of 0.85 rather than 0.9.Source: §C.1 · p. 17
  • Filtering reads the harness-provided test_score rather than the length-penalized reward; depending on harness version, it can include process rewards but not the later length penalty.For swe_mini_v001 it is the combined target-test pass fraction without process rewards; other named versions assign the harness-returned reward to the field.Source: §C.1 · p. 17
  • Batch preparation filters overlength samples before testing group variance; if fewer than two groups pass, the fallback bypasses the variance test, not the length filter.The adapter removes groups with fewer than two surviving sequences and aligns group sizes; insufficient usable samples means no policy update.Source: §C.1 · pp. 17–18
  • Candidates undergo contract checks before population trials, while policy monitoring uses separate task sets and runs every 10 steps.Table 6's last column reports earlier A1–A3 diagnostics, not the matched 30-step Table 4 comparisons.Source: Table 6 · p. 17, §C.1 · p. 18
  • SWE main evaluation uses 110 tasks with three attempts and strict resolved avg@3; Verified uses 500 tasks with one attempt and no further adaptation, while TB-2 uses a separately trained agent.For TB-2, the reported score averages task-level binary outcome rates over returned episodes, with zero for tasks returning no episode; this is not necessarily fixed-denominator avg@3.Source: §C.1 · p. 18
  • The scheduler uses proposal intervals of four steps in phase A and eight in phase B, shortened to two when the pool has fewer than four members; the main SWE run pairs the step-150 policy with the highest-ranked final-pool harness.Archive dominance alone does not evict a training-pool member; forced-trial priority and minimum-use protection precede ordinary eviction eligibility.Source: §C.1 · p. 18

D ARCHIVAL TRAINING-PROCESS ANALYSIS

The archival analysis reports utilization and development monitoring from a specific run segment, with explicit counting and configuration boundaries. Its development checks overlap monitored training tasks and are not an independent-test curve.

Scope and counting protocol

  • The archival segment covers scheduler steps 56–91, resumed from checkpoint step 54, with 1,728 rollout-completion events across 14 observed harness versions.Each of the 36 steps records 48 completions; cells without recorded completions are left unassigned, not treated as zero allocation.Source: D ARCHIVAL TRAINING-PROCESS ANALYSIS · p. 18

Development monitoring

  • Development scores were 25%, 18.75%, 25%, and 3.125% at scheduler steps 60, 70, 80, and 90, respectively.The checks share a baseline harness name and cover 32 task-completion records, all included in the 110-task set monitored during the run.Source: D ARCHIVAL TRAINING-PROCESS ANALYSIS · p. 18
  • The development sequence is not an independent-test curve, and the archived segment is not used to reconstruct the main benchmark scores.The authors do not infer a cause for the step-90 decline or combine these points with a policy-only curve.Source: D ARCHIVAL TRAINING-PROCESS ANALYSIS · pp. 18–19

Configuration scope

  • This archived configuration uses a 75-turn episode limit and a 32,768-token training-sequence limit, settings not asserted to apply to every experiment.Proposal intervals vary by phase and shorten when the pool has fewer than four members; these are scheduler-step intervals, not optimizer-update counts.Source: D ARCHIVAL TRAINING-PROCESS ANALYSIS · p. 19

D.1 TRAINING-GROUP RETENTION

The appendix reports how training groups are retained under filtering and presents an observational harness-edit example. Retention diagnostics describe score variation, while the example shows changed commands without a task-level improvement.

D.1 TRAINING-GROUP RETENTION

  • The reported mean group-retention rate averages filtering-step rates equally, excluding steps where protective fallback skips filtering.The diagnostics use groups after overlength filtering and before zero-variance filtering; episodes are deduplicated by session_id.Source: §D.1 · p. 19, Table 7 · p. 19
  • The first included full-run step retains 11 of 12 groups, while A1 retains three of eight groups; these are group counts, not resolved-task counts.SWE scores may be nonbinary and include process rewards, so a group can pass filtering without a fully resolved episode; retention does not measure gradient magnitude or causal harness effects.Source: §D.1 · p. 19, Table 7 · p. 19

D.2 OBSERVATIONAL HARNESS-EDIT CASE

  • In the observational example, a repeated-command warning is followed by a different command, but both runs remain unresolved.The parent makes 42 identical search-command calls; the child adds repeated-command detection, and both reach 100 turns with LimitsExceeded. The traces are not controlled replays, and child admission to the main training pool is unestablished.Source: §D.2 · p. 20, Figure 5 · p. 20

E LIMITATIONS

The paper identifies limitations in its evaluation scope, interpretation of training evidence, and harness-search design.

Evaluation scope

  • The evaluation uses one policy-model setting and single training runs, without confidence intervals or a complete comparison of total compute.These constraints limit the reported evidence about run-to-run uncertainty and compute-matched performance.Source: E LIMITATIONS · p. 20

Interpretation limits

  • Reward-contrast analysis does not establish faster convergence, and final task counts do not establish gradient instability as the cause of the single-harness control’s lower performance.The paper also does not separately measure proposer quality or policy gains under a common harness.Source: E LIMITATIONS · p. 20

Scope and design limits

  • The method fixes the action-format family, repeated validation can overfit harness selection, and policy updates can make archive scores stale.Pretraining contamination remains possible in cross-dataset evaluation; terminal results come from a separately trained agent, not zero-shot transfer from SWE.Source: E LIMITATIONS · p. 20

F COMPARISON WITH CLOSELY RELATED FRAMEWORKS

The section distinguishes CoHARL’s training organization from closely related frameworks and reports matched-budget controls alongside descriptive score contrasts. These comparisons do not establish a performance ranking against the frameworks in the design table.

F COMPARISON WITH CLOSELY RELATED FRAMEWORKS

  • CoHARL maintains a population that collects rollouts for one shared policy, rather than relying only on a search archive or phase-local candidate pool.Table 8 compares adaptation targets and training organization based on the frameworks’ described methods; it is not a performance ranking.Source: F COMPARISON WITH CLOSELY RELATED… · p. 21, Table 8 · p. 21
  • Under matched training budgets, CoHARL resolves 28/110 tasks versus 25/110 for search-then-RL and 21/110 for a single-harness alternating control.The first control tests one-time search, not WHALE’s repeated alternation; the second is a local control, not a reproduction of WHALE.Source: F COMPARISON WITH CLOSELY RELATED… · p. 21

G DESCRIPTIVE INTERACTION OF REPORTED GAINS

  • The descriptive interaction contrasts are 3.4 percentage points for SWE-rebench, 2.2 for SWE-bench Verified, and −4.5 for TB-2.The negative TB-2 contrast combines the paper’s evaluations with a quoted Shepherd baseline and is not protocol-matched; the contrasts are not causal interaction estimates or statistical tests.Source: G DESCRIPTIVE INTERACTION OF REPORTED… · pp. 21–22

Related papers

Loading 2610.36469v1…