PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolving Coding Agents via Harness-Policy Optimization

Zhepei Wei, Wenlin Yao, Baolin Peng, Danqing Wang, Qianhui Wu, Hao Cheng, Yu Meng, Jianfeng Gao

arXiv:2610.22791v1cs.CLcs.AI

TL;DR

Coding agents depend on both their policy and harness, yet optimizing one at a time limits improvement and joint approaches may rely on external proposers. Self-HPO alternates failure-targeted harness search with RL policy updates, using the evolving policy as proposer. Across three coding benchmarks, it outperforms one-sided baselines, raising average resolve rate from 57.8% to 64.8%.

  • Problem

    Harness-only optimization cannot improve the underlying model, policy-only learning is limited to trajectories elicited by a fixed harness, and joint optimization has relied on external proposers despite evolving policy failures.

    Source: §1 · pp. 1–2

  • Method

    Self-HPO alternates harness revisions proposed by the current policy from failure evidence with RL under the selected harness, using hard training tasks to target the next search.

    Source: §1 · p. 2

  • Results

    57.8% to 64.8%: Self-HPO improves average resolve rate over policy-only and harness-only baselines across three coding benchmarks, with gains generalizing out of distribution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The optimized harness and policy each retain gains when paired with their base counterpart, while their joint gain exceeds the sum of their individual gains.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large language model (LLM) agents have shown promising potential in coding tasks, and their performance is jointly determined by the policy model and the harness that mediates their interaction with the environment. Indeed, harness and policy are mutually reinforcing and evolving only one component is suboptimal---a harness tuned for one checkpoint may not target the failure modes of an updated policy, while policy learning remains limited to the trajectories elicited by a fixed harness. In light of this, we propose Self-HPO, a recursive self-improving framework for coding agents that alternates harness and policy optimization, using the evolving policy as its own harness proposer. Specifically, in each round, the current policy first analyzes its own failures and proposes new harness candidates (H-step). The policy is then updated through reinforcement learning (RL) under the selected harness (P-step), and hard tasks identified during training form the next harness search set, keeping harness optimization targeted to the policy's evolving failure modes. Across three coding benchmarks, Self-HPO consistently outperforms harness-only and policy-only baselines, improving Qwen3.5-35B-A3B's average resolve rate from 57.8% to 64.8%, with gains generalizing to out-of-distribution settings. Further analysis shows that both optimized components retain gains even when paired with their base counterparts, while their joint gain exceeds the sum of their individual gains. Self-HPO also improves reliability (Pass@1) and expands the capability frontier (Pass@k) over the base agent. Additionally, we show that with a stronger external model (e.g., Opus 4.8) as the harness proposer, the same framework yields a further performance boost.

1 Introduction

Coding-agent performance depends on both the policy and its harness, but optimizing only one limits what can improve. Self-HPO alternates harness and policy optimization, using the evolving policy to target its own failures, and outperforms one-sided baselines across three coding benchmarks.

1 Introduction

  • Harness-only optimization cannot improve the underlying model, while policy-only training is limited to trajectories elicited by its fixed harness; self-driven joint optimization remains underexplored.Prior approaches that optimize both components rely on an external harness proposer and face a moving target as policies and failure modes evolve.Source: §1 · pp. 1–2
  • Self-HPO alternates harness revisions proposed by the current policy with RL under the selected harness, then uses hard tasks to target the next search.The updated policy becomes the next harness proposer, aligning search with its evolving failure modes.Source: §1 · p. 2
  • 57.8% to 64.8%: Self-HPO improves average resolve rate over policy-only and harness-only baselines across three coding benchmarks, with gains generalizing out of distribution.The right panel of Figure 1 tracks average resolve rate across the three benchmarks along the H- and P-steps.Source: §1 · p. 2, Figure 1 · p. 2
  • Both optimized components retain gains when paired with their base counterparts, while their joint gain exceeds the sum of their individual gains.Self-HPO also improves Pass@1 reliability and expands the Pass@k capability frontier over the base agent.Source: §1 · p. 2

2 Self-HPO: Self-Evolving via Harness-Policy Optimization

Self-HPO alternates harness search with policy learning under the selected harness. Its evolving policy proposes targeted harness changes using failure evidence, and training failures refresh the search set for the next round.

2.1 Problem Setup

  • Each round holds the policy fixed during harness search, then freezes the selected harness while RL updates the policy; hard training tasks seed the next search set.The framework is instantiated as Self-HPO, where the evolving policy proposes candidates, or with Opus 4.8 as a fixed external proposer.Source: §2.1 · p. 3

2.2 H-step: Failure-Conditioned Harness Search

  • The frozen current policy acts as both task solver and harness proposer, inspecting failures to revise prompts or executable runtime logic and evaluating candidates on the same search set.That set evolves between rounds: after initialization, it contains tasks never resolved in any rollout of the preceding policy-learning step, where fresh failure evidence is gathered.Source: §2.2 · p. 3

2.3 P-step: Policy Optimization under the Optimized Harness

  • Under the optimized harness, the policy learns through RL with verifier-based binary rewards; tasks never resolved during that step become the next hard-task set.The default optimizer combines GRPO with BAR, which draws additional trajectories to form reward-balanced groups and reduce uninformative zero-variance groups under sparse outcome rewards.Source: §2.3 · p. 4

3 Experiments

Experiments compare alternating harness–policy optimization with single-component baselines across three coding benchmarks, then examine ablations, component interactions, reliability, training signals, and an evolved harness. Self-HPO outperforms the single-component baselines, with gains also reported on out-of-distribution benchmarks.

3.2 Main Results

  • 64.8% average resolve rate for Self-HPO versus 57.8% for Base, with Self-HPO outperforming policy-only and harness-only methods across three benchmarks.On SWE-Verified, Self-HPO reaches 69.5%, beating RLVR by 2.0 points and Self-Harness by 4.3 points.Source: §3.2 · p. 5, Table 1 · p. 5
  • Self-HPO reaches 59.7% on SWE-Multi and 65.2% on SWE-Pro, exceeding Base by 5.5 and 6.9 points, respectively.The gains transfer beyond the Python training domain to benchmarks covering nine non-Python languages and broader engineering tasks.Source: §3.2 · p. 5
  • Meta-Harness beats SFT by 4.1 points on SWE-Verified and 8.9 points on average, while HPO further improves on Meta-Harness.SFT falls below Base on both out-of-domain benchmarks, while Meta-Harness improves on both.Source: §3.2 · p. 5
  • Across all three benchmarks, Self-HPO gains after its second harness step and following policy step, while another Self-Harness update drops SWE-Multi from 55.0% to 54.3%.The final Self-HPO pair surpasses both single-component baselines on every benchmark.Source: §3.2 · p. 5, Figure 3 · p. 6

3.3 Ablations

  • Replacing RL with SFT lowers resolve rate from 71.6% to 67.2%, and RFT reaches 70.5%; reusing the initial search set lowers performance to 70.1%.Self-proposal reaches 69.5%, a 2.1-point decrease from HPO, but still exceeds both self-improving single-component baselines.Source: §3.3 · p. 6

3.4 Analysis

  • H2 with the base policy improves resolve rate from 61.0% to 64.5%, P2 with the seed harness reaches 64.4%, and their combination reaches 69.5%.The combined 8.5-point gain over Base exceeds the sum of the individual gains, 3.5 + 3.4 points, by 1.6 points.Source: §3.4 · p. 6, Table 3 · p. 6
  • Self-HPO raises pass@8 from 80.0% to 84.8% and improves pass@1 on Hard, Medium, and Easy tasks, but decreases it by 2.2 points on Resolved tasks.Zero-success rollout groups fall to 6.5% over the final ten Self-HPO P-step updates, versus mostly 11%–17% over RLVR’s last 50 updates.Source: §3.4 · p. 7, Figure 4 · p. 7

3.5 Case Studies

  • The evolved harness preserves the seed agent loop and adds four checks for impending budget exhaustion, empty patches, interface validation, and failed patch application.The checks combine runtime safeguards with behavioral prompts to address ways useful repository edits may fail to become valid submissions.Source: §3.5 · p. 7

4 Related Work

Related work covers harness optimization, policy learning, and joint adaptation of agent components. Self-HPO is distinguished by updating the harness proposer policy and by refreshing the harness search set with tasks that remain unsolved during RL.

Joint harness–policy evolution

  • Prior harness-optimization work adapts prompts, programs, contexts, or workflows, while policy-learning methods train under a fixed harness.Self-HPO combines harness changes with updates to the policy that the harness serves.Source: §4 · p. 8
  • Self-HPO uses the policy trained in each P-step as the next harness proposer, unlike WHALE’s frozen external proposer and Evo2’s separately trained proposer.HASE also uses a self-proposing policy but puts harness edits in the RL action space rather than alternating harness search with RL.Source: §4 · p. 8
  • Self-HPO forms the next harness search set from tasks still unsolved during RL, whereas WHALE, Evo2, and HASE use a fixed task set.Source: §4 · p. 9

5 Conclusion

Self-HPO alternates harness and policy optimization, with the evolving policy proposing harness revisions and hard tasks from training guiding the next search. Across three coding benchmarks, it outperforms policy-only and harness-only baselines and improves reliability and capability over the base agent.

5 Conclusion

  • Across three coding benchmarks, Self-HPO raises average resolve rate from 57.8% to 64.8% and outperforms policy-only and harness-only baselines.The analyses also find complementary harness and policy gains, improved Pass@1 reliability, and an expanded Pass@k capability frontier over the base agent.Source: §5 · p. 9
  • Self-HPO alternates harness and policy optimization, using the evolving policy to propose harness revisions and training-identified hard tasks to guide the next search.The policy learns through RL under the selected harness.Source: §5 · p. 9

A Implementation Details

The implementation uses Qwen3.5-35B-A3B with a mini-SWE-agent seed harness, and alternates policy-proposed harness search with RL updates. Training-driven hard tasks guide subsequent harness searches.

Agent implementation

  • The system uses Qwen3.5-35B-A3B with SGLang serving and runs RL and SFT through slime on one node with eight B200 GPUs.The seed harness follows the mini-SWE-agent interface and executes bash commands in task-specific repository environments.Source: A Implementation Details · p. 16

Optimization data

  • Subsequent H-steps search tasks never resolved in any preceding P-step rollout, reusing their IDs to gather fresh failure evidence with the updated policy.The initial search set contains 300 tasks randomly sampled from training data.Source: A Implementation Details · p. 16

Harness optimization

  • The current policy serves as both task solver and harness proposer; each H-step evaluates six candidates per task and holds the selected harness fixed during the next policy update.HPO instead uses Opus 4.8 as proposer, while Self-Harness and Meta-Harness use the same search budget.Source: A Implementation Details · p. 16

Policy optimization

  • Each P-step collects on-policy trajectories under the updated harness and optimizes the policy with verifier rewards, GRPO, and Balanced Adaptive Rollout (BAR).BAR generates eight rollouts per task and selects four into reward-balanced groups with success fractions between 0.25 and 0.75.Source: A Implementation Details · p. 16
  • BAR filtering yields 32 reward-balanced groups, or 128 training trajectories per batch; each batch receives one epoch and one optimizer update.The policy uses Adam at a constant learning rate of 10^-6, with no KL penalty; RLVR under the seed harness uses 50 updates per P-step.Source: A Implementation Details · p. 16

B Additional Evaluation Details

Additional evaluations specify shared evaluation settings, compare search-frontier trajectories, and describe pass@k measurement. The supplied table captions identify comparison scope but do not provide outcome values.

Evaluation configurations

SFT configuration

  • The Table 1 SFT baseline trains for two epochs on resolved Opus 4.8 trajectories from RL training tasks, while Table 2 substitutes SFT for RL in HPO’s P-step.The Table 1 setup uses batch size 64, a 5 × 10^-6 learning rate, and 10% warmup.Source: B Additional Evaluation Details · p. 16

Search-frontier curves

  • After a shared first H-step reaches 64.3%, harness-only search reaches 65.2%, while Self-HPO reaches 67.8%, then 68.6% after its second H-step and 69.5% after its second P-step.Figure 6 marks candidate scores, Self-HPO P-steps, and Base and final RLVR reference lines.Source: B Additional Evaluation Details · p. 16, Figure 6 · p. 17

Comparison with frontier models

  • Table 4 compares frontier-model resolve rates on SWE-Bench Verified with Qwen3.5-35B-A3B methods from Table 1; its caption marks Self-HPO in bold.The caption supplies comparison scope, not the table’s outcome values.Source: Table 4 · p. 17, B Additional Evaluation Details · p. 17

Pass@k calculation

  • Pass@k averages task-level success estimates from eight attempts; Pass@1 is mean success frequency, while Pass@8 records whether any attempt succeeds.Figure 4 compares Base with the final Self-HPO pair and displays 95% bootstrap confidence intervals.Source: B Additional Evaluation Details · p. 17

Conceptual comparison

  • Table 5 compares methods by coding-domain coverage, policy learning, harness proposer, and harness-search task selection.Its caption states that Self-HPO uses its evolving policy to propose revisions and training-identified hard tasks for the next search set.Source: Table 5 · p. 18

C Conceptual Comparison with Related Methods

Self-HPO differs from related methods in how it couples policy updates, harness proposal, and search-task selection. Its updated solver becomes the next proposer, while RL-exposed hard tasks guide subsequent harness searches.

C Conceptual Comparison with Related Methods

  • The comparison covers methods’ coding-domain coverage, policy-learning approach, harness proposer, and search-task selection.The listed methods are Meta-Harness, Self-Harness, Task-CoEvolve, Evo2, WHALE, and HASE.Source: C Conceptual Comparison with Related… · p. 18
  • Self-HPO reuses the updated task-solving policy as the next harness proposer, unlike Self-Harness’s fixed policy, WHALE’s frozen external proposer, and Evo2’s separately trained proposer.Its H-step holds policy weights fixed, and its P-step performs RL under the selected harness.Source: C Conceptual Comparison with Related… · p. 18
  • Self-HPO selects the next search set from hard tasks exposed during policy training, whereas Task-CoEvolve selects tasks using outcome variance across candidate harnesses.Self-HPO updates task IDs between H-steps and keeps them fixed within each H-step.Source: C Conceptual Comparison with Related… · p. 18

D Harness Details and Prompt Templates

The appendix describes the seed harness, the space of harness revisions, and the checks added by an optimized harness. The optimized harness retains the seed loop and prompts while intervening through an override of step().

Seed harness

Harness search space

  • Candidates subclass the seed agent and may alter its step() behavior, while policy and sampling settings remain fixed.The search space allows changes such as inspecting agent state, returning feedback before submission, replacing a patch, or intervening during a task.Source: D Harness Details and Prompt Templates · p. 18

Harness proposer prompt

  • The proposer prompt uses the current best harness as parent, starting from the seed harness in round one.The first-round search set is random S0; brace-delimited values change across iterations and rounds.Source: Figure 7 · p. 19

Optimized harness

E Case Study: Where the Harness Intervenes

For the same task and fixed policy, the seed harness accepts a prose summary instead of a patch, while the evolved harness returns control after detecting an empty submission. The policy then writes and submits a valid patch under the evolved harness.

Case study

Loading 2610.22791v1…