PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories

Shuai Shao, Kangning Zhang, Qingyao Li, Shijian Wang, Hao Wang, Wenxiang Jiao, Yuan Lu, Yi Guo, Weiwen Liu, Weinan Zhang

arXiv:2608.02276v1cs.AI

TL;DR

Deployed agents accumulate interaction trajectories, but their behavior typically remains fixed, motivating ways to improve the harness around them. Harness-R1 trains a dedicated engineer to patch a frozen target from observed failures and same-task reruns. It improves average success across three benchmarks, including after direct target-agent fine-tuning.

  • Problem

    Deployed agents accumulate interaction trajectories, but their behavior typically remains fixed, leaving open how to use that experience for continual improvement.

    Source: §1 · p. 1

  • Method

    Harness-R1 trains a dedicated engineer on batches of target failures, using validated executable patches and same-task reruns to reward the engineer while keeping the target frozen.

    Source: §1 · p. 2

  • Results

    Harness-R1 improves all three benchmarks, raising vanilla Qwen3.5-9B average success from 44.3% to 53.6% (+9.3 points).

    Source: §6 · p. 10

  • Takeaways & Limitations

    Patch quality is judged by realized task outcomes, since reasonable-looking executable edits need not benefit the running agent.

    Source: §5 · p. 9

    Same-batch outcome rewards tie the training signal to the tasks used to mine failures.

    Source: §5 · p. 9

Abstract

from arXiv · show

Agents built around large language models continually accumulate interaction trajectories during deployment, yet their behavior typically remains fixed. Beyond updating model weights, these trajectories can improve the agent harness that constructs context, mediates tools, validates actions, and recovers execution. We introduce Harness-R1, the first method, to our knowledge, that makes failure-conditioned, lifecycle-wide editing of an existing executable runtime a learned capability. It post-trains a dedicated harness engineer with online reinforcement learning so that its edits are optimized for the realized task success they produce, rather than proposed by a fixed editor. A separate 9B engineer converts batches of target-agent failures into validated executable patches; fresh same-batch reruns of the frozen target provide outcome rewards, so training updates only the engineer. Cold-start supervised fine-tuning initializes this editing policy, which is then trained online with group-relative policy optimization. Across WebShop, ALFWorld, and DBBench, Harness-R1 raises vanilla Qwen3.5-9B success from 44.3% to 53.6% (+9.3 percentage points). After direct target-agent fine-tuning, a target-specific engineer raises the average further from 59.2% to 64.2% (+5.0 points); because these gains hold both before and after fine-tuning the target, Harness-R1 points toward co-evolving the harness engineer and the target agent.

1 Introduction

Deployed agents accumulate interaction trajectories and failures, but their behavior typically remains fixed; Harness-R1 learns to improve the executable harness around a frozen target. It trains a dedicated engineer on the realized outcomes of its patches, improving success across three benchmarks before and after target-agent fine-tuning.

1 Introduction

  • Agent interaction trajectories record observations, actions, environment feedback, and task outcomes, including failures such as tool misuse, lost state, and failed recovery.The paper asks whether this experience can support continual improvement after deployment.Source: §1 · p. 1
  • Harness optimization complements model training by changing runtime mechanisms around a model without changing its weights.Context construction, memory, tool mediation, action validation, and recovery logic shape what the model observes and how execution proceeds.Source: §1 · p. 1
  • Fixed, prompted harness editors are unreliable: a fixed Self-Refine rule lowers reward on all three benchmarks, while frontier-editor gains are unstable or limited.Some prompted editors even reduce WebShop reward.Source: §1 · p. 2
  • Prior harness-optimization systems generally leave the proposer fixed or use outcomes for patch selection, while learning to install failure-conditioned edits into an existing runtime remains less studied.Harness-R1 frames that editing capability as online reinforcement learning for a dedicated engineer.Source: §1 · p. 2
  • Training an engineer is challenging because unrestricted edits can break interdependent runtime stages, while patch quality can only be judged through the target’s post-edit behavior.The required feedback path connects observed failures, constrained executable edits, and resulting performance gains.Source: §1 · p. 2
  • 44.3% to 53.6% (+9.3 points) for the vanilla target and 59.2% to 64.2% (+5.0 points) after direct target-agent fine-tuning across WebShop, ALFWorld, and DBBench.Both comparisons report average success.Source: §1 · p. 2
  • Harness-R1 trains a dedicated engineer with online reinforcement learning to produce executable patches from target-agent failures, while keeping the target frozen.Cold-start supervised fine-tuning initializes the policy before online GRPO, and same-task reruns provide the realized performance reward.Source: §1 · p. 2

2 Related Work

Prior work searches or edits harnesses and model-external artifacts, and some learned editors target narrower structures or task-solving workflows. Harness-R1 instead trains a dedicated policy from the realized effects of executable patches across an existing runtime.

LLM-Based Harness Evolution

Algorithmic Optimization of Harness Components

Learned Harness Editors

3 Method

Harness-R1 treats editing the executable runtime around a frozen target as a batch-conditioned learning problem. It initializes a dedicated harness engineer with supervised fine-tuning, then trains it online using the realized task utility of its patches.

3 Method

  • Harness-R1 post-trains a dedicated engineer to improve the executable runtime around a frozen target, evaluating each modification by rerunning the same tasks.The framework uses cold-start supervised fine-tuning followed by GRPO.Source: §3 · p. 4, §3.2 · p. 5

3.1 Problem Setup

  • The frozen target interacts through a base runtime that assembles context, forwards actions, and relays feedback; the engineer receives compact packets containing only failed episodes.The engineer generates a batch-conditioned executable overlay but does not answer tasks or participate in their rollouts.Source: §3.1 · p. 4
  • The overlay can intervene at episode initialization, pre-decision, pre-action, and post-feedback, while leaving the target's weights untouched.Patches are validated before installation, and invalid patches have no effect.Source: §3.1 · p. 4
  • The patched target reruns every task in the same batch, including prior successes; the resulting objective is transductive and carries no persistent patch memory across batches.The same-batch evaluation controls task composition, but there is no iterative refinement within an instance.Source: §3.1 · pp. 4–5

3.2 Outcome-Grounded Post-Training

  • Cold-start supervised fine-tuning uses teacher-proposed edits from disjoint task batches, retaining at most one executable, complete, non-regressive response per failure packet.The retained responses initialize the engineer through teacher-forced next-token prediction.Source: §3.2 · p. 5
  • Online GRPO samples eight candidate patches per failure packet, reruns the frozen target on the full batch for each valid patch, and updates only the engineer.Invalid, no-op, or incomplete evaluations receive zero reward; WebShop uses shaped environment reward, while ALFWorld and DBBench use binary success.Source: §3.2 · p. 5

4 Experiments

The experiments test whether outcome-trained harness editing improves task success across three interactive environments, transfers across target agents and held-out tasks, and benefits from interventions at different lifecycle positions.

4 Experiments

  • The experiments ask whether outcome-trained edits outperform supervised editing and fixed editors, remain useful after target-agent training, and transfer across models and tasks.The three benchmarks span web navigation, embodied household tasks, and relational-database tasks.Source: §4 · p. 6, §4.1 · p. 6

4.1 Experimental Setup

  • The evaluation spans WebShop, ALFWorld, and DBBench, comparing a frozen Qwen3.5-9B target with its SFT counterpart and fixed, frontier, and trained editors.Success is measured over 500 WebShop, 500 ALFWorld, and 300 DBBench tasks; the separate Reflection results use a two-episode protocol.Source: §4.1 · p. 6

4.2 Main Results

  • 53.6% average success, up from 44.3% (+9.3 points), with Harness-R1 improving the frozen target on all three benchmarks.The equal-weight average is based on WebShop, ALFWorld, and DBBench.Source: Table 1 · p. 7, §4.2 · p. 7
  • 53.6% average success for Harness-R1 exceeds GLM-5.2’s 48.8%; fixed prompt strategies are mixed, with ReAct up 3.2 points and Self-Refine down 2.5.Reflection’s 55.8% cumulative success uses a two-episode protocol and is not directly comparable to single-episode results.Source: §4.2 · p. 7
  • 64.2% average success, up from 59.2% after direct agent SFT, as a target-specific Harness-R1 engineer adds 5.0 points.Some individual metrics dip slightly, but overall success improves, with gains especially concentrated on ALFWorld.Source: §4.2 · p. 7

4.4 Held-Out Task Generalization

  • 8.9 ± 1.5 percentage points: Harness-R1 improves pooled held-out success after each engineer uses 10 failures and applies one patch to the remaining tasks.The held-out set contains 1,270 tasks across the three benchmarks; the gain is positive for all three matched seeds.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 Where in the Lifecycle Do Modifications Matter?

  • 53.1% average success for the full patch is 8.9 points above no intervention; removing pre-action mediation or post-feedback recovery costs 3.9 or 3.3 points.The dominant position varies by environment: pre-action mediation matters most on WebShop and post-feedback recovery on ALFWorld; effects are conditional and should not be added into a universal ranking.Source: Figure 4 · p. 8, §4.5 · p. 9

4.3 Generalization across Target Agents

  • Across 20 unseen target configurations, the benchmark-averaged gain is 7.06 points, and every target-level average is positive.Across 21 targets and three benchmarks, 56 of 63 combinations improve, four are unchanged, and three regress by no more than 2.0 points.Source: Figure 3 · p. 8, §4.3 · p. 8

5 Discussion

The discussion contrasts outcome-verified, failure-conditioned editing with plausible but unreliable fixed edits, and identifies limits in the adaptation schedule and reward signal.

5 Discussion

  • A valid, well-formed patch is necessary but insufficient: the patched target’s realized outcome, rather than textual plausibility, is the test of utility.The discussion notes that a plausible frontier-editor diagnosis can compile into behavior that lowers success.Source: §5 · p. 9
  • Failure-conditioned edits avoid betting on one universal rule; fixed Self-Refine lowers reward by an average of 2.6 points, while the dominant lifecycle position varies by environment.Pre-action mediation matters most on WebShop, whereas post-feedback recovery matters most on ALFWorld.Source: §5 · p. 9
  • The study covers one adaptation from a vanilla target to a fine-tuned target, and its same-batch reward remains tied to tasks used to mine failures.Held-out performance and inference-efficiency terms are proposed as future reward additions; multi-round co-evolution remains future work.Source: §5 · p. 9

6 Conclusion

Harness-R1 trains a dedicated engineer to edit an executable harness around a frozen target, using cold-start SFT followed by online GRPO. It improves benchmark success and transfers to unseen target models.

6 Conclusion

  • Harness-R1 raises vanilla Qwen3.5-9B average success from 44.3% to 53.6% (+9.3 points) and, after target fine-tuning, from 59.2% to 64.2% (+5.0 points).The engineer is trained with cold-start supervised fine-tuning followed by online GRPO; the target agent remains frozen during engineer training.Source: §6 · p. 10
  • The learned editor transfers to twenty unseen target models, gains on every model, and improves 1,270 held-out tasks.Together, the results support harness construction as a learnable capability that complements weight updates and permits the engineer and target to co-evolve.Source: §6 · p. 10

A Prompts

  • SFT and online GRPO use the same response protocol, with GRPO failure packets built from current frozen-target rollouts.The appendix presents fixed prompt instructions separately from per-example input and notes that released SFT JSON contains materialized messages for all three benchmarks.Source: A Prompts · p. 13

A.1 Prompt for the Harness Engineer

  • The harness-engineer prompt’s failure packet contains a group of failed trajectories from the frozen target agent.Source: Figure A.1 · p. 13

A.2 Benchmark-Specific Prompt Content

  • Benchmark-specific prompt fields expose runtime evidence rather than hidden task answers.Source: Figure A.2 · p. 14

A.3 Assistant Response Format

  • The response template prefills the opening <think> sequence, and training targets include the remaining analysis plus exactly one executable patch.Source: Figure A.3 · p. 15

B Implementation Details

The implementation pairs teacher-filtered cold-start examples with online GRPO, while separately fine-tuning the target for sequential adaptation experiments. Executable patches use lifecycle hooks and are rewarded by complete rerun performance differences.

B.1 Harness-Engineer Cold-Start SFT

  • The 877 cold-start SFT examples were GPT-5.5-proposed patches filtered for executability, same-batch rerun completion, and non-negative task-reward change.The Qwen3.5-9B engineer was trained for two epochs with full-parameter fine-tuning; the reported configuration uses 32,768 context length and learning rate 10−5.Source: §B.1 · p. 15

B.2 Online GRPO

  • Online GRPO samples eight candidates per prompt, with four rollout prompts per batch and 32 sequences globally; the listed learning rate is 10−6.The configuration also specifies four rollout iterations per update and maximum policy staleness of eight.Source: §B.2 · p. 16

B.3 Direct Target-Agent SFT

  • Direct target-agent SFT uses 2,515 successful, task-deduplicated trajectories across WebShop, ALFWorld, and DBBench, with two epochs and a 24,576 sequence cutoff.The trajectories comprise 901 WebShop, 774 ALFWorld, and 840 DBBench episodes.Source: §B.3 · p. 16

B.4 Evaluation and Reward

  • Engineer reward is the full-batch mean reward change; invalid, no-op, or incomplete patches receive zero reward.WebShop uses continuous environment reward, while ALFWorld and DBBench use binary success.Source: §B.4 · p. 16

C Executable Patch Interface

  • Patches can intervene at four execution-lifecycle positions, and the host runtime interprets their structured effects.DBBench v1 supports soft guidance and blocking but does not execute SQL rewrites or forced commits.Source: C Executable Patch Interface · p. 17, Table C.1 · p. 17
  • The engineer generates patches before reruns and does not join subsequent task interaction; valid complete patches receive the full-batch performance difference, including regressions.Uninstallable, behaviorally inert, or unevaluated patches count as no intervention and receive zero reward.Source: C Executable Patch Interface · p. 17

D Data Construction and Task Splits

Tasks are split before trajectory collection or patch generation, keeping SFT and RL training task identities disjoint. Validation selects checkpoints, while test tasks are reserved for final evaluation.

D Data Construction and Task Splits

  • SFT and RL use disjoint task partitions; validation is reserved for checkpoint selection and test sets for final evaluation.Table D.1 counts distinct tasks, not trajectories, failure packets, generated patches, or optimizer samples.Source: D Data Construction and Task Splits · p. 17, Table D.1 · p. 18
  • WebShop uses task indices 0–499 as its fixed test set and 5,190 RL training tasks after reserving 100 RL tasks for validation.The RL-side division follows a task-batch-level split; baseline and patched execution use goal seed 233.Source: D Data Construction and Task Splits · p. 17
  • ALFWorld’s 500-task test set combines all 109 official new_std tasks with a stratified 391-task sample from train_valid; 1,280 tasks remain for RL training.Tasks are stratified by six task families, with 99 RL-side tasks reserved for validation.Source: D Data Construction and Task Splits · p. 17
  • DBBench assigns 2,401 tasks to SFT and 2,402 to the RL side, leaving 2,302 RL training tasks after validation; its separate 300-task standard test set is evaluation-only.Source: D Data Construction and Task Splits · p. 17

E Target-Agent Generalization

A single learned editing policy transfers to unseen target agents, with target-specific patches evaluated on each target’s own failure traces. Gains are positive on average across targets despite a few small benchmark-level regressions.

E Target-Agent Generalization

  • The same policy is tested on targets never used during training, generating target-specific patches from each target’s own failure traces rather than reusing one fixed patch.WebShop uses a fixed-seed 500-task rerun; ALFWorld and DBBench use their respective test sets.Source: E Target-Agent Generalization · p. 18
  • 56 of 63 target–benchmark combinations improve, four are unchanged, and three regress; the benchmark-averaged gain across twenty unseen targets is 7.06 points.All target-level averages are positive, and the regressions are small and occur on WebShop for Llama-3.1-70B and Gemma-4-31B-it, and on ALFWorld for Qwen2.5-72B.Source: E Target-Agent Generalization · p. 18

F Held-Out Task Generalization

Harness-R1 patches inferred from ten failures improve held-out success, while frontier engineers average negative and vary substantially across seeds.

F Held-Out Task Generalization

  • For each benchmark and seed, engineers used the same ten frozen-target failures to generate one patch for remaining tasks; evaluation pooled 1,270 held-out tasks across three seeds.The full-split reference included 1,300 tasks, and the error term was sample standard deviation across seeds.Source: F Held-Out Task Generalization · p. 18
  • 8.9 ± 1.5 points: Harness-R1 improves pooled held-out success and is positive on every seed, while both frontier engineers average negative and straddle zero.The larger frontier spreads, ±2.5 and ±3.6, reflect swings between marginal gains and sizable regressions.Source: F Held-Out Task Generalization · p. 18
  • The result suggests that converting sparse failures into a broadly useful edit is a capability scale alone does not confer.Both frontier engineers averaged negative held-out changes and straddled zero across seeds.Source: F Held-Out Task Generalization · p. 18

G Lifecycle-Position Ablation

The lifecycle ablation finds the largest average drops when pre-action mediation or post-feedback recovery is removed. Selected case studies illustrate distinct intervention mechanisms and limits across environments.

G Lifecycle-Position Ablation

  • 53.1% average success: the full patch exceeds no intervention by 8.9 points; removing pre-action mediation or post-feedback recovery causes the largest drops, 3.9 and 3.3 points.The dominant position varies by environment, and effects are conditional because one patch can coordinate positions rather than provide a universal importance ranking.Source: G Lifecycle-Position Ablation · p. 18, Table G.1 · p. 19

H Qualitative Case Studies

  • The qualitative cases inspect three validation-selected engineer evaluations, checking runtime traces as well as code to confirm intended edits actually executed.Each uses the same frozen Qwen3.5-9B target and the same ten tasks as baseline evidence; aggregate claims come from full evaluations.Source: H Qualitative Case Studies · p. 20

H.1 WebShop: Correcting a Premature Purchase

  • Across ten WebShop tasks, a narrow pre-action guard raises full successes from 2 to 5 and mean reward from 0.682 to 0.768, preserving prior full successes.The guard blocks a premature purchase when budget or required-option predicates fail, but cannot repair an earlier wrong-product choice.Source: §H.1 · p. 20

H.2 ALFWorld: Coordinating Multiple Lifecycle Positions

  • In ALFWorld, coordinated lifecycle guidance changes batch success from 1/10 to 6/10, rescuing six baseline failures but regressing one success.One two-object trajectory still loops, illustrating imperfect stage tracking.Source: §H.2 · pp. 20–21

H.3 A Failure Case of Direct Harness Editing

  • On ALFWorld, Gemini-3.5-Flash edits reduce success from 208/500 (41.6%) to 177/500 (35.4%); broad stage rules prematurely placed one object in two-object tasks.The regression example shows that plausible diagnosis and executable code do not ensure safe runtime behavior.Source: §H.3 · p. 21

H.4 DBBench: Preserving Schema and Stored-Value Conventions

In DBBench, Harness-R1 improves a batch result over both no intervention and a valid competitor patch by preserving schema and stored-value conventions. Cross-case examples show environment-specific editing policies, without guaranteeing regression-free rules.

H.4 DBBench: Preserving Schema and Stored-Value Conventions

  • On DBBench batch 022, Harness-R1’s stage-aware patch raises the frozen target from 4/10 to 6/10; on the same baseline evidence and tasks, a valid GLM-5.2 patch reaches 5/10.The patch recommends schema inspection after identifier errors, row inspection before mutation, and stored-value verification before commit.Source: §H.4 · p. 21

H.5 Cross-Case Interpretation

  • Across cases, Harness-R1 learns environment-dependent edits rather than one fixed prompt, but useful executable interventions are not guaranteed complete or regression-free.Its WebShop guard cannot fix an earlier wrong-product choice; ALFWorld routing and two-object state can cause regressions or loops, while DBBench value conventions may need stronger neighborhood checks.Source: §H.5 · p. 22

Related papers

Loading 2608.02276v1…