Source-linked AI summary
LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents
Yiming Du, Yuxin Jiang, Tao Yuan, Jianbo Dai, Shaowei Wang, Jierun Chen, Chaofan Tao, Xianzhi Yu, Lifeng Shang, Kam-Fai Wong, Xiaohui Li, Haoli Bai
TL;DR
Native coding-agent harnesses complicate policy-gradient training because rollout tokens, execution outcomes, and trainer signals can diverge. LEGO-RL bridges these harnesses to scalable optimization, improving SWE-bench Verified resolve rates across OpenHands SDK, Claude Code, and OpenCode while maintaining rollout–training probability correlation above 0.99.
Problem
Native harnesses can rewrite contexts and histories while execution failures and reward hacking compromise faithful probability recomputation and reliable reward signals during policy-gradient training.
Method
LEGO-RL connects existing coding-agent harnesses to policy-gradient optimization through lightweight adapters while preserving their workflows and integrating faithful rollout capture, sandboxed execution, and observability.
Results
Across three harnesses, LEGO-RL improved SWE-bench Verified resolve rates, including OpenHands SDK from 64.0% to 70.4%, while maintaining rollout–training probability correlation above 0.99.
Takeaways & Limitations
The results support combining scalable optimization with reliable execution, faithful trajectory capture, and trajectory-level observability for coding-agent reinforcement learning.
Takeaways & Limitations
Evaluation used one model architecture, separate harness training, and single runs, leaving generalization and run-to-run variance unquantified.
Abstract
from arXiv · showhide
Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train-inference discrepancies decouple rollout behavior from policy updates. To address this, we present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. LEGO-RL is built upon three pillars: (1) faithful optimization via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; (2) reliable execution via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and (3) observable training through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses. LEGO-RL improves Qwen3.5-35B-A3B across OpenHands SDK (64.0% to 70.4%), Claude Code (62.4% to 68.2%), and OpenCode (57.2% to 66.6%) on SWE-bench Verified, while maintaining a rollout-training probability correlation above 0.99.
1 Introduction
LEGO-RL connects native coding-agent harnesses to scalable policy-gradient optimization without modifying their workflows, addressing trajectory-fidelity and execution-reliability challenges. Across three harnesses, it improves SWE-bench Verified resolve rates while maintaining rollout–training probability correlation above 0.99.
- Motivation: Coding-agent reinforcement learning optimizes long-horizon trajectories spanning repositories, tools, code changes, dependencies, and test execution rather than isolated responses.Policies are updated from complete trajectories and verifier rewards.
- Challenges: Harness-side prompt construction, context compaction, history rewriting, sparse-MoE routing, sandbox failures, and dependency errors can break faithful training and execution signals.Reconstructed trajectories may differ from sampled tokens, while unreproduced expert routing and execution failures introduce additional mismatches.
- Framework: LEGO-RL connects existing harnesses to policy-gradient optimization without modifying agent workflows, preserving model APIs, tool interfaces, prompts, and context-management policies.It is built on verl and Harbor, and new harnesses require only a lightweight adapter.
- Framework: LEGO-RL combines exact rollout-token capture and routing replay with sandbox orchestration, image caching, termination-aware filtering, recovery, validation, monitoring, and trajectory diagnosis.These mechanisms target faithful log-probability recomputation, reward integrity, asynchronous training stability, and failure visibility.
- Results: 64.0% → 70.4% on OpenHands SDK, 62.4% → 68.2% on Claude Code, and 57.2% → 66.6% on OpenCode are LEGO-RL’s SWE-bench Verified resolve-rate results.The evaluation uses a sparse mixture-of-experts policy and maintains rollout–training probability correlation above 0.99.
2 Related Work
Related work spans agentic RL frameworks that differ in where rollout generation occurs and coding-agent benchmarks that evaluate repository-level issue resolution through executable validation. These coding tasks involve many interleaved model calls, tool actions, and environment transitions, motivating infrastructure beyond single-step LLM-RL.
- Agentic RL training frameworks: Agentic RL frameworks differ in whether rollout generation remains inside the RL framework or is managed through trainer-controlled pipelines.Early infrastructure such as verl placed rollout generation within a trainer-managed pipeline, while newer frameworks include slime, MOLT, SkyRL, and AReaL.
- Coding-agent benchmarks, harnesses, and training tasks: SWE-bench introduced repository-level issue resolution with executable validation for coding-agent evaluation.Subsequent benchmarks extended this setting to harder tasks, multilingual repositories, and continuously refreshed task sets.
- Coding-agent benchmarks, harnesses, and training tasks: Coding-agent training tasks require many interleaved model calls, tool actions, and environment transitions rather than single-step LLM-RL.
3 The LEGO-RL Framework
LEGO-RL treats the unmodified coding-agent harness as part of the environment while optimizing only its policy, and builds training around faithful trajectory capture, reliable sandbox execution, and scalable asynchronous orchestration. Its infrastructure preserves policy-generated tokens and verifier signals despite harness history rewriting, execution failures, and long-tailed rollout durations.
- Problem setup: LEGO-RL models each task as a problem, initialized repository, and executable verifier, treating the native harness as environment and optimizing only policy πθ.The harness supplies each turn’s context, the policy generates assistant tokens, and the verifier returns a terminal binary reward.
- Faithfulness: Faithful optimization requires capturing exact rollout contexts, policy tokens, response masks, and routing decisions at the model-serving boundary rather than reconstructing trajectories afterward.Harness compaction, re-serialization, or history rewriting can prevent the recorded transcript from reproducing sampled tokens; sparse MoE training also requires reuse of behavior-policy expert routing.
- Sandbox execution: LEGO-RL runs every trial in a fresh isolated sandbox with shared-storage lazy image streaming, read-only runtime mounts, and backend support for Docker and Kubernetes.These mechanisms target environment setup, isolation, and efficient preparation across large-scale concurrent execution.
- Sandbox execution: Stage-specific timeouts and asynchronous scheduling prevent pathological long-running trials from delaying synchronized rollout batches.Agent execution dominates duration, while setup and verification contribute relatively little on average but have substantial tail latency.
- Faithfulness: The in-process proxy connects unmodified harnesses to the optimizer through OpenAI-compatible and Anthropic APIs while preserving generation-time policy spans.It captures token IDs, log-probabilities, response masks, and generation metadata directly from serving sessions, while message-level alignment handles rewritten histories.
- Rollout orchestration: Fully asynchronous rollout generation keeps inference slots productive under heavy right-tailed durations by decoupling trajectory generation from optimization and recovering partial trajectories across weight synchronization.Configurable limits bound exceptionally long sessions, while completed trajectories and verifier rewards flow into the data buffer for trainer consumption.
4 Experiments
LEGO-RL improves Qwen3.5-35B-A3B across three native coding-agent harnesses, while preserving rollout–training probability alignment. Experiments also show that execution failures, task difficulty, harness-specific control flow, and long-horizon interaction materially affect training reliability and efficiency.
- End-to-end effectiveness: 70.4%, 68.2%, and 66.6% are the peak SWE-bench Verified scores for OpenHands SDK, Claude Code, and OpenCode, up from 64.0%, 62.4%, and 57.2%.All three runs used the same Qwen3.5-35B-A3B initial checkpoint, 2,699-task training index, and 200k-token context budget.
- Baseline comparisons: LEGO-RL-Qwen3.5-35B-A3B is strongest across all three harnesses, exceeding Qwen3.6-35B-A3B by 3.0, 4.8, and 6.0 points.The comparison uses the same measurement protocol and also includes KAT-Coder-V2.5-Dev.
- Baseline comparisons: KAT-Coder-V2.5-Dev gains 3.4 points under Claude Code but loses 0.4 under OpenHands SDK, where Qwen3.6-35B-A3B scores higher.The passage states that the cause cannot be isolated, while demonstrating harness-specific transfer differences.
- Trajectory validity: 7.1% of Claude Code, 2.4% of OpenHands SDK, and 6.4% of OpenCode trajectories are excluded when infrastructure failures terminate rollouts before optimization.Valid trajectories reaching configured turn or token limits retain their verifier outcomes.
- Task difficulty: 51.4% of OpenHands SDK groups have zero variation, while OpenCode remains approximately unchanged at 43.3%, indicating that fixed task pools can provide less informative supervision.Under OpenHands SDK, increased all-solved groups outweigh the decline in all-wrong groups; under OpenCode, the increase in all-solved groups is smaller.
- Rollout–training alignment: 0.998 is the median Pearson correlation across every scaffold, never falling below 0.989 at any training step, while p99 mean token log-probability discrepancy stays below 3 × 10−3.Trainer-side recomputation closely reproduces rollout-time generation across agent scaffolds and context-management policies.
5 Conclusion
LEGO-RL connects native coding-agent harnesses to scalable policy-gradient optimization while preserving their original control flows. Its integrated execution, rollout, training, reward-integrity, and observability mechanisms support reliable and faithful training across three harnesses.
- Framework: LEGO-RL connects native coding-agent harnesses to scalable policy-gradient optimization without modifying their original control flows.The framework is designed to preserve native harness behavior during optimization.
- Reliability: Sandboxed execution, verification, and reward-integrity safeguards support reliable training.These components are integrated into the framework to improve execution and reward reliability.
- Training and observability: Token-faithful rollout capture, asynchronous training, and trajectory-level observability support faithful training across OpenHands SDK, Claude Code, and OpenCode.The framework combines rollout fidelity, training infrastructure, and trajectory diagnostics across all three named coding-agent harnesses.
Limitations and Future Work · A Reward-Integrity Failure Modes
LEGO-RL is limited by evaluation scope, single-run cost constraints, and coarse executable verification, while future work targets mixed training, broader harness support, richer credit assignment, and automated diagnosis. Its training reliability also depends on verifier rewards faithfully reflecting task completion, with six systematic failure modes categorized by root cause and addressed by production defenses.
- Limitations and Future Work: All experiments use Qwen3.5-35B-A3B, with each coding-agent harness trained separately, leaving generalization to other architectures and mixed-harness training unevaluated.The limitation concerns both model-architecture generalization and training across multiple harnesses.
- Limitations and Future Work: Production-scale training cost limits each main configuration to a single run, leaving run-to-run variance in training gains and system-level efficiency unquantified.The paper identifies this as a limitation of its evaluation design.
- Limitations and Future Work: Executable verification provides a reliable but coarse binary reward signal.The supplied passage introduces executable verification as a limitation but is truncated after “binary r”.
- Limitations and Future Work: Future work extends LEGO-RL to mixed training across repository repair and other verifiable software tasks, and across several harnesses within one policy.These directions address the current separate-harness training setup.
- Limitations and Future Work: Additional plans include adapters for further harnesses, richer credit assignment than terminal binary rewards, and automated diagnosis in the Live UI.The project also plans continued releases of framework updates, harness adapters, trained checkpoints, and task indices.
- A Reward-Integrity Failure Modes: Training reliability depends on verifier rewards faithfully reflecting task completion, but development identified six systematic failure modes violating that assumption.Table 6 categorizes failures by whether agents exploit the reward mechanism or environments decouple rewards from agent behavior.
- A Reward-Integrity Failure Modes: Table 6 organizes reward-integrity failures by root cause and lists the corresponding defenses deployed in production.The table distinguishes reward-mechanism exploitation from environment-generated rewards decoupled from agent behavior.
B Routing-Replay Negative Control and Capture Coverage
Routing replay has two independent risks: assigning decisions to the wrong tokens and failing to capture decisions at all. Misalignment silently degrades metrics, while incomplete capture reached only 24% coverage before diagnosis and is handled fail-soft rather than through incorrect assignments.
- Failure modes: Routing replay can fail through either incorrect token-decision alignment or incomplete capture, and these failure modes are independent.A system may have perfect alignment with incomplete capture, or complete capture with systematic misalignment.
- Misaligned replay: Systematic one-position misalignment degrades all metrics relative to disabling replay and cannot be detected merely by confirming that replay is active.The misaligned system appears to function normally, so diagnosis requires a more specific diagnostic than replay activation.
- Incomplete capture: 24% overall capture coverage was reached before diagnosis when the buffer formula underestimated hybrid-attention requirements by roughly fourfold.An out-of-range guard silently recorded excess decisions as zero, causing coverage to decay with sequence length.
- Incomplete capture: Fail-soft handling replays tokens without recorded decisions unconstrained, preventing incomplete capture from actively harming the policy through incorrect assignments.This behavior contrasts with misalignment, which routes tokens through experts selected for neighboring tokens.
C Assisted-Analysis Case Study: A Collapsed Run · D How Validation Attempts Fail
The assisted-analysis case study diagnoses a collapsed Qwen3-30B-A3B run, locating failure in action cessation and degraded optimization signals rather than infrastructure. Validation-attempt analysis separately classifies how unsuccessful attempts progress and reports a rise in resolved attempts across the production run.
- C Assisted-Analysis Case Study: A Collapsed Run: Training reward fell from 0.351 to 0.050, while held-out validation reward fell from 0.230 to 0.014 over 29 logged steps.The case uses Qwen3-30B-A3B with the OpenHands SDK on a 449-task pool.
- C Assisted-Analysis Case Study: A Collapsed Run: Step 17 marked learning cessation, as training reward decline (t = −7.5) was mirrored by validation reward.This ruled out a training-metric artifact.
- C Assisted-Analysis Case Study: A Collapsed Run: Turns per trajectory fell from 9.0 to 1.0 between steps 17 and 25, while response length shrank to one-fifth and reward dropped to a tenth.Trajectory inspection identified the proximate failure as the policy ceasing to act.
- C Assisted-Analysis Case Study: A Collapsed Run: Advantage bounds reached zero at two of the last three steps, while the KL term rose to 150× its initial level.Nearly every group became all-wrong, leaving updates dominated by penalty rather than reward.
- C Assisted-Analysis Case Study: A Collapsed Run: Rollout–training agreement declined from 0.995 to 0.851 at step 22, while setup-failure share never exceeded 1.6% across steps 17–28.The evidence does not establish whether capture regression contributed to the collapse or reflected distance from the reference policy; agreement crossed below 0.95 at step 21.
- D How Validation Attempts Fail: Validation trajectories were classified by whether they edited the right file, edited without fixing the issue, produced no edit, or exhausted their budget.The verifier reward records only whether a validation attempt succeeded.
- D How Validation Attempts Fail: 63.8% to 68.6%: the resolved share rose across the first and last of 26 validation events in the OpenHands SDK production run.The comparison used the same 500-task set.
E Reasoning–Action Composition Across Training · F Tool Allocation
Across training, reasoning occupies an increasingly common share of responses while interaction horizons lengthen and tool allocation shifts differently across OpenHands SDK and Claude Code. These changes describe trajectory composition and tool-use dynamics, not evidence that reasoning capability improved.
- E Reasoning–Action Composition Across Training: Reasoning share measures characters outside tool-call delimiters, treating tool-call arguments as acting, on a fixed 120-task validation cohort reused at each validation event.Scoring visible prose alone would roughly double the reported levels.
- E Reasoning–Action Composition Across Training: 0.22 to 0.39 (OpenHands SDK) and 0.18 to 0.39 (Claude Code): reasoning share rises over training and converges to the same final level.The passage characterizes this as an observational policy-level trend rather than an isolated scaffold effect.
- E Reasoning–Action Composition Across Training: By the last epoch, tasks grouped by eight-rollout success have nearly equal reasoning shares, although early training shows lower shares on more difficult tasks.Hard tasks gain the most because they start lowest, and outcome-based grouping does not establish causality.
- E Reasoning–Action Composition Across Training: The observed pattern is interpreted as trajectory shape drifting toward a common reasoning ratio, not as evidence of improved reasoning capability.Direct intervention on reasoning length would be required to establish that reasoning itself caused success.
- F Tool Allocation: 69.9 to 106.7 (OpenHands SDK) and 63.6 to 76.9 (Claude Code): tool-calling turns per validation task increase under both scaffolds.Test-suite invocation becomes more frequent and malformed calls decline across both scaffolds.
- F Tool Allocation: 20.1% to 9.7%: OpenHands SDK reduces shell-based file inspection while structured view commands rise from 14.5% to 24.6% of calls.The breakdown compares the first and last 420 trajectories of each run.
- F Tool Allocation: 37.8% to 32.0%: Claude Code’s structured file-tool use falls while shell inspection rises from 6.5% to 7.3%.The comparison is approximate because the two harnesses expose different tool sets.
G Agent Behavior Changes Across Training
Training substantially improves self-verification and exploration behaviors, while failure handling changes little. The evaluation also tracks protocol compliance and separates coverage from reliability using deterministic trajectory scans over production OpenHands SDK runs.
- Behavior changes: Self-verification and exploration improve substantially after training, according to deterministic scans of agent transcripts.These behaviors form the upper block of Table 9’s trajectory-level analysis.
- Behavior changes: Failure-handling behaviors barely move across training.Failure handling is the middle block in the trajectory-level behavior analysis.
- Behavior changes: Protocol compliance and the coverage/reliability split are also evaluated as lower-block behavior categories.The categories are read from agent transcripts rather than judged by a separate evaluator.
- Evaluation setup: 420 trajectories at each endpoint of the production OpenHands SDK run provide the before-and-after behavior comparison.Pass@k and passk rows are computed over prompt groups in the first and last thirds of the run.
H Live Observability Dashboard
The Live Observability Dashboard provides granular views of coding-agent training, linking rewards and standard RL metrics to trajectories, task-level progress, consistency, and failure causes. Supporting mechanisms make live-run diagnostics actionable by exposing per-step progress and distinguishing verifier artifacts from policy failures.
- Live training overview: 94.1% of 34,816 rollouts complete and carry learning signal, while the dashboard tracks solve rates, reward changes, trajectory shape, and termination reasons.The per-instance grid colors tasks by solve rate; accompanying views decompose improved versus regressed tasks and track turn counts, token budgets, and chain-of-thought share.
- Diagnostic views: The dashboard combines standard RL curves with per-instance checkpoint transitions, group-relative update composition, failure causes, validation progress, and optional AI-assisted run analysis.Its panel inventory includes entropy, KL, gradient norm, throughput, model-flops utilization, task-grid trend significance, overlong ratios, stage timing, tool frequency, validation-failure classification, and language-model diagnostic summaries.
- Diagnostic views: Every trial’s full agent transcript appears beside its verifier reward and termination reason, enabling suspicious training curves to be traced to concrete behavior.The trajectory viewer connects aggregate signals with individual agent behavior.
- Diagnostic views: The consistency panel reports probability Pearson, per-sample mean/max absolute log-ratio, batch log-ratio histograms, and ESS fraction under importance weighting.These measures form the per-step fidelity suite described in §4.4.
- Live-run trust mechanisms: A central per-step progress aggregator collects one-line summaries from finished trials and renders live counts with per-outcome tallies, while offline checks investigate suspected false-zero validation scores.The live summary includes task, termination reason, reward, turns, and wall-clock information; the passage notes verifier artifacts as a suspected source of false zeros.
I Run Configuration · J Data Formats
The production runs share clarified training configurations, while each task moves from a SWE-bench-style raw record to a sandboxed Harbor task with verifier-only grading state. A lightweight task index then points to shared-storage Harbor tasks for trainer sampling.
- I Run Configuration: The three production runs share hyperparameters, with KL applied only in the loss, split prompt-response truncation, and importance-sampling correction disabled because fidelity remained high.The context budget truncates prompt and response independently when either exceeds its allocated share.
- J Data Formats: The documented data flow uses the real instance 12rambau__sepal_ui-814 to illustrate successive raw-record and executable-task representations.The raw-record example includes repository, commit, patch, test-patch, and problem-statement fields.
- J Data Formats: Each upstream instance contains a repository snapshot, issue text, gold patch, test patch, and test lists defining the verifier outcome.The documented example uses the OpenSWE-derived instance 12rambau__sepal_ui-814.
- J Data Formats: Data preparation converts each raw instance into a self-contained Harbor task whose manifest specifies the environment image, stage resources, timeouts, and sandbox verifier.The verifier reproduces official SWE-bench grading inside the sandbox.
- J Data Formats: During agent execution, only the initialized repository is visible; tests, gold-patch metadata, and grading scripts enter the sandbox afterward during verification.This delayed upload implements the tests-withheld-until-grading defense.
- J Data Formats: The trainer samples from a thin task index whose rows point to Harbor tasks, keeping task contents on shared storage while enabling inexpensive filtering and remixing.The index stores pointers rather than full task contents.