Long-Horizon RL Needs Muon

1 Introduction

• Issue resolution requires interpreting an issue, navigating and editing a codebase, and validating a candidate patch through tests.Executable tests provide reinforcement-learning feedback, while trajectory lengths vary and asynchronous collection separates rollout and learner policies.Source: §1 · p. 1
• The study compares Adam and Muon within SPO, then examines Muon with SPO++, reward shaping, and importance correction.Ablations cover length and tool-error penalties and update rules using training success rate.Source: §1 · p. 1
• The full recipe achieves higher training success than SPO baselines in selected runs across both model families.Figure 2 compares the selected runs.Source: §1 · pp. 1–2, Figure 2 · p. 2
• The training recipe uses test outcomes as binary rewards and combines length and tool-error penalties, TIS reweighting, token-level loss averaging, and a Muon update.The overview depicts the agent inspecting, editing, and testing a repository before receiving the final-test outcome.Source: Figure 1 · p. 1

2 Related Work

• Execution-based feedback for software-engineering agents can come from software evolution, test execution, or simulation; this study uses execution-based rewards.Prior work includes SWE-Gym, R2E-Gym, and SWE-smith [Pan et al., 2024; Jain et al., 2025; Yang et al., 2025; JLULLM LAB, 2026].Source: §2 · p. 2
• SPO and SPO++ provide the single-rollout foundation, while other work addresses asynchronous or off-policy training through value learning, importance sampling, trust regions, turn-level correction, or variance control.Muon has also been studied in mathematical and interactive RL; here it is studied in single-rollout asynchronous issue resolution.Source: §2 · p. 2
• Prior agentic and reasoning RL work studies reward shaping and loss aggregation, including length penalties and trajectory weighting; this paper examines their combination with Muon in multi-turn software repair.Source: §2 · p. 2

3.1 Muon-based policy optimization

• Muon’s spectral-norm update preserves gradient singular vectors and equalizes nonzero singular values under an update budget.The implementation uses approximate orthogonalization for hidden weight matrices; embeddings, output heads, normalization parameters, and vector parameters use AdamW.Source: §3.1 · p. 2
• The SPO baseline derives advantages from outcome rewards and a task-specific baseline, standardizes them across trajectories, and uses PPO-style clipping with asymmetric importance-ratio bounds of 0.2 and 0.28.Separately, the recipe follows SPO++ in standardizing advantages across valid tokens included in the policy loss.Source: §3.1 · pp. 2–3

3.2 Outcome reward and auxiliary penalties

• Binary test outcomes are supplemented with turn-level length and tool-error penalties, plus a trajectory-level penalty for predefined command-rejection messages.The tool-error penalty is −0.5 when triggered; the length penalty reaches −0.5 above 2,048 tokens and is omitted on the final turn contributing to the policy loss.Source: §3.2 · p. 3

3.3 Importance correction and loss aggregation

• Truncated importance sampling corrects for differences between training and rollout policies, applying weights directly to masked token-level PPO losses without batchwise normalization.The weight cap is C = 2; the no-TIS ablation removes these weights while retaining PPO ratio clipping.Source: §3.3 · p. 3
• Token-level averaging weights trajectory means by valid-token count, while sequence-level averaging weights trajectory means equally; environment-observation tokens are excluded.Advantage standardization is unchanged in the sequence-level averaging ablation.Source: §3.3 · pp. 3–4

4.1 Experimental setup

• Training uses R2E-Gym issue-resolution tasks with the Claude Code harness, and success means the fraction of evaluated rollouts whose final patches pass all required tests in each update.MiniCPM5-2B and Qwen3.5-4B use the same training-update and rollout budgets across independently initialized runs; auxiliary penalties are excluded from the success metric.Source: §4.1 · p. 4
• The full-recipe comparison holds Muon fixed while jointly changing advantage normalization, task-specific outcome estimation, clipping, and auxiliary penalties.Ablations remove either or both penalties, or vary truncated importance sampling and loss aggregation with the length penalty disabled; configurations collect their own asynchronous rollouts and pair same-seed runs.Source: §4.1 · pp. 4–5

4.2 Main results

• The full recipe reaches higher later-training success than both SPO baselines across both model families; success continues improving with the full recipe while several Adam configurations decline.Without the length penalty, token-level and sequence-level averaging vary in relative performance across seeds.Source: §4.2 · p. 5, Figure 5 · p. 5

• Combining Muon with single-rollout asynchronous updates, importance correction, and auxiliary reward shaping yields higher training success than SPO baselines in evaluated runs from both model families.Source: §5 · p. 5