PaperlayerView exact revision on arXiv

Source-linked AI summary

Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents

Liming Pu, Xiaoxia Li, Yifu Liu, Teng Cao, Bin Yang

arXiv:2609.01245v1cs.LGcs.AI

TL;DR

The paper asks whether outcome-only RL truly reaches a ceiling on small open models for long-horizon interactive tasks. It introduces CANOPY, which expands same-task exploration and anchors strictly on-policy updates, and reports leaderboard-leading AppWorld performance alongside gains on SWE-bench Verified.

  • Problem

    The paper examines whether apparent limitations of outcome-only RL arise from sparse rewards themselves or from distorted training procedures that cause signal starvation and policy drift.

  • Method

    CANOPY restores outcome variation with larger same-task groups and retained hard tasks, then uses on-policy updates, KL anchoring, and action-token-only loss.

  • Results

    86.9 Test-Normal TGC and 67.6 Test-Challenge TGC placed a CANOPY-trained Qwen3-14B at the top of the AppWorld leaderboard, while the same principles improved Qwen3.5-9B by 16.6 points on SWE-bench Verified.

  • Takeaways & Limitations

    The results support internalizing some long-horizon domain capabilities into the weights of a small open model rather than assembling them through inference-time scaffolding.

  • Takeaways & Limitations

    The AppWorld training pool contains only 90 standardized tasks, so larger, more diverse, and more complex distributions are needed to test whether the gains persist.

Abstract

from arXiv · show

Reinforcement learning is a natural way to post-train LLM agents for long-horizon interactive tasks judged only by end-of-task verification, yet a shared belief holds that outcome-only RL soon hits a ceiling on small open models. Recent work therefore compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration. We argue the ceiling is an artifact of two failures of common practice. Signal starvation: group-relative RL with sparse outcome-only rewards yields a gradient only when a task's rollout group mixes successes and failures, so under-scaled exploration silences exactly the hardest, most instructive tasks. Policy drift: squeezing many updates out of a small task pool degrades the policy itself, as an unanchored objective lets the sampling distribution collapse exactly when saturation has already made informative groups rare. We present CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens, then cash in an enlarged interaction budget at test time. On AppWorld, a long-horizon interactive coding benchmark, a Qwen3-14B policy trained with CANOPY through environment interaction alone--without task-specific supervision, auxiliary credit signals, or elaborate agent scaffolding--topped the public leaderboard (Feb. 2026; Test-Normal TGC 86.9, Test-Challenge 67.6), and the same design principles lift Qwen3.5-9B on SWE-bench Verified by 16.6 points. Agentic RL alone thus internalizes long-horizon capability directly into a small open model; we plan to release the complete training stack at https://github.com/AlibabaResearch/SignalCoverageRL.

1 Introduction

The paper argues that outcome-only RL can train small open models for long-horizon interactive tasks, and attributes the apparent ceiling to signal starvation and policy drift rather than an inherent limit of the reward.

  • Motivation: Outcome-only RL is presented as a route for internalizing domain interaction skills into a lightweight policy without external machinery.The setting involves long-horizon interaction and end-of-task verification.
  • Diagnosis: The paper identifies distorted training trajectories as the source of the apparent ceiling, including shortened horizons, filtered hard tasks, few rollouts, stale reuse, and substituted rewards.These practices deprive the policy of complete self-generated trajectories and can collapse the learning signal.
  • Signal starvation: Group-relative outcome RL produces useful gradient only when a task’s rollout group mixes successes and failures, leaving small groups on hard tasks mostly silent.The paper defines signal coverage as Psig(p, n) = 1 − p^n − (1 − p)^n.
  • Policy drift: Repeatedly updating on a limited task pool can narrow the sampling distribution and reduce exploration when informative groups are already rare.The paper names this failure policy drift.
  • CANOPY: CANOPY restores coverage with larger same-task groups and retained hard tasks, then limits drift through on-policy, KL-anchored updates over action tokens.The protocol uses no optimizer change or auxiliary module.
  • Results: 86.9 Test-Normal TGC and 67.6 Test-Challenge TGC were achieved by Qwen3-14B on AppWorld, while Qwen3.5-9B improved by 16.6 points on SWE-bench Verified.The AppWorld result was reported at the top of the public leaderboard in February 2026.

2 Related Work

Related work largely modifies rewards, estimators, or inference-time scaffolding around interactive agents; this paper instead treats exploration coverage as the missing ingredient in group-relative RL.

  • Coding agents and agentic RL: Interactive coding agents span software engineering, web and research tasks, and application operation, with AppWorld serving as the primary testbed here.Software repair is used as the transfer domain.
  • RL algorithms and failure modes: PPO- and GRPO-family methods dominate agent post-training, with variants changing objectives, importance ratios, grouping, or credit assignment.GRPO replaces a learned critic with group-relative rollouts.
  • Novelty: Existing methods often fix estimators or rewards while leaving exploration treated as given.The paper claims that scaling exploration can remove the need for such fixes.
  • Policy training on AppWorld: AppWorld policy-training routes include plain outcome RL, refined credit signals, and other methods that alter reward or credit assignment.Examples include trajectory-graph redistribution, execution-process signals, and self-generated curricula with step-level judges.
  • Training-free AppWorld systems: Training-free systems compensate around fixed frontier models using orchestration, retrieval, skills, memory, playbooks, or test-time debugging.The paper contrasts these systems with capability internalized in model weights.

3 The CANOPY Protocol

CANOPY is a minimal agentic RL protocol that restores sparse-reward signal through broader same-task exploration and controls repeated-update drift with strictly on-policy, anchored training.

  • 3.1 Preliminaries: The Agentic RL Loop: The agentic RL loop samples trajectories for tasks, scores them with held-out unit tests, converts rewards to group-relative advantages, and updates the policy that generated them.No learned critic is required.
  • 3.2 Signal Starvation, and Explore More: Sparse outcome rewards provide a nonzero group-relative gradient only when rollouts contain both successes and failures.All-success and all-failure groups contribute no gradient.
  • 3.2 Signal Starvation, and Explore More: Signal coverage rises with group size on hard tasks but collapses when per-rollout success probability is near either extreme.Figure 2 reports 34% coverage at n=8 and 81% at n=32 for p=0.05.
  • 3.2 Signal Starvation, and Explore More: CANOPY sizes groups from pilot-estimated difficulty, retains the hardest tasks, and avoids per-turn generation caps so error-recovering episodes are not truncated.The sizing rule is explicitly a first-order hardware-budget heuristic rather than an optimal prescription.
  • 3.3 Policy Drift, and Drift Less: Policy drift arises when repeated updates on a small task pool narrow the sampling distribution, reuse stale rollouts, and distort contributions from differently sized trajectories.An unanchored objective compounds signal starvation as saturation makes informative groups rare.
  • 3.3 Policy Drift, and Drift Less: CANOPY answers drift by using one update over the whole rollout batch, a KL anchor to the base model, pooled action-token normalization, and an action-token mask.Environment tokens receive no policy gradient, and the sparse reward is retained rather than replaced by a proxy.
  • 3.3 Policy Drift, and Drift Less: Fault quarantine excludes only serving-layer failures from training treatment, while agent-induced terminations remain genuine behavioral failures scored zero.This separates infrastructure faults from failures caused by the agent.

4 Experiments

Experiments evaluate CANOPY on AppWorld and SWE-bench Verified, showing strong leaderboard performance, interpretable training dynamics, budget transfer toward difficult tasks, and cross-domain improvement.

  • Main results: Qwen3-14B reaches 86.9 Test-Normal TGC and 67.6 Test-Challenge TGC as a single policy without external agent scaffolding.The policy uses one checkpoint at inference without orchestration, skill libraries, retrieved memory, or test-time debugging.
  • Training dynamics: Rollout logs show informative groups shrinking as reward saturates, while L3 remains informative after easier tiers become silent.Figure 3 uses n=32 groups across 90 training steps and reports bootstrap confidence intervals.
  • Training dynamics: The KL anchor preserves late-stage exploration: anchored entropy remains 0.217 with Dev reaching 87.3, whereas unanchored entropy falls to 0.038 and Dev stalls at 81.6.The divergence appears past approximately step 70.
  • Budget transfer: Increasing the interaction budget raises Test-Normal performance from 79.5 to 83.2 mean@4, with gains concentrated on L3 and unseen applications.The base also improves from 22.8 to 32.4, but remains below the trained policy at the smaller budget.
  • Ablations: Ablations identify rollout-group coverage and strictly on-policy updates as the largest contributors, costing −16.4 and −17.4 when weakened.Removing the KL anchor costs −7.0, token-level loss costs −5.4, and densifying reward costs −1.8.
  • Transfer to software repair: The same design principles improve Qwen3.5-9B on SWE-bench Verified from 31.3 to 47.9 mean@4 and from 43.8 to 58.0 best@4.Budget transfer adds +2.3 in the reported transfer setting.

5 Conclusion and Future Work

The paper argues that outcome-only RL can internalize long-horizon interaction capabilities in small open models when training restores informative outcome variation and limits policy drift. It reports leaderboard leadership on AppWorld and identifies broader environment scaling, algorithmic efficiency, and domain mid-training as directions for testing and extending the approach.

  • Conclusion: CANOPY addresses informative outcome variation and policy drift with a simple explore-more, drift-less recipe.The approach uses interaction-based post-training rather than elaborate inference-time machinery.
  • Conclusion: A single open 14B policy trained with CANOPY reached the top of the AppWorld leaderboard at submission time.The policy internalized capabilities in its weights rather than relying on external orchestration, skill libraries, or retrieved memory.
  • Conclusion: The same design principles improved software repair, supporting capability internalization beyond application operation.The passage states this as a result without specifying the software-repair score.
  • Future work: AppWorld training uses only 90 tasks, so larger, more diverse, and more complex task distributions are needed to test whether the gains persist and expose new limits.The proposed expansions include multilingual software engineering and harder benchmarks.
  • Future work: More sample-efficient RL could make environment scaling affordable while adapting group size or task sampling to the current success probability p.The paper frames this as a way to manage the exploration–exploitation trade-off more directly.
  • Future work: Domain mid-training could improve base-model coverage and raise the attainable ceiling of outcome-based RL while complementing interaction-based post-training.

A Full AppWorld Training Configuration

The main AppWorld configuration collects large same-task rollout groups and performs one gradient update without reusing rollouts, while sparse-group advantages explain why isolated successes can dominate gradients. The implementation also verifies strict on-policy behavior through unit importance ratios and logged PPO statistics.

  • Configuration: The rollout phase collects 90 × 32 = 2,880 trajectories, followed by exactly one gradient step over all of them.Because the rollout batch and PPO mini-batch are both 90 tasks, no rollout is reused.
  • Advantage magnitudes: For a group with k successes out of n, a successful rollout receives a standardized advantage based on the group’s sparse success outcomes.The derivation uses the empirical mean and population standard deviation of the binary rewards.
  • Advantage magnitudes: At k = 1, the successful rollout receives Â+ = √n −1 while each failure receives Â−= −1/√n −1.The single success is weighted n −1 times more heavily than each failure and carries essentially the whole group gradient.
  • On-policy verification: With one pass over a gradient batch equal to the rollout batch, the sampling and learning policies coincide, making ρi,t exactly 1 and measured policy KL exactly 0.Under this configuration, the PPO clip is never active.
  • On-policy verification: Across all 90 main-run steps, actor/pg_clipfrac and actor/ppo_kl were identically zero.The clipped surrogate therefore reduced to the plain policy gradient throughout the main run.

D The “w/o Strict On-Policy” Variant

The non-strict-on-policy variant consumes each rollout batch in two sequential half-batch updates while keeping most listed training settings unchanged. Its interpretation is limited because halving the mini-batch also changes update count, gradient-estimate variance, optimizer dynamics, and effective step size.

  • Configuration: The variant halves the gradient mini-batch, consuming each rollout batch in two sequential updates instead of one.It uses a train batch of 88 tasks and a mini-batch of 44, with no replay buffer, cross-iteration rollout reuse, or asynchronous generation.
  • Interpretation: The variant is not a single-variable contrast because it doubles optimizer updates, halves trajectories per gradient estimate, and changes Adam moment and effective-step trajectories.The paper therefore does not attribute the endpoint gap to policy lag alone.
  • Interpretation: Separating policy lag from the other changes would require same-mini-batch, double-update and matched-update-count controls, which were not run.
  • Diagnostics: The variant’s importance-ratio diagnostics were small: clip fraction averaged 1.5 × 10−3, peaked at 2.7 × 10−3, and measured policy KL averaged 1.6 × 10−4.Under 0.3% of tokens were ever clipped, so stale-ratio throttling was not the binding constraint.
  • Results: At step 90, variant entropy fell to 0.036 versus 0.217 for the main run, while Test-Normal mean@4 was 62.1 versus 79.5.The entropy collapse was quantitatively indistinguishable from removing the KL anchor, which produced entropy 0.038.

E Per-Variant Step-90 Diagnostics

Table A2 links late-training performance to policy stability: entropy tracks final score, while removing KL produces severe optimization instability. The ablation evidence is limited by one training run per variant and substantial evaluation noise for small differences.

  • Final entropy orders the ablation variants almost exactly as final score does, supporting policy drift as a diagnostic explanation.
  • 4.7 × 10^4 gradient norm in the no-KL run is four orders of magnitude above every anchored run at step 90.
  • One training run per variant prevents confidence-level separation of the reported ablation deltas.
  • Approximately 0.5 TGC points is the standard error for a 4-run mean over 168 Test-Normal tasks, so one- or two-point differences approach evaluation noise.
  • The dense-reward gap of −1.8 is suggestive rather than established, whereas −16 and −17 entries lie far outside the estimated evaluation noise.

F Compute and Sampling Cost

CANOPY’s minimalist design reduces moving parts, not compute: the main AppWorld run used hundreds of thousands of trajectories and substantial multi-node GPU time. The n=8 comparison does not isolate coverage from total sampling because it uses one quarter as many trajectories at matched steps.

  • The main AppWorld run consumed 259,200 trajectories across 90 steps, 90 tasks, and rollout group size n=32.
  • 3.47 × 10^9 tokens were processed across generation and training.
  • 42.0 hours on 8 nodes × 8 GPUs corresponded to approximately 2,690 GPU-hours, with generation dominating early step time.
  • Mean trajectories contained 10,979 response tokens over 42.5 assistant turns, while the 32k response cap bound only 0.18% on average.
  • The n=8 ablation used one quarter of the trajectories at matched steps, so its −16.4 cannot separate coverage from total sampling.

G Environment Reliability and Fault Quarantine

The training environment uses server-side isolation, repair, and quarantine rules to distinguish agent-induced failures from infrastructure faults. These reliability choices materially affect measured performance, but their attribution and exclusion rates remain imperfectly characterized.

  • The serving architecture uses per-worker memory caps, automatic recycling, request timeouts, worker repair, and episode-level state isolation.
  • Agent-induced timeouts, memory exhaustion, and episode-local corruption are scored as failures rather than quarantined.
  • Quarantine is applied before group scoring, so excluded episodes contribute neither rewards nor tokens to the loss denominator.
  • The serving-layer attribution is heuristic: misclassification can bias groups optimistically or conservatively, and exact attribution is unavailable for arbitrary agent code.
  • No per-step quarantine count was logged, preventing an exclusion-rate report or the proposed sensitivity retraining experiment.
  • Comparable runs reached 73–74 after stabilization versus approximately 47 TGC before silent environment corruption, but the shift cannot be decomposed by mechanism.

H SWE-bench Configuration and Data Construction

The SWE-bench transfer constructs a repository-disjoint training set from SWE-rebench and evaluates all 500 Verified instances conservatively. It preserves CANOPY’s core principles while adapting rollout and hyperparameters to the more expensive software-repair setting.

  • Data construction: Training filters require locally materializable Docker images, repository-level disjointness from SWE-bench Verified, and prompts of at most 4,096 tokens.
  • Data construction: The filtered SWE-rebench training set contains 5,639 tasks spanning 1,667 repositories with mean prompt length 1,821 tokens.
  • Evaluation: SWE-bench Verified evaluation uses all 500 instances with mini-swe-agent, scoring five unavailable Docker images as zero rather than excluding them.
  • Configuration: The transfer retains strictly on-policy updates, outcome-only sparse rewards, KL anchoring, and token-level pooled loss, while using rollout group size n=16.

I Evaluation Protocol

Evaluation separates training, scaled, and leaderboard configurations, using official harnesses and held-out tests across code-executing environments with automatic verifiers.

  • AppWorld configurations: AppWorld evaluations use the fixed step-90 checkpoint and distinguish training, scaled, and leaderboard configurations.The training setup uses 50 turns and 32k responses; the scaled and leaderboard setups use 100 turns and 61k via YaRN.
  • AppWorld configurations: The leaderboard configuration uses the scaled setup with mean@1, the official AppWorld harness, and unit tests.
  • Evaluation scope: Both evaluated domains are code-executing environments with automatic state- or test-based verifiers.The outcome-only RL evidence is therefore scoped to interactive tasks whose success can be checked programmatically.
  • Evaluation scope: All evaluation uses the official AppWorld and SWE-bench harnesses with held-out unit tests.
Loading 2609.01245v1…