PaperlayerView exact revision on arXiv

Source-linked AI summary

Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents

Liming Pu, Xiaoxia Li, Yifu Liu, Teng Cao, Bin Yang

arXiv:2609.01245v1cs.LGcs.AI

TL;DR

The paper asks whether outcome-only RL truly reaches a ceiling on small open models for long-horizon interactive tasks. It introduces CANOPY, which expands same-task exploration and anchors strictly on-policy updates, and reports leaderboard-leading AppWorld performance alongside gains on SWE-bench Verified.

  • Problem

    The paper examines whether apparent limitations of outcome-only RL arise from sparse rewards themselves or from distorted training procedures that cause signal starvation and policy drift.

    Source: §1 · p. 2, §3.3 · p. 4

  • Method

    CANOPY restores outcome variation with larger same-task groups and retained hard tasks, then uses on-policy updates, KL anchoring, and action-token-only loss.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4

  • Results

    86.9 Test-Normal TGC and 67.6 Test-Challenge TGC placed a CANOPY-trained Qwen3-14B at the top of the AppWorld leaderboard, while the same principles improved Qwen3.5-9B by 16.6 points on SWE-bench Verified.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The results support internalizing some long-horizon domain capabilities into the weights of a small open model rather than assembling them through inference-time scaffolding.

    Source: §4.2 · p. 6, §5 · p. 7

    The AppWorld training pool contains only 90 standardized tasks, so larger, more diverse, and more complex distributions are needed to test whether the gains persist.

    Source: §5 · p. 7

Abstract

from arXiv · show

Reinforcement learning is a natural way to post-train LLM agents for long-horizon interactive tasks judged only by end-of-task verification, yet a shared belief holds that outcome-only RL soon hits a ceiling on small open models. Recent work therefore compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration. We argue the ceiling is an artifact of two failures of common practice. Signal starvation: group-relative RL with sparse outcome-only rewards yields a gradient only when a task's rollout group mixes successes and failures, so under-scaled exploration silences exactly the hardest, most instructive tasks. Policy drift: squeezing many updates out of a small task pool degrades the policy itself, as an unanchored objective lets the sampling distribution collapse exactly when saturation has already made informative groups rare. We present CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens, then cash in an enlarged interaction budget at test time. On AppWorld, a long-horizon interactive coding benchmark, a Qwen3-14B policy trained with CANOPY through environment interaction alone--without task-specific supervision, auxiliary credit signals, or elaborate agent scaffolding--topped the public leaderboard (Feb. 2026; Test-Normal TGC 86.9, Test-Challenge 67.6), and the same design principles lift Qwen3.5-9B on SWE-bench Verified by 16.6 points. Agentic RL alone thus internalizes long-horizon capability directly into a small open model; we plan to release the complete training stack at https://github.com/AlibabaResearch/SignalCoverageRL.

1 Introduction

The paper argues that outcome-only RL can train small open models for long-horizon interactive tasks, and attributes the apparent ceiling to signal starvation and policy drift rather than an inherent limit of the reward.

Motivation

  • Outcome-only RL is presented as a route for internalizing domain interaction skills into a lightweight policy without external machinery.The setting involves long-horizon interaction and end-of-task verification.Source: §1 · p. 1

Diagnosis

  • The paper identifies distorted training trajectories as the source of the apparent ceiling, including shortened horizons, filtered hard tasks, few rollouts, stale reuse, and substituted rewards.These practices deprive the policy of complete self-generated trajectories and can collapse the learning signal.Source: §1 · p. 2

Signal starvation

  • Group-relative outcome RL produces useful gradient only when a task’s rollout group mixes successes and failures, leaving small groups on hard tasks mostly silent.The paper defines signal coverage as Psig(p, n) = 1 − p^n − (1 − p)^n.Source: §1 · p. 2

Policy drift

  • Repeatedly updating on a limited task pool can narrow the sampling distribution and reduce exploration when informative groups are already rare.The paper names this failure policy drift.Source: §1 · p. 2

CANOPY

  • CANOPY restores coverage with larger same-task groups and retained hard tasks, then limits drift through on-policy, KL-anchored updates over action tokens.The protocol uses no optimizer change or auxiliary module.Source: §1 · p. 2

Results

  • 86.9 Test-Normal TGC and 67.6 Test-Challenge TGC were achieved by Qwen3-14B on AppWorld, while Qwen3.5-9B improved by 16.6 points on SWE-bench Verified.The AppWorld result was reported at the top of the public leaderboard in February 2026.Source: §1 · p. 2

2 Related Work

Related work largely modifies rewards, estimators, or inference-time scaffolding around interactive agents; this paper instead treats exploration coverage as the missing ingredient in group-relative RL.

Coding agents and agentic RL

  • Interactive coding agents span software engineering, web and research tasks, and application operation, with AppWorld serving as the primary testbed here.Software repair is used as the transfer domain.Source: §2 · p. 2

RL algorithms and failure modes

  • PPO- and GRPO-family methods dominate agent post-training, with variants changing objectives, importance ratios, grouping, or credit assignment.GRPO replaces a learned critic with group-relative rollouts.Source: §2 · p. 2

Novelty

  • Existing methods often fix estimators or rewards while leaving exploration treated as given.The paper claims that scaling exploration can remove the need for such fixes.Source: §2 · p. 3

Policy training on AppWorld

  • AppWorld policy-training routes include plain outcome RL, refined credit signals, and other methods that alter reward or credit assignment.Examples include trajectory-graph redistribution, execution-process signals, and self-generated curricula with step-level judges.Source: §2 · p. 3

Training-free AppWorld systems

  • Training-free systems compensate around fixed frontier models using orchestration, retrieval, skills, memory, playbooks, or test-time debugging.The paper contrasts these systems with capability internalized in model weights.Source: §2 · p. 3

3 The CANOPY Protocol

CANOPY is a minimal agentic RL protocol that restores sparse-reward signal through broader same-task exploration and controls repeated-update drift with strictly on-policy, anchored training.

3.1 Preliminaries: The Agentic RL Loop

  • The agentic RL loop samples trajectories for tasks, scores them with held-out unit tests, converts rewards to group-relative advantages, and updates the policy that generated them.No learned critic is required.Source: §3.1 · p. 3

3.2 Signal Starvation, and Explore More

  • Sparse outcome rewards provide a nonzero group-relative gradient only when rollouts contain both successes and failures.All-success and all-failure groups contribute no gradient.Source: §3.2 · pp. 3–4
  • Signal coverage rises with group size on hard tasks but collapses when per-rollout success probability is near either extreme.Figure 2 reports 34% coverage at n=8 and 81% at n=32 for p=0.05.Source: Figure 2 · p. 3, §3.2 · p. 4
  • CANOPY sizes groups from pilot-estimated difficulty, retains the hardest tasks, and avoids per-turn generation caps so error-recovering episodes are not truncated.The sizing rule is explicitly a first-order hardware-budget heuristic rather than an optimal prescription.Source: §3.2 · p. 4

3.3 Policy Drift, and Drift Less

  • Policy drift arises when repeated updates on a small task pool narrow the sampling distribution, reuse stale rollouts, and distort contributions from differently sized trajectories.An unanchored objective compounds signal starvation as saturation makes informative groups rare.Source: §3.3 · p. 4
  • CANOPY answers drift by using one update over the whole rollout batch, a KL anchor to the base model, pooled action-token normalization, and an action-token mask.Environment tokens receive no policy gradient, and the sparse reward is retained rather than replaced by a proxy.Source: §3.3 · pp. 4–5
  • Fault quarantine excludes only serving-layer failures from training treatment, while agent-induced terminations remain genuine behavioral failures scored zero.This separates infrastructure faults from failures caused by the agent.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate CANOPY on AppWorld and SWE-bench Verified, showing strong leaderboard performance, interpretable training dynamics, budget transfer toward difficult tasks, and cross-domain improvement.

Main results

  • Qwen3-14B reaches 86.9 Test-Normal TGC and 67.6 Test-Challenge TGC as a single policy without external agent scaffolding.The policy uses one checkpoint at inference without orchestration, skill libraries, retrieved memory, or test-time debugging.Source: §4.2 · p. 6

Training dynamics

  • Rollout logs show informative groups shrinking as reward saturates, while L3 remains informative after easier tiers become silent.Figure 3 uses n=32 groups across 90 training steps and reports bootstrap confidence intervals.Source: §4.3 · p. 6, Figure 3 · p. 6
  • The KL anchor preserves late-stage exploration: anchored entropy remains 0.217 with Dev reaching 87.3, whereas unanchored entropy falls to 0.038 and Dev stalls at 81.6.The divergence appears past approximately step 70.Source: §4.3 · p. 6, Figure 4 · p. 6

Budget transfer

  • Increasing the interaction budget raises Test-Normal performance from 79.5 to 83.2 mean@4, with gains concentrated on L3 and unseen applications.The base also improves from 22.8 to 32.4, but remains below the trained policy at the smaller budget.Source: §4.3 · p. 6, Figure 5 · p. 7

Ablations

  • Ablations identify rollout-group coverage and strictly on-policy updates as the largest contributors, costing −16.4 and −17.4 when weakened.Removing the KL anchor costs −7.0, token-level loss costs −5.4, and densifying reward costs −1.8.Source: §4.3 · pp. 6–7

Transfer to software repair

  • The same design principles improve Qwen3.5-9B on SWE-bench Verified from 31.3 to 47.9 mean@4 and from 43.8 to 58.0 best@4.Budget transfer adds +2.3 in the reported transfer setting.Source: §4.4 · p. 7, Table 4 · p. 7

5 Conclusion and Future Work

The paper argues that outcome-only RL can internalize long-horizon interaction capabilities in small open models when training restores informative outcome variation and limits policy drift. It reports leaderboard leadership on AppWorld and identifies broader environment scaling, algorithmic efficiency, and domain mid-training as directions for testing and extending the approach.

Conclusion

  • CANOPY addresses informative outcome variation and policy drift with a simple explore-more, drift-less recipe.The approach uses interaction-based post-training rather than elaborate inference-time machinery.Source: §5 · p. 7
  • A single open 14B policy trained with CANOPY reached the top of the AppWorld leaderboard at submission time.The policy internalized capabilities in its weights rather than relying on external orchestration, skill libraries, or retrieved memory.Source: §5 · p. 7
  • The same design principles improved software repair, supporting capability internalization beyond application operation.The passage states this as a result without specifying the software-repair score.Source: §5 · p. 7

Future work

  • AppWorld training uses only 90 tasks, so larger, more diverse, and more complex task distributions are needed to test whether the gains persist and expose new limits.The proposed expansions include multilingual software engineering and harder benchmarks.Source: §5 · p. 7
  • More sample-efficient RL could make environment scaling affordable while adapting group size or task sampling to the current success probability p.The paper frames this as a way to manage the exploration–exploitation trade-off more directly.Source: §5 · p. 7
  • Domain mid-training could improve base-model coverage and raise the attainable ceiling of outcome-based RL while complementing interaction-based post-training.Source: §5 · p. 7

A Full AppWorld Training Configuration

The main AppWorld configuration collects large same-task rollout groups and performs one gradient update without reusing rollouts, while sparse-group advantages explain why isolated successes can dominate gradients. The implementation also verifies strict on-policy behavior through unit importance ratios and logged PPO statistics.

Configuration

  • The rollout phase collects 90 × 32 = 2,880 trajectories, followed by exactly one gradient step over all of them.Because the rollout batch and PPO mini-batch are both 90 tasks, no rollout is reused.Source: A Full AppWorld Training Configuration · p. 10

Advantage magnitudes

  • For a group with k successes out of n, a successful rollout receives a standardized advantage based on the group’s sparse success outcomes.The derivation uses the empirical mean and population standard deviation of the binary rewards.Source: B Advantage Magnitudes in a Sparse,… · p. 10
  • At k = 1, the successful rollout receives Â+ = √n −1 while each failure receives Â−= −1/√n −1.The single success is weighted n −1 times more heavily than each failure and carries essentially the whole group gradient.Source: B Advantage Magnitudes in a Sparse,… · p. 10

On-policy verification

  • With one pass over a gradient batch equal to the rollout batch, the sampling and learning policies coincide, making ρi,t exactly 1 and measured policy KL exactly 0.Under this configuration, the PPO clip is never active.Source: C Verifying That Updates Are Strictly… · p. 10
  • Across all 90 main-run steps, actor/pg_clipfrac and actor/ppo_kl were identically zero.The clipped surrogate therefore reduced to the plain policy gradient throughout the main run.Source: C Verifying That Updates Are Strictly… · p. 10

D The “w/o Strict On-Policy” Variant

The non-strict-on-policy variant consumes each rollout batch in two sequential half-batch updates while keeping most listed training settings unchanged. Its interpretation is limited because halving the mini-batch also changes update count, gradient-estimate variance, optimizer dynamics, and effective step size.

Configuration

  • The variant halves the gradient mini-batch, consuming each rollout batch in two sequential updates instead of one.It uses a train batch of 88 tasks and a mini-batch of 44, with no replay buffer, cross-iteration rollout reuse, or asynchronous generation.Source: D The “w/o Strict On-Policy” Variant · p. 10

Interpretation

  • The variant is not a single-variable contrast because it doubles optimizer updates, halves trajectories per gradient estimate, and changes Adam moment and effective-step trajectories.The paper therefore does not attribute the endpoint gap to policy lag alone.Source: D The “w/o Strict On-Policy” Variant · p. 10
  • Separating policy lag from the other changes would require same-mini-batch, double-update and matched-update-count controls, which were not run.Source: D The “w/o Strict On-Policy” Variant · p. 10

Diagnostics

  • The variant’s importance-ratio diagnostics were small: clip fraction averaged 1.5 × 10−3, peaked at 2.7 × 10−3, and measured policy KL averaged 1.6 × 10−4.Under 0.3% of tokens were ever clipped, so stale-ratio throttling was not the binding constraint.Source: D The “w/o Strict On-Policy” Variant · p. 10

Results

  • At step 90, variant entropy fell to 0.036 versus 0.217 for the main run, while Test-Normal mean@4 was 62.1 versus 79.5.The entropy collapse was quantitatively indistinguishable from removing the KL anchor, which produced entropy 0.038.Source: D The “w/o Strict On-Policy” Variant · p. 10

E Per-Variant Step-90 Diagnostics

Table A2 links late-training performance to policy stability: entropy tracks final score, while removing KL produces severe optimization instability. The ablation evidence is limited by one training run per variant and substantial evaluation noise for small differences.

F Compute and Sampling Cost

CANOPY’s minimalist design reduces moving parts, not compute: the main AppWorld run used hundreds of thousands of trajectories and substantial multi-node GPU time. The n=8 comparison does not isolate coverage from total sampling because it uses one quarter as many trajectories at matched steps.

G Environment Reliability and Fault Quarantine

The training environment uses server-side isolation, repair, and quarantine rules to distinguish agent-induced failures from infrastructure faults. These reliability choices materially affect measured performance, but their attribution and exclusion rates remain imperfectly characterized.

H SWE-bench Configuration and Data Construction

The SWE-bench transfer constructs a repository-disjoint training set from SWE-rebench and evaluates all 500 Verified instances conservatively. It preserves CANOPY’s core principles while adapting rollout and hyperparameters to the more expensive software-repair setting.

Data construction

Evaluation

Configuration

I Evaluation Protocol

Evaluation separates training, scaled, and leaderboard configurations, using official harnesses and held-out tests across code-executing environments with automatic verifiers.

AppWorld configurations

  • AppWorld evaluations use the fixed step-90 checkpoint and distinguish training, scaled, and leaderboard configurations.The training setup uses 50 turns and 32k responses; the scaled and leaderboard setups use 100 turns and 61k via YaRN.Source: I Evaluation Protocol · p. 12
  • The leaderboard configuration uses the scaled setup with mean@1, the official AppWorld harness, and unit tests.Source: I Evaluation Protocol · p. 12

Evaluation scope

  • Both evaluated domains are code-executing environments with automatic state- or test-based verifiers.The outcome-only RL evidence is therefore scoped to interactive tasks whose success can be checked programmatically.Source: J Scope of the Claims · pp. 12–13
  • All evaluation uses the official AppWorld and SWE-bench harnesses with held-out unit tests.Source: K Reproducibility · p. 13

Related papers

Loading 2609.01245v1…