PaperlayerView exact revision on arXiv

Source-linked AI summary

SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning

Qirui Liu, Wei Cao, Yu Mi, Yichen Sun, Xuekun Zhang, Lei Liu, Yue Shen, Zhixuan Chu

arXiv:2610.38391v1cs.AI

TL;DR

Outcome rewards offer little information about intermediate decision quality, and trajectory-level skills can affect unrelated steps. SCOPE generates corrective skills for selected errors and distills them selectively; results show improvements across tasks, including unseen ALFWorld.

  • Problem

    Outcome-based RL gives little indication of which intermediate decisions were sound or need correction, and trajectory-level skills can shift teacher preferences at unrelated steps.

    Source: §1 · p. 2

  • Method

    After supervised warm-up, SCOPE analyzes its own trajectories, generates step-specific corrective skills, and distills their guidance only at selected error steps.

    Source: §1 · pp. 2–3

  • Results

    SCOPE improves consistently across ALFWorld and WebShop; on unseen ALFWorld, it achieves 89.6% success, 23.2 points above GRPO with 46.7% fewer interactions.

    Source: §1 · p. 2, §5 · p. 15

  • Takeaways & Limitations

    SCOPE aligns corrective supervision with decisions that require it, without additional guidance at inference time.

    Source: §5 · pp. 14–15

Abstract

from arXiv · show

Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.

1 Introduction

SCOPE addresses coarse outcome feedback and off-target trajectory-level distillation with self-generated, step-specific correction, bounded updates, and an efficiency reward. Experiments report improvements across tasks and backbones, including on unseen ALFWorld tasks.

1 Introduction

  • Outcome rewards indicate task completion but give little signal about intermediate decision quality, while trajectory-level skills can shift probabilities at unrelated steps.The paper frames identifying critical errors independently of trajectory outcomes as its key question.Source: §1 · p. 2
  • SCOPE attains 97.7% and 85.2% success, outperforming SEED by 5.9 and 6.3 percentage points, respectively.The reported improvements are consistent across ALFWorld and WebShop and across three backbones.Source: §1 · p. 2
  • On unseen ALFWorld, SCOPE achieves 89.6% success (+23.2 points) with 46.7% fewer interactions; Qwen3-32B reaches 99.2% seen and 94.0% unseen success.The Qwen3-32B seen-task result is reported as 14.0 points above GRPO.Source: §1 · pp. 2–3
  • After supervised warm-up, SCOPE analyzes its own successful and failed trajectories, generates step-specific corrective skills, and distills guidance only at selected error steps.The policy learns to locate errors and generate corrective skills before online reinforcement learning.Source: §1 · pp. 2–3
  • A bounded signed distillation coefficient limits extreme auxiliary updates while preserving positive and negative corrections, and a success-relative pairwise length reward encourages efficiency.The paper reports occasional abrupt performance collapses with unbounded OPSD updates.Source: §1 · pp. 2–3

2 Related Work

Related work spans outcome-based reinforcement learning, hindsight skills, and on-policy self-distillation. SCOPE complements outcome-based RL by localizing errors and restricting corrective distillation to selected steps.

2.1 Reinforcement Learning for LLM Agents

  • GiGPO [6] and GraphGPO [5] refine credit assignment, while SCOPE complements outcome-based RL with hindsight error localization rather than re-estimating step-level RL advantages.GRPO [22] compares trajectory outcomes within a sampled group.Source: §2.1 · p. 3

2.2 Hindsight Skills for LLM Agents

  • Reflexion [23], Voyager [25], and ExpeL [41] demonstrate reflection, reusable skills, or memory; unlike SEED [29], SCOPE restricts self-generated skill guidance to identified errors.SCOPE applies this approach to errors in both successful and failed trajectories.Source: §2.2 · p. 3

2.3 On-Policy Self-Distillation

  • Unlike divergence-based weighting, SCOPE identifies critical errors through self-generated hindsight analysis, restricts distillation to those steps, and bounds signed teacher–student updates.Figure 2 depicts supervised initialization followed by online rollout analysis and selective self-distillation with efficiency-aware GRPO.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

3 Method

SCOPE combines outcome-based reinforcement learning with corrective self-distillation at selected error steps. Its method uses self-bootstrapped trajectory analysis, bounded signed updates, and a success-relative reward for efficient task completion.

3 Method

  • SCOPE adds corrective supervision only at selected error steps, alongside outcome-based reinforcement learning and a success-relative length reward.A supervised warm-up prepares the policy to analyze trajectories and generate corrective skills; the policy then serves as actor and analyzer online.Source: §3 · p. 4

3.1 Agent Interaction and Policy Snapshots

  • The agent generates a response for each observation from its interaction history, and a completed trajectory records its environment-interaction steps and task reward.Interaction length and response-token length are distinct quantities; the policy snapshot supplies the actor, analyzer, and privileged teacher for an outer iteration.Source: §3.1 · p. 4

3.2 Self-Bootstrapped Error Analysis

  • The policy analyzes successful and failed trajectories, identifies errors, and assigns step-specific corrective skills; unselected steps receive no auxiliary distillation signal.For failed trajectories, the analyzer can also use successful action sequences from the same on-policy task group as reference context.Source: §3.2 · p. 5

3.3 Bounded Signed Self-Distillation

  • Teacher scores are cached before actor optimization, while student scores are recomputed as parameters change; this can turn moderate initial gaps into extreme training discrepancies.Unselected steps do not enter the distillation loss or its token count; if no tokens are selected, the loss is zero.Source: §3.3 · pp. 5–6
  • A bounded signed coefficient limits extreme teacher–student-gap updates while retaining both positive reinforcement and negative correction.The coefficient’s magnitude is bounded, and its sign preserves the direction of the gap; the loss averages over selected tokens only.Source: §3.3 · pp. 6–7

3.4 Success-Relative Pairwise Efficiency

  • A success-relative pairwise reward compares successful trajectories for the same task by environment-interaction length, providing a policy-relative efficiency reference.The comparison is scale-invariant, and groups with no successes receive no efficiency bonus.Source: §3.4 · p. 7

3.5 Joint Policy Optimization

  • GRPO normalizes each task group’s task-and-length rewards, then assigns each trajectory’s resulting advantage to all its steps and tokens.The joint objective adds selective OPSD to the RL objective, balancing outcome-based learning and corrective supervision at selected error steps.Source: §3.5 · pp. 7–8

4 Experiments

The experiments evaluate SCOPE on ALFWorld and WebShop across model backbones, testing overall performance, error localization, component effects, training behavior, generalization, and scaling.

4.2 Main Results

  • Across backbones, SCOPE leads WebShop success and leads or ties on ALFWorld; gains over GRPO reach 46.1 ALFWorld points and 40.6 WebShop points.Against trajectory-level SEED, SCOPE gains 5.9 and 2.3 ALFWorld points on Qwen2.5-3B and Qwen2.5-7B, respectively, and 6.3 WebShop points on both.Source: Table 1 · p. 9, §4.2 · pp. 9–10

4.3 Diagnosing Trajectory-Level Supervision

  • WebShop trajectories show outcome–step mismatch: 29.8% of successful-trajectory steps are erroneous, while 42.4% of failed-trajectory steps are correct.Skill-unrelated steps account for 57.9% of nonzero sampled-token teacher–student gaps, whose magnitudes are similar to gaps at skill-relevant steps.Source: §4.3 · p. 10

4.4 Ablation Studies

  • Removing SFT initialization lowers overall success from 98.4% to 92.2%, while trajectory-level guidance and skill-bank guidance reach 96.1% and 95.3%, respectively.The trajectory-level variant jointly changes skill granularity and coverage; its result supports their design together.Source: Table 2 · p. 11, §4.4 · p. 11

4.4 Ablation Studies; 4.5 Training Dynamics

  • Removing the pairwise length reward lowers success from 98.4% to 93.0%; SCOPE also reduces interaction steps early and maintains low counts later, outperforming GRPO in efficiency.SCOPE reaches 65.1% WebShop success at step 60, exceeding GRPO’s final 63.0% at step 150, with 60.0% fewer online updates.Source: Figure 5 · p. 11, §4.4 · p. 11, §4.5 · p. 12

4.6 Parameter Sensitivity

  • At step 180, success falls to 89.8% with α = 1, 94.5% with c = 0.1, and 93.8% with τlen = 0.05, versus 98.4% for the default configuration.The sweeps also report 97.7% with α = 5, 98.4% with c = 10, and 96.1% with τlen = 0.2.Source: Figure 6 · p. 12, §4.6 · p. 12

4.5 Training Dynamics

  • Unbounded SCOPE collapses after initial improvement, whereas bounded signed distillation maintains stable training by limiting gap-driven amplification while preserving positive and negative corrections.Sharply reduced student log-probabilities produce extreme teacher–student gaps that amplify raw-gap OPSD updates during minibatch optimization.Source: §4.5 · p. 12

4.8 Generalization to Unseen Tasks; 4.9 Scaling SCOPE to Qwen3-32B

  • On unseen ALFWorld tasks, SCOPE reaches 89.6% success versus GRPO’s 66.4%, with 46.7% fewer interactions; on Qwen3-32B, it reaches 99.2% seen and 94.0% unseen success.For Qwen3-32B unseen tasks, SCOPE records 94.0% success versus 81.3% for GRPO, with 47.2% fewer interactions.Source: Figure 8 · p. 13, §4.8 · p. 13, §4.9 · pp. 13–14

4.7 Error-Localization Capability

  • SFT improves error localization for both backbones, and the 3B SFT model surpasses 7B Base in micro-F1; the SEED-style analyzer is weaker, particularly on ALFWorld.The comparison evaluates base and SFT-initialized models alongside a SEED-style analyzer on WebShop and ALFWorld.Source: Figure 7 · p. 13, §4.7 · p. 13

5 Conclusion

SCOPE addresses coarse trajectory-level supervision by localizing errors and applying corrective guidance only at selected steps. Experiments show consistent improvements, including on unseen ALFWorld tasks with fewer interactions.

5 Conclusion

  • SCOPE internalizes error localization and applies self-generated, step-specific corrective guidance only at selected error steps.Bounded signed distillation controls gap-driven updates, while success-relative pairwise rewards encourage efficient task completion.Source: §5 · p. 14
  • Experiments across ALFWorld and WebShop and three backbones show consistent improvements over outcome-based RL and strong performance against skill-guided methods.Source: §5 · p. 15
  • On unseen ALFWorld tasks, SCOPE achieves 89.6% success, outperforming GRPO by 23.2 points while reducing reported interactions by 46.7%.The conclusion links these results to aligning corrective supervision with decisions requiring correction, without additional guidance at inference time.Source: §5 · p. 15

A Theoretical Analysis

The analysis addresses bounded signed updates, irrelevant distillation tokens, and the success-relative length reward, while limiting its distillation characterization to a fixed training microbatch.

A Theoretical Analysis

A.1 Bounded Signed Distillation

The bounded signed distillation update has a log-cosh potential whose gradient matches the auxiliary update, with bounded influence under stated assumptions. Separately, irrelevant selected tokens contribute to a worst-case bound that tightens as their share falls.

A.1 Bounded Signed Distillation

  • The detached distillation update is the gradient of a log-cosh potential, giving small gaps approximately linear signed coefficients and bounded influence for large gaps.The potential is nonnegative, symmetric, and strictly convex.Source: §A.1 · pp. 17–18
  • The auxiliary gradient is bounded by cB when every token-score gradient norm is at most B.The coefficient bounds follow from the range and monotonicity of tanh.Source: §A.1 · p. 18
  • The bounds assume bounded token-score gradients and do not establish global stability for RL training.α controls local sensitivity, while c limits gap-dependent amplification.Source: §A.1 · p. 18

A.2 Influence of Irrelevant Selected Tokens

  • The off-target gradient contribution is bounded under bounded token-score gradients, and reducing the irrelevant-token fraction tightens a worst-case bound on its effect.The analysis does not assume that the analyzer always selects correctly.Source: §A.2 · pp. 18–19

A.3 Properties of Success-Relative Efficiency Rewards

The success-relative pairwise reward assigns larger bonuses to shorter successful trajectories without imposing a fixed length target. Its scores are scale-invariant, and its mean bonus across successes is conserved.

A.3 Properties of Success-Relative Efficiency Rewards

  • Shorter trajectories receive larger pairwise scores among successes, while scaling all interaction lengths by the same positive factor leaves the scores unchanged.Equal lengths receive equal scores.Source: §A.3 · p. 19
  • When every successful trajectory has reward at least as high as every failed trajectory, the shaped reward preserves success-over-failure ordering for λlen > 0.Among successes with equal original rewards, shorter trajectories receive larger shaped rewards.Source: §A.3 · p. 19
  • The pairwise reward favors shorter successes without a fixed length target, while its mean bonus remains λlen/2 across successes.The relative allocation changes with the current rollout group.Source: §A.3 · p. 19

B.1 Training Configuration

At the step-150 checkpoint, SCOPE achieves high success rates on ALFWorld and WebShop with Qwen2.5-3B and Qwen2.5-7B. The appendix also specifies the training hardware and checkpoint choices used for the reported comparisons.

B.1 Training Configuration

  • The smaller-backbone runs use two NVIDIA H200 GPUs on WebShop and four on ALFWorld; Qwen3-32B experiments use 16 NVIDIA H200 GPUs.The policy snapshot supplies both the actor and online analyzer, and teacher scores are cached before student optimization.Source: §B.1 · p. 20
  • The main comparison uses step 150, while ALFWorld ablations and parameter sweeps use step 180.Training-step savings compare the first observed checkpoints reaching GRPO’s final mean success rate.Source: §B.1 · p. 20

B.2 Validation Curves Across Backbones

  • 97.7% and 98.4% overall ALFWorld success, and 85.2% and 84.4% WebShop success, are reported for SCOPE-3B and SCOPE-7B at step 150.WebShop normalized task scores are 87.9% and 89.6%, respectively; mean interaction counts are 8.24 and 7.73 on ALFWorld, and 6.02 and 5.95 on WebShop.Source: §B.2 · p. 20

B.3 Supervised Data and Hindsight Analysis

The supervised-data pipeline annotates policy trajectories with errors and corrective skills, then uses structured summaries and skill entries as the assistant target. Failed trajectories can draw on successful action sequences as reference context.

B.3 Supervised Data and Hindsight Analysis

  • GLM-5.2 generates offline SFT annotations from task descriptions, episode outcomes, recorded observations and responses, and action-validity feedback when available.Successful trajectories are analyzed for avoidable errors; failed trajectories may also receive up to three successful action sequences from the same task as reference context.Source: §B.3 · p. 20
  • Each annotation contains a trajectory summary and at most five skill entries, with target-step indices, an error explanation, and a reusable corrective instruction.Entries may cover multiple steps when they require the same correction; the annotator is asked to preserve reasonable exploration.Source: §B.3 · p. 20
  • The data pipeline checks JSON structure, required fields, and valid, unique target indices before export.Successful reference action lists remain in the input rather than the assistant prediction target.Source: §B.3 · pp. 20–21
  • The assistant target contains only the structured summary and skill entries; when a failed group has no success, the reference block is omitted.The online analyzer uses the same analysis instructions and output schema.Source: §B.3 · p. 21

B.4 Error-Localization Evaluation Metrics

This material specifies how error-step predictions are evaluated, how motivation diagnostics are constructed, and how analyzer prompts operationalize skill extraction.

B.4 Error-Localization Evaluation Metrics

  • Predicted error-step sets are extracted from generated skill entries’ target_steps and compared with reference annotations.The evaluation reports step-level and trajectory-level metrics as percentages.Source: §B.4 · pp. 21–22

B.5 Motivation Diagnostics

  • The WebShop diagnostic covers 1,400 trajectories, separating local decision quality from episode outcome and using GLM-5.3 for error labels.GLM-5.2 supplies the offline SFT annotations; GLM-5.3 supplies diagnostic labels.Source: §B.5 · p. 22
  • 57.9% of nonzero-gap tokens occur at skill-unrelated steps; the reported positive and negative shares among those tokens are 46.1% and 53.9%.The analysis scores 33,602 steps and 4,035,655 response tokens across 177 task groups; uncertainty intervals use 2,000 task-group bootstrap resamples.Source: §B.5 · p. 22

C SFT and Analyzer Prompt Templates

  • The analyzer prompt uses shared instructions and a JSON schema, with task and trajectory placeholders replaced by recorded context.It includes a reference block only for failures with available successful references.Source: C SFT and Analyzer Prompt Templates · p. 22

System message

  • The system message directs the analyzer to extract concise, policy-facing skills for specific error steps and return valid JSON.The required output is JSON only.Source: System message · p. 22

Shared analysis instructions and output schema

  • Shared instructions request a trajectory summary and up to five step skills, with errors grounded in information available at each target step and reasonable exploration excluded.Skills should be concise, reusable, and concrete; if no clear error is supported, return an empty step_skills list using the specified schema.Source: Shared analysis instructions and… · pp. 22–23

C.1 Outcome-Specific User Template

  • Successful and failed trajectories use the same template, with outcome-specific instructions and successful references included only for failures when available.For failures, reference actions are hints; differences alone do not establish mistakes.Source: §C.1 · p. 23, User template: successful / failed… · p. 23

User template: successful / failed trajectory

  • The episode context supplies the task description, success or failure status, and recorded interaction steps with observations, responses, and action-validity flags.Failure prompts may also include successful reference action sequences.Source: User template: successful / failed… · p. 23
Loading 2610.38391v1…