PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Distilled Agentic Reinforcement Learning

Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

arXiv:2605.15155v1cs.LGcs.AIcs.CL

TL;DR

Long-horizon agent RL offers only coarse trajectory-level supervision, while OPSD becomes unstable and unreliable when transferred to multi-turn training. SDAR makes OPSD a gated auxiliary objective, preserving RL as the backbone, and improves over GRPO across ALFWorld, Search-QA, and WebShop while avoiding naive GRPO+OPSD instability.

  • Problem

    OPSD does not transfer cleanly to multi-turn agents because supervision becomes unstable and negative privileged-teacher signals require asymmetric treatment.

  • Method

    SDAR preserves verifier-driven RL as the primary objective and selectively gates token-level OPSD distillation with a smooth adaptive mechanism.

  • Results

    +9.4% on ALFWorld, +7.0% on Search-QA, and +10.2% on WebShop-Acc over GRPO for 7B models, while avoiding naive GRPO+OPSD instability.

  • Takeaways & Limitations

    SDAR consistently outperforms pure RL and hybrid RL–OPSD baselines across three benchmarks and three model scales.

  • Takeaways & Limitations

    The privileged training context is available only to the teacher branch and not accessible at test time.

Abstract

from arXiv · show

Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.

1 Introduction

SDAR positions RL as the primary optimization backbone and OPSD as a selectively gated auxiliary objective to address multi-turn instability and asymmetric privileged guidance. Across Qwen2.5 and Qwen3 agents, it improves over GRPO, avoids naive GRPO+OPSD instability, and outperforms hybrid RL–OPSD methods.

  • Motivation: Multi-turn agents face compounding interaction errors because each action changes future observations and responses remain in context for later decisions.This extended-horizon setting makes agentic post-training different from static single-turn reasoning.
  • Challenges: OPSD supplies dense token-level teacher guidance, but student drift makes supervision unreliable, causing rising per-turn KL divergence and catastrophic task degradation.TCOD addresses this with curriculum learning, but relies on rigid temporal schedules or trajectory-depth thresholds.
  • Challenges: Negative teacher–student gaps require cautious treatment because retrieved skills may be irrelevant, incomplete, or redundant, and teachers may fail to ground relevant skills into reliable token preferences.Positive gaps provide endorsement, whereas negative gaps can reflect either suppression-worthy tokens or instability in privileged context.
  • SDAR: SDAR leaves the verifier-driven RL policy loss untouched and selectively distills tokens through an adaptive, smooth gate instead of a rigid hand-crafted schedule.This preserves the semantics and unbiasedness of the RL advantage while controlling OPSD as an auxiliary objective.
  • Results: +9.4% on ALFWorld, +7.0% on Search-QA, and +10.2% on WebShop-Acc for 7B are SDAR’s improvements over GRPO across Qwen2.5 and Qwen3 evaluations.The method was validated on ALFWorld, WebShop, and Search-QA.
  • Results: SDAR entirely avoids the catastrophic instability of naive GRPO+OPSD and consistently outperforms Skill-SD and RLSD across all three model-agent benchmarks.These comparisons establish SDAR’s advantage over both naive OPSD augmentation and hybrid RL–OPSD methods.

2 Method

SDAR augments standard GRPO with an auxiliary on-policy self-distillation objective that uses privileged teacher context during training. A detached sigmoid gate modulates token-level guidance, emphasizing teacher-endorsed positive-gap tokens while softly attenuating negative-gap tokens.

  • Teacher Context: Privileged training-only context c+ is available to the teacher branch but not the test-time policy.Examples include reference answers, skills, and other auxiliary information unavailable at test time.
  • Objective: SDAR trains multi-turn agents with a standard GRPO policy loss plus an auxiliary on-policy self-distillation objective.The overall objective combines L_GRPO with L_SDAR.
  • Token-Level Gating: A token-level trust gate converts privileged teacher guidance into weights while leaving the verifier-driven RL objective unchanged.The gate modulates the OPSD signal on each student-sampled token.
  • Token-Level Gating: The gate uses a logistic sigmoid with sharpness β > 0, producing smooth bounded weights and controlling conservative attenuation versus strong activation.The sigmoid maps raw scores into (0, 1).
  • Gating Strategies: Gap gating strengthens distillation on positive-gap tokens endorsed by the privileged teacher and softly attenuates negative-gap tokens.The method also instantiates entropy gating for high-uncertainty positions and soft-OR gating that combines uncertainty with teacher-student gap.

3 Experiment

Experiments across ALFWorld, Search-QA, and WebShop show that SDAR improves over GRPO, internalizes privileged skills without inference-time retrieval, and generalizes better than hybrid baselines. Ablations support gap-based token gating, λ = 0.01, β = 5, and reverse-KL distillation.

  • Overall Performance: SDAR improves Qwen2.5-3B over GRPO by +9.4% on ALFWorld, +7.0% on Search-QA, and +4.7% on WebShop-Acc.On ALFWorld, the reported scores are 84.4 versus 75.0.
  • Skills Internalization: SDAR requires no external skills during inference yet surpasses skill-augmented Skill-GRPO* in most settings.Skill-GRPO falls from 80.5 to 60.2 on ALFWorld-3B when tested without skills.
  • Strong Generalization: SDAR outperforms Skill-SD and RLSD on Qwen2.5-3B ALFWorld, scoring 84.4 versus 73.4 and 79.7, respectively.On Qwen3-1.7B, Skill-GRPO drops to 21.1% on ALFWorld, below GRPO’s 46.1%, while RLSD reaches 42.2%.
  • Token-Level Gating Strategy: Teacher-Student Gap gating outperforms entropy and soft-OR gating, reaching an asymptotic success rate of ∼0.84 with a steeper climb after 100 steps.The teacher-student gap directly measures disagreement with the student’s selected token, while the alternatives are less selective.
  • Ablations: β = 5 and λ = 0.01 provide the best balance between selective distillation and the primary RL objective.Small β values include indiscriminate distillation, while λ = 0.1 lets the distillation gradient dominate policy updates.
  • Ablations: Reverse KL clearly outperforms forward KL and Jensen–Shannon divergence by concentrating probability on teacher-supported modes.This selectivity down-weights tokens assigned low probability by the frequently uncertain teacher.

4 Related Work

Prior work has advanced RL for LLM reasoning and extended agents to dynamic environments, while on-policy distillation offers token-level supervision with distinct distributional and variance trade-offs. Hybrid RL-distillation methods address complementary strengths but can suffer from rigid scheduling or unstable updates, motivating adaptive auxiliary gating.

  • RL for LLM Agents: RL for LLMs has shown strong effectiveness on verifiable reasoning tasks and is increasingly applied to autonomous agents in dynamic, open-world environments.Examples include GUI automation, gameplay, and embodied control.
  • On-Policy Distillation: On-policy distillation supervises student-generated sequences, while GKD-style methods require full-vocabulary teacher distributions and PG-style methods risk high-variance updates.These approaches minimize token-level divergences or convert discrepancies into token-level rewards, respectively.
  • Hybrid Methods: Hybrid RL-distillation methods exploit complementary strengths but can suffer from rigid hand-crafted scheduling or substantially unstable updates.The proposed contrast is a separate auxiliary objective with adaptive, bounded, token-level gating that preserves unbiased RL advantages while selectively injecting beneficial teacher signals.

5 Conclusion

SDAR reconciles RL and OPSD for multi-turn agent training with a sigmoid gate that regulates distillation intensity token by token. Across three benchmarks and three model scales, it consistently improves over pure RL and hybrid baselines while preserving RL as the optimization backbone.

  • 5 Conclusion: SDAR reconciles RL and OPSD for multi-turn agent training through a sigmoid gate that autonomously regulates each token’s distillation intensity.The gate enables selective use of teacher signals during training.
  • 5 Conclusion: SDAR preserves RL as the unbiased optimization backbone while selectively extracting beneficial teacher signals.This design combines RL’s primary optimization role with targeted distillation.
  • 5 Conclusion: Across three benchmarks and three model scales, SDAR consistently improves over both pure RL and hybrid baselines.The reported gains are consistent across the evaluated settings.

A Theoretical Analysis · A.1 Design Rationale

SDAR uses a reverse-KL-aligned divergence gap evaluated on student-sampled tokens, then bounds and detaches it to provide stable token-level weighting. This design avoids expensive full-vocabulary matching, limits excessive gradients, preserves token-importance ordering, and prevents an additional optimization pathway.

  • A.1 Design Rationale: SDAR adopts a reverse-KL-aligned gap to determine how divergence enters optimization.The design question concerns how the divergence signal should enter the optimization objective.
  • A.1 Design Rationale: Reverse KL evaluates divergence on student-sampled tokens instead of requiring computationally expensive full-vocabulary matching.This motivates choosing reverse KL rather than forward KL.
  • A.1 Design Rationale: Using the raw gap directly as a coefficient would produce overly strong, unbounded token-level gradients during early training or severe teacher-student mismatch.The instability arises when training is immature or the teacher and student disagree substantially.
  • A.1 Design Rationale: Wrapping the gap in a sigmoid transforms raw discrepancy into a bounded and monotone importance weight.The sigmoid addresses the instability of directly applying the raw gap.
  • A.1 Design Rationale: The sigmoid preserves token-importance ordering while strictly preventing gradient explosion.Bounding the transformed discrepancy retains relative importance among tokens.
  • A.1 Design Rationale: Applying stop-gradient to g_t makes the gate a confidence weight and yields a stable, first-order weighted likelihood update.Detachment prevents an additional self-referential optimization pathway.

A.2 Theoretical Properties

Theoretical propositions characterize SDAR’s auxiliary objective as a detached, gate-weighted likelihood with an adaptive token-level curriculum and bounded gradient influence. They also show why detaching the gate is necessary: otherwise, the objective introduces an unstable self-referential gradient coupling.

  • Equivalent Weighted Likelihood Form: With detached teacher signals and gate, minimizing LSDAR is equivalent up to an additive constant to maximizing token-weighted student log-likelihood.The additive constant is independent of θ.
  • Gradient Form: The auxiliary gradient is modulated directly by the bounded scalar gate gt.Under the same detachment assumptions, the gradient takes the gate-weighted likelihood form.
  • Monotonicity and Smoothness of the Gate: Larger discrepancies receive stronger weights because gt = σ(β∆t) is strictly increasing in ∆t, creating an online token-level curriculum.The gate is also described as smooth through its derivative.
  • Bounded Auxiliary Gradient: The gate cannot amplify the auxiliary gradient beyond the unweighted likelihood gradient when each token gradient norm is bounded by Bt.The bound follows from the triangle inequality and 0 < gt < 1.
  • Effect of Not Detaching the Gate: Without stop-gradient on the gate, the token loss introduces an unstable self-referential coupling term into the gradient.The resulting gradient includes both the gate derivative contribution and the ordinary gate-weighted likelihood term.

B Algorithm

SDAR combines on-policy rollout rewards and GRPO’s clipped policy loss with token-level gated distillation from a skill-conditioned teacher. Its joint update minimizes L(θ) = LGRPO(θ) + λ · LSDAR(θ), alongside comparisons to five baseline algorithms.

  • Baselines: The comparison set includes GRPO, OPSD, Skill-SD, GRPO+OPSD, and RLSD.GRPO uses group-relative advantages; OPSD distills from frozen πref; Skill-SD uses importance-weighted K3 divergence; GRPO+OPSD adds OPSD to GRPO; RLSD reweights GRPO advantages with self-teacher gaps.
  • SDAR procedure: SDAR retrieves a skill, samples G on-policy responses, obtains environment rewards, and computes sequence-level advantages before policy optimization.The procedure supports UCB, KM, Full, and Random skill retrieval.
  • SDAR procedure: SDAR computes teacher logits using each task, retrieved skill, and sampled response, then applies token-level gated distillation.The algorithm labels this stage “Token-level gated distillation.”
  • SDAR procedure: SDAR jointly updates the policy by minimizing L(θ) = LGRPO(θ) + λ · LSDAR(θ).GRPO remains the policy-loss component while LSDAR is added with mixing coefficient λ.

C Hyperparameters

Table 3 reports the method-specific hyperparameters used for all baselines and SDAR across the experiments. It defines the optimization, distillation, gating, KL, and skill-retrieval notation used in the table.

  • Hyperparameters: Table 3 summarizes method-specific hyperparameters for all baselines and SDAR across the experiments.
  • Hyperparameters: η denotes learning rate, G group size, and ϵ PPO clip ratio.
  • Hyperparameters: λ denotes the distillation loss coefficient, β sigmoid gate sharpness, αKL KL penalty coefficient, and SRS skill retrieval strategy.KM is the specified abbreviation for keyword matching.

D Training Dynamics · E Prompt

The paper reports SDAR’s training dynamics across model scales and environments using five diagnostic metrics, and provides complete prompt templates for ALFWorld, Search-based QA, and WebShop. Skill context is included during training through retrieved skills but omitted at inference.

  • D Training Dynamics: Gate Active Ratio is measured for Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B across ALFWorld, WebShop, and Search-QA.These measurements are presented in Figure 10.
  • D Training Dynamics: Gate Mean is measured for Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B across ALFWorld, WebShop, and Search-QA.These measurements are presented in Figure 11.
  • E Prompt: The {skill context} field is populated with the retrieved skill during training and left empty at inference time.This training-versus-inference treatment is specified for the full SDAR prompt templates.
  • D Training Dynamics: OPSD Loss and Teacher-Student Gap are tracked across Qwen2.5-3B, Qwen2.5-7B, Qwen3-1.7B, ALFWorld, WebShop, and Search-QA.The OPSD Loss appears in Figure 12, while the Teacher-Student Gap appears in Figure 13.
  • D Training Dynamics: Reward Curve is tracked during training for Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B on ALFWorld, WebShop, and Search-QA.The reward trajectories are presented in Figure 14.
  • E Prompt: SDAR uses complete prompt templates for the ALFWorld, Search-based QA, and WebShop evaluation environments.The templates are provided in Figures 15–17.
  • E Prompt: The Search-based QA prompt frames the model as an expert agent answering step-by-step and includes task description, step count, and interaction history.Past search queries are wrapped in <search> tags, while returned search results are wrapped in <information> tags.
Loading 2605.15155v1…