PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, Honggang Qi

arXiv:2604.10674v1cs.LGcs.AIcs.CL

TL;DR

Multi-turn agent RL suffers from sparse rewards and long horizons, while fixed privileged information misses diverse valid strategies and naive RL-distillation coupling can collapse. Skill-SD converts trajectories into teacher-only skills, uses importance-weighted reverse-KL distillation, and synchronizes the teacher dynamically. It improves vanilla GRPO on AppWorld and Sokoban and also substantially exceeds vanilla OPD.

  • Problem

    Multi-turn agent RL has sparse, delayed rewards and high sample complexity, while fixed privileged information cannot represent diverse valid strategies and naive self-distillation-RL coupling can collapse.

  • Method

    Skill-SD summarizes completed trajectories into natural-language skills for teacher-only conditioning, while the plain-prompt student learns through importance-weighted reverse-KL distillation and dynamic teacher synchronization.

  • Results

    Skill-SD outperforms vanilla GRPO on AppWorld and Sokoban and surpasses vanilla OPD by large margins.

  • Takeaways & Limitations

    Skill-SD provides per-token supervision without constraining student exploration or requiring skills at inference time.

  • Takeaways & Limitations

    The method uses lightweight UCB retrieval and sampled-token rather than full-vocabulary distillation, and trajectory-level KL guarantees remain an open question.

Abstract

from arXiv · show

Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language skills that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%). Project page: https://k1xe.github.io/skill-sd/

1 Introduction

Skill-SD addresses sparse, delayed rewards and diverse valid strategies in multi-turn agent tasks by using trajectory-derived skills for teacher-only self-distillation, with gradient correction and dynamic synchronization for stability.

  • RL training for multi-turn agents is limited by sparse, delayed rewards and high sample complexity.
  • Prior self-distillation methods use privileged information effectively mainly when tasks have unique ground-truth answers, unlike agent tasks with diverse valid strategies.
  • Skill-SD summarizes completed trajectories into compact skills describing successful behaviors, mistakes, and workflows, conditioning only the teacher while the student acts under the plain prompt.
  • The importance-weighted reverse-KL loss corrects per-token gradient bias caused by cross-prompt distribution mismatch.
  • Periodic teacher synchronization is essential for stable training: off-policy teacher rollouts collapse mid-training, whereas frozen teachers converge to lower plateaus.
  • 64.9% accuracy on AppWorld and 62.5% on Sokoban outperform vanilla GRPO by +14.0% and +10.9%, respectively.

2 Related Work

Related work spans self-distillation, KL-estimator design, multi-turn agent RL, and trajectory-based experience or memory methods; Skill-SD is positioned across these lines.

  • On-policy distillation and self-distillation for LLMs: Prior self-distillation methods either use distillation alone or integrate distillation with reinforcement learning through dense targets, unified objectives, or selective imitation.
  • KL-regularized policy optimization and estimator design: KL-estimator research distinguishes unbiased KL value estimation from gradient optimization, because directly differentiating k3 can produce biased gradients.
  • Reinforcement learning for multi-turn LLM agents: Multi-turn agent RL work develops methods for long-horizon optimization, two-level advantage estimation, instability diagnosis, and fine-grained credit assignment.
  • Experience, reflection, and memory in agents: Experience and memory methods store reflections, reusable experience, rewritten hindsight trajectories, strategic principles, or self-improving feedback loops.

3 Method

Skill-SD combines GRPO with skill-conditioned self-distillation: the student generates plain-prompt rollouts, while a dynamically synchronized teacher uses retrieved trajectory skills to provide token-level guidance.

  • 3.1 Problem Setup and GRPO Backbone: Each task produces multi-turn student rollouts whose action-token sequences receive completion-rate rewards and fine-grained task verification.
  • 3.1 Problem Setup and GRPO Backbone: GRPO samples trajectory groups, normalizes rewards within each group, and applies token-mean reduction with asymmetric clipping bounds.
  • 3.2 Skill-Conditioned Teacher and Dynamic Self-Evolution: Because tasks admit many valid action sequences, Skill-SD stores compact summaries of successes, mistakes, and workflows instead of canonical trajectories.
  • 3.2 Skill-Conditioned Teacher and Dynamic Self-Evolution: The student acts on the plain prompt, while the teacher receives retrieved skills through a skill-augmented prompt; teacher parameters can synchronize from the latest student checkpoint.
  • 3.2 Skill-Conditioned Teacher and Dynamic Self-Evolution: Skill retrieval selects one task-local skill with UCB statistics, and an auxiliary LLM summarizes trajectories without entering the optimization objective.
  • 3.3 Importance-weighted reverse-KL loss: The sampled-token reverse-KL objective re-scores student-generated tokens under current student and skill-conditioned teacher distributions, using importance correction for gradient correctness.
  • 3.3 Importance-weighted reverse-KL loss: Sampled-token distillation avoids storing full-vocabulary logits at every position, reducing memory overhead for long-horizon traces.
  • 3.4 Training Objective and Procedure: The SDL loss decreases by 59.3% during representative AppWorld training as student and teacher token distributions largely converge.

4 Experiments

Skill-SD is evaluated across AppWorld and Sokoban using on-policy student rollouts, dynamic teacher synchronization, and trajectory-derived skills. It achieves strong benchmark performance while ablations show that student-owned rollouts and dynamic synchronization are central to stable training.

  • Benchmarks: Skill-SD evaluates API coordination and adaptive replanning in AppWorld alongside irreversible spatial planning in Sokoban.AppWorld uses real-world API coordination and recoverable errors, whereas Sokoban tests long-horizon planning where a single wrong push can make a puzzle unsolvable.
  • Main results: 64.9% AppWorld accuracy and 62.5% Sokoban accuracy surpass Vanilla GRPO by +14.0% and +10.9%, respectively.Skill-SD also reaches 84.9% AppWorld completion and 71.1% Sokoban completion.
  • Training dynamics: Skill-SD improves steadily on both benchmarks, while Skill-Augmented GRPO overfits on AppWorld and Vanilla OPD remains at low performance.Skill-Augmented GRPO has the highest AppWorld training score but only 42.1% validation accuracy, compared with Skill-SD’s 64.9%.
  • Training setup: Skill-SD uses student-owned on-policy trajectories, teacher re-scoring under skill-augmented prompts, and periodic teacher synchronization.The algorithm samples student trajectories under the plain prompt, computes rewards and advantages, re-scores them with the teacher, and combines GRPO with SDL.
  • Ablation analysis: Teacher-owned off-policy rollouts collapse during mid-training, reaching 12.5% Sokoban accuracy and 45.6% AppWorld accuracy.Distribution mismatch makes importance ratios unstable; the effect is more severe in Sokoban because errors are less recoverable.
  • Ablation analysis: Dynamic synchronization improves on-policy training by +15.8% AppWorld accuracy and +12.5% Sokoban accuracy over a frozen teacher.The frozen on-policy configuration remains stable but converges to a lower plateau.
  • Ablation analysis: λ = 0.001 gives the best validation completion, whereas λ = 0.01 overregularizes and λ = 0.0005 provides insufficient teacher guidance.The SDL coefficient balances RL and distillation; λ = 0.001 acts as a mild shaping term without dominating the combined gradient.
  • Overhead: Trajectory summarization adds external LLM API calls as Skill-SD’s primary computational overhead relative to Vanilla GRPO.A naïve synchronous implementation would block rollout and parameter-update operations.

5 Conclusion

Skill-SD combines skill-conditioned self-distillation with GRPO for multi-turn LLM agents, using trajectory-derived skills to guide the teacher while the student acts under the plain task prompt.

  • Skill-SD provides per-token supervision without constraining the student’s exploration by separating teacher conditioning from student rollout prompts.The teacher uses analytical skills distilled from trajectories, while the student generates on-policy trajectories under the plain task prompt.
  • Dynamic teacher synchronization keeps the distillation signal calibrated as the student improves.
  • Importance-weighted SDL provides per-token unbiased gradients for the distillation loss.The paper attributes this property to Proposition 2.
  • Skill-SD substantially outperforms vanilla GRPO, skill-augmented GRPO, and pure on-policy distillation on AppWorld and Sokoban.The experiments use 90 AppWorld tasks and 96 Sokoban levels.
  • Off-policy variants suffer mid-training collapse, while frozen teachers converge to lower plateaus.

Limitations

The paper identifies three improvement directions: more semantic skill retrieval, fuller-vocabulary distillation, and trajectory-level theoretical analysis.

  • Skill retrieval uses a lightweight UCB bandit criterion rather than embedding-based semantic retrieval.A learned retrieval component may improve skill selection as the skill bank grows, but would introduce an additional model.
  • Sampled-token distillation evaluates only the generated token instead of using full-vocabulary distillation.The cheaper approximation may be supplemented by partial-vocabulary methods to improve self-evolution signal fidelity.
  • The gradient analysis is token-level and per-update rather than a trajectory-level KL analysis under evolving teacher references.Extending the analysis to trajectory-level KL bounds remains an open theoretical question.
  • Each skill is a structured JSON object with three fields summarizing a completed trajectory.During training, skills are prepended to the teacher prompt to provide task-specific guidance.
  • The skill fields include success analysis, mistake analysis, and a golden workflow.The examples describe successful behavior, API mistakes, and an idealized workflow.

B Trajectory Example and Skill Generation

The trajectory example shows an AppWorld agent retrieving Spotify data, correcting API-call mistakes, exporting a deduplicated CSV, and terminating the account before generating a reusable skill.

  • The AppWorld task requires exporting all Spotify library data to a CSV file and terminating the account afterward.The requested CSV contains unique songs with Title and Artists headers.
  • The agent discovers Spotify APIs, logs in, and retrieves song, album, and playlist libraries before fetching song details.The available endpoints include library, song-detail, and account-deletion APIs.
  • The agent encounters an invalid access_token parameter because show_song accepts only song_id.It consults the API specification, removes the extra parameter, and deduplicates songs across sources.
  • The agent extracts 59 unique songs successfully before writing the CSV.
  • A missing username causes a file_system.login validation error, after which the agent supplies both required credentials and creates the CSV.The file-system API documentation confirms that username and password are required.
  • The completed trajectory terminates the Spotify account and marks the task complete.The generated skill records the successful sequencing, API mistakes, and ideal workflow for reuse.

D.1 Estimator family

The reverse-KL estimator family distinguishes estimating KL values from optimizing their gradients. Although k3 is non-negative and low variance, directly differentiating it can produce biased gradients.

  • Estimator distinction: The estimator family separates KL value estimation from KL gradient optimization.This distinction is central to selecting a valid self-distillation loss.
  • k3 estimator: k3 is attractive because it is non-negative and widely used as a low-variance reverse-KL estimator.
  • Gradient bias: Directly differentiating k3 does not generally yield the desired reverse-KL gradient.The resulting gradient can therefore be biased despite the estimator’s usefulness for KL value estimation.

D.2 General sampled-token SDL

Importance-weighted sampled-token SDL corrects gradient estimation under an arbitrary sampling distribution. The construction uses stop-gradient treatment and distribution-matched importance weights to recover the intended reverse-KL gradient.

  • Definition: Importance-weighted SDL defines a sampled-token self-distillation term for tokens drawn from an actual sampling distribution.The formulation is stated for a fixed prefix and uses sampled tokens yt ∼ µ(· | y<t).
  • Gradient construction: Stop-gradient treatment freezes the teacher-related term while the student log-probability contributes the score function.The proof uses the fact that the teacher log-probability is computed at pre-update parameters and does not depend on current θ.
  • Importance correction: The importance-weight denominator must match the actual sampling distribution µ for the correction to close under expectation.
  • Importance correction: The numerator must use the trainable student so the product-rule gradient reduces to the intended reverse-KL form.
  • Gradient identity: The k3 bracket simplifies to ℓt, and taking expectation under µ proves conditional unbiasedness of the per-token gradient.
  • On-policy corollary: In the main on-policy branch, ρon_t equals the GRPO ratio numerically but weights the auxiliary reverse-KL term rather than the clipped RL surrogate.

D.4 Why SDL and GRPO need distinct interpretations

Skill-SD uses separate importance and clipping ratios because SDL and GRPO serve different optimization roles. The GRPO ratio remains centered on the old student policy, while SDL uses the distribution needed for its correction.

  • Distinct quantities: Skill-SD assigns distinct quantities to self-evolution weighting and GRPO clipping.
  • Off-policy branch: The off-policy SDL weight uses the teacher sampling distribution, whereas the GRPO clipping ratio uses the old student sampling distribution.
  • GRPO trust region: Student-centered GRPO clipping must remain centered at the old student policy, including asymmetric clip-higher bounds.
  • GRPO trust region: Reusing the teacher-denominator ratio for clipping would mis-center the trust region because teacher and student differ by prompt conditioning.

E Additional Experimental Details

The additional details define the AppWorld ablation protocol, its evaluation metrics, and the loss-reduction and clipping settings used for GRPO and SDL.

  • Ablation protocol: The AppWorld study varies the exact axes summarized in its ablation matrix.
  • Evaluation: AppWorld uses the official train/dev split, with accuracy measured as pass@1 on the dev set.
  • Evaluation: Completion rate is the fraction of task-specific unit tests satisfied by the final environment state.
  • Loss and clipping: Both LGRPO and LSDL use token-mean reduction over valid action tokens, while GRPO applies DAPO asymmetric clipping with εh > εl.

F Training Hyperparameters

The section lists implementation choices for Skill-SD and identifies the training-hyperparameter table for AppWorld and Sokoban experiments.

  • Skill-SD is implemented with the rLLM framework and verl backend.
  • Trajectory summarization into skills uses Seed1.8.
  • Multi-turn rollouts use Token-In-Token-Out mode, with raw-token reading and generation instead of the chat completion API.
  • GRPO follows Shao et al. (2024).
  • Table 5 presents training hyperparameters for the AppWorld and Sokoban experiments.
  • UCB1 is described as theoretically optimal for rewards bounded in [0, 1], with logarithmic regret guaranteed via Hoeffding’s inequality.
Loading 2604.10674v1…