Source-linked AI summary
PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
Chunji Lv, Yangguang Wei, Junlin Liu, Yang Gao, Ming Liu, Xinming Wang, Jinyang Wu, Guoren Wang, Changsheng Li
TL;DR
Sparse, delayed rewards make credit assignment difficult in multi-turn agentic reinforcement learning, while teacher guidance can vary across token positions. PCSD weights token-level self-distillation by persistent local teacher support and achieves the strongest ALFWorld results across both backbones, while remaining competitive on WebShop.
Problem
Sparse, delayed trajectory-level rewards make it difficult to assign credit to individual intermediate tokens or actions in multi-turn agentic reinforcement learning.
Method
PCSD estimates teacher credibility from persistent local support using adaptive aggregation, exponential decay, trend modulation, and sigmoid-gated token-level weighting jointly optimized with GRPO.
Results
PCSD achieves the highest ALFWorld Overall success rates on both backbones, exceeding GRPO by 15.6 and 13.3 percentage points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop.
Takeaways & Limitations
The results support persistent local teacher support as a basis for combining dense token-level supervision with sparse environmental feedback in agentic reinforcement learning.
Takeaways & Limitations
Fixed aggregation and gating hyperparameters and a frozen teacher reduce adaptation to evolving trajectory statistics and variations in teacher reliability.
Abstract
from arXiv · showhide
Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.
Introduction
PCSD addresses sparse, delayed feedback in multi-turn agent reinforcement learning by weighting on-policy self-distillation according to persistent local teacher support rather than isolated discrepancies. It combines adaptive, locality-preserving aggregation and trend-aware gating, improving performance across ALFWorld and WebShop while generalizing to unseen scenarios.
- Motivation: PCSD targets difficult credit assignment in multi-turn agent tasks, where sparse and delayed rewards provide little guidance about which decisions caused success or failure.
- Motivation: On-policy self-distillation (Zhao et al. 2026) provides dense token supervision from a privileged teacher, but noisy context, imperfect retrieval, and ambiguity can make teacher guidance unreliable.
- Method: PCSD estimates teacher credibility from persistent local teacher-favoring signals, combining fine-grained positional discrimination with robustness to pointwise noise.
- Method: PCSD adaptively selects aggregation ranges, exponentially emphasizes nearby discrepancies, attenuates declining support, and converts the resulting signal into token-level weights.
- Experiments: PCSD consistently outperforms outcome-only GRPO and existing self-distillation weighting schemes on ALFWorld, remains competitive on WebShop, and generalizes to unseen scenarios.
Related Work
RL has become a dominant paradigm for LLM post-training and is increasingly applied to agentic interaction trajectories. On-policy distillation complements sparse trajectory-level rewards with dense teacher guidance, while OPSD uses a privileged-context-augmented copy of the student.
- RL has become a dominant paradigm for LLM post-training and has extended to agentic interaction trajectories, including code generation and tool use.
- On-policy distillation complements sparse trajectory-level rewards with dense teacher guidance on policy-sampled trajectories, improving fine-grained credit assignment.
- OPSD realizes this approach using a frozen, privileged-context-augmented copy of the student.
Preliminaries
The paper formulates agentic RL as multi-turn interaction between a student policy and an external environment, where sparse trajectory-level rewards hinder token-level credit assignment. OPSD supplements this feedback with privileged-context teacher guidance, while PCSD uses teacher–student support gaps and their local persistence for token-level weighting.
- Agentic RL formulation: Agentic RL models a trajectory as alternating environment states and student-generated responses across multiple interaction turns.At turn k, the student samples response y_k from state s_k, which can trigger the next state; the trajectory is τ = (s_1, y_1, s_2, y_2, …, s_k, y_k).
- Agentic RL formulation: Sparse, delayed environmental rewards—often only terminal success or failure—make credit assignment to intermediate tokens or actions difficult.
- On-policy self-distillation: OPSD adds dense supervision to on-policy student trajectories by having a privileged-context teacher condition on task-relevant skills unavailable to the student.The teacher uses an augmented context, whereas the student predicts from its visible history; prior OPSD studies report higher teacher trajectory-level success on average (Zhao et al. 2026).
- Teacher support: Teacher guidance can vary by token position, so PCSD treats the teacher–student sampled log-probability gap as relative token support and uses its local persistence for continuous weights.The privileged-context teacher is frozen, initialized from the same base checkpoint as the student; larger gap values indicate stronger relative support.
Method
PCSD allocates continuous, position-specific teacher supervision from persistent teacher support across nearby tokens rather than isolated gaps. It adaptively combines multi-scale aggregation, trend modulation, and sigmoid gating, then jointly optimizes the resulting distillation objective with GRPO.
- Persistent consistency: PCSD estimates token-level teacher support by exponentially aggregating teacher–student sampled log-probability gaps across nearby positions, reducing reliance on unstable pointwise observations.Larger persistent-consistency estimates indicate stronger teacher support while retaining position-specific values for weighting.
- Adaptive windowing: PCSD softly adapts smoothing to local gap variability by interpolating between short-window estimates for low variance and long-window estimates for high variance.Intermediate variance produces continuous interpolation, avoiding discrete position-specific window choices.
- Trend modulation: PCSD attenuates positions with declining local teacher support using one-sided trend modulation, while exponential decay limits contributions from distant positions.Negative local slopes are attenuated; nonnegative slopes receive no additional trend-based attenuation.
- Continuous weighting: A sigmoid gate maps adaptive persistent consistency to bounded continuous weights, assigning stronger distillation to positions with stronger support and further reducing weights under declining trends.The gate’s sharpness is controlled by β_gate, while the trend factor attenuates weights associated with decreasing local support.
- Joint objective: PCSD forms a weighted negative-log-likelihood auxiliary objective on student-sampled tokens and combines it with GRPO, which supplies shared trajectory-level advantages and policy optimization.Teacher log-probabilities and weights are detached, and normalization by valid-token count preserves both weight allocation and effective distillation magnitude.
Experiments
Across ALFWorld and WebShop experiments, PCSD achieves the strongest reported ALFWorld Overall results on both backbones, including on unseen environments, while its ablations and training dynamics support adaptive persistent weighting.
- Overall Performance: 90.6% and 59.4% ALFWorld Overall success with Qwen2.5-3B-Instruct and Qwen3-1.7B-Instruct exceed GRPO by 15.6 and 13.3 percentage points and SDAR by 6.2 and 5.5.Table 1 reports the main comparison across ALFWorld and WebShop; PCSD remains competitive on WebShop and achieves the best aggregate ALFWorld results on both models.
- Training Dynamics: 20%–28% of valid tokens receive wk,i > 0.5 while the teacher–student gap gradually increases, indicating concentrated, adaptive rather than hard-selective distillation.The training dynamics show supervision being reallocated as the student policy evolves.
- Component Ablation: Replacing adaptive aggregation, trend modulation, or exponential decay with fixed or uniform weighting consistently reduces Overall performance, while complete PCSD performs best.The component ablation supports complementary contributions from adaptive aggregation, trend modulation, and proximity-aware weighting.
- Generalization to Unseen Environments: 86.7% ALFWorld unseen-split Overall success exceeds GRPO’s 70.9% and SDAR’s 72.7%, with PCSD leading nearly every subtask without privileged evaluation skills.The unseen-split comparison covers category-wise and Overall success rates across unseen environment configurations.
- Sensitivity to the Distillation Coefficient: 75.0% without distillation improves to 87.5% at λPCSD = 0.005 and 90.6% at λPCSD = 0.01, then falls to 83.6% at λPCSD = 0.05.Performance is non-monotonic, indicating that excessive distillation can weaken the balance between teacher supervision and reward optimization.
Discussion
PCSD isolates persistent-consistency weighting by using fixed aggregation and gating hyperparameters with a frozen privileged teacher, but this limits adaptation to changing trajectory statistics and teacher reliability.
- Discussion: Fixed aggregation and gating hyperparameters plus a frozen privileged teacher isolate PCSD’s weighting effect but reduce adaptation to evolving trajectory statistics and teacher reliability.Future work could learn context-dependent aggregation and gating parameters from trajectory statistics and teacher uncertainty.
Conclusion
PCSD is an on-policy self-distillation framework that weights token-level supervision by persistent local teacher support, yielding consistent improvements over outcome-only RL and existing self-distillation baselines.
- Conclusion: PCSD weights token-level supervision using persistent local teacher support, combining adaptive exponential-decay aggregation, one-sided trend modulation, and continuous gating.This integrates evidence across multiple tokens while preserving positional specificity.
- Conclusion: PCSD consistently improves over outcome-only reinforcement learning and existing self-distillation baselines.
Appendix
The appendix supplies additional details on PCSD’s formulation, implementation, and analysis. It covers teacher construction and environment setup, training objectives and procedures, and analyses of PCSD’s aggregation and trend mechanisms.
- Implementation Details: The appendix details privileged-teacher construction, skill retrieval, information isolation, and prompt templates for both environments.
- Training Procedure: It presents the complete GRPO objective and the PCSD training procedure.
- Method Analysis: It analyzes PCSD’s local bias–variance trade-off, exponential aggregation, effective window length, and one-sided trend modulation.
Teacher and Privileged Skills
PCSD uses a frozen teacher copy that scores the student’s own generated tokens while accessing reusable task skills unavailable to the student. Retrieval and prompting isolate privileged information from instance-specific, future, and hidden-state information.
- Teacher construction: The privileged teacher is a separate frozen copy of the student’s base checkpoint and receives no reinforcement-learning gradient or optimizer updates.For each student-sampled trajectory, it evaluates the same generated tokens under teacher forcing.
- Privileged skills and retrieval: The skill repository provides reusable interaction rules and task-specific procedures without evaluation-instance solutions, expert trajectories, target locations, future observations, or hidden states.Retrieved skills cover strategies such as object manipulation and navigation.
- Prompt construction and information isolation: The student uses only the task instruction, observable feedback, and causal interaction history, whereas the teacher additionally receives retrieved skills with the shared observable prefix.Privileged skills are excluded from student inputs during training and evaluation, while the teacher scores the student-generated tokens.
- Prompt construction and information isolation: Skill retrieval uses only the initial observable task description and excludes rewards, future observations, hidden states, and the student’s free-form reasoning.These constraints prevent instance-specific privileged or future information from entering the teacher context beyond the observable context shared with the student.
Prompt Templates
The prompt templates provide bounded interaction context and enforce structured reasoning followed by one executable action in ALFWorld and WebShop. During PCSD training, the teacher shares the student’s environment context and tokens while receiving additional privileged skill information.
- ALFWorld: ALFWorld prompts include the task, bounded recent observation–action history, current observation, admissible actions, and require <think> reasoning followed by exactly one <action>.The initial prompt omits interaction history; later prompts include the interaction index and most recent H observation–action pairs.
- ALFWorld: The ALFWorld template exposes long-horizon causal structure through objectives, observations, action history, and admissible actions.The described sequence can involve locating, acquiring, transforming, navigating, and placing an object.
- WebShop: WebShop prompts provide the shopping request, bounded page history, current rendered page, and available operations, requiring reasoning before one admissible action.At the initial step history is omitted; subsequent prompts include the step index and most recent H observation–action pairs.
- WebShop: WebShop normalizes actions into search[<your query>] and click[<item>], covering retrieval, attribute verification, variant selection, navigation, and purchase.If the assembled prompt exceeds the configured length threshold, the interaction history is removed and a no-history prompt is used.
- Student and Teacher Inputs: During PCSD training, the student receives the standard environment prompt, while the teacher receives identical context plus task-relevant privileged skill information.The teacher evaluates the student-generated response tokens rather than sampling a separate action sequence, so the comparison uses identical context and tokens.
GRPO Objective
The GRPO objective samples groups of complete interaction trajectories, assigns each trajectory’s group-relative advantage to its valid response tokens, and combines clipped policy optimization with token-level KL regularization. Prompt, environment, and padding tokens are masked, while the rollout policy updates between iterations and the frozen reference policy remains distinct from the privileged teacher.
- Trajectory sampling and masking: GRPO samples G complete interaction trajectories per input and uses response-token masks to identify valid student-generated tokens.The mask excludes prompt, environment, and padding tokens.
- Group-relative advantage: The objective computes a group-relative advantage from trajectory rewards and assigns the same trajectory-level advantage to every valid response token in that trajectory.A positive numerical-stability constant ϵnum is used in the advantage computation.
- Clipped policy optimization: The policy term uses token-level importance ratios with clipping through ϵclip for numerical stability and constrained policy updates.The clipped ratio is formed from the student-visible context preceding each token.
- KL regularization: The resulting loss adds a token-level KL penalty toward the frozen reference policy, weighted by βKL, to the clipped GRPO objective.The reference policy remains fixed throughout training, whereas the rollout policy πθold is updated between rollout iterations.
- Policy roles: The frozen reference policy is distinct from the privileged teacher used elsewhere in the method.This distinction separates GRPO regularization from teacher-based self-distillation.
Training Procedure … Weight Robustness and Teacher-Quality Alignment
PCSD combines skill-free student rollouts, frozen-teacher evaluation, detached local weighting, and GRPO optimization with adaptive aggregation, trend attenuation, and sigmoid gating. Across diagnostics, it preserves token priorities under isolated noise and avoids SDAR’s inverse teacher-quality associations, while remaining subject to local smoothing and limited alignment strength.
- Training Procedure: Each training update copies the student to the rollout policy, samples G skill-free trajectories, computes group-normalized GRPO advantages, and evaluates the same tokens with a frozen skill-augmented teacher.No teacher response is resampled, and PCSD weights are computed independently within responses without crossing interaction-turn boundaries.
- Bias–Variance Trade-off in High-Variance Regions: PCSD balances short- and long-window estimates by local variance, using exponential decay to reduce noise while retaining temporal locality and continuous interpolation across variance regimes.Low-variance regions retain more token detail, whereas high-variance regions favor the stable long-window estimate; Nmax = 8 and α = 0.8 yield an effective sample size of about 6.41.
- Further Analysis of Local Aggregation: Exponential aggregation suppresses isolated fluctuations with distance, while one-sided trend modulation attenuates weights when teacher support declines without amplifying nonnegative trends.For n = 8, the slope variance is σ2/42; with γ = 0.3, a normalized slope of −1 gives ηt = 0.7.
- Qualitative Rollout: A recorded valid_unseen rollout completes the ALFWorld acquire–heat–place task in eight valid actions with won=True and reward 10.The trajectory recovers from an unsuccessful fridge search, finds the potato on a countertop, and preserves the required action ordering.
- Training and Hyperparameter Settings: The experimental PCSD configuration uses Nmin = 1, Nmax = 8, α = 0.8, τlow = 0.05, τhigh = 0.5, γ = 0.3, βgate = 5.0, and λPCSD = 0.01.Trainable methods use AdamW with learning rate 1 × 10−6, KL coefficient 0.01, gradient clipping at 1.0, and 150 update steps.
- Benchmark Protocols: Experiments train ALFWorld on the official train split and evaluate valid_seen and valid_unseen, while WebShop uses 1,000 training tasks and 128 fixed validation instances.ALFWorld updates process 16 tasks with eight rollouts each, and WebShop uses different fixed training and validation seeds.
- Weight Robustness and Teacher-Quality Alignment: PCSD shows weak nonnegative teacher-quality alignment, with ρ = 0.174 for skill removal and ρ = 0.052 for shuffled skills, unlike SDAR’s negative correlations.SDAR records −0.489 and −0.208 under the same perturbations; the analysis does not establish strong alignment for every sample or downstream task gains.
- Weight Robustness and Teacher-Quality Alignment: PCSD preserves token-weight rankings better than SDAR under stronger isolated-gap perturbations, reaching 0.934 versus 0.929 at 5% and 0.887 versus 0.864 at 10%.At 1%, correlations are 0.985 for PCSD and 0.986 for SDAR, making the light-perturbation difference negligible.