Group-Reflective Self-Distillation for Agentic Reinforcement Learning

1 Introduction

• Terminal rewards provide coarse trajectory-level supervision that does not identify which behaviors should be reinforced or corrected.Successful behaviors, recurring mistakes, and incidental choices remain entangled in the outcome signal.Source: §1 · p. 1
• Retrieved skills may mismatch the current prompt, trajectory, or state, leaving guidance loosely aligned with on-policy experience.The passage describes retrieval as enabling reuse of general behavioral knowledge, but notes that selected skills may not fit the current interaction.Source: §1 · p. 1
• Single-trajectory skills analyzed by stronger external models may exceed the policy’s capability and capture path-specific details instead of success–failure patterns.A successful rollout can include unnecessary detours, while a failed rollout can still contain useful behaviors.Source: §1 · pp. 1–2
• GRSD reflects on each verified rollout and contrasts successful and failed reflections to create policy-native, group-level DO/AVOID guidance.A stop-gradient snapshot synthesizes the contrasts; a self-teacher then modulates turn-level advantages without changing their verifier-determined signs.Source: §1 · p. 2
• GRSD combines policy-native reflection, outcome-contrastive guidance, and sign-preserving turn-level credit refinement, with evaluations reporting leading aggregate performance and unseen-task generalization.The contributions describe skill generation without external models and evaluations across diverse agentic environments and model scales.Source: §1 · p. 2

2.1 Agentic Reinforcement Learning for LLMs

• Sparse, delayed terminal rewards indicate trajectory success or failure without identifying the intermediate decisions responsible for the outcome.GRSD retains outcome-based RL while refining trajectory-level advantages into turn-specific credit without changing the verifier-determined direction.Source: §2.1 · p. 2

2.2 On-Policy Distillation and Self-Distillation

• On-policy self-distillation uses one model as privileged-context teacher and plain-context student, while agentic variants distill skills from external memories or external trajectory analyzers.External memories may mismatch the current state or interaction history, and stronger analyzers may produce guidance beyond the optimized policy’s capability.Source: §2.2 · p. 2
• GRSD contrasts the policy’s own reflections on successful and failed rollouts for the same prompt, deriving guidance without external skill libraries or external skill-generation models.The guidance is not required as privileged context at inference.Source: §2.2 · p. 2

3.1 Preliminaries

• Vanilla GRPO assigns the same trajectory-level advantage to every action token, so it does not distinguish how intermediate decisions contribute to the outcome.The method formulates a multi-turn tool-use agent whose actions are followed by environment observations and verified task rewards.Source: §3.1 · pp. 2–3

3.2 Group-Reflective Guidance Construction

• A fixed external judge scores reflection quality but does not generate or revise reflection content; reflection tokens receive a separate learning signal.The judge checks outcome consistency and whether reflections are specific, grounded, and actionable.Source: §3.2 · pp. 3–4
• GRSD builds guidance in two stages: the policy reflects on verified trajectories, then contrasts successful and failed skills from the same prompt.Stage A learns policy-native reflection skills; Stage B synthesizes compact DO/AVOID guidance, with sampled guidance detached from gradient flow.Source: §3.2 · pp. 3–4

3.3 Turn-Level Self-Distillation

• A stop-gradient teacher compares sampled turns under plain and guidance-augmented contexts, then aligns likelihood changes with the outcome-derived update direction.Conflicting teacher signals reduce update magnitude; guidance is enabled only for groups containing both successful and failed rollouts.Source: §3.3 · p. 4, §3.2 · p. 4

3.4 Overall Optimization Objective

• Turn-specific advantages modulate the trajectory advantage without reversing its sign, while the joint objective trains task and reflection tokens with separate losses.The reflection loss trains reflection tokens; privileged guidance affects task optimization only through detached credit modulation, not as a generation target.Source: §3.3 · p. 4, §3.4 · p. 4

4.1 Experimental Setup

• Evaluation covers ALFWorld, in-domain NQ and HotpotQA plus five out-of-domain QA datasets, and 128 held-out WebShop tasks across three Qwen model backbones.The reported metrics are success rate for ALFWorld, answer accuracy for Search-based QA, and normalized score and success rate for WebShop.Source: §4.1 · p. 5

4.2 Main Results

• Averaged across backbones, GRSD exceeds GRPO by 5.5% on ALFWorld and 2.7% on Search-based QA, and improves WebShop score and success by 4.8% and 7.2%, respectively.It also achieves the best aggregate result among GRPO, OPSD, and RLSD for every environment and backbone, and outperforms the stronger of Skill-SD and SDAR by 4.2% on ALFWorld overall success and 5.5% on WebShop success on average.Source: §4.2 · p. 6

4.3 Training Dynamics

• On ALFWorld, GRSD continues improving after baselines begin to plateau, reaches the highest validation success rate, and maintains a smaller teacher–student gap than Skill-SD and SDAR across nearly all steps.The smaller gap is consistent with privileged context causing less shift from native action likelihoods and smoother late-stage optimization.Source: §4.3 · p. 6, Figure 3 · p. 6

4.4 Ablation Studies

• Removing group-reflective guidance, group aggregation, or failed reflections degrades performance, with failed reflections providing information beyond successful behavior.Replacing group-reflective guidance drops ALFWorld and WebShop success by 7.0% and WebShop score by 5.5%; excluding failed reflections drops ALFWorld and WebShop success by 3.9% and WebShop score by 2.3%.Source: §4.4 · p. 6, Table 2 · p. 6
• Turn-level modulation outperforms token-level and trajectory-level alternatives, which degrade metrics by introducing token-level noise or coarse attribution.Token-level modulation reduces ALFWorld and WebShop success by 3.1% and 2.3%; trajectory-level modulation drops ALFWorld, SearchQA, and WebShop success by 6.3%, 0.9%, and 5.4%.Source: §4.4 · pp. 6–7, Table 2 · p. 6
• Policy-native reflection generation achieves higher ALFWorld success and a lower mean teacher–student gap than external-model reflection generation.Its success-rate advantage is clear in the second half of training, and its lower gap persists throughout training.Source: §4.4 · p. 7, Figure 4 · p. 7
• The strongest late-stage performance occurs at α = 0.01 and λ = 0.5, while excessive reflection-loss weight competes with task optimization and excessive distillation strength reshapes credit using potentially noisy discrepancies.A small λ leaves turn-level advantages close to the trajectory-level signal; λ = 0.5 adds turn-level credit while preserving the verifier-grounded signal.Source: §4.4 · p. 7, Figure 5 · p. 7

4.5 Generalization on ALFWorld Unseen

• On ALFWorld Unseen, GRSD averages 83.6% success, exceeding GRPO by 8.2% and SDAR by 6.0%, with largest gains on Pick2 and Cool.The Pick2 and Cool gains over GRPO are 23.5% and 14.3%, respectively.Source: §4.5 · p. 7, Figure 6 · p. 7

5 Conclusion

• Across environments and model scales, GRSD achieves the strongest aggregate performance, stronger unseen-task generalization, and sustained late-stage improvements.It adds no inference overhead and frames guidance from verified experience as an experience-driven loop for agentic self-evolution.Source: §5 · p. 7

A Theoretical Analysis of GRSD

• The analysis studies how group-reflective guidance produces bounded turn-wise reweighting of the GRPO policy surrogate.The guidance, teacher–student likelihood scores, and turn weights use a frozen policy snapshot and are constants in the task-policy surrogate.Source: A Theoretical Analysis of GRSD · p. 10
• Verifier reward R_i and binary outcome label b_i characterize each rollout, with b_i used to partition successful and failed reflections.The reflections are sampled from the same frozen snapshot used to compute the guidance.Source: A Theoretical Analysis of GRSD · p. 10
• The reflections’ separate learning path through the reflection objective is distinct from their role in the task-policy surrogate.The task-policy surrogate treats guidance-related quantities as constants.Source: A Theoretical Analysis of GRSD · p. 10

A.1 Group-Reflective Contrast

• Under the idealized model, aggregating multiple successful and failed reflections reduces contrast-estimator error relative to a single success–failure pair.The reduction relies on conditional uncorrelatedness; the natural-language guidance generator does not explicitly compute the estimator or guarantee exact recovery of the mean contrast.Source: §A.1 · p. 10

A.2 Turn-Level Likelihood Signal

• The same frozen policy scores the same sampled action in plain and guidance-augmented contexts, isolating the effect of adding group-level guidance.A positive likelihood signal indicates that the guidance-induced change agrees with the reward-relative update direction encoded by the advantage.Source: §A.2 · p. 10

A.3 Bounded Advantage Modulation

• When δ = λb_G < 1, turn-level modulation preserves the sign of each nonzero GRPO advantage and bounds its relative perturbation by δ.Aligned likelihood evidence increases the corresponding advantage’s magnitude, while conflicting evidence decreases it.Source: §A.3 · p. 11

A.4 Positive Reweighting of the GRPO Surrogate

• GRSD’s policy-surrogate component is exactly a detached, positively reweighted clipped GRPO surrogate; KL penalties and the reflection objective are not reweighted.Source: §A.4 · p. 11
• Positive per-term reweighting bounds absolute perturbation relative to the sum of individual gradient norms, but does not ensure aggregate GRSD and GRPO gradients are collinear.Different sample gradients may point in different directions and partially cancel.Source: §A.4 · p. 11

A.5 Gradient Routing and Reflection Learning

• The stop-gradient snapshot fixes turn weights for the plain-context task update, while the separate reflection loss trains the reflection generator through shared policy parameters.The task gradient does not directly modify the mechanism producing the weights.Source: §A.5 · p. 12
• The analysis establishes optimization-mechanism properties, not semantic correctness, monotonic return improvement, or convergence of joint task–reflection optimization.Its statistical claim about reduced trajectory-specific variation relies on Proposition 1’s representation assumptions.Source: §A.5 · p. 12

• ALFWorld uses 3,553 training tasks and 274 evaluation tasks, including 140 seen and 134 unseen tasks across six task families.Success rates are reported for each family and as a weighted average.Source: B Dataset Details · p. 12
• Search-based QA trains on 169,615 NQ and HotpotQA questions and evaluates on two in-domain and five out-of-domain datasets.The seven fixed evaluation sets contain 51,713 questions in total; results use exact-match accuracy and a micro average.Source: B Dataset Details · p. 12, Table 4 · p. 12
• WebShop uses 1,000 training tasks and 128 fixed held-out evaluation tasks, reporting normalized task score and success rate as percentages.Source: B Dataset Details · pp. 12–13
• Each sampled task produces eight rollouts, and pooled evaluation metrics weight each evaluated instance equally while subset columns are reported separately.Training draws are cumulative, so tasks may be repeated.Source: B Dataset Details · p. 13

C Implementation Details

• Implementation settings include environment-specific batch sizes and token limits, eight rollouts per prompt, and GRPO optimization with PPO clipping and a reference KL penalty.The ALFWorld Qwen3-1.7B run uses 600 updates; other runs use 240.Source: C Implementation Details · p. 13, Table 5 · p. 13

D GRSD Algorithm

• The policy generates one reflection per verified rollout, and a detached snapshot contrasts successful and failed reflections into DO/AVOID guidance when both outcomes are represented.A fixed rubric judge scores reflections during training; the guidance is used as text context and receives no gradient.Source: C Implementation Details · p. 13, D GRSD Algorithm · pp. 13–14

E Computational Cost

• GRSD raises observed training time per update by 17% over GRPO, versus 8% for SDAR; its teacher pass takes 12.0 seconds, compared with SDAR’s 25.1 seconds.The reported additional operations add no inference-time overhead, and the methods use the same plain-prompt inference procedure.Source: E Computational Cost · p. 14, Table 6 · p. 15

F Group-Reflective Skill Examples

• For the recorded mug-cooling group, synthesized guidance retains the locate–cool–place sequence while identifying wrong-object actions and goal-free retries as recurring failure patterns.The group contains three successful and five failed rollouts; Stage B guidance is constructed from the full set of reflections.Source: F Group-Reflective Skill Examples · pp. 14–15

G Additional Results

• The analyses examine Stage A reflection-learning dynamics and Stage B mixed-outcome group availability, alongside validation performance across environments and model scales.Translucent curves show raw measurements and solid curves show smoothed trends.Source: G Additional Results · p. 15

G.1 Policy-Native Reflection Learning

• Reflection reward rises rapidly early in training and then converges to a substantially higher level, indicating sustained improvement in policy-native reflection quality.The reflection reward assesses outcome consistency, causal grounding, and actionability.Source: §G.1 · p. 15

G.2 Valid-Group Dynamics

• The valid-group ratio rises from a low initial level and remains within a nontrivial intermediate range despite fluctuations, preserving success–failure contrasts as the policy improves.For invalid groups, GRSD retains the original GRPO advantage without guidance-based modulation.Source: §G.2 · p. 15

H Prompt Templates

• During training, the privileged self-teacher prepends group-reflective guidance only for detached teacher log probabilities; inference uses plain prompts.The guidance prefix does not change sampled responses or trajectories.Source: H Prompt Templates · pp. 15–16

G.3 Training Dynamics Across Model Scales

• At both 3B and 7B scales, GRSD achieves the best final validation results across ALFWorld, Search-based QA, and WebShop, with particularly pronounced gains on ALFWorld and WebShop.Figures 7 and 8 show validation success over 240 training steps across three environments.Source: §G.3 · p. 15, Figure 7 · p. 16, Figure 8 · p. 16

H.1 Agent Prompts

• Environment-facing prompts pair initial and subsequent observations, retaining shared action-format requirements while filling environment-specific observation and history slots..Source: §H.1 · p. 16

H.2 Self-Reflection Prompt (pref)

• Each reflection uses one verified trajectory’s outcome and history to produce three grounded imperative points about successful behavior or failures to avoid..Source: §H.2 · p. 16, §H.4 · p. 19

H.3 Guidance Construction Prompt (pguide)

• For mixed-outcome groups, a detached policy branch contrasts successful and failed trajectory reflections to produce separate DO and AVOID guidance..Source: §H.3 · p. 16, §H.4 · p. 19

H.4 Reflection Judge Prompt (J )

• An external judge scores reflection quality—not trajectory success—on causal correctness, outcome consistency, faithfulness, and actionability using a 0–3 rubric.It returns only one digit.Source: §H.4 · pp. 16, 20