PaperlayerView exact revision on arXiv

Source-linked AI summary

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen

arXiv:2609.20784v1cs.CLcs.AI

TL;DR

Multi-turn agents receive sparse trajectory-level rewards, while privileged-context distillation raises questions about teacher reliability and when its supervision remains useful. RetireOPD reward-trains a skill-conditioned teacher, then trains a skill-free student and retires the teacher online. Across tasks and model scales, it improves performance over RL and surpasses its own teacher.

  • Problem

    Multi-turn agents receive a single outcome reward per trajectory, and privileged-context distillation leaves open whether its teacher is reliable and how long its supervision remains useful.

    Source: §1 · p. 2

  • Method

    RetireOPD reward-trains a skill-conditioned teacher, then jointly trains a skill-free student with GRPO and OPD until discrepancy stagnates and student competence reaches a target fraction of the teacher’s.

    Source: §1 · p. 3

  • Results

    Across three model scales on ALFWorld and WebShop, RetireOPD improves performance over RL and surpasses its own skill-conditioned teacher.

    Source: §5 · p. 10

  • Takeaways & Limitations

    RetireOPD internalizes privileged skills into agents without requiring that information at inference time.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.

1 Introduction

RetireOPD addresses two weaknesses in privileged-information distillation: skill context alone does not ensure a reliable teacher, and teacher supervision is useful only during part of training. It trains a skill-conditioned teacher, then retires supervision online when the student is ready to continue with RL alone.

1 Introduction

  • Privileged context alone does not ensure a reliable teacher: under shared-policy GRPO+OPSD, the skill-conditioned branch does not consistently outperform the student it supervises.The shared policy is optimized mainly through the skill-free student objective, so it is not trained to act on skill context.Source: §1 · p. 2
  • Under GRPO+OPD, the teacher-student discrepancy first narrows and then widens; continued matching can conflict with reward optimization after the student internalizes the teacher’s behavior.The discrepancy’s stagnation indicates locally opposed gradients, and continued matching holds the student near the teacher’s performance ceiling.Source: §1 · p. 2, Figure 2 · p. 2
  • Fixed schedules can withdraw guidance too early or too late because the discrepancy’s stopping point varies from step 50 to step 90 across models and tasks.RetireOPD instead uses discrepancy and student competence to determine the transition online.Source: §1 · pp. 2–3
  • RetireOPD first trains a skill-conditioned teacher with environment rewards, then trains a skill-free student with GRPO and OPD, and retires the teacher when discrepancy stops decreasing and student success reaches a target fraction of the teacher’s.After retirement, training proceeds with GRPO alone.Source: §1 · p. 3
  • Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over GRPO by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, while surpassing its teacher in every setting.It also outperforms RL, distillation, and hybrid baselines.Source: §1 · p. 3

2 Related Work

Prior agentic distillation methods use privileged context or reduce teacher influence, but generally retain shared teacher-student policies or predetermined transitions. RetireOPD separately trains its teacher and makes the global transition online using discrepancy and student competence.

2.1 On-Policy Distillation with Privileged Teachers

  • Agentic privileged-teacher methods distill between contexts of one shared policy, controlling teacher signals through token gating, synchronization, or advantage shaping; RetireOPD instead trains its teacher separately with environment rewards.These methods commonly use retrieved or hindsight skills as privileged information.Source: §2.1 · p. 3

2.2 Combining On-Policy Distillation with Reinforcement Learning

  • Prior hybrid approaches reduce teacher influence through annealing, expanding student exposure, withdrawing skills, or sequential RL and distillation, but set the transition before training.Signal-dependent approaches act per prompt or token, and none removes the teacher globally; RetireOPD uses discrepancy and relative competence for that transition.Source: §2.2 · pp. 3–4
  • RetireOPD’s overview comprises reward-based teacher construction, student learning with both GRPO and OPD, and adaptive retirement of OPD supervision.The retirement decision is made by the current policy.Source: Figure 3 · p. 4

3 Method

RetireOPD constructs a skill-conditioned teacher with environment rewards, trains a skill-free student jointly with teacher supervision and environment rewards, then adaptively retires the teacher when transfer stalls and the student reaches sufficient competence.

3.1 Privileged Teacher Construction

  • The skill-conditioned teacher receives privileged context and is optimized with GRPO; after environment-reward training, it is frozen to provide dense student supervision.The student is initialized from the same base model but has no access to the skill context, including at inference.Source: §3.1 · p. 4

3.2 Joint GRPO-OPD Optimization

  • The skill-free student learns jointly from environment rewards and teacher supervision on student-generated trajectories, with the teacher conditioned on privileged skills.The teacher supervises states visited by the student rather than generating a separate trajectory.Source: §3.2 · pp. 4–5

3.3 Adaptive Teacher Retirement

  • Adaptive Retirement monitors whether the teacher-student gap is still shrinking and whether student success reaches a threshold relative to teacher success, then removes OPD and continues with GRPO alone.Relative competence averages student performance over two consecutive windows to reduce short-term fluctuations; combining the criteria guards against premature retirement while the student is weak.Source: §3.3 · pp. 5–6

4 Experiments

Across ALFWorld and WebShop, RetireOPD performs best across three model scales and surpasses its skill-conditioned teacher. Experiments examine teacher construction, training dynamics, retirement criteria, and retirement timing.

4.1 Experiment Setup

  • ALFWorld tests multi-step household tasks, while WebShop tests product search and selection; experiments use Qwen2.5-1.5B, 3B, and 7B-Instruct.The student and teacher share architecture and initialization; the default retirement thresholds are γ = 0.9 and δ = 0.Source: §4.1 · p. 6

4.2 Main Results

  • 18.8 points on Qwen2.5-3B ALFWorld (93.8% vs. 75.0%) and 19.0 points on Qwen2.5-1.5B WebShop (75.8% vs. 56.8%) are RetireOPD’s gains over GRPO.RetireOPD achieves the best performance across all three model scales on both benchmarks.Source: §4.2 · p. 6, Table 1 · p. 7
  • On Qwen2.5-3B-Instruct, RetireOPD reaches 93.8% ALFWorld success and 77.3% WebShop accuracy, above its teacher’s 79.7% and 64.8%.RetireOPD also consistently outperforms GRPO+OPD, which retains teacher distillation throughout training.Source: §4.2 · pp. 7–8

4.4 Ablation Studies

  • Adaptive retirement continues improving while linear annealing plateaus, suggesting teacher utility does not necessarily decay on a predefined schedule.Figure 6 compares adaptive retirement with ATOD’s linear annealing strategy.Source: §4.4 · p. 9, Figure 6

4.3 Training Dynamics

  • 76.6% to 93.8% success follows removal of OPD at retirement; continued GRPO+OPD plateaus, while switching to OPD reduces the gap without improving performance.The teacher-student gap first narrows and then widens as training progresses.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 Ablation Studies

  • 92.2% with both retirement signals exceeds 89.0% with either signal alone and 82.8% when OPD is retained throughout training.The results indicate that alignment progress and relative competence are complementary criteria.Source: Table 2 · p. 9, §4.4 · p. 9

4.5 Retiring Timing Robustness

  • Retiring steps stay within ±5 steps of the default for γ ∈ [0.80, 0.96] and δ ∈ [−0.10, 0.04]; success rates across tested thresholds range from 89.1% to 92.2%.The selected transition step shifts from 55 to 95 across threshold settings, while performance remains relatively stable.Source: §4.5 · p. 9, §4.4 · p. 9, Figure 7 · p. 9

5 Conclusion

RetireOPD transfers privileged skills to a skill-free student through reward-optimized teaching, joint GRPO and OPD training, and adaptive teacher retirement. Across three model scales on ALFWorld and WebShop, it outperforms RL, continuous GRPO+OPD, and its corresponding skill-conditioned teachers.

5 Conclusion

  • RetireOPD internalizes privileged skills without requiring them at inference and consistently improves performance across three model scales on ALFWorld and WebShop.The method optimizes a skill-conditioned teacher with environment rewards, then trains a skill-free student with GRPO and OPD before retiring supervision when transfer stagnates and competence is sufficient.Source: §5 · p. 10

A Theoretical Analysis

The local analysis relates GRPO reward optimization to OPD’s reduction of teacher–student discrepancy. It shows why discrepancy stagnation can signal conflict or diminishing guidance, and why teacher exit can improve returns when the objectives conflict.

Interaction between GRPO and OPD

  • OPD accelerates reward optimization when ⟨g, h⟩<0 and conflicts with it when ⟨g, h⟩>0.If λ⟨g, h⟩>∥g∥², the joint update decreases expected return to first order.Source: A Theoretical Analysis · p. 13

Discrepancy as a Conflict Signal

  • A non-decreasing discrepancy indicates local conflict between the GRPO direction and the OPD direction, while a near-zero OPD signal offers little additional first-order guidance.Discrepancy stagnation therefore signals either emerging conflict or diminishing OPD guidance.Source: A Theoretical Analysis · p. 13

Benefit of Teacher Exit

  • When ⟨g, h⟩>0, teacher exit yields a larger first-order reward improvement than continued joint optimization for sufficiently small η.This result assumes GRPO and OPD are locally conflicting.Source: A Theoretical Analysis · p. 13

Implication for Adaptive Exit

  • The adaptive-exit rule uses discrepancy stagnation as its signal and also requires sufficient student competence to avoid premature exit from early training noise.The analysis associates stagnation with diminishing OPD benefit or emerging conflict with GRPO.Source: A Theoretical Analysis · p. 13

B Implementation Details

The experiments use a unified setup, with shared configurations grouped for compact presentation. ATOD differs from the main method in its loss-weight schedule, which shifts optimization from distillation toward reward-driven GRPO.

Implementation Details

  • The experiments use a unified training setup, and Table 4 groups methods with identical configurations.The table summarizes optimization, data, and method-specific hyperparameters; its caption defines G, λ, and αKL and describes the method groupings.Source: B Implementation Details · p. 14, Table 4

ATOD Linear Annealing

  • ATOD keeps the main method’s experimental settings except for its loss-weight schedule.The annealing progress is defined using the current training step.Source: Hyperparameter RetireOPD… · p. 14
  • ATOD starts with GRPO and OPD weights of 0.1 and 1.0, then holds them at 1.1 and 0.1 from step 80 onward.The schedule gradually increases the GRPO weight and decreases the OPD weight, shifting toward reward-driven GRPO while retaining some distillation regularization.Source: Hyperparameter RetireOPD… · p. 14

C Case Study

The token-level case study contrasts the teacher’s value during early exploration with its potential to constrain a competent student. The examples show both stronger skill-informed guidance at a critical action and teacher disapproval of a successful student action.

Case study

  • The case study examines teacher scores on student-generated tokens in two situations: early exploration and after the student has acquired task competence.The two examples focus on privileged-skill guidance and potentially restrictive teacher supervision, respectively.Source: C Case Study · p. 14

Skill-informed supervision during early exploration

  • At the sink, the skilled teacher assigns the key token clean a likelihood of −0.826, compared with −16.625 from the OPSD teacher.The retrieved skill says to clean the lettuce before placement; routine navigation and object-manipulation actions receive similar scores from both teachers.Source: C Case Study · pp. 14–15, Figure 8 · p. 15

Teacher conflict after competence improves

  • In a successful trajectory, the teacher assigns a low score to the student’s task-effective transition toward the microwave.The example illustrates how continued teacher matching can conflict with reward-driven optimization after the student becomes sufficiently competent.Source: C Case Study · p. 15, Figure 9 · p. 15

D Algorithm

RetireOPD constructs a privileged skill-conditioned teacher before training a skill-free student with GRPO and, while active, OPD. It monitors the student during training and exits teacher supervision when both exit conditions are met.

Algorithm

  • The pipeline first trains a skill-conditioned teacher, then initializes a skill-free student for joint GRPO–OPD training with adaptive exit.The teacher is trained from the base policy using task-relevant skill context and environment rewards.Source: D Algorithm · p. 16
  • During student training, the algorithm performs skill-free on-policy rollouts, updates with GRPO, and applies OPD while the teacher has not exited.After exit, the update uses the GRPO objective alone.Source: D Algorithm · p. 16
  • At the adaptive-exit check, the algorithm sets EXITED to true when both monitored conditions, m ≥ δ and η_m ≥ γ, hold.The procedure then returns the skill-free student policy.Source: D Algorithm · p. 16

E Training Dynamics

The appendix presents RetireOPD training dynamics across three Qwen2.5 model sizes and the ALFWorld and WebShop environments, covering validation scores, relative discrepancy, entropy, and reward curves.

E Training Dynamics

F Prompt

The paper uses a unified interaction-prompt format across environments: students receive observations and action history, while teachers additionally receive privileged skill information during OPD.

F Prompt

  • The unified prompt gives the student task observations and action history, while the teacher also receives privileged skill information during OPD.Source: F Prompt · p. 21, Figure 16 · p. 21
  • The ALFWorld student prompt frames the agent as operating in the ALFRED Embodied Environment and inserts the task description.Source: F Prompt · p. 21, Figure 14 · p. 21
  • The WebShop student prompt frames the agent as an autonomous agent in the e-commerce environment and inserts the task description.Source: F Prompt · p. 21, Figure 15 · p. 21
  • The shared interaction history includes the step count and the most recent observations with their corresponding actions.Source: F Prompt · p. 21

Related papers

Loading 2609.20784v1…