Source-linked AI summary
Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
Youling Huang, Tiankuo Xu, Jiaji Liu, Tong Zheng, Shuo Zhou, Shaotong Qi, Junchi Yao, Shiyang Liu, Hao Xu, Pengcheng Xu, Bo Huang, Hongyi Fu, Lin Lin
TL;DR
Sparse outcome rewards can leave long-horizon agents without useful learning signals, and it is unclear how long teacher guidance should persist. GATS adapts on-policy distillation to the teacher–student performance gap; across three configurations, it achieves the highest average success rate and improves over reward-only GRPO.
Problem
In long-horizon sparse-reward RL, failed rollout groups can yield zero reward-driven policy-gradient signal, while the appropriate duration of teacher guidance remains unresolved.
Source: §1 · pp. 1–2Method
GATS adds token-level on-policy teacher supervision on student-generated rollouts, adapting its weight to the measured performance gap and withdrawing the teacher at the reference.
Source: §1 · p. 2, §3.3 · p. 4Results
Across all three teacher–student configurations, GATS attains the highest average success rate; with a Qwen2.5-1.5B teacher and Qwen2.5-7B student, it raises GRPO from 52.97% to 64.84% (+11.87 points).
Source: §4.2 · p. 6Takeaways & Limitations
Under matched student rollout budgets, students continue beyond their smaller, task-trained teachers after guidance is withdrawn.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Reinforcement learning for long-horizon agents typically relies on sparse outcome-based rewards. This leads to a severe cold-start problem, as early-stage policies often fail to solve sampled tasks, leaving little useful reward signal for learning. To mitigate this problem, we use on-policy distillation (OPD) to provide token-level guidance on the student's own rollouts. We find that the benefit of this guidance depends on the performance gap between the teacher and the student. When the teacher substantially outperforms the student, distillation helps guide the student through the early training stage where outcome rewards provide little learning signal. As the gap narrows and eventually reverses, however, continued distillation becomes less beneficial and may hinder further improvement. Motivated by this observation, we propose Gap-Adaptive Teacher Scheduling (GATS), which augments the student's RL objective with an OPD term whose weight adapts to the teacher-student performance gap. Specifically, GATS gradually reduces teacher guidance as the student approaches the teacher's reference performance and withdraws it once that reference is reached. This enables GATS to leverage task-trained teachers smaller than the student, since teacher guidance is primarily needed during early training. Across ALFWorld, WebShop, and ScienceWorld with three Qwen2.5 teacher-student configurations, GATS achieves the highest average success rate among the compared methods in all three configurations, improving over reward-only GRPO by 4.37%-11.87% under matched student rollout budgets. Code is available at https://github.com/Ricardo-H/guide-then-let-go.
1 INTRODUCTION
Sparse-reward RL can leave early long-horizon policies without a useful learning signal, while fixed or manually timed teacher guidance creates a separate scheduling trade-off. GATS adapts on-policy distillation to the teacher–student performance gap and withdraws guidance at crossover.
1 INTRODUCTION
- When every trajectory in a sampled group fails, equal rewards produce zero group-relative advantages and no reward-driven policy-gradient signal.This creates a cold-start bottleneck for outcome-based RL under limited rollout budgets.Source: §1 · p. 1
- Off-policy expert trajectories impose rigid token-level imitation that can encourage expert-pattern memorization, poor transfer, and overfitting.The cited work contrasts this with supervision on the student's own rollouts.Source: §1 · p. 2
- On-policy distillation provides token-level teacher supervision on student rollouts and can reduce all-failure groups during early training.The figure caption describes representative WebShop dynamics for the Qwen2.5-3B→7B configuration.Source: §1 · p. 2, Figure 1 · p. 2
- Guidance withdrawn too early can return the student to sparse-reward learning, while persistent guidance may restrict later improvement.Fixed-weight OPD is reported to plateau near the teacher reference, motivating guidance tied to student progress.Source: §1 · p. 2
- GATS scales its on-policy distillation weight with the teacher–student performance gap and withdraws guidance when that gap closes.The proposed schedule is monotone and requires neither fixed expert trajectories nor a hand-designed annealing schedule.Source: §1 · pp. 2–3
- Experiments across ALFWorld, WebShop, and ScienceWorld report that GATS outperforms strong baselines under the same student rollout budget.The authors also report that ablations attribute the gains to the gap-adaptive schedule.Source: §1 · p. 3
2 RELATED WORK
Prior work studies agentic reinforcement learning and teacher guidance for sparse-reward settings. The related-work section identifies adaptive annealing and eventual teacher withdrawal as an open problem.
2 RELATED WORK
- In agentic RL, sparse outcome rewards can leave many costly long-horizon rollouts without effective learning signals when weaker policies fail to find successful trajectories.The passage links failed trajectories to collapsed within-group reward variance and zero group-relative advantage (Xi et al., 2025; Yu et al., 2026).Source: §2 · p. 3
- Prior OPD methods select samples for teacher supervision using external criteria or gradually reduce the teacher signal with a predefined schedule.Examples include incorrect groups (Li et al., 2026b), all-failure groups (Ding, 2026), disagreement (Zhong et al., 2026), and linear annealing (Tan et al., 2026).Source: §2 · p. 3
- How to adapt teacher-signal annealing to actual training dynamics and eventually withdraw the teacher so the student can surpass it remains open.The passage contrasts this gap with approaches that retain the expert throughout training or use manually predefined schedules.Source: §2 · p. 3
3 PRELIMINARIES
The section defines student-generated trajectory groups and their outcome feedback, then frames GRPO and OPD as objectives on these rollouts, with GATS combining them under gap-adaptive guidance.
3.1 AGENTIC SETTING
- For each training prompt, the student samples trajectories represented as tokens with prompt, prior-token, action, and environment-feedback context, and each trajectory receives a task-specific outcome reward.Source: §3.1 · p. 3
- GATS combines the GRPO and teacher-guided OPD losses, reducing OPD weight as the capability gap narrows until training continues with GRPO alone.Source: Figure 2 · p. 4
3.2 GROUP RELATIVE POLICY OPTIMIZATION
- GRPO optimizes the student from relative outcome feedback within a sampled group, using a clipped importance-ratio objective.The trajectory reward is centered by the group mean and scaled by the group standard deviation to yield the advantage.Source: §3.2 · p. 4
3.3 ON-POLICY DISTILLATION
- OPD aligns the student with a frozen teacher through dense token-level supervision on student-generated trajectories.Teacher log-probabilities are detached from gradient computation.Source: §3.3 · p. 4
4 GAP-ADAPTIVE TEACHER SCHEDULING
GATS adapts on-policy distillation to the teacher–student performance gap, reducing and ultimately withdrawing guidance as the student approaches the teacher reference. Across three configurations, it outperforms reward-only GRPO, while training dynamics and schedule comparisons support gap-adaptive guidance over persistent or fixed-horizon alternatives.
4 GAP-ADAPTIVE TEACHER SCHEDULING
- GATS scales the OPD weight with the gap between lagged student performance and a fixed teacher reference, then permanently withdraws the teacher at the reference.Before withdrawal, the objective combines GRPO and gap-adaptive OPD; afterward, training continues with GRPO alone.Source: §4 · pp. 5–6
4.1 EXPERIMENTAL SETUP
- Evaluation covers ALFWorld, WebShop, and ScienceWorld, with three Qwen2.5 teacher–student configurations and matched student rollout budgets across methods.The reported average success rate is the unweighted mean of five benchmark–split metrics.Source: §4.1 · p. 6
4.2 OVERALL PERFORMANCE
- GATS achieves the highest average success rate in all three teacher–student configurations, improving over GRPO by 4.37–11.87 points.For the Qwen2.5-1.5B teacher and Qwen2.5-7B student, average success rises from 52.97% to 64.84% (+11.87 points).Source: §4.2 · p. 6, Table 1 · p. 7
4.3 TRAINING DYNAMICS: FROM COLD START TO TEACHER WITHDRAWAL
- Teacher guidance improves early rollout success and lowers all-failure rates, while GATS continues improving after withdrawal and exceeds teacher references in all three environments.Persistent GRPO+OPD plateaus around teacher reference values, and GRPO eventually catches up; GATS instead continues learning after withdrawal.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
4.4 ANNEALING SCHEDULE COMPARISON
- GATS beats linear, cosine, and step decay by 6.16%, 7.55%, and 12.50% in average success rate, respectively, and fixed horizons are sensitive to task.The best fixed horizon varies across datasets, while matching GATS’s withdrawal steps still falls substantially behind it.Source: §4.4 · pp. 8–9, Table 2 · p. 9
4.5 TRAINING EFFICIENCY
- At the GRPO 150-update compute budget, GATS reaches 77.1% WebShop and 46.0% ScienceWorld smoothed training success, exceeding GRPO and GRPO+OPD in both comparisons.GPU-hour accounting includes student training and, for teacher-assisted methods, teacher training and online inference.Source: §4.5 · pp. 9–10, Table 3 · p. 9
5 CONCLUSION
GATS uses the teacher–student task-performance gap to adapt temporary OPD guidance, withdrawing the teacher when the student's smoothed success rate reaches the teacher reference. Experiments show gains over GRPO and distillation baselines under matched student rollout budgets.
5 CONCLUSION
- GATS gains over GRPO and distillation baselines under matched student rollout budgets.Experiments span three interactive environments.Source: §5 · p. 10
- GATS adjusts the auxiliary OPD weight using the measured teacher–student task-performance gap and withdraws the teacher at the reference score.Withdrawal occurs when the smoothed student success rate reaches the teacher reference.Source: §5 · p. 10
- Students exceed their smaller, task-trained teachers in the reported experiments.Source: §5 · p. 10
- Controlled schedule comparisons support adapting guidance to task progress rather than setting its duration in advance.Future work could explore task-specific scheduling and teacher reactivation under changing task distributions.Source: §5 · p. 10
REPRODUCIBILITY STATEMENT
The reproducibility statement directs readers to the paper's method, evaluation, training, diagnostic, and interaction details. The authors also describe how generative AI tools assisted manuscript preparation and state that the authors are responsible for the final manuscript and its scientific claims.
REPRODUCIBILITY STATEMENT
- Sections 3 and 4 specify the learning objectives and scheduling rule; the appendices report evaluation protocols, training configurations, baseline implementations, checkpoint diagnostics, and interaction templates.Source: REPRODUCIBILITY STATEMENT · p. 10
AI USE STATEMENT
- Generative AI tools assisted with manuscript revision, literature lookup, notation and consistency checks, and LaTeX and figure preparation; the authors take responsibility for the final manuscript and its scientific claims.AI-generated suggestions also informed methodological caveats and experimental interpretation.Source: AI USE STATEMENT · p. 10
A IMPLEMENTATION DETAILS
The appendix describes the experimental resources and implementations used in the paper. Its stated scope covers benchmarks and evaluation protocols, model and training configurations, and baseline implementations.
A IMPLEMENTATION DETAILS
- The appendix describes the benchmarks used in the experiments.Source: A IMPLEMENTATION DETAILS · p. 13
- The appendix describes the evaluation protocols used in the experiments.Source: A IMPLEMENTATION DETAILS · p. 13
- The appendix describes model and training configurations as well as baseline implementations.Source: A IMPLEMENTATION DETAILS · p. 13
A.1 BENCHMARKS AND EVALUATION
The experiments use ALFWorld, WebShop, and ScienceWorld with specified evaluation splits and interaction limits. The appendix also details the Qwen2.5 teacher–student configurations and shared training setup.
A.1 BENCHMARKS AND EVALUATION
- The evaluation covers ALFWorld ID and OOD, WebShop, and ScienceWorld ID and OOD, with task-pool sizes reported in Table 4.ALFWorld uses valid seen and unseen splits; ScienceWorld uses seen and unseen task pools, while WebShop has one evaluation split.Source: §A.1 · p. 13, Table 4 · p. 13
- Episodes are limited to 50 turns in ALFWorld, 15 in WebShop, and 30 in ScienceWorld, which also has a 100-tick simulator budget.Evaluation success depends solely on task completion within these budgets; training-time invalid-action penalties are excluded from the reported success rate.Source: §A.1 · p. 13
- Final checkpoints are evaluated three times with 128 episodes per evaluation, and Avg. SR is the unweighted mean of five benchmark–split metrics.Source: §A.1 · p. 13
A.2 MODELS AND TRAINING
- The three teacher–student configurations are 1.5B→7B, 1.5B→14B, and 3B→7B, using Qwen2.5-Instruct models with teachers trained by GRPO and then frozen.Source: §A.2 · p. 13
- Student methods run for 150 updates with training data, student rollout budgets, and common optimization hyperparameters held fixed within each environment and student size.Source: §A.2 · p. 13
A.3 BASELINE IMPLEMENTATIONS
The section compares GATS with five baselines, including prompt-only and trained students. Three distillation-based baselines combine GRPO with OPD, with different supervision or weighting schemes.
A.3 BASELINE IMPLEMENTATIONS
- The five baselines are prompt-only, GRPO, GRPO + OPD, ATOD, and SOD.All trained baselines follow the shared student-training protocol; the three distillation-based baselines combine GRPO with OPD.Source: §A.3 · pp. 13–14
- Prompt-only evaluates the instruction-tuned student without additional training, while GRPO trains the student without teacher supervision.Prompt-only uses the evaluation protocol described in Appendix A.1.Source: §A.3 · pp. 13–14
- GRPO + OPD uses a fixed OPD weight λt = 1 throughout training, while ATOD uses an annealed OPD–RL schedule and T-DUR.SOD applies step-level divergence-based reweighting without permanently withdrawing teacher supervision.Source: §A.3 · p. 14
A.4 SCHEDULING ABLATIONS
The scheduling ablations vary the scalar OPD weight while holding other training settings fixed, then describe GATS’s gap-based weighting and permanent withdrawal rule.
A.4 SCHEDULING ABLATIONS
- The fixed-time ablation varies only the scalar OPD weight while holding the teacher, student, objective, rollout budget, and other training settings fixed.It considers a horizon N over updates t = 1, . . . , 150.Source: §A.4 · p. 14
- Linear and cosine schedules reach zero at update N, whereas the step schedule retains unit weight through N and switches to GRPO alone at N + 1.All schedules use GRPO alone whenever the OPD weight is zero.Source: §A.4 · p. 14
- The decay-shape comparison uses N = 80; linear decay also tests N ∈ {20, 40, 60, 80}, with environment-specific matched settings of 52, 29, and 29.The matched settings correspond to ALFWorld, WebShop, and ScienceWorld and match observed GATS withdrawal steps in this configuration.Source: §A.4 · p. 14
- Unlike fixed-time schedules, GATS uses teacher–student success-rate feedback to adapt supervision.The algorithm initializes an empty success-rate queue and uses an initial OPD weight of one.Source: §A.4 · pp. 14–15
A.5 GAP-ADAPTIVE TEACHER SCHEDULING
- GATS sets the OPD weight from preceding student success-rate measurements relative to the teacher reference, then permanently withdraws teacher guidance when the estimate reaches that reference.The reference and student estimate use five measurements; after withdrawal, training proceeds with GRPO alone.Source: §A.5 · p. 15, §A.4 · p. 15
B.1 TASK-PERFORMANCE-GAP DIAGNOSTIC
The ALFWorld diagnostic finds that OPD’s gain decreases as the teacher–student performance gap narrows and turns negative after the student overtakes the teacher.
B.1 TASK-PERFORMANCE-GAP DIAGNOSTIC
- OPD’s gain decreases as the teacher–student gap narrows and becomes negative after the student overtakes the teacher.The ALFWorld table reports the gap as teacher SR minus student SR before continuation and OPD gain as the final SR difference between matched GRPO+OPD and GRPO continuations, both in percentage points.Source: §B.1 · p. 15, Table 7 · p. 16
C PROMPTS AND INTERACTION TEMPLATES
- The interaction templates require reasoning in <think> tags followed by an action in <action> tags; later steps also include recent interaction history.The first-step template omits the prior step count and recent interaction history included in the history-augmented template.Source: C PROMPTS AND INTERACTION TEMPLATES · p. 16
C.1 ALFWORLD
- The ALFWorld prompt supplies the task, recent observations and actions, current observation, and admissible actions.It asks the agent to reason step-by-step before presenting an admissible action.Source: §C.1 · p. 16
C.2 WEBSHOP
- The WebShop prompt supplies the shopping task, recent interaction history, current observation, and available actions.It asks the agent to reason about which admissible action best advances the shopping goal.Source: §C.2 · pp. 16–17
C.3 SCIENCEWORLD
- The ScienceWorld prompt frames tasks in a text-based elementary-science environment and supplies recent history, the current observation, and admissible actions.It asks the agent to reason step-by-step before presenting an appropriate action.Source: §C.3 · p. 17