Source-linked AI summary
Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents
Tong Zhang, Zhou Liu, Yihao Liu, Jiahua Bao, Xuchen Li, Honglin Lin, Tao Cheng, Zhihan Yu, Kai Tang, Xiaoxi Jiang, Guanjun Jiang
TL;DR
Teacher–student token-level gaps alone do not show whether guidance benefits a multi-turn agent’s final outcome. OG-OPD calibrates supervision with paired student continuations, and achieves the highest mean success rate across the evaluated settings.
Problem
In multi-turn agent OPD, token-level gaps identify local differences but do not establish whether teacher guidance improves the current student’s final task outcome.
Source: §1 · p. 2Method
Results
OG-OPD achieves the highest mean task success rate in every evaluated setting, improving over vanilla OPD by 3.6–17.7 percentage points.
Source: §4.2 · p. 7, §6 · p. 10Takeaways & Limitations
Whether teacher guidance helps depends on the current student’s ability to complete the task from the resulting state, so local gaps alone are insufficient.
Source: §1 · p. 2The benchmark evaluations do not establish real-world safety; deployment requires separate safety evaluations and appropriate safeguards.
Source: Ethics statement · p. 10
Abstract
from arXiv · showhide
On-policy distillation (OPD) trains a student on its own trajectories with dense teacher supervision. Recent work on OPD for multi-turn autonomous agents often treats large teacher-student token-level distributional gaps as promising intervention points, linking larger gaps to a greater need for correction. Yet, our empirical analysis reveals a supervision-benefit mismatch: large gaps can be benign, while small gaps can be outcome-critical. Teacher-student gaps capture differences at the current turn, whereas the benefit of teacher guidance depends on how the current student interacts with the environment afterward. The student may still succeed despite choosing an action that differs from the teacher's, while a teacher-preferred action may lead to a state from which the student cannot complete the task. Local gaps alone are therefore not enough to determine whether teacher guidance benefits the current student. Effective supervision should instead emphasize guidance that the current student can translate into better final task outcomes. Accordingly, we propose Outcome-Guided On-Policy Distillation (OG-OPD), which applies trajectory-relative weighting to teacher supervision and calibrates these weights using final task outcomes from paired student continuations. This calibration selectively strengthens supervision on the student's original trajectories at turns where teacher guidance benefits the current student. Across ALFWorld, ScienceWorld, and WebShop, OG-OPD consistently outperforms baselines under diverse settings. It improves task success rates by 3.6-17.7 percentage points over vanilla OPD and by up to 7.0 percentage points over the strongest baseline.
1. Introduction
In multi-turn agents, local teacher–student gaps do not establish whether guidance improves the student’s eventual task outcome. OG-OPD calibrates supervision using paired student continuations, and outperforms baselines across the evaluated settings.
1. Introduction
- Prior agentic OPD methods organize supervision through rollout construction, interleaved teacher interventions, or gaps to identify candidate intervention turns.Because actions shape later observations and states, the introduction asks whether guidance at those turns improves final outcomes.Source: §1 · p. 2
- Large teacher–student gaps can be benign, while small gaps can be outcome-critical because guidance benefits depend on the student’s continuation from the resulting state.Assessing benefit requires comparing the current student’s final outcomes after its original action and after the teacher action.Source: §1 · p. 2
- OG-OPD uses trajectory-relative weights and paired student continuations’ final outcomes to strengthen supervision only at turns where teacher guidance improves the student’s outcome.Teacher responses and continuations calibrate weights but are not additional distillation targets.Source: §1 · p. 2
- OG-OPD achieves the highest mean task success rate among compared methods in every evaluated setting across ALFWorld, ScienceWorld, and WebShop.On WebShop, the Qwen3-1.7B student achieves 45.7% success, versus 28.0% for vanilla OPD and 38.7% for the strongest competing baseline.Source: §1 · p. 2
- Across the two analyzed benchmarks, teacher harm averages 10.8% of large-gap decisions, exceeding teacher rescue at 8.8%.The paper’s contribution is to use task outcomes, rather than gaps alone, to decide where to strengthen supervision.Source: §1 · p. 2
2. Preliminaries
The preliminaries formalize agent interaction as trajectories whose final outcomes indicate benchmark success, then describe on-policy distillation and turn-level supervision weights.
2.1. Multi-turn Interaction
- A multi-turn trajectory consists of student responses parsed into actions, environment transitions, and appended observations, and ends at termination or the interaction limit.Its final outcome is a binary benchmark-success indicator.Source: §2.1 · p. 3
2.2. On-policy Distillation
- On-policy distillation applies reverse-KL teacher supervision to contexts sampled from a frozen rollout student, held fixed while the trainable student is optimized.At a fixed context, the expected signed log-probability ratio under the rollout student equals negative reverse KL.Source: §2.2 · p. 3
- The token-level update direction determines whether a sampled token’s probability is favored upward or downward, while OG-OPD changes only turn-level weights.The batch loss uses valid teacher-scored generated tokens and a clipped policy surrogate.Source: §2.2 · p. 3
- The sampled turn-level gap magnitude is distinct from signed token-level update directions and full-vocabulary KL divergence.The gap is defined over turns retaining at least one valid scored token.Source: §2.2 · p. 3
3. Outcome-Guided On-Policy Distillation
The empirical analysis finds that local teacher–student gaps do not reliably indicate whether teacher guidance benefits the current student. OG-OPD uses trajectory-relative weights and paired student continuations to selectively strengthen supervision when the teacher action improves final task outcomes.
3.1. Empirical Analysis
- Large gaps were benign in 5.4% of decisions, while teacher rescue and harm occurred in 8.8% and 10.8%; small-gap outcome-critical decisions were 7.6%.Teacher harm occurred more often than rescue on WebShop, while the reverse held on ScienceWorld.Source: §3.1 · p. 4, Figure 2 · p. 4
- In 20.6% of large-gap decisions on average, the teacher frequently completed the task while the current student rarely succeeded from the same resulting state.This teacher-only completion pattern motivates evaluating guidance under the current student’s continuation policy.Source: §3.1 · p. 4
3.2. Trajectory-Relative Weighting and Candidate Selection
- Trajectory-relative weighting uses the first valid turn as a reference, reducing weights for larger gaps and increasing weights for smaller gaps up to βmax.With negligible ε0 and an inactive cap, the weighted gap magnitude is approximately the first valid turn’s gap magnitude.Source: §3.2 · p. 4
- Candidate turns require a sufficiently large gap and, after the initial turn, a thresholded positive increase; only the first replay-valid, legal, differing teacher action is tested within budget.Paired continuations are evaluated only at selected turns to limit additional environment execution.Source: §3.2 · p. 5
3.3. Outcome-Based Calibration
- At a selected turn, paired continuations use the same frozen student, state, history, settings, interaction budget, and seed schedule to test whether the student can finish after either action.Only failure under the original action paired with success under the teacher action permits added distillation weight.Source: §3.3 · pp. 5–6
- Outcome-based calibration adds weight only at the selected turn on the student’s original response, leaving other turns’ pre-calibration weights unchanged.Teacher responses and paired continuations calibrate weights but are not additional distillation targets.Source: §3.3 · p. 6
4. Experiments
Across three benchmarks and multiple teacher–student configurations, OG-OPD achieves the highest task success rate in every evaluated setting. The experiments also show that outcome-based calibration, trajectory-relative weighting, and gap-based candidate selection each contribute to performance.
4.1. Experimental Setup
- Experiments covered ALFWorld, ScienceWorld, and WebShop with Qwen3-32B/Qwen3-1.7B and, on ALFWorld and WebShop, Qwen3-8B-RL/Qwen3-4B configurations.Methods shared training tasks, student initialization, teacher checkpoints, interaction protocol, and evaluation protocol within each benchmark and model setting; reported results were averaged over three training seeds.Source: §4.1 · pp. 6–7
4.2. Main Results and Training Dynamics
- OG-OPD had the highest SR in every evaluated setting; on WebShop it reached 45.7% versus FutureBridge-OPD’s 38.7%, and on ALFWorld it reached 64.2% Overall SR versus 62.7%.Across settings, gains over vanilla OPD ranged from 3.6 to 17.7 percentage points.Source: §4.2 · pp. 7–8, Table 1 · p. 7, Table 2 · p. 7
- With Qwen3-32B/Qwen3-1.7B, OG-OPD’s SR gain over vanilla OPD was 17.7 percentage points on WebShop and 3.6 points on ScienceWorld.The paper relates this contrast to teacher harm being more frequent than teacher rescue within WebShop’s large-gap group, with the reverse pattern on ScienceWorld.Source: §4.2 · p. 8
- On ScienceWorld and WebShop, OG-OPD achieved the highest Score and lowest Rounds across evaluated settings; for Qwen3-1.7B on WebShop, Score rose from 27.5 to 44.3 while Rounds fell from 6.8 to 5.6.On ScienceWorld, Score rose from 48.1 to 52.0 and Rounds fell from 12.6 to 11.6; neither Score nor Rounds triggers additional upweighting.Source: §4.2 · p. 8
- On WebShop, OG-OPD showed earlier improvements in teacher likelihood and action parseability than vanilla OPD, while fewer than 1% of recorded turns received additional weight.Per-step training cost was 1.42× vanilla OPD on ScienceWorld and 1.48× on WebShop for the Qwen3-1.7B student.Source: Figure 4 · p. 8, §4.2 · p. 9
4.3. Ablation Study
- Directly upweighting candidate turns improved SR over the stated starting values, but remained below OG-OPD: 14.9% versus 16.6% on ScienceWorld and 35.0% versus 45.7% on WebShop.Removing outcome-based calibration while retaining trajectory-relative weights also underperformed the full procedure.Source: §4.3 · p. 9
- Replacing trajectory-relative weights with uniform unit weights reduced SR by 11.1 percentage points on ScienceWorld and 12.4 points on WebShop.Outcome-based calibration and trajectory-relative weighting are described as complementary components.Source: §4.3 · p. 9
- Random turn sampling instead of gap-based candidate selection reduced SR by 1.5 percentage points on ScienceWorld and 8.0 points on WebShop.The ablation retained teacher response evaluation and outcome-based calibration.Source: §4.3 · p. 9
5. Related Work
Related work addresses on-policy distillation through objective design, rollout construction, selective weighting, and teacher-guidance evaluation. OG-OPD is distinguished by using paired student continuations’ final task outcomes to calibrate supervision on the original student trajectory.
On-Policy Distillation for Multi-Turn Agents
- Multi-turn OPD research shapes training through rollout-depth curricula, turn-level teacher participation schedules, and replay of student continuations from teacher trajectories.Related objective work studies reverse KL, adaptive intermediate distributions, entropy-dependent objectives, and skew KL with adaptive reuse of student-generated outputs.Source: §5 · p. 9
Selective Supervision and Reweighting
- Selective supervision methods adjust weights using entropy, teacher–student gaps, final outcomes, or perplexity, while trajectory-relative weighting treats relative gap magnitude as a reliability signal.The cited trajectory-relative rule does not distinguish turns where teacher guidance would help the current student from those where it would not.Source: §5 · p. 9
Downstream Evaluation and Teacher Guidance
- Unlike Chen et al. (2026), which uses teacher-preferred tokens in paired continuations and adds accepted teacher responses as targets, OG-OPD uses final task outcomes to adjust supervision on the original student trajectory.Other downstream work uses sampled continuations for step-level supervision or credit assignment, and policy optimization uses future KL or repeated-state action groups.Source: §5 · pp. 9–10
6. Conclusion
The paper concludes that local teacher–student gaps alone do not determine whether guidance benefits a multi-turn agent, and presents OG-OPD as an outcome-calibrated approach. Its reported evaluation is benchmark-based, with supporting materials and an AI-use disclosure.
6. Conclusion
- Across three benchmarks and two teacher–student pairings, OG-OPD achieved the highest mean task success rate among compared methods in every evaluated setting.The conclusion characterizes token-level distributional gaps as insufficient to determine whether teacher guidance benefits the current student.Source: §6 · p. 10
AI use statement
- Generative AI assisted with language polishing and experimental code development; authors reviewed the work and take responsibility for the final content.The statement says AI was not used for other tasks requiring disclosure.Source: AI use statement · p. 10
Ethics statement
- Evaluation used benchmark environments, without live purchases or physical deployments, so the reported performance does not establish real-world safety.The paper says deployment requires separate safety evaluations and appropriate safeguards.Source: Ethics statement · p. 10
Reproducibility statement
- The source code and supplementary materials document the method and experimental procedures needed to reproduce the reported results.The materials include optimization and implementation details, model configurations, evaluation metrics, and the training-cost measurement protocol.Source: Reproducibility statement · p. 10
A. Optimization details
OG-OPD applies outcome-calibrated turn weights to the student’s original response losses while retaining the shared token-level surrogate. The supplied analysis also states that gradient bounds do not establish task-success improvement after optimization.
A. Optimization details
- Vanilla OPD and OG-OPD use the same clipped surrogate on the same valid original response tokens; only turn weights differ.Teacher responses and paired continuations calibrate weights, not distillation targets.Source: A. Optimization details · p. 14
- A selected turn receives additional weight only when the paired outcome indicates teacher rescue, with the increment also limited by the distance to the weight floor.The expected increment depends on both rescue probability and that distance.Source: A. Optimization details · pp. 14–15
- Positive fixed turn weights rescale token loss gradients without reversing their directions, but reweighting across turns can change the batch-gradient direction.The derivation applies to fixed recorded coefficients and weights.Source: A. Optimization details · p. 14
- A bound on the additional loss gradient does not imply that a small number of upweighted turns yields a small gradient change or improved task success.The coefficient floor bounds supervision coefficients, not gradient norms.Source: A. Optimization details · p. 15
B. Empirical analysis details
The empirical analysis compares paired student continuations after student and teacher actions, then classifies decisions and compares their frequencies across gap groups. Decisions are sampled and replay-checked on ScienceWorld and WebShop.
B. Empirical analysis details
- Each sampled decision compares student continuations after the original action and teacher action, with a teacher continuation also used to assess what the teacher can complete.Only replay-valid positions with a legal teacher action differing from the student’s are included.Source: B. Empirical analysis details · p. 15
- Five matched trials classify outcomes using thresholds of at most one and at least three successes, and outcome-critical decisions require differing outcomes in at least three pairs.The model stays fixed and continuations share seed schedules, decoding settings, and remaining budgets.Source: B. Empirical analysis details · pp. 15–16
- The retained analysis includes 329 ScienceWorld decisions after one replay failure and all 274 selected WebShop decisions.Gap groups use benchmark-specific empirical quantiles; the large- and small-gap groups contain 172 and 37 ScienceWorld decisions, and 73 and 33 WebShop decisions.Source: B. Empirical analysis details · p. 16
- Figure 2 confidence intervals resample retained decisions within each benchmark and gap group, while the zero observed small-gap critical count is not evidence of a zero population proportion.The intervals use 2,000 bootstrap resamples.Source: B. Empirical analysis details · p. 16
- Table 4 reports category decision counts divided by gap-group size, with percentages and an average that weights the two benchmarks equally.Category rates are averaged across benchmarks before rounding.Source: Table 4 · p. 16, B. Empirical analysis details · p. 16
B.1. Ranking by Gap Magnitude
Ranking decisions by gap magnitude does not consistently identify where teacher guidance benefits the current student. The execution contract specifies how candidates are selected and how outcomes calibrate supervision on the student’s own trajectories.
B.1. Ranking by Gap Magnitude
- WebShop net rescue is negative over much of the coverage range, while ScienceWorld stays near zero or slightly positive.Figure 5 plots rescue precision and net rescue for decisions ranked by decreasing gap magnitude; its curves show category frequencies, not expected outcome differences.Source: §B.1 · p. 16, Figure 5 · p. 17
C. Execution contract
- Teacher and student scores are evaluated on the same valid original response tokens, while gaps drive trajectory-relative weights and candidate selection.Missing or nonfinite scores and turns without valid tokens are excluded.Source: C. Execution contract · p. 17
- Candidates are checked chronologically within a bounded proposal budget, and the first replay-valid turn with a legal, different teacher action is selected for paired continuations.Both continuations use the same frozen student, remaining budget, decoding settings, and seed schedule.Source: C. Execution contract · p. 17
- Only student-trajectory tokens enter the loss; paired continuation outcomes set the additional weight while the original token-level supervision remains the signed loss coefficient.Training uses one matched pair per selected position, compared with five matched trials per decision in the empirical analysis.Source: C. Execution contract · pp. 17–18, Table 5 · p. 18
D. Experimental details
The experimental details specify model configurations and comparison scope, including training checkpoints and evaluation procedures. Performance and ablation comparisons assess complete training procedures at step 200.
D.1. Model configurations and comparison scope
- The empirical analysis, gap ranking, ablations, and timing use a Qwen3-32B teacher and Qwen3-1.7B student, while WebShop training dynamics cover both configurations.Runs use one node with 8×A800 GPUs.Source: §D.1 · p. 18
- Figure 4 shows WebShop training curves separately from table results averaged over three seeds.Entropy and Teacher NLL are averaged over five steps; remaining rates are pooled over five batches.Source: §D.1 · p. 18
- ALFWorld and WebShop RL teachers are separately trained Qwen3-8B models, with fixed checkpoints used across compared methods.The ALFWorld checkpoint is from step 150 and the WebShop checkpoint from step 100.Source: §D.1 · p. 18
- Performance and ablation comparisons evaluate complete training procedures at step 200 rather than matching variants on extra upweighted turns or total supervision weight.OG-OPD adds environment execution and final-task-outcome feedback; actual training costs are reported separately.Source: §D.1 · p. 18
D.2. Evaluation tasks, metrics, and aggregation
The evaluation specifies task collections and interaction limits for three benchmarks, then defines score, success-rate, and round aggregates. Trained-model metrics use a fixed endpoint and are summarized across three seeds.
D.2. Evaluation tasks, metrics, and aggregation
- ALFWorld uses 140 Seen, 134 Unseen, and 121 Hard tasks; ScienceWorld evaluates 1,308 canonical tasks, and WebShop uses 100 held-out sessions.Interaction limits are 30 steps for ALFWorld and ScienceWorld and 15 for WebShop.Source: §D.2 · p. 18
- Score uses the maximum ScienceWorld score and terminal WebShop reward, while Rounds averages evaluated episodes’ interaction lengths, including failures.ScienceWorld scores range from 0–100 and WebShop terminal rewards from 0–1.Source: §D.2 · p. 18
- Overall SR pools all 395 ALFWorld tasks with equal weight, alongside SR for Seen, Unseen, and Hard collections.ScienceWorld success requires full completion; WebShop success requires terminal reward above 0.5.Source: §D.2 · pp. 18–19
- Trained-model results use optimizer step 200 and report means and sample standard deviations across three seeds; zero-shot references are evaluated without training.Metrics are computed over each benchmark’s episodes separately for each seed before aggregation.Source: §D.2 · p. 19
E. Training efficiency
Training efficiency compares vanilla OPD and OG-OPD using per-step elapsed time and normalized cost. OG-OPD’s measured overhead includes replay, teacher generation, and paired student continuations.
E. Training efficiency
- Timing averages elapsed time per optimization step through step 200 across three seeds, including state replay, teacher response generation, and paired student continuations.Final synchronization and evaluation are excluded; deployment uses only the trained student.Source: E. Training efficiency · p. 19
- Cost normalizes OG-OPD time to vanilla OPD using the ratio of their mean times per step.Table 6 reports this comparison on 8×A800 GPUs with a Qwen3-32B teacher and Qwen3-1.7B student.Source: E. Training efficiency · p. 19, Table 6 · p. 19
- 66.61 seconds per step on ScienceWorld and 32.49 seconds on WebShop, or approximately 42% and 48%, are OG-OPD’s reported increases over vanilla OPD.These are end-to-end overheads under the reported hardware and execution settings, not separate measurements of calibration operations.Source: E. Training efficiency · p. 19, Table 6 · p. 19