On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

1 Introduction

• Large models’ inference cost motivates transferring agent capabilities to smaller students, making stable teacher-to-student distillation a practical question.The introduction situates this question in multi-turn agents that plan, use tools, and interact with environments.Source: §1 · pp. 1–2
• Vanilla OPD’s teacher PPL grows with interaction turn, as small student errors compound and move rollouts beyond the teacher’s familiar distribution.The elevated PPL drops only marginally from training step 100 to 200, suggesting the problem is not naturally alleviated by longer training.Source: §1 · pp. 2–3
• Guided-OPD mixes teacher- and student-generated turns, decaying teacher intervention over training to keep early trajectories familiar and later training on-policy.The method reports lower teacher PPL across turns and more reliable supervision.Source: §1 · pp. 2–3
• +21.1% Score and +25.5% Success Rate over vanilla OPD on ALFWorld, ScienceWorld, and WebShop; gains are larger for smaller students.Across three Qwen3 student sizes distilled from Qwen3-30B-A3B, Guided-OPD also reduces average interaction turns by 6.0%.Source: §1 · pp. 2–3

2 Preliminary

• A multi-turn agent cycles through an observation, an agent response, and the environment’s next observation; its state is the interaction history up to the current observation.The response contains a reasoning trace followed by an executable action.Source: §2 · p. 3
• A trajectory comprises successive states and agent responses, ending at a termination action or the finite horizon; the agent is modeled as an autoregressive language model.The policy is written as πθ(at | ht).Source: §2 · p. 3
• OPD rolls out the student policy and aligns it with the frozen teacher through token-level divergence at each visited state.Source: §2 · p. 3
• Token-level alignment can use forward KL, reverse KL, or Jensen–Shannon divergence.Source: §2 · p. 3

3 Method

• Guided-OPD injects teacher guidance at turn granularity, then withdraws it until training returns to pure OPD, consistent with student-only inference.Source: §3 · p. 3

3.1 Turn-level Guidance

• Turn-level guidance assigns roles independently per turn: student turns use reverse KL to favor teacher-endorsed actions, while teacher turns use forward KL to cover teacher demonstrations.Source: §3.1 · pp. 3–5

3.2 Curriculum Guidance

• During the curriculum stage, β decays from βstart to zero; cosine is the default schedule, and the remaining training steps use standard OPD.Source: §3.2 · p. 5

3.3 Final Objective

• The objective weights reverse-KL distillation on student-induced states by 1−βt and forward-KL distillation on teacher-induced states by βt, recovering standard OPD as βt approaches zero.Source: §3.3 · p. 5

4.1 Experimental Setup

• Experiments distill 0.6B, 1.7B, and 4B Qwen3 students from Qwen3-30B-A3B, evaluating them on ALFWorld, ScienceWorld, and WebShop against Zero-shot, Vanilla OPD, and TCOD.ALFWorld uses official seen/unseen splits as IID/OOD evaluations; metrics are SR and Round, with Score unavailable on ALFWorld and reported on ScienceWorld and WebShop.Source: §4.1 · pp. 5–6

4.2 Main Results

• AdaSwitch’s average Score is below Vanilla OPD on 1.7B and 4B, while TCOD’s Score gain shrinks from +23.4% on 0.6B to +2.2% on 4B.Guided-OPD achieves higher average Score and SR than either baseline at every student size.Source: §4.2 · p. 6
• Guided-OPD’s relative Score gain over Vanilla OPD grows from +10.3% on the 4B student to +35.3% on the 0.6B student; the 4B student scores 42.64 versus the teacher’s 41.40.The 4B student also exceeds the teacher on ScienceWorld (29.50 vs. 27.73) and WebShop (55.78 vs. 55.07).Source: §4.2 · p. 6
• Against Vanilla OPD, average Score gains were +35.3%, +17.6%, and +10.3%, average SR gains +32.8%, +23.7%, and +20.1%, and average Round fell 7.9%, 7.3%, and 2.8% for 0.6B, 1.7B, and 4B students.Guided-OPD also strictly outperformed all baselines on the three aggregate metrics within every student-model group.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Ablation Study

• Swapping the default KL roles lowers Score/SR from 37.94/19.93 to 33.32/15.47 and raises Round from 18.73 to 23.30; asymmetric pairing beats both symmetric variants.Reverse KL on both turns yields 36.58 Score and 17.83 SR, versus 34.56 and 17.02 for forward KL on both.Source: §4.3 · pp. 7–8
• Fixed β schedules trail the cosine schedule’s 37.94 Score by 2.1–3.5 points; fixed β = 0.25 reaches 16.09 SR versus 19.93.As fixed β rises from 0.25 to 0.75, Score rises from 34.41 to 35.85 while SR falls from 16.09 to 14.82; cosine achieves the best Score, SR, and Round.Source: §4.3 · p. 8
• Cosine decay reaches 37.94 Score, above linear at 36.12 and exponential at 34.44.The cosine schedule decays slowly near both endpoints and faster in the middle; linear is a reasonable but suboptimal alternative, while exponential loses the most on Score and SR.Source: §4.3 · p. 8

• Multi-turn OPD work adds reward-aligned signals or uses temporal curricula, but the former depends on a reliable verifier or critic and the latter does not repair states already entered.Source: §5 · p. 8
• Rollout-guidance methods switch bidirectionally at token level or hand generation to the teacher after one deviation; the paper describes chimeric trajectories or limited student recovery, respectively.These methods retain the teacher throughout training, leaving a training-inference mismatch when it is removed at inference.Source: §5 · p. 8
• Guided-OPD intervenes at turn level, aligned with environment state transitions, and decays intervention to zero so late training returns to pure OPD.The related-work passage contrasts this with trajectory-level scheduling and persistent teacher guidance.Source: §5 · p. 8

• Small student errors compound across turns and push rollouts beyond the teacher’s familiar state distribution, making supervision least reliable where needed most.Source: §6 · p. 8
• Guided-OPD mixes teacher- and student-generated turns within rollouts and schedules teacher intervention along a curriculum that smoothly decays to zero.Turn-level mixing aligns role transitions with environment state transitions, while asymmetric KL pairing matches each divergence to the role driving generation.Source: §6 · p. 9
• Across three benchmarks and three Qwen3 student sizes, Guided-OPD consistently outperforms vanilla OPD and strong guidance and curriculum baselines, with largest gains on the smallest student.Source: §6 · p. 9

Limitations

• The evaluation covers only ALFWorld, ScienceWorld, and WebShop, leaving richer observation and action spaces such as GUI and continuous-control settings untested.Whether Guided-OPD remains effective in those settings is an open question.Source: Limitations · p. 9
• All distillation uses Qwen3 students and a Qwen3 teacher, so the findings reflect intra-family transfer rather than cross-family transfer.Tokenizer mismatches and distributional gaps in cross-family transfer may require curriculum adjustments.Source: Limitations · p. 9
• The initial mixing probability, curriculum ratio, and cosine decay are fixed and shared across student sizes and benchmarks.The study suggests that adapting the schedule to a training signal may reduce tuning cost and raise the performance ceiling.Source: Limitations · p. 9
• Experiments use one 8-GPU node, students up to 4B parameters, and a 30B-A3B teacher; substantially larger scales and longer-horizon tasks remain unverified.The authors leave verification at those scales and horizons to future work because it requires more resources.Source: Limitations · p. 9

A Experimental Setup Details

• The appendix provides full experimental details complementing Section 4.1, organized into five subsections.These cover benchmark environments, evaluation metrics, baselines, training and evaluation hyperparameters, and environment-specific prompt templates.Source: A Experimental Setup Details · p. 11
• Benchmark environments and evaluation metrics are documented in separate subsections.The appendix also describes the baselines and training and evaluation hyperparameters.Source: A Experimental Setup Details · p. 11
• Environment-specific prompt templates are included among the appendix’s experimental details.Source: A Experimental Setup Details · p. 11

A.1 Benchmark Environments

• The suite spans embodied navigation, scientific reasoning, and e-commerce in three text-based interactive environments.Table 3 summarizes the benchmarks and defines “Max Turns” as the maximum agent–environment interaction rounds per task.Source: §A.1 · p. 12, Table 3 · p. 12
• ALFWorld tests six household-task categories through natural-language observations and high-level commands, with up to 30 interaction steps per episode.Its seen and unseen splits serve as IID and OOD evaluations, respectively.Source: §A.1 · p. 12
• ScienceWorld tests scientific reasoning across 30 elementary-school-science task types, using structured observations and up to 30 interaction steps per episode.Its tasks commonly require long-horizon planning, multi-step tool use, and cause–effect reasoning.Source: §A.1 · p. 12
• WebShop tests natural-language shopping instructions through a structured web interface, with search and click actions and up to 15 steps per episode.The environment returns a product-match score and a success signal after purchase.Source: §A.1 · p. 12

A.2 Metrics

• Score is a continuous 0–100 task measure for ScienceWorld and WebShop, but is not reported for ALFWorld.It captures partial credit through subgoal completion in ScienceWorld and product-attribute match in WebShop.Source: §A.2 · p. 12
• Success Rate is the binary task-completion rate, with ScienceWorld and WebShop requiring a score of 100 for success.ALFWorld success is determined by the environment’s terminal success flag.Source: §A.2 · p. 12
• Round measures average environment interaction steps per task; when Success Rate is similar, fewer steps indicate greater efficiency.Source: §A.2 · pp. 12–13
• ALFWorld reports Success Rate and Round separately on IID seen and OOD unseen splits; all metrics use validation sets and shared decoding settings.Source: §A.2 · p. 13

A.3 Baselines

• All baselines use the Qwen3-30B-A3B teacher and the same training hyperparameters and evaluation script as Guided-OPD, except for method-specific settings.These shared conditions are used to ensure a fair comparison.Source: §A.3 · p. 13
• Zero-Shot Student evaluates the untrained student without task-specific fine-tuning or distillation, establishing its starting capability as a lower bound.It also serves as a sanity-check lower bound for distillation methods.Source: §A.3 · p. 13
• Vanilla OPD lets the student sample the full trajectory and applies teacher KL to student-generated tokens, without teacher intervention during rollout.It is the direct counterpart for measuring turn-level guidance and curriculum scheduling.Source: §A.3 · p. 13
• AdaSwitch hands the remainder of a generation to the teacher when instantaneous divergence first exceeds a threshold based on its recent sliding-window average.The handover is one-shot, and switching is triggered by sample-level divergence rather than training progress.Source: §A.3 · p. 13
• TCOD uses rollout depth as a difficulty proxy, increasing the allowed interaction turns linearly during training and accumulating KL over those turns.Its curriculum changes trajectory depth; it introduces no turn-level teacher intervention within a rollout.Source: §A.3 · p. 13

A.4 Hyperparameters

• Training configuration shared across methods, system-level settings, and method-specific hyperparameters are summarized in Table 4; evaluation hyperparameters are summarized in Table 5.The method-specific entries cover Guided-OPD, AdaSwitch, and TCOD.Source: §A.4 · p. 13, Table 4 · p. 14, Table 5 · p. 14

A.5 Prompt Templates

• Prompts for training and evaluation share a structure comprising task description, observation–action history, current observation, admissible actions, and a required thought/action format.Reasoning and executable actions are syntactically separated with <thought> and <action> tags.Source: §A.5 · p. 13
• ALFWorld prompts require step-by-step reasoning followed by an executable action selected from a dynamically updated admissible-action set.The available actions reflect the current situation.Source: §A.5 · pp. 13, 15
• WebShop prompts frame shopping goals and recent interaction history, then ask the agent to reason and choose an admissible action.Actions include searching with a free-text query when the search bar is available and clicking elements available on the current page.Source: §A.5 · p. 15