PaperlayerView exact revision on arXiv

Source-linked AI summary

TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents

Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, James Cheng

arXiv:2604.24005v3cs.LGcs.AI

TL;DR

Vanilla on-policy distillation becomes unstable in multi-turn agents as compounding errors drive students beyond the teacher’s effective support. TCOD progressively expands the student’s trajectory depth, mitigating KL instability and improving performance by up to 15.71 success-rate points across three benchmarks.

  • Problem

    Vanilla on-policy distillation remains underexplored in multi-turn agents, where sequential interactions create a more challenging setting than static single-turn reasoning.

  • Method

    TCOD controls the trajectory depth exposed to students and progressively expands it from short to long using a curriculum schedule.

  • Results

    Across four student–teacher pairs and three benchmarks, TCOD alleviates KL instability and improves performance by up to 15.71 success-rate points over vanilla OPD.

  • Takeaways & Limitations

    TCOD can recover small students, improve larger ones, and generalize beyond the teacher’s capability boundary in challenging tasks.

  • Takeaways & Limitations

    TCOD-B2F requires pre-collected successful teacher trajectories, while the optimal fixed curriculum pace may vary across environments and student–teacher pairs.

Abstract

from arXiv · show

On-policy distillation (OPD) has shown strong potential for transferring reasoning ability from frontier or domain-specific models to smaller students. While effective on static single-turn tasks, its behavior in multi-turn agent settings remains underexplored. In this work, we identify a key limitation of vanilla OPD in such settings, which we term Trajectory-Level KL Instability. Specifically, we observe that KL divergence increases together with a drop in success rate, and even after convergence, the KL remains high, leading to unstable training. This instability arises from inter-turn error compounding: as errors accumulate, the student is driven beyond the teacher's effective support, rendering the supervision signal unreliable. To address this, we propose TCOD (Temporal Curriculum On-Policy Distillation), a simple yet effective framework that controls the trajectory depth exposed to the student and progressively expands it from short to long with a curriculum schedule. Experimental results across four student-teacher pairs on three multi-turn agent benchmarks (ALFWorld, WebShop, ScienceWorld) show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD. Further evaluations show that TCOD can even surpass the teacher's performance and generalize to tasks on which the teacher fails. Our code is available at https://github.com/kokolerk/TCOD.

1 Introduction

Vanilla on-policy distillation becomes unstable in multi-turn agents because compounding errors push students beyond the teacher’s effective support. TCOD addresses this by progressively expanding the trajectory depth exposed to students, improving performance while reducing action rounds.

  • On-policy distillation transfers reasoning through token-level KL minimization over student-generated rollouts but is designed primarily for static, single-turn tasks.
  • Vanilla OPD exhibits Trajectory-Level KL Instability, combining KL escalation, success-rate collapse, and prohibitively high KL before convergence.
  • Compounding errors across turns push students outside the teacher’s effective support, lowering teacher-assigned token probabilities and making supervision unreliable.
  • TCOD controls trajectory depth and progressively expands it from short to long using a configurable curriculum growth rate.
  • 15.71 success rate points and 2.97 average action rounds are the reported TCOD improvements for the larger Qwen2.5-7B student.

2 Related Work

Prior work studies multi-turn LLM agents, on-policy distillation, and curriculum learning. However, existing OPD improvements primarily target single-turn settings, while curriculum-based reinforcement learning methods still rely on external mechanisms.

  • LLM-based Multi-turn Agents: LLMs have demonstrated strong multi-turn agent capabilities across embodied planning, web navigation, and other interactive environments.A common paradigm interleaves reasoning and action generation through frameworks such as ReAct.
  • On-Policy Distillation and its Limitations: On-policy distillation replaces sparse scalar rewards with dense distillation signals, improving sample efficiency.Existing methods refine OPD through objective design, including balancing forward and backward KL terms and incorporating reward clipping.
  • On-Policy Distillation and its Limitations: Existing OPD improvements primarily target single-turn settings and do not directly address multi-turn agent environments.The cited approaches improve training stability and convergence through KL-objective design and RL-style heuristics such as reward clipping.
  • Curriculum Learning: Curriculum learning progressively exposes models to more difficult examples as competence grows, with applications spanning LLM training and reinforcement learning.Curriculum-based reinforcement learning methods such as GRPO still rely on an external mechanism.

3 Preliminary

The paper models multi-turn autonomous agents as finite-horizon interactions and represents partially observable states through complete interaction histories. It formulates on-policy distillation as aligning a student policy with a teacher under the student’s own state distribution.

  • Multi-turn Agent Setting: Agents interact with an environment over a finite horizon, receiving observations, generating responses, and obtaining subsequent observations at each turn.The turn index ranges from 0 to T−1, where T is the maximum number of interaction steps.
  • History State for Multi-turn Agent: Because environments are generally partially observable, the agent state is defined as the full interaction history up to the current observation.The history includes prior observations and actions, together with the current observation.
  • History State for Multi-turn Agent: A complete trajectory consists of alternating histories and actions and ends either with a termination action or when the horizon is reached.This trajectory representation captures the entire multi-turn interaction sequence.
  • On-Policy Distillation for Multi-turn Agent: On-policy distillation aligns the student policy with the teacher policy under the student’s own state distribution, using KL divergence to measure their discrepancy.The setup considers teacher policy πϕ and student policy πθ.

4 TCOD: Temporal Curriculum On-Policy Distillation

TCOD addresses trajectory-level KL instability in multi-turn on-policy distillation by progressively controlling the interaction depth exposed to the student. It provides forward and backward curriculum variants that reduce compounding errors while expanding training toward end-to-end task completion.

  • Motivation: Vanilla OPD becomes unstable in long-horizon interactions as compounding errors escalate KL divergence and degrade performance.The instability is specific to multi-turn trajectories, where student actions and observations are appended to history and causally affect later turns.
  • Motivation: KL escalation co-occurs with success-rate collapse during multi-turn OPD training.This contrasts with reported single-turn settings, where KL typically converges and decreases.
  • Motivation: Per-turn KL divergence increases with turn index, revealing compounding error amplification across the trajectory.The student can enter states beyond the teacher’s effective support, making supervision unreliable.
  • TCOD-F2B: TCOD-F2B restricts the student to k interaction steps, increasing k from a small initial value toward a larger horizon during training.The curriculum first emphasizes early-turn signals and then progressively enables end-to-end completion.
  • TCOD-B2F: TCOD-B2F uses the teacher as a navigator by executing the initial prefix of a successful trajectory before the student completes the remaining interaction.The student’s horizon k monotonically expands during training, avoiding early-turn error accumulation.

5 Experiments

Experiments across ALFWorld, WebShop, and ScienceWorld show that TCOD improves performance and training stability over vanilla OPD, while remaining robust to curriculum growth-rate choices and reducing training cost. TCOD can also surpass teacher performance on challenging or unseen ALFWorld tasks.

  • ALFWorld performance: 15.71 points: TCOD improves success rate over OPD while reducing average action steps by 2.97 on ALFWorld.The comparison uses Qwen2.5-3B and Qwen2.5-7B students with a GRPO-trained Qwen2.5-7B teacher.
  • Benchmark and model-scale evaluation: Across ALFWorld, WebShop, and ScienceWorld, TCOD-F2B and TCOD-B2F achieve performance comparable to or stronger than vanilla OPD.The benchmarks span reasoning levels from simple to complex, and ALFWorld includes seen and unseen splits.
  • Training dynamics: 18.67 points: TCOD improves average success rate while mitigating KL escalation across different benchmarks and model sizes.For Qwen3 students, TCOD-F2B with η = {3, 6} maintains stable KL and an increasing success rate; it also recovers from response-length explosion while policy-gradient loss decreases smoothly.
  • Beyond-teacher generalization: Up to 2.5 points: TCOD surpasses the teacher in success rate on ALFWorld’s unseen split.The hard split contains 121 challenging tasks where the teacher performs poorly; TCOD-B2F also slightly surpasses the teacher by 0.7 points under the domain-specific teacher setting.
  • Curriculum growth-rate ablation: Less than 2% variation: TCOD performance remains stronger than vanilla OPD across η ∈ {2, 4, 6}, showing low sensitivity to curriculum growth rate.The reported robustness reduces the need for extensive hyperparameter tuning.
  • Training efficiency: Nearly 32%: TCOD-F2B and TCOD-B2F reduce total training time versus vanilla OPD on ALFWorld and ScienceWorld.The efficiency gain comes from shorter early-training trajectories; TCOD-F2B is more efficient than TCOD-B2F.

6 Conclusion · Appendix

The conclusion identifies Trajectory-Level KL Instability as a fundamental failure mode of vanilla OPD in multi-turn agents and introduces TCOD to control student trajectory depth. The appendix includes a training-time comparison.

  • Appendix: The appendix reports a training time comparison.
  • 6 Conclusion: Vanilla OPD suffers from Trajectory-Level KL Instability in multi-turn agents.The conclusion characterizes this as a fundamental failure mode.
  • 6 Conclusion: Compounding errors across turns cause escalating KL divergence.
  • 6 Conclusion: Escalating KL divergence makes teacher supervision unreliable.
  • 6 Conclusion: TCOD controls the trajectory depth exposed to the student during training.The framework is described as simple and principled.
  • 6 Conclusion: TCOD is instantiated through two practical variants: Forward-to-Backward (F2B) and Backward-to-Forward (B2F).

A Limitations and Future Work

TCOD has practical limitations involving demonstration collection and curriculum scheduling. TCOD-F2B avoids demonstration requirements, while the fixed schedule has only been shown robust across the studied benchmarks and model sizes.

  • Limitations: TCOD-B2F relies on pre-collected successful teacher trajectories, creating additional trajectory collection overhead.This requirement motivates consideration of alternatives that do not use demonstrations.
  • Future Work: TCOD-F2B provides a drop-in alternative that requires no demonstrations.The forward-to-backward variant addresses the demonstration requirement of TCOD-B2F.
  • Limitations: TCOD’s fixed curriculum schedule was empirically robust across the three studied benchmarks and model sizes.The evidence is limited to the benchmarks and model sizes evaluated in the study.

B Additional Observation

The additional observations show that vanilla OPD becomes unstable for small students in multi-turn environments, while teacher–student capacity matching affects outcomes more than teacher strength alone. For 3B students, both 30B and 7B teachers produce comparable KL and success-rate trends.

  • Observation 1: KL escalation and success-rate collapse co-occur in small models below 3B under vanilla OPD.This behavior contrasts with prior single-turn math and QA settings, where KL typically decreases and stabilizes during training.
  • Observation 1: Vanilla OPD causes trajectory-level KL divergence to increase sharply during training for Qwen3-0.6B, Qwen3-1.7B, Qwen2.5-0.5B, and Qwen2.5-1.5B students.The observation covers small students from both the Qwen3 and Qwen2.5 model families.
  • Observation 2: For a 3B student, a 30B teacher and a 7B RL teacher yield steadily decreasing KL divergence and comparable success-rate improvements.The similar outcomes indicate that increasing teacher strength beyond a certain point does not provide additional benefits.
  • Observation 2: Teacher–student matching matters because stronger teachers are not always better.The comparison in Figure 8 examines the impact of teacher–student pairing, including a 3B student trained with both a strong 30B teacher and a 7B RL teacher.

C Algorithm for TCOD-F2B/B2F … D.2 Baselines

TCOD uses temporal curricula that gradually expose students to longer trajectories, either by expanding student rollouts or reducing teacher prefixes. Experiments compare these procedures across multi-turn benchmarks against teacher, zero-shot, SFT, and vanilla OPD baselines.

  • C Algorithm for TCOD-F2B/B2F: TCOD-F2B progressively expands the student rollout horizon k according to a linear pacing schedule.The student policy rolls out k steps at each iteration, with k increased during training.
  • C Algorithm for TCOD-F2B/B2F: By focusing distillation on early-turn states before extending the horizon, TCOD-F2B builds a foundation that mitigates compounding errors and prevents KL collapse.The curriculum gradually exposes the student to the full trajectory.
  • C Algorithm for TCOD-F2B/B2F: TCOD-B2F starts from a vetted checkpoint by replaying the teacher’s initial L − k steps, then increases the student-controlled suffix until k = L.Teacher actions in the prefix do not contribute to the gradient, and the student ultimately executes the complete trajectory end-to-end.
  • D.1 Benchmark Environments: ALFWorld evaluates navigation and object manipulation across six household-task categories using seen, unseen, and teacher-failure Hard splits.The Hard set contains 121 tasks where the teacher fails under pass@10 sampling.
  • D.1 Benchmark Environments: Webshop tests multi-turn product search and selection against user instructions in a simulated e-commerce environment.The agent must search for and select matching products.
  • D.2 Baselines: The baselines establish performance boundaries using the teacher as an upper bound and the untuned student as a lower bound.The teacher is evaluated directly, while the zero-shot student performs interactive tasks without task-specific fine-tuning or distillation.
  • D.2 Baselines: SFT imitates two epochs of teacher-collected successful trajectories with NLL loss, whereas vanilla OPD minimizes token-level KL over full student rollouts without horizon constraints or curriculum.SFT suffers from exposure bias in multi-turn settings; OPD is the direct baseline for Trajectory-Level KL Instability.

D.3 Training Hyperparameters · D.4 Evaluation Hyperparameters

TCOD training is conducted across ALFWorld, ScienceWorld, and WebShop, with configurations summarized in Table 4. Evaluation uses three test sets and consistent hyperparameters across environments, as shown in Table 5.

  • D.3 Training Hyperparameters: Training covers three text-based interactive environments: ALFWorld, ScienceWorld, and WebShop.The training configuration is summarized in Table 4.
  • D.3 Training Hyperparameters: ALFWorld is one of the environments included in TCOD training.The paper explicitly lists ALFWorld among the three training environments.
  • D.3 Training Hyperparameters: ScienceWorld is one of the environments included in TCOD training.The paper explicitly lists ScienceWorld among the three training environments.
  • D.3 Training Hyperparameters: WebShop is one of the environments included in TCOD training.The paper explicitly lists WebShop among the three training environments.
  • D.4 Evaluation Hyperparameters: Evaluation uses the test unseen, test, and train hard sets, with train hard available for ALFWorld only.These three test sets define the reported evaluation coverage.
  • D.4 Evaluation Hyperparameters: Evaluation hyperparameters are consistent across all environments and are presented in Table 5.The evaluation configuration is summarized separately from training hyperparameters.

D.5 More experiments results · E Environment Prompts

TCOD-B2F exhibits a controlled non-monotonic training dynamic as its curriculum expands, while maintaining stable generalization across unseen and hard splits. The environment prompts use a consistent history-and-observation structure, with task-specific reasoning and action formats for ALFWorld, ScienceWorld, and WebShop.

  • D.5 More experiments results: TCOD-B2F success initially starts high, declines as the curriculum expands to longer trajectories, and recovers as the student adapts.The same mid-training decrease and later improvement appears on the valid seen split.
  • D.5 More experiments results: Valid unseen and train hard success rates remain relatively stable without pronounced drops, indicating controlled curriculum transition rather than overfitting or instability.The curriculum introduces temporary difficulty while ultimately improving performance and maintaining stable generalization.
  • E Environment Prompts: Environment prompts consistently include task description, observation-action history, current observation, admissible actions, and thinking/action format requirements.This structure is used during both training and evaluation.
  • E.1 ALFWorld Prompts: ALFWorld prompts support household navigation and object manipulation with step-by-step reasoning in <thought> tags followed by executable actions in <action> tags.The template supplies task context, recent history, current state, and admissible actions.
  • E.2 ScienceWorld Prompts: ScienceWorld prompts guide multi-step laboratory experiments requiring scientific knowledge and procedural reasoning through task context, history, observations, valid actions, and tagged responses.The response format requires reasoning within <thought> tags and the selected action within <action> tags.
  • E.3 WebShop Prompts: WebShop prompts support e-commerce navigation, filtering, and purchasing by matching natural-language preferences to product attributes and selecting dynamically available actions.Its action format includes search[<query>] and click[<button name>] operations, depending on page state.
Loading 2604.24005v3…