PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?

Ya Gao, Zhengtao Zou, Pekka Marttinen

arXiv:2610.31918v1cs.AI

TL;DR

Interactive-agent distillation raises the question of whether training should follow teacher-generated demonstrations or student-generated experience, and at which level. The paper formalizes history and action-prefix choices separately and studies them in controlled self-distillation. Student histories paired with teacher prefixes generally perform best alone, while pairing and scheduling also affect mixed-training performance.

  • Problem

    Teacher-generated trajectories and student-generated experience are both used for agent distillation, but the relevant choice may differ between the interaction history and the current-action prefix.

    Source: §1 · p. 1

  • Method

    The framework independently selects teacher or student sources for histories and action prefixes, then studies configurations, mixtures, and schedules through self-distillation with a frozen, task-guided teacher.

    Source: §1 · p. 2, §2.1 · p. 2, §2.2.1 · p. 3

  • Results

    Across tested settings, ST has the strongest standalone performance in most cases, including the highest success rate in 10 of 12 settings, while SS is weakest standalone.

    Source: §1 · p. 2, §4.1 · p. 5

  • Takeaways & Limitations

    Pairing matters beyond marginal on-policiness, and mixtures and schedules can benefit from configurations that perform weakly alone, including delayed SS exposure.

    Source: §1 · p. 2, §8 · p. 9

    The experiments use privileged-information self-distillation with frozen teachers initialized from student checkpoints; generalization to substantially larger models remains unestablished.

    Source: §8 · p. 9

Abstract

from arXiv · show

Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule. We study this problem through self-distillation, where a frozen teacher has access to task guidance unavailable to the student, and conduct a three-stage controlled study across three models and four interactive environments. We find that student-induced histories paired with teacher-generated action prefixes generally provide the strongest standalone training signal, whereas following the student at both levels generally performs worst. Nevertheless, fully on-policy data becomes beneficial when incorporated into a mixture with other configurations. Moreover, mixtures with identical teacher/student proportions at each level can yield substantially different performance, showing that how the two levels are paired matters beyond the overall amount of on-policy data. Finally, front-loading fully teacher-generated contexts and delaying fully on-policy contexts improves over a static mixture. These findings provide practical principles for deciding where, how much, and when agent distillation should follow the student.

1 INTRODUCTION

The paper argues that interactive-agent distillation is not simply on- or off-policy: history generation and current-action-prefix generation are separate choices. It studies these choices, their mixtures, and their timing through controlled self-distillation.

1 INTRODUCTION

  • Prior approaches broadly use teacher-generated trajectories or student-generated experience, but student errors can leave histories or partial actions from which teachers struggle to continue.This motivates asking at which level, and to what extent, distillation should follow the student.Source: §1 · pp. 1–2
  • The framework independently specifies the history source and action-prefix source, yielding four configurations and control over pairing, mixture proportions, and training schedule.The study uses self-distillation with a frozen, task-guided teacher across three backbones and four interactive environments.Source: §1 · p. 2
  • Student-induced histories paired with teacher-generated action prefixes (ST) generally perform best alone, while fully student-generated contexts (SS) perform worst across tested settings.Student histories expose the teacher to more opportunities for corrective supervision, whereas student-generated prefixes can constrain useful continuations.Source: §1 · p. 2
  • Mixture performance depends on how history and action-prefix sources are paired, not just their proportions; hybrid configurations are strongest tested, and modest SS allocation can help.The strongest tested mixtures retain contributions from both fully teacher-generated and fully on-policy contexts.Source: §1 · p. 2
  • Front-loading fully teacher-generated contexts and delaying fully on-policy contexts outperforms a static allocation and achieves the strongest final validation performance among tested schedules.TT accelerates early learning, whereas delaying SS improves later and final validation performance.Source: §1 · p. 2

2 HIERARCHICAL AGENT DISTILLATION

The framework treats history generation and current-action-prefix generation as independently controllable sources. A common distillation objective supports four configurations, mixtures, and schedules.

2.1 PRELIMINARIES

  • An interactive agent alternates actions with environment feedback; the student operates without privileged guidance, while the frozen teacher receives it and supplies distillation supervision.The study initializes teacher and student from the same checkpoint.Source: §2.1 · p. 2

2.2 HIERARCHICAL AGENT DISTILLATION FRAMEWORK

  • The framework separates the policy inducing the interaction history from the policy generating the current-action prefix, allowing these sources to be controlled independently.The four configurations are TT, ST, TS, and SS; the PI-conditioned teacher provides the target in each.Source: §2.2 · pp. 2–3, §2.2.1 · p. 3, Figure 1 · p. 3

2.2.1 TWO LEVELS OF ON-POLICINESS

  • At the history level, H selects the preceding trajectory source; at the action-prefix level, A selects tokens within the current action, conditioned on that history.ST uses teacher continuations at student-induced histories, whereas TS uses student prefixes from teacher-induced histories.Source: §2.2.1 · p. 3

2.2.2 A COMMON TRAINING OBJECTIVE

  • For each configuration, the student minimizes token-level divergence from the PI-conditioned teacher over histories and action prefixes sampled from the selected sources.The sampled context and teacher outputs are fixed during each update, and gradients update only the student's next-token distribution.Source: §2.2.2 · p. 3

2.2.3 MIXTURES AND CURRICULA

  • Mixture weights control each source pairing; changing joint weights while holding student-history and student-prefix marginals fixed changes how the sources are paired.Allowing weights to vary with training progress defines a curriculum, separating where, how much, and when to follow the student.Source: §2.2.3 · p. 4

3 SETUP OF CONTROLLED STUDY

The controlled study uses three model backbones and four interactive environments, with task guidance available to frozen teachers. Training refreshes student-dependent examples, and evaluation measures task success and related outcomes.

3.1 MODELS AND BENCHMARKS

  • The study uses Qwen3-8B, Qwen3-14B, and Ministral-3-14B-Instruct-2512 on ToolQA, OfficeBench, ALFWorld, and WebShop.Teacher and student share an initial checkpoint, and the teacher remains frozen.Source: §3.1 · p. 4

3.2 TRAINING SETUP

  • The teacher receives task guidance on solution strategies, tool use, and failure modes, while training collects and refreshes student-dependent trajectories and continuations between rounds.Each exercise samples an interaction history and one current action; eight action positions are sampled per source trajectory.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.3 EVALUATION

  • Students are evaluated in three trials using mean success rate, pass@3, and mean action count; WebShop also reports its official Task Score.Maximum action counts vary by benchmark.Source: §3.3 · p. 5

4 STAGE 1: SEPARATING THE TWO LEVELS OF ON-POLICINESS

Across the tested settings, student histories paired with teacher-generated actions (ST) are the strongest standalone configuration, while fully student-generated contexts (SS) are weakest. The results and analyses show that history and action-prefix sources interact, and that configurations differ in corrective supervision, error recovery, and learning dynamics.

4 STAGE 1: SEPARATING THE TWO LEVELS OF ON-POLICINESS

  • Stage 1 matches initialization, training tasks, and exercises per task within each backbone–benchmark setting, while the analysis examines source interactions and complementary behaviors.The analysis motivates the later mixture and scheduling studies.Source: §4 · p. 5, §4.2 · p. 6

4.1 MAIN RESULTS

  • Replacing teacher histories with student histories generally helps with teacher prefixes but hurts with student prefixes; teacher prefixes outperform student prefixes by 3.6–14.4 points on student histories.On teacher histories, teacher prefixes offer at most a 1.9-point advantage. Student histories paired with teacher-generated actions are most effective.Source: §4.1 · p. 5
  • ST leads overall, achieving the highest success rate in 10 of 12 settings and highest pass@3 in all 12, while SS is weakest among distilled variants.ToolQA is the exception to ST’s success-rate lead for Qwen3-8B and Ministral-3, though ST retains higher pass@3; SS’s average success-rate deficit relative to ST is 10.2 points for Qwen3-8B and 7.7 for Qwen3-14B.Source: §4.1 · p. 5
  • The main source-level trends persist under reverse KL, while forward KL is generally similar or better except in some SS settings.The study uses forward KL in its remaining experiments.Source: §4.1 · p. 6

4.2.1 SUPERVISION DURING DATA COLLECTION

  • Student histories expose the teacher to more unresolved errors and corrective-supervision opportunities, while retaining student reasoning or operation in action prefixes reduces continuation executability and appropriateness.Unresolved errors occur in 82.7–85.0% of ST exercises versus 14.0–19.1% of TT exercises; with half the operation retained, appropriateness falls from 80.2% to 56.2% for Qwen3-8B and from 83.8% to 60.2% for Qwen3-14B. TS also concentrates disagreement on error spans, whereas SS distributes relatively more disagreement before errors.Source: §4.2.1 · p. 6

4.2.2 BEHAVIOR OF TRAINED STUDENTS

  • The configurations show distinct behavioral strengths: SS has fewer repeated failures and planning mistakes, TT follows explicit rules more reliably, and ST achieves the strongest recovery from student errors.For Qwen3-14B, ST has the highest error-correction and eventual-success rates from erroneous histories; TT leads learning in rounds 1–3, ST overtakes by round 5, and SS continues gaining later.Source: §4.2.2 · p. 6

5 STAGE 2: PAIRING AND TRAINING-MIXTURE STUDY

At a fixed training budget, mixtures with the same marginal amounts of student-generated histories and prefixes can perform differently depending on how those sources are paired. The comparisons favor emphasizing hybrid configurations while retaining both fully teacher-generated and fully on-policy examples.

5 STAGE 2: PAIRING AND TRAINING-MIXTURE STUDY

  • The mixture study matches total exercises and training rounds across configurations and reuses the initial Stage 1 exercise pools.The resulting principle is to emphasize student histories paired with teacher actions without excluding configurations that are weaker in isolation.Source: §5 · p. 7, §5.2 · p. 8

5.1 PAIRING MATTERS AT FIXED ON-POLICINESS

  • At identical marginal on-policiness, the independent mixture performs best across the shown Qwen3-14B metrics and benchmarks, improving success rate over standalone ST by 0.5–3.3 points.The compared mixtures assign 50% of examples to student histories and 50% to student action prefixes, but differ in how sources are paired.Source: §5.1 · p. 7

5.2 FROM COMPLEMENTARITY TO MIXTURE COMPOSITION

  • The independent mixture outperforms both alternatives across all settings, and retaining both diagonal configurations is more effective than assigning their combined allocation entirely to TT or SS.With ST and TS each held at 0.25, TT-more beats SS-more; adding some SS improves over TT-more despite SS’s weak standalone performance.Source: §5.2 · p. 7
  • Increasing the combined hybrid weight to 0.75 improves success rate over the independent mixture in every setting, but removing TT and SS entirely performs worse.The hybrid-heavy mixture retains TT and SS at 0.125 each.Source: §5.2 · p. 8
  • With TT and SS each at 0.125, ST-more yields higher success rate than TS-more in every setting, favoring student histories over student action prefixes within the hybrid allocation.The two mixtures exchange the student-history and student-prefix proportions while preserving their combined hybrid allocation.Source: §5.2 · p. 8

6 STAGE 3: SCHEDULING THE TRAINING MIXTURE

The scheduling study compares static and staged allocations of a balanced mixture across eight rounds. Combining early TT emphasis with later SS emphasis outperforms static allocation throughout training and finishes with the highest validation success among tested schedules.

6 STAGE 3: SCHEDULING THE TRAINING MIXTURE

  • The study reallocates a balanced independent mixture over eight rounds, front-loading TT and delaying SS while preserving each targeted configuration’s total allocation.It measures validation success after every round on ToolQA, ALFWorld, and WebShop.Source: §6 · p. 8
  • By round 4, TT-first exceeds the static baseline by 8.7–13.2 percentage points on all three benchmarks, while its final advantage narrows to 0.7–2.0 points.SS-later initially trails or tracks the baseline, then overtakes it in the second half and finishes 2.0–2.9 points higher.Source: §6 · p. 8
  • The combined TT-first + SS-later schedule beats the static baseline at every evaluated round and achieves the highest final validation success on all three benchmarks.Although TT-first alone often leads early, the combined schedule ultimately surpasses it.Source: §6 · p. 8
  • Across the three stages, effective on-policiness depends on where, how much, and when student-generated context is used, rather than maximizing on-policy data.The conclusion favors scheduling TT earlier and SS later within a mixture over maintaining a uniform allocation.Source: §6 · p. 8

7 RELATED WORK

Prior work uses distillation to transfer agent behavior and privileged guidance, including through on-policy supervision. This paper positions its contribution as separating history-level from action-prefix-level source choices and analyzing their interactions in mixtures and schedules.

Distillation and privileged information for agents

  • Prior agent distillation includes supervised training on interaction trajectories and context distillation that transfers behavior elicited by additional instructions or reasoning context (Snell et al., 2022; Gao et al., 2026).The passage identifies knowledge distillation as transfer from a teacher model to a student model (Hinton et al., 2015).Source: §7 · p. 8
  • The paper addresses how teacher- and student-generated contexts should be constructed, combined, and scheduled when distilling privileged guidance.This is framed as complementary to work that internalizes task guidance and corrective feedback or uses trajectory-derived skills as privileged teacher context for on-policy self-distillation (Wang et al., 2026).Source: §7 · p. 9

On-policy distillation and imitation learning

  • The framework distinguishes history-level and action-prefix-level on-policiness as coupled design choices, enabling controlled analysis of their effects, interactions, and allocation over training.It unifies teacher-generated, student-generated, and hybrid contexts under a common objective.Source: §7 · p. 9

8 CONCLUSION

The conclusion presents hierarchical agent distillation as a way to distinguish where, how, and when training follows the student. Across three backbones and four environments, it reports benefits from hybrid pairings and staged allocation, while noting limits on generalization and mixture selection.

8 CONCLUSION

  • Student histories paired with teacher-generated actions generally provide the strongest standalone performance; pairing and timing also matter in mixtures.Emphasizing fully teacher-generated contexts early and fully on-policy contexts later improves validation performance over a static uniform mixture.Source: §8 · p. 9
  • The experiments focus on privileged-information self-distillation with frozen teachers initialized from the students’ checkpoints; generalization to substantially larger models remains unestablished.The mixture study identifies composition principles through targeted comparisons, not a universally optimal allocation.Source: §8 · p. 9
  • The authors report using LLMs to assist with manuscript editing, trajectory annotation, and task-guidance construction, with outputs reviewed or validated as appropriate.They retain responsibility for study design, analysis, interpretation, and the final manuscript.Source: AI USE STATEMENT · p. 10

A.1.1 PRIVILEGED INFORMATION EXAMPLES

The appendix describes how privileged guidance is constructed, which benchmarks and training settings are used, and how trajectories are annotated. It also identifies the analyses and comparisons reported for reverse KL, Stage 1, and hybrid configurations.

A.1.1 PRIVILEGED INFORMATION EXAMPLES

  • The privileged-information prompt revises seed hints using contrasts between successful and failed trajectories, seeking reusable strategies and addressing mistakes or incomplete hints.The guidance requirements also call for actionable, concise rules that prioritize preventable failures.Source: §A.1.1 · p. 12

A.1.2 BENCHMARKS

  • The benchmarks span tool-augmented question answering, office workflows, embodied household tasks, and web interaction; WebShop requires searching, inspecting, comparing, and purchasing products.ToolQA uses heterogeneous external sources, while OfficeBench involves workflows across six applications.Source: §A.1.2 · pp. 13–14

A.1.3 TRAINING DATA STATISTICS AND HYPERPARAMETER SETTINGS

  • The appendix points to tables for training-data statistics and shared hyperparameters, and reports using Adam with 8 H100 and 4 AMD MI250X GPUs.The supplied passages do not provide the tables’ numerical data or hyperparameter values.Source: §A.1.3 · p. 14, Table 2 · p. 14, Table 3 · p. 14

A.1.4 TRAJECTORY ANNOTATION PROTOCOLS

  • Trajectory annotation first screens for unresolved issues, then targets issue recognition, corrective behavior, or correction quality; uncertain cases receive GPT-5.4 adjudication.For correction-quality judgments, annotators first derive an appropriate next step independently, then compare the agent’s action with that solution.Source: §A.1.4 · pp. 14–15

A.2.1 REVERSE KL

  • The reverse-KL appendix compares its results with the forward-KL results in Table 1, but the supplied passage does not state the comparison’s outcomes.Source: §A.2.1 · p. 15

A.2.2 STAGE 1: ANALYSIS

  • Stage 1 materials cover corrective supervision, error profiles, validation success across training rounds, recovery from erroneous histories, and regional teacher–student disagreement.The supplied passages identify the tables and figures but do not give their numerical findings.Source: §A.2.2 · p. 15, Table 6 · p. 16, Table 7 · p. 16

A.2.4 STAGE 2 MAIN RESULTS (CONT.)

  • Table 9 and Table 10 report results for Qwen3-8B and Ministral-3, respectively.The supplied passage identifies the tables and models but does not provide their result values.Source: §A.2.4 · p. 16

A.2.7 STAGE 2: HYBRID CONFIGURATION

  • Hybrid-configuration comparisons vary how teacher/student histories and action prefixes are paired; retaining both diagonal configurations improves success rate in Qwen3-8B and Ministral-3.Figures 11 and 12 hold ST and TS at 0.25 each and vary the remaining allocation; Table 9 compares the independent and hybrid-heavy mixtures.Source: Table 9 · p. 17, Figure 9 · p. 18, Figure 10 · p. 18

Related papers

Loading 2610.31918v1…