PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Teacher Continuations at Student States

Haojin Wang, Dylan Zhang, Huaibo Chen, Suhao Yu, Yihang Sun, Zhanyang Jin, Jiaying Ye, Dianqi Li, Prasanna Sattigeri, Kamal Youcef-Toumi, Hao Peng

arXiv:2609.36246v1cs.CL

TL;DR

Offline SFT and token-level OPD have limitations in where and how they provide teacher supervision. OLIVE trains on teacher continuations from refreshed student-generated prefixes, and the paper reports improvements across hard reasoning and agentic tasks.

  • Problem

    Offline SFT uses fixed teacher trajectories, while token-level OPD leaves later targets conditioned on the student’s original continuation and distribution matching requires teacher token probabilities.

    Source: §1 · pp. 1–2

  • Method

    OLIVE repeatedly samples prefixes from the evolving student, has the teacher continue them, and trains with cross-entropy on the teacher-generated tokens.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    OLIVE improves hard-reasoning pass@8 by 6% to 8% and agentic avg@4 by 7% to 22%, with a 0.9% average drop on general benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OLIVE continues improving where offline distillation plateaus while better preserving student capabilities and plasticity.

    Source: §1 · p. 3

Abstract

from arXiv · show

We present OLIVE (OnLine InterVEntion). At each iteration, the evolving student policy generates a new prefix, the teacher continues it autoregressively, and the student is updated using cross-entropy computed on the teacher-generated tokens. Each design choice targets a corresponding limitation of existing distillation methods: (1) sequential covariate shift in offline supervised fine-tuning (SFT) on fixed teacher trajectories, (2) fragmented supervision under prefix failure in token-level on-policy distillation (OPD), and (3) the need for access to teacher token probabilities in distribution-matching distillation. OLIVE achieves higher reasoning performance than OPD (with a top-16 KL approximation) at comparable GPU-hour cost. Our asynchronous implementation further reduces OLIVE's total training time by 23.8\%. We evaluate OLIVE on both hard reasoning tasks and agentic tasks which reflects modern post-training scenarios, and it consistently outperforms existing distillation methods under the same training budget. By regenerating prefixes from the evolving student, OLIVE continues improving after offline distillation plateaus while better preserving the general capabilities and plasticity of the student. Using only text from GPT-5.4-mini, continuously training with OLIVE outperforms offline SFT from the same teacher by 13\% on ScienceWorld. These results support OLIVE as an effective and efficient approach to online language-model distillation.

1. Introduction

Offline SFT uses fixed teacher trajectories, while token-level OPD supervises student rollouts without showing how to continue after a correction. OLIVE refreshes student-generated prefixes and has reported gains across reasoning and agentic tasks.

1. Introduction

  • Offline SFT trains on fixed teacher trajectories, creating sequential covariate shift, while token-level OPD leaves later targets conditioned on the student’s original continuation.The introduction notes that OPD also assumes the teacher assigns meaningful probability mass to student-reached states and requires teacher token probabilities for distribution matching.Source: §1 · pp. 1–2
  • OLIVE refreshes prefixes from the evolving student, then trains on teacher continuations using cross-entropy only on teacher-generated tokens.For agentic tasks, the student first acts in the environment and the teacher then takes over.Source: §1 · p. 2, Figure 1 · p. 2
  • OLIVE improves hard-reasoning pass@8 by 6% to 8% and agentic avg@4 by 7% to 22%, with a 0.9% average drop on general benchmarks.The introduction also reports that ScienceWorld success reaches 7.5% from a near-zero student that OPD fails to improve.Source: §1 · p. 2
  • OLIVE continues improving where offline distillation plateaus, while better preserving student capabilities and plasticity.The paper presents refreshing supervision as an important axis of distillation design alongside the form supervision takes.Source: §1 · p. 3

2. Motivation

The section asks how teacher supervision can connect a student’s own attempts to behavior it cannot yet reliably generate. It motivates teacher takeover, refreshed intervention contexts, and learning from generated text.

2. Motivation

  • The central question is how teacher supervision can connect the student’s own attempts to behavior it does not yet generate reliably.The motivation places this question in difficult reasoning and interaction tasks.Source: §2 · p. 3
  • Teacher takeover lets the teacher continue from a student-generated prefix, demonstrating later decisions from contexts the student actually reaches.Unlike OPD targets conditioned on the student’s earlier decisions, teacher continuations can follow the teacher’s corrective choices; in interactive settings, later observations also follow those choices.Source: §2 · p. 3
  • Because student states change during training, the paper proposes regenerating prefixes and interventions rather than reusing contexts collected from the initial policy.This follows the learner-state supervision principle of DAgger (Ross et al., 2011).Source: §2 · p. 3
  • Teacher interventions are text, so cross-entropy on the generated continuation can train the student without teacher token probabilities.The teacher continuation can be tokenized with the student’s tokenizer; it determines the trajectory to learn from, while suffix cross-entropy makes text-only learning possible.Source: §2 · p. 3
  • The paper evaluates refreshed interventions on complex reasoning and long-horizon interaction tasks, alongside the cost of generating them.The section frames these choices as a continuously refreshed procedure for learning from continuations of student attempts.Source: §2 · p. 4

3. OLIVE: OnLine InterVEntion

OLIVE collects prefixes from the evolving student, has the teacher continue them, and updates the student on the teacher-generated text. The section extends this procedure to multi-turn interaction and describes an asynchronous implementation.

3.1. Main Algorithm

  • OLIVE samples student prefixes online, asks the teacher to continue them for at most M tokens, and updates the student with masked cross-entropy.Prefixes are sampled again after each update; the teacher continuation conditions on the prompt and student prefix and may be partial to bound generation cost.Source: §3.1 · p. 4
  • The prompt and student prefix remain in context but are masked from the loss, which is computed on teacher-generated text and supports black-box teachers and different tokenizers.The teacher continuation is tokenized using the student’s tokenizer, and its resulting length may differ from the teacher-token length.Source: §3.1 · pp. 4–5

3.2. Multi-turn OLIVE

  • For multi-turn tasks, the student first produces actions and observations; the teacher then acts for up to M turns, and cross-entropy is applied only to teacher actions.The method retains the interaction history as context and masks student turns and environment observations; reported settings use k = 5 or 10 student turns and up to M = 5 teacher turns.Source: §3.2 · p. 5

3.3. Asynchronous OLIVE for Scalable Training

  • Asynchronous OLIVE overlaps student prefix sampling for later batches with teacher generation, bounding stale-prefix lag by depth d.Larger d allows more overlap but permits greater mismatch between the prefix-generating policy and the policy being trained; reasoning experiments use d = 3.Source: §3.3 · p. 5, Figure 2 · p. 5

4. Experiments

The experiments evaluate OLIVE on reasoning and multi-turn agentic tasks, then compare its training efficiency with OPD.

4.2. Multi-turn Agentic Task

  • Across five agentic environments, OLIVE consistently outperformed two online distillation methods, achieving higher success rates in fewer turns.ScienceWorld showed the biggest gain and was the most complex environment by initial student performance.Source: Table 2 · p. 7, §4.2 · pp. 7–8

4.1. Reasoning Task

  • +4.42 and +5.35 avg@8 points were OLIVE’s gains across Qwen3-1.7B and Qwen3-4B, the largest among compared distillation methods.RLVE supplies 18 tunable reasoning environments with 9K hard training problems; comparisons cap distilled tokens per rollout at 7168.Source: §4.1 · pp. 6–7

4.3. Training Efficiency

  • 23.8% lower total training time than synchronous OLIVE and 28% lower than OPD were reported for asynchronous OLIVE, with minimal performance degradation versus synchronous OLIVE.The comparison used Qwen3-4B as student and Qwen3-4B-Thinking-2507 as teacher on 9K RLVE training examples across 8 H200 GPUs.Source: Figure 4 · p. 8, §4.3 · p. 8

5. Analysis

Across reasoning and agentic settings, OLIVE’s online prefixes are associated with stronger task performance and less forgetting than offline distillation. As training continues or tasks are learned sequentially, OLIVE keeps improving where offline training plateaus.

5.1. Online Prefixes Learn More while Forgetting Less

  • The 5.1 comparison filters correct reasoning from four teacher runs per prompt, four teacher continuations per student prefix, and four OLIVE online rollouts per problem.OLIVE computes cross-entropy only on correct reasoning content after filtering.Source: §5.1 · p. 9
  • OLIVE outperforms both offline baselines while achieving less forgetting on general capabilities and maintaining higher task performance.Figure 6 encodes RLVE pass@8 after finetuning in purple and change in average general-benchmark accuracy relative to the plain student in grey.Source: §5.1 · pp. 9–10, Figure 6 · p. 9

5.2. Online Interventions Keep the Student Learning

  • OLIVE surpasses offline distillation by 13% after 5 epochs on ScienceWorld, despite trailing during the first two epochs.The comparison uses Qwen3-1.7B as student and GPT-5.4-mini as teacher; offline training plateaus after two epochs.Source: §5.2 · p. 10, Figure 7 · p. 10
  • OLIVE with multiple rollouts performs better than corresponding offline epoch checkpoints.Figure 7 compares ScienceWorld success rate over five training epochs and across rollout numbers.Source: Figure 7 · p. 10, §5.2 · p. 11
  • On the final SciWorld task after sequential training, OLIVE improves over offline distillation by 13.0 points, with the largest gap on the last environment.The analysis attributes the difference to fixed offline trajectories increasingly mismatching states reached after earlier training stages.Source: §5.2 · p. 11, Figure 8 · p. 11

6. Related Work

Related work moves from distribution matching to teacher-generated text and student-context distillation. OLIVE uses online teacher interventions to construct supervision from student states.

6. Related Work

  • Knowledge distillation matches teacher and student token distributions, while SeqKD trains on teacher-generated text and Symbolic KD selectively elicits demonstrations with designed prompts.The passage attributes these approaches to Hinton et al. (2015), Kim and Rush (2016), and West et al. (2022), respectively.Source: §6 · p. 11
  • OPD uses teacher distributions over student generations to mitigate distribution mismatch common in offline distillation.The passage cites Agarwal et al. (2024), Gu et al. (2024), and Lu and Lab (2025) for OPD.Source: §6 · p. 11
  • OLIVE constructs supervision through online teacher interventions rather than scoring student generations, targeting settings with unreliable rollouts or unavailable logit-level supervision.The passage contrasts this with scoring student generations, which can fail with large capability gaps or long-horizon tasks.Source: §6 · p. 11

7. Conclusion

OLIVE distills teacher continuations elicited from online student prefixes, using refreshed student context and teacher-generated text as supervision. The paper reports gains on reasoning and agentic benchmarks, alongside continued learning and less forgetting than offline distillation.

7. Conclusion

  • OLIVE collects online student-prefix rollouts, has the teacher continue them, and masks the student prefix when calculating the loss.The student policy is refreshed at each gradient step, and only teacher-generated text supplies supervision.Source: §7 · p. 12
  • OLIVE improves performance by 6% to 8% on hard reasoning tasks and by up to 22% on agentic benchmarks.The conclusion also reports reduced offline-distillation plateauing and less forgetting on general benchmarks.Source: §7 · p. 12
  • The findings suggest that where and how supervision is placed is an important axis of distillation design.The conclusion connects this point to refreshing student context and online intervention.Source: §7 · p. 12

A. Implementation Details

The appendix identifies separate default configurations for OPD and OLIVE in reasoning experiments. OLIVE’s configuration distinguishes its roadside supervision procedure from the optimization setup for training on stitched traces.

A. Implementation Details

B.1. Multi-turn Agentic Task

In a ScienceWorld trajectory, the student loops on an incorrect action sequence and remains at score 0. Starting from that student-generated prefix, the teacher corrects the action and completes the task.

B.1. Multi-turn Agentic Task

  • The trajectory moves from score 0 at handoff to a final score of 1.00 after the teacher continues from the student’s prefix.The student acts for ten turns before the teacher takes over for five turns.Source: §B.1 · p. 18
  • For ten turns, the student repeats attempts to go outside and open the location, while the door remains unopened.The student treats “outside” as an openable object and repeats the same failed sequence.Source: §B.1 · p. 18
  • The teacher first opens the door, then reaches the outside and focuses on the giant tortoise egg and baby dragonfly to finish the task.The teacher’s first action corrects the student’s mistaken attempt to open the location itself.Source: §B.1 · pp. 18–19

B.2. Single-turn Reasoning on RLVE

The RLVE case study illustrates a teacher correcting a student’s reasoning after a 4096-token prefix. In a separate ALFWorld ablation, ten student turns before handoff perform best among the tested prefix lengths.

B.2. Single-turn Reasoning on RLVE

  • A 1024-token teacher continuation can complete the student’s half-written reasoning path and correct its error within the first few sentences.The RLVE example cuts the student prefix at token 4096.Source: §B.2 · p. 19
  • In the cycle-count example, the teacher identifies the missed triangle and lists three cycles, matching the gold answer.The student’s prefix considers an answer of two cycles before the teacher continuation redirects the reasoning.Source: §B.2 · pp. 19–20

C.1. Intervention Time

  • 40.0% success rate is the best tested result with 10 student turns before handoff and 5 teacher turns on ALFWorld.With a student prefix, success rates range from 35.6% to 40.0%, and performance is not monotonic in prefix length.Source: §C.1 · p. 21

C.2. Generalization to Other Tasks

After sequential training on BabyAI, TextCraft, SearchQA, and ScienceWorld, OLIVE is evaluated on held-out ALFWorld. It retains plasticity on trained environments and shows a positive success-rate difference over offline distillation on the held-out task.

C.2. Generalization to Other Tasks

  • OLIVE’s average success-rate gain over offline distillation on held-out ALFWorld is +2.8 after sequential training on four other environments.The student is trained sequentially on BabyAI, TextCraft, SearchQA, and ScienceWorld, with ALFWorld withheld from training.Source: §C.2 · p. 21, Figure 10 · p. 21

C.3. Additional Results on Terminal Agents

On a sampled TBLite evaluation, OLIVE improves pass@4 over both OPD and the base student. The appendix also describes the agentic prompt protocol and states that prompts and action parsers are shared across training and evaluation.

C.3. Additional Results on Terminal Agents

  • The terminal-agent comparison uses 500 TMax training examples, with OLIVE’s 10 student turns and 10 teacher turns matched to OPD’s 20-turn budget.Both methods use eight rollouts per prompt and a batch size of 16.Source: §C.3 · p. 21
  • 12.5% pass@4 with OLIVE exceeds the 10.0% pass@4 reported for both OPD and the base student on TBLite.The evaluation samples 50 tasks and runs four attempts per task; the authors call the result preliminary because the subset is small.Source: §C.3 · pp. 21–22, Figure 11 · p. 21

D. Prompts for Agentic Environments

  • The five agentic environments share a ReAct interaction protocol, and the same prompts and action parsers are used for rollouts and evaluation.The household prompt, for example, requires a thought and an action in the response.Source: D. Prompts for Agentic Environments · pp. 22, 24

Related papers

Loading 2609.36246v1…