PaperlayerView exact revision on arXiv

Source-linked AI summary

Stochastic Teacher Intervention for Agentic On-Policy Distillation

Junnan Liu, Linhao Luo, Zhijun Chen, Qianren Mao, Thuy-Trang Vu, Gholamreza Haffari

arXiv:2610.10878v1cs.CLcs.AI

TL;DR

In multi-turn agentic OPD, student actions alter later observations, allowing early errors to move rollouts into histories where teacher supervision is less reliable. STI-OPD uses discrepancy-guided stochastic teacher intervention and importance-weighted reverse KL; it outperforms evaluated distillation baselines across benchmarks and student sizes, with its largest margins on long-horizon tasks.

  • Problem

    In multi-turn OPD, student actions shape later observations, and accumulated errors can move rollouts into histories where teacher token-level supervision is less reliable or counterproductive.

    Source: §1 · p. 1

  • Method

    STI-OPD maps a k3 estimate of teacher–student policy discrepancy to stochastic turn-level intervention and uses importance-weighted reverse KL to correct teacher-token sampling mismatch.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    STI-OPD outperforms standard OPD and every evaluated intervention-based baseline at both student scales, with its largest margins on the longest-horizon tasks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Ablations show that discrepancy-guided intervention and importance weighting both contribute to the reported gains.

    Source: §1 · p. 3

    Importance correction recovers the conditional reverse-KL objective at a fixed context but does not correct the intervention-induced distribution over contexts; the sampling argument assumes πθ is absolutely continuous with respect to πT.

    Source: §3.3 · pp. 6–7

Abstract

from arXiv · show

On-policy distillation (OPD) efficiently transfers capabilities from a stronger teacher to a student language model through dense token-level supervision on student-generated rollouts and has shown promise on complex tasks such as mathematical reasoning. However, in multi-turn agentic tasks, student decisions shape subsequent observations, causing early errors to accumulate across turns. The resulting trajectories can drift away from the teacher's rollout distribution, making the teacher's token-level supervision less reliable or even counterproductive for OPD training. To address this issue, we introduce STI-OPD, a stochastic teacher intervention framework for multi-turn agentic OPD. During multi-turn interaction, STI-OPD uses teacher intervention guided by teacher-student policy discrepancy to replace the student's proposed action with a teacher-generated one to maximize the acquisition of reliable supervision. We further develop a stochastic intervention strategy, addressing the limitations of previous threshold-based or fixed-schedule approaches, that estimates policy discrepancy using KL divergence and maps it to an intervention probability. By sampling whether to intervene from this probability, STI-OPD adaptively balances teacher control with student exploration. To learn from the resulting mixed-policy trajectories, we introduce an Importance-Weighted Reverse KL objective that corrects the token sampling mismatch between teacher-generated responses and the student policy to preserve the original OPD objective. Across tool-integrated reasoning and long-horizon interaction, STI-OPD outperforms the strongest prior OPD baseline on every evaluated benchmark and student size. Ablations further show that both discrepancy-guided intervention and importance weighting contribute to these gains.

1 Introduction

Multi-turn interaction can push student rollouts into histories where teacher supervision is unreliable, while existing approaches do not adaptively correct the current action. STI-OPD uses discrepancy-guided stochastic intervention and importance weighting, and outperforms evaluated distillation baselines across the reported settings.

1 Introduction

  • In multi-turn tasks, student actions shape later observations, so early errors can move rollouts into off-teacher-distribution histories where token-level supervision is less reliable or counterproductive.These deviations can accumulate across turns, making supervision on student-generated tokens alone insufficient to control distribution shift.Source: §1 · pp. 1–2
  • Prior approaches filter supervision or reshape trajectories, but loss filtering cannot correct the action driving the next transition, and schedules do not assess current correction needs.Threshold-based adaptive intervention can amplify noisy discrepancy estimates into abrupt control changes, risking missed corrections or unnecessary teacher dependence and cost.Source: §1 · p. 2
  • STI-OPD maps a k3 estimate of teacher–student policy discrepancy to a stochastic intervention probability, increasing teacher intervention as discrepancy grows while preserving student exploration.Importance-weighted reverse KL corrects the token-sampling mismatch from teacher-generated responses while preserving the original OPD objective.Source: §1 · pp. 2–3
  • STI-OPD outperforms all distillation baselines on each benchmark and at both student scales, with its largest margins on long-horizon tasks.The experiments span tool-integrated reasoning and long-horizon agentic interaction.Source: §1 · p. 3

2 Related Work

Prior multi-turn OPD methods manage instability by altering supervision or trajectory construction, while intervention-based approaches also face a token-sampling mismatch. STI-OPD combines stochastic discrepancy-based control with importance sampling to address these issues.

On-policy Distillation for Multi-Turn Agents

  • Prior OPD methods either reweight or mask supervision, reshape student training trajectories, or hand generation to the teacher; intervention methods may supervise teacher-sampled tokens with an integrand assuming student sampling.The trajectory approaches include depth expansion, curricula, turn-wise allocation, replayed teacher prefixes, and teacher bridges after failed prefixes.Source: §2 · p. 3

Expert Intervention and Distribution Shift

  • Interactive imitation learning connects learner actions to future state distributions; related LLM methods use adaptive boundaries, supported reference states, curricula, continuation triggers, future-rollout validation, or hysteretic thresholds.The passage characterizes these intervention rules as hand-specified and sensitive to noisy turn-level estimates.Source: §2 · p. 3
  • STI-OPD uses instantaneous teacher–student KL to set stochastic intervention probabilities rather than thresholds, and importance sampling corrects the mismatch from teacher-generated tokens.Figure 2 depicts retaining the student proposal or replacing it with a teacher response before execution, then using importance ratios in training.Source: §2 · p. 3, Figure 2 · p. 4

3 Methodology

STI-OPD intervenes during multi-turn rollouts when estimated teacher–student discrepancy is high, then corrects teacher-generated tokens so training can retain the conditional reverse-KL objective. Its methodology connects the evolving interaction history, stochastic intervention, and importance-weighted learning.

3.1 Problem Formulation

  • Because student actions shape later observations, errors can compound into histories where teacher guidance is less reliable; intervening before execution can influence the next environment transition.The task is modeled as repeated responses and environment observations conditioned on the interaction history.Source: §3.1 · pp. 4–5

3.2 Stochastic Teacher Intervention

  • STI-OPD scores a student proposal with token-level k3 estimates, aggregates a length-normalized discrepancy score, and samples whether to replace the full response with a teacher response.The k3 estimate is non-negative and unbiased under student sampling; in the near-policy regime, it has lower-order variance than k1. Larger estimated discrepancies raise intervention probability, while stochastic sampling preserves a chance of student execution.Source: §3.2 · pp. 5–6

3.3 Learning from Teacher Intervention

  • For teacher-generated tokens, importance weighting corrects the sampling mismatch to recover the conditional reverse-KL expectation used by OPD.Retained student tokens need no correction because they were sampled from the student policy.Source: §3.3 · pp. 6–7
  • The token-sampling correction is defined at a fixed context and does not correct the context distribution induced by intervention, including teacher-generated prefixes and later histories.Thus, it corrects the teacher branch’s token distribution without undoing adaptive intervention’s effect on visited contexts.Source: §3.3 · p. 7

4 Experiments

The experiments evaluate STI-OPD on tool-integrated reasoning and long-horizon agentic tasks, across two student sizes and a range of distillation baselines. STI-OPD leads the baselines across reported benchmarks; ablations and analyses examine its components, dynamics, scaling, and cost.

4.1 Experimental Setup

  • The evaluation covers tool-integrated reasoning and long-horizon interaction, using Qwen3-0.6B and Qwen3-1.7B students with Qwen3-4B and, in scaling experiments, Qwen3-32B teachers.Evaluation uses the student policy alone, without teacher intervention.Source: §4.1 · p. 7

4.2 Main Results

  • Across both student sizes, STI-OPD leads all distillation baselines in every reported cell, with its largest gains on long-horizon tasks where errors compound.The results span mathematical and scientific reasoning, code generation, and interactive decision making.Source: §4.2 · pp. 7–8, Table 2 · p. 8, Table 1 · p. 8

4.3 Ablation Study

  • Removing importance weighting causes the largest reported task degradation; using k1 or forward KL instead of k3 or reverse KL also reduces performance.Stochastic intervention outperforms deterministic thresholding across ALFWorld and ScienceWorld without a manually selected decision boundary.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 Mechanism, Scaling, and Cost Analysis of STI-OPD

  • STI-OPD shifts from teacher to student control as discrepancy falls, retains its lead with a Qwen3-32B teacher, and lies on the performance–cost Pareto frontier.At both student sizes, it improves substantially over standard OPD with modest additional training time and has higher performance at lower computational cost than FUTUREBRIDGE-OPD.Source: §4.4 · pp. 9–10, Figure 4 · p. 10, Figure 5 · p. 10

5 Conclusion

STI-OPD combines stochastic, discrepancy-based teacher intervention with importance weighting for multi-turn agentic distillation. Its full algorithm collects mixed-policy trajectories and trains the student with a corrected reverse-KL objective.

5 Conclusion

  • STI-OPD outperforms standard OPD and every evaluated intervention baseline across tool-integrated reasoning and long-horizon benchmarks, at both student scales.Its largest margins occur on the longest-horizon tasks, with only modest wall-clock overhead over standard OPD.Source: §5 · p. 10

A Algorithmic Details of STI-OPD

  • At each turn, a non-negative k3 estimate of teacher–student discrepancy sets a stochastic intervention probability.The algorithm forms a turn-level score from token-level estimates, maps it to a probability, then samples an intervention decision.Source: §5 · p. 10, A Algorithmic Details of STI-OPD · p. 15

5 Conclusion

  • Teacher intervention controls the histories visited, while importance sampling corrects token mismatch on teacher-generated responses to preserve the conditional reverse-KL objective.The resulting mixed-policy trajectories provide the student tokens used for optimization.Source: §5 · p. 10, A Algorithmic Details of STI-OPD · p. 15

B.1 Datasets

The training corpus combines mathematical, coding, and scientific prompts, while evaluation spans reasoning benchmarks and multi-turn environments. The section also specifies six distillation baselines and controls comparisons by matching data and training budgets.

B.1 Datasets

  • The tool-integrated reasoning corpus contains 34,501 prompts: 14,116 from DAPO-Math, 10,385 from Skywork-OR1-RL-Data, and 10,000 from MegaScience.Each prompt starts a fresh rollout under the method being trained; original dataset responses are not fixed distillation targets.Source: §B.1 · p. 16
  • Evaluation covers AIME 2024–2026 and HMMT February 2026 for mathematical reasoning, GPQA for scientific transfer, and LiveCodeBench v6 for code generation.GPQA comprises 448 expert-written questions; LiveCodeBench v6 uses recent contest problems with executable tests.Source: §B.1 · p. 16
  • ALFWorld and ScienceWorld test interactive, multi-turn behavior in which actions change later observations and early errors can alter the remaining trajectory.ScienceWorld tasks require exploration and experiments; ALFWorld tasks involve household actions such as navigation and object manipulation.Source: §B.1 · pp. 16–17

B.2 Baselines

  • The baselines include standard student-rollout OPD, SAGE-OPD's selective turn supervision, and GUIDED-OPD's decaying teacher-intervention curriculum.The comparison uses shared student initialization, teacher, prompts, rollout budget, and evaluation protocol within each setting.Source: §B.2 · p. 18
  • RELAY-OPD inserts short teacher relays at detected prefix failures, TCOD progressively increases student-controlled depth, and FUTUREBRIDGE-OPD validates interventions through subsequent student continuation.These methods vary how intervention locations or interaction depth are selected.Source: §B.2 · p. 18

B.3 Training Configuration and Parameters

Training uses Qwen3-0.6B and Qwen3-1.7B students with Qwen3-4B as the default teacher, plus a larger-teacher study. The prompt templates specify task-specific response formats and interaction protocols.

B.3 Training Configuration and Parameters

  • The experiments use Qwen3-0.6B and Qwen3-1.7B students with Qwen3-4B as the default teacher, and also test Qwen3-32B.Training lasts one epoch for tool-integrated reasoning and 200 steps for ALFWorld and ScienceWorld, at a learning rate of 1 × 10−6.Source: §B.3 · pp. 18–19
  • Each prompt receives one rollout at temperature 1.0 and top-p 1.0, with trajectories capped at five turns for tool-integrated reasoning and 30 turns for ALFWorld and ScienceWorld.Responses are capped at 32,768 tokens and model context length at 40,960 tokens.Source: §B.3 · p. 18
  • During training, the teacher scores student-proposed turns, intervention is sampled, and the student learns from teacher-generated tokens with importance weighting; evaluation runs the student alone.Reported results are means across independent runs, with standard deviations across those runs.Source: §B.3 · p. 19

B.4 Prompt Templates

  • The reasoning prompt requests stepwise analysis and verification, allows helpful tool use, and requires the final answer in boxed format.Interactive-task templates prescribe think/action turns and environment-specific valid action spaces, while code generation requires Python-only output.Source: §B.4 · pp. 19–20

C.1 STI-OPD Results with Larger Teacher

With Qwen3-32B as teacher, STI-OPD achieves the highest reported success rates across the listed long-horizon tasks at both student sizes. Its largest stated margins over the best competing method are on ScienceWorld and LOOK.

C.1 STI-OPD Results with Larger Teacher

  • STI-OPD has the highest success rate in every reported ALFWorld category and on ScienceWorld at both student sizes with Qwen3-32B as teacher.The advantage holds over standard OPD and the intervention-based baselines, so the reported benefit is not specific to the Qwen3-4B teacher.Source: §C.1 · p. 21
  • +10.1 and +7.4 points are STI-OPD's margins over the best competing method on ScienceWorld and LOOK, respectively, for the 1.7B student.The passage reports positive margins on every task category and identifies these as the most pronounced.Source: §C.1 · p. 21
  • The largest gains are reported on tasks with longer dependencies, where information and action outcomes must propagate across multiple turns.The passage describes this pattern as consistent with the intended role of adaptive intervention at histories where student errors would otherwise affect later decisions.Source: §C.1 · p. 21

C.2 Preserving the Reverse-KL Objective Helps Mitigate Forgetting

The section tests whether preserving reverse KL helps retain previously acquired capabilities during teacher intervention. STI-OPD improves over the original student checkpoint, unlike the forward-KL variant.

C.2 Preserving the Reverse-KL Objective Helps Mitigate Forgetting

  • The evaluation tests whether preserving reverse KL mitigates forgetting during distillation with teacher intervention.The forward-KL variant is compared with STI-OPD on capability-retention benchmarks.Source: §C.2 · p. 21
  • STI-OPD improves over the original checkpoint on both IFEval and Arena-Hard at both student sizes, while the forward-KL variant underperforms it.The comparison uses the original student checkpoint as the reference for performance changes.Source: §C.2 · pp. 21–22

C.3 Stochastic Intervention v.s. Deterministic Threshold

STI-OPD is compared with deterministic discrepancy thresholds and with other intervention methods under a matched teacher budget. It outperforms the tested alternatives in both comparisons.

C.3 Stochastic Intervention v.s. Deterministic Threshold

  • The threshold comparison holds models, data, training settings, rollout budget, and evaluation protocol constant while testing thresholds 0.5, 1.0, and 2.0.It focuses on stochastic probability sampling versus a hard decision boundary.Source: §C.3 · p. 22
  • STI-OPD beats all tested deterministic thresholds across ALFWorld categories and ScienceWorld; deterministic performance peaks at 1.0 and degrades on either side.The stochastic strategy avoids a manually tuned decision boundary while preserving opportunities for student execution.Source: §C.3 · pp. 22–23, Table 7 · p. 23

C.4 Teacher Computation Matched Comparison

  • The matched-budget comparison caps and equalizes teacher interventions across methods, keeping other settings identical to the main experiments.This controls for intervention quantity and tests how methods allocate and learn from the fixed budget.Source: §C.4 · p. 23
  • Under an equal teacher-intervention budget, STI-OPD improves 1.4–7.0 points over FUTUREBRIDGE-OPD across seven tasks, averaging approximately 4.2 points.The largest reported gains are LOOK (+7.0), ScienceWorld (+5.5), and PICK TWO/PICK & HEAT (+4.8 each).Source: Table 8 · p. 23, §C.4 · p. 23

C.5 Case Studies

The cases trace intervention-assisted behavior becoming student-controlled at local and full-trajectory scales. Later successful trajectories show fewer ineffective actions or no teacher-generated turns across the illustrated tasks.

Case C.1: Local Transfer — The Microwave Subtask Becomes Student-Controlled

  • On the microwave subtask, navigation and opening scores fall from 0.845/0.846 to 0.231/0.287 as the student learns the corrected sequence.The later trajectory still contains two earlier teacher actions, so this case is local rather than end-to-end transfer.Source: §C.5 · p. 24

Case C.2: End-to-End Transfer — Search and Placement without Intervention

  • The student’s pen-search and placement trajectory reaches reward 1 with 9→0 teacher interventions, 4→0 ineffective actions, and 24→21 turns.The later trajectory completes the multi-stage task autonomously, including locating both pens and returning them to the desk.Source: §C.5 · pp. 24–25

Case C.3: Compositional Transfer — The Full Skill Chain Becomes Student-Controlled

  • In the bowl-cooling task, the student later completes the required acquire–cool–place chain in eight valid actions, with teacher use falling from seven turns to zero.The early trajectory detours and stalls; the later trajectory succeeds without intervention.Source: §C.5 · pp. 25–26

D Limitations and Future Work

The evaluation is limited to text-only models and interactive tasks, and the method requires teacher probabilities unavailable through sampling-only APIs. Whole-turn regeneration also constrains how selectively teacher supervision can be applied.

D Limitations and Future Work

  • Experiments cover text-only LLMs, two student and teacher scales, and two families of text-based interactive tasks; multimodal, embodied, and irreversible-side-effect settings remain untested.Whether discrepancy-calibrated intervention transfers to those settings is open.Source: D Limitations and Future Work · p. 26
  • STI-OPD requires per-token teacher probabilities on student-proposed and teacher-generated tokens, excluding teachers available only through sampling APIs.Source: D Limitations and Future Work · p. 26
  • When intervening, STI-OPD regenerates the entire turn, spending supervision on the full response even when discrepancy is concentrated in one decision.The method does not correct only part of a turn.Source: D Limitations and Future Work · p. 26

Related papers

Loading 2610.10878v1…