PaperlayerView exact revision on arXiv

Source-linked AI summary

DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

Qijia He, Zequn Li, Spencer Luo, Tianle Chen, Pradyumna Narayana, Zixian Ma, Ranjay Krishna, Lan Nie, Sugato Basu

arXiv:2610.19302v1cs.LGcs.AI

TL;DR

OPD supervises student-visited states, but its signal may be unreliable when teacher and student diverge or errors compound across long reasoning and multi-turn interaction. DTDD triggers teacher takeover using segment-level student–teacher divergence, retaining OPD on the student prefix and behavior-cloning the teacher-generated recovery suffix. Action-level takeover improves best-observed performance by 4.8 and 4.1 percentage points on ALFWorld with LoRA and agentic code, respectively.

  • Problem

    OPD supervises student-visited states, but its signal may be unreliable when teacher and student diverge or errors compound across long reasoning and multi-turn interaction.

    Source: §1 · pp. 1–2

  • Method

    DTDD triggers teacher takeover using segment-level student–teacher divergence, retaining OPD on the student prefix and behavior-cloning the teacher-generated recovery suffix.

    Source: §1 · p. 2

  • Results

    Action-level takeover improves best-observed performance by 4.8 and 4.1 percentage points on ALFWorld with LoRA and agentic code, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across 23 experimental arms, takeover tends to help when the teacher outperforms the OPD student, can recover from student prefixes, and provides learnable continuations at an intervention rate that is neither excessive nor insufficient.

    Source: §1 · p. 3

    Takeover can route training to an invalid supervision source when the teacher underperforms OPD, fail when the student cannot learn recoveries, or replace too much on-policy interaction when intervention is excessive.

    Source: §4.5 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use. We introduce Divergence-Triggered Dynamic Distillation (DTDD), an adaptive OPD framework that uses segment-level student-teacher divergence to trigger teacher takeover during training. When divergence triggers an intervention during reasoning, the student prefix continues to receive standard OPD supervision. The teacher then generates a recovery suffix, which the student learns through behavior cloning, using cross-entropy on the teacher-generated tokens. We evaluate DTDD in three regimes: long-horizon reasoning, distillation from a base initialization without offline supervised fine-tuning (SFT), and multi-turn agents. We also compare alternative trigger designs and teacher-intervention rates. DTDD reduces the post-warmup peak gradient norm by $8.6\times$ and, from a base initialization, reaches $21.3\%$ held-out math accuracy in $2.6\times$ fewer optimizer updates than OPD; both methods finish near $22\%$. It improves best-observed performance by $4.8$ and $4.1$ percentage points on ALFWorld with LoRA and agentic code, respectively. Ablations show that segment-averaged triggers outperform token-spike triggers. Behavior cloning is also more effective than the tested OPD surrogate for training teacher-generated recoveries. Across 23 experimental arms, takeover tends to help when the teacher substantially outperforms the OPD-trained student, provided that the teacher can recover from student-visited states, the student can learn those recoveries, and teacher intervention is neither excessive nor insufficient.

1. Introduction

OPD supervises student-visited states, but its guidance can be unreliable when the teacher signal mismatches the student or errors compound. DTDD triggers teacher takeover at divergence, retaining OPD on the student prefix and training a teacher-generated recovery suffix; evaluations show more stable or faster learning and improvements in agent tasks.

1. Introduction

  • OPD can provide ineffective guidance when teacher and student thinking patterns mismatch, long reasoning amplifies deviations, or incorrect tool calls compound errors.Student-state coverage alone does not ensure effective learning; the signal must also be reliable and learnable at those states.Source: §1 · pp. 1–2
  • DTDD intervenes when segment-level student–teacher divergence is high, keeping OPD on the student prefix and behavior-cloning the teacher-generated recovery suffix.Without a trigger, DTDD reduces to standard OPD; in multi-turn settings, takeover replaces the current reasoning-and-tool-call action.Source: §1 · p. 2
  • 8.6× lower post-warmup peak gradient norms with 15K- and 18K-token budgets, while preserving held-out accuracy.From Qwen3-1.7B-Base, high-dose DTDD reaches 21.3% held-out accuracy around update 70, versus around update 180 for OPD; both finish near 22%.Source: §1 · pp. 2–3
  • At comparable accuracy, DTDD requires 22.0 serial hours and 43M teacher-written tokens, versus 42.8 hours and 95M for offline SFT followed by OPD.Source: §1 · p. 3
  • Action-level takeover improves best-observed performance by 4.8 and 4.1 percentage points on ALFWorld with LoRA and agentic code, respectively.Across 23 arms in 13 settings, DTDD tends to help when the teacher outperforms the OPD student, can recover from student prefixes, and supplies learnable continuations at an appropriate intervention rate.Source: §1 · p. 3

2. Related Work

Prior work addresses unreliable OPD either by modifying its learning signal along student trajectories or by introducing teacher-generated corrections. DTDD combines discrepancy-triggered trajectory intervention with OPD on the retained prefix and behavior cloning on a regenerated suffix.

2. Related Work

  • Prior OPD work modifies supervision within student trajectories or introduces teacher-generated corrections, motivating choices about intervention boundaries and training objectives.DTDD complements methods that improve initialization, augment training data, or reshape the learning signal by regenerating an unreliable region of the current student trajectory.Source: §2 · p. 3
  • Unlike trajectory interventions with fixed or independently scheduled boundaries, DTDD uses student–teacher discrepancy to select the intervention point.Prior approaches include complete-rollout rewriting, per-step targets, random handoffs, scheduled student control, and state-dependent teacher segments.Source: §2 · p. 3
  • Compared with AdaSwitch’s token-spike trigger and unified KD objective, DTDD averages divergence over segments and trains the retained prefix with OPD and regenerated suffix with behavior cloning.DTDD rewinds to the first divergent segment and asks the teacher to regenerate the remaining suffix from the retained student prefix.Source: §2 · p. 3
  • DTDD’s distinctive training signal is teacher recovery used as supervision during student training.In reasoning, the student prefix receives OPD and the teacher-generated suffix receives behavior cloning.Source: §2 · pp. 3–4

3. Divergence-Triggered Dynamic Distillation

DTDD uses on-policy student trajectories and segment-averaged divergence to decide when to replace a trajectory suffix with teacher-generated recovery. It trains retained student prefixes with OPD and recovered suffixes with behavior cloning, while no-trigger rollouts remain standard OPD.

3.1. On-Policy Distillation Preliminaries

  • OPD evaluates frozen-teacher next-token distributions along student-generated histories, using a top-K approximation of reverse-KL supervision.The student samples a rollout; at each position the student and teacher distributions are defined on the same student-visited history.Source: §3.1 · p. 4

3.2. DTDD Overview

  • DTDD selects the first segment whose aggregated student–teacher divergence crosses a prespecified threshold, rewinds to its preceding boundary, and regenerates the suffix with the teacher.The retained student prefix receives OPD and the teacher suffix receives behavior cloning; without a crossing, the rollout receives standard OPD.Source: §3.2 · p. 4, Figure 2 · p. 5
  • The intervention dose controls whether takeover supplies frequent coherent guidance to a base student or sparsely repairs failures in effective OPD.In multi-turn tasks, takeover occurs at action boundaries from the triggering state before interaction continues.Source: §3.2 · p. 4

3.3. Segment-Level Trigger and Recovery

  • The trigger uses a nonnegative discrepancy measuring disagreement magnitude, averaged over ordered segments rather than individual token spikes.Reasoning segments contain fixed-length token blocks; agentic segments contain one model-generated action, excluding environment observations.Source: §3.3 · p. 5
  • The first eligible threshold crossing sets the rewind point; if none crosses, the trajectory is unchanged and receives standard OPD.Averaging divergence within a segment prevents isolated token-level spikes from causing intervention.Source: §3.3 · p. 5

3.4. Role-Split Training Objective

  • For recovered reasoning rollouts, OPD trains the retained prefix and token-level cross-entropy trains the teacher-generated suffix, conditioned on the preceding hybrid trajectory.If no trigger occurs, the objective reduces to OPD; the main experiments set the suffix-loss weight to 1.Source: §3.4 · p. 6

4. Experiments

The experiments test DTDD on long-horizon reasoning, base-model learning without offline SFT, and multi-turn agents, comparing controlled OPD and DTDD runs across these settings. They examine stability, learning efficiency, agent performance, and conditions associated with takeover gains.

4. Experiments

  • The experiments address long-horizon stability, base-model learning efficiency, transfer to interactive tasks, and which student–teacher regimes support intervention.The design ablates triggering mechanisms and teacher-suffix losses, alongside alternative intervention configurations.Source: §4 · pp. 6–7, §4.1 · p. 7

4.1. Experimental Setup

  • The evaluation pairs controlled OPD and DTDD runs across math reasoning, ALFWorld, and agentic code, reporting task-specific accuracy or success metrics.Within settings, paired runs share student initialization, teacher, prompts, batch order, generation budget, and optimizer.Source: §4.1 · p. 7

4.2. Q1: Does DTDD Stabilize Long-Horizon OPD?

  • At 15K tokens, DTDD reduces the post-warmup gradient peak from 12.02 to 1.40 (8.6×) while keeping held-out accuracy within one point of OPD.At 18K, gradient peaks are 12.81 for OPD and 1.84 for DTDD (7.0×); segment-averaged triggers outperform token-spike triggers, and behavior cloning outperforms the tested OPD surrogate on teacher suffixes.Source: §4.2 · pp. 7–8, §4.1 · p. 7

4.3. Q2: Can Takeover Warm Up a Base Student Online?

  • From Qwen3-1.7B-Base, DTDD reaches 21.3% held-out accuracy near update 70 versus OPD near update 180, with both later near 22%.DTDD reaches 21.34% in 22.0 hours using 43M teacher-written tokens, versus 21.55% in approximately 42.8 hours and 95M tokens for offline SFT followed by OPD.Source: §4.3 · pp. 8, 10, Figure 4 · p. 9

4.4. Q3: Does Divergence-Guided Routing Transfer to Agentic Tasks?

  • Best-observed results improve from 76.49% to 81.32% on ALFWorld and from 36.49% to 40.59% on agentic code with sparse takeover.On ALFWorld, takeover falls near zero as the student improves but reactivates during a later discrepancy rise; in agentic code the teacher-token dose is 2.8%.Source: §4.4 · p. 10, Table 3 · p. 11

4.5. Q4: What Determines Whether Teacher Takeover Helps?

  • Across 23 arms in 13 settings, gains concentrate in regimes where OPD retains more headroom below the teacher.The controls identify three constraints: the teacher must be a valid supervision source, the student must learn recoveries, and intervention must not be excessive.Source: §4.5 · p. 10, Figure 6 · p. 11

5. Conclusion and Limitations

DTDD routes supervision between standard OPD and teacher-generated recovery using segment-level student–teacher divergence. The conclusion reports benefits across several settings while identifying manual intervention control and unassessed recovery usefulness as limitations.

5. Conclusion and Limitations

  • DTDD's action-level takeover improves ALFWorld and agentic code; relative to offline SFT followed by OPD, it reaches comparable accuracy with 1.9× less measured serial time and 55% fewer teacher-written tokens.Source: §5 · p. 11
  • Across paired experiments, sparse takeover suppresses long-budget optimization extremes, accelerates learning from a base initialization, and improves ALFWorld and agentic code.Controlled comparisons favor segment-level triggering with behavior cloning.Source: §5 · p. 11
  • DTDD uses manually specified thresholds or target takeover rates, and its divergence trigger does not assess whether teacher recovery is useful for the student.Automatic dose calibration and recovery-quality assessment are identified as future work.Source: §5 · p. 11

A. Measured Limitations of Vanilla OPD

This appendix documents measured OPD behavior and the training, controller, and interactive-agent protocols used in the experiments. It also specifies a limitation of the ALFWorld and WebShop teacher-action surrogate.

A. Measured Limitations of Vanilla OPD

B. Experimental Details and Reporting Protocols

  • The appendix expands experimental settings with task-specific generation budgets, optimization and controller settings, interactive objectives, and evaluation protocols.The budget specifies maximum generated tokens for reasoning tasks or maximum environment/tool turns for interactive tasks.Source: B. Experimental Details and Reporting… · p. 15, Table 4 · p. 15

B.1. Reasoning Training Protocol

  • Reasoning runs use 64 prompts with four rollouts per prompt, one policy update per batch, a frozen teacher, and no sequence-level outcome reward.The protocol uses AdamW, a constant 10^-6 learning rate without warm-up, global gradient clipping at 1.0, and student top-K distillation with K=16.Source: §B.1 · p. 16

B.1.1. Controller and Recovery Details

  • Reasoning takeover is retrospective: DTDD evaluates completed student rollouts, retains the prefix, and replaces the suffix with teacher continuation.This changes the training sequence without reducing student rollout compute; fixed or rate-controlled thresholds determine triggers, with the first crossing setting the boundary.Source: §B.1 · p. 16, §B.1.1 · p. 16

B.2.1. ALFWorld and WebShop

  • The interactive protocol defines action-token objectives using student- and teacher-written tokens, with implementations differing in recovery horizon and teacher-action loss.For ALFWorld and WebShop, teacher actions use a sampled-token multi-turn OPD surrogate; the reported implementation is not an unbiased estimator of either KL direction.Source: §B.2 · p. 16, §B.2.1 · p. 17
  • ALFWorld uses a Qwen2.5-3B-Instruct student and GiGPO-7B teacher, while WebShop uses the same student and a WebShop-specific GiGPO-7B teacher.In both, the teacher replaces the triggering action and the student resumes at the next turn.Source: §B.2.1 · pp. 17–18

B.2.2. Agentic Code

  • Agentic code applies OPD to student-written tokens and behavior cloning to teacher-written tokens, with takeover online at an action boundary.After takeover, the teacher generates all remaining turns; the runs use Qwen3-1.7B-SFT with DemyAgent-GRPO-4B on the code-only portion of Open-AgentRL-30K.Source: §B.2.2 · p. 18

B.3. Evaluation and Checkpoint Selection

The appendix explains checkpoint selection for reported performance envelopes and defines matched-checkpoint summaries for comparing learning curves. It also clarifies that checkpoints from one run are correlated rather than independent training replicates.

B.3. Evaluation and Checkpoint Selection

  • Table 3 selects each arm’s checkpoint by the Average within its block, using only checkpoints with results for every benchmark in that block.The selected updates vary by method and task; for example, ALFWorld uses updates 280 and 140 for OPD and DTDD, respectively.Source: §B.3 · p. 18

B.4. Matched-Checkpoint Comparisons

  • Table 7 evaluates trained methods within each block at a shared update: 200 for math, 400 for ALFWorld, and 60 for agentic code.Its subscripts show absolute percentage-point changes from OPD, and bold marks the best trained method.Source: Table 7 · p. 19, §B.4 · p. 19

B.3. Evaluation and Checkpoint Selection

  • Every curve represents one training run, so checkpoints from the same run are correlated and are not treated as independent training replicates.Average@k measures evaluation-time sampling variation rather than training-seed uncertainty.Source: §B.3 · p. 19

B.4. Matched-Checkpoint Comparisons

  • Last Δ, last-five Δ, and all-shared Δ distinguish endpoint differences from average DTDD–OPD accuracy gaps across shared checkpoints.All differences are in percentage points of held-out average@16 accuracy; OPD late range measures its final-five-checkpoint range within one run.Source: §B.4 · p. 19

C.1. Training Dynamics across Response Budgets

Training-dynamics analyses distinguish a stable 7K control from coupled OPD instability at longer budgets, and test clipping and intervention behavior. The reported results also examine trigger timing and the exploration floor.

C.1. Training Dynamics across Response Budgets

  • At 7K, OPD and DTDD follow almost the same entropy, gradient, truncation, teacher-overlap, and train-batch reward trajectories.This setting serves as the stable control.Source: §C.1 · p. 20, Figure 8 · p. 20
  • At 15K, OPD’s gradient and entropy rise with increasing truncation and declining teacher overlap; DTDD limits the event while preserving held-out accuracy.The 18K run reproduces the coupled event; final-five DTDD–OPD accuracy differences are −0.73 and −0.20 points at 15K and 18K, respectively.Source: §C.1 · p. 20, Figure 9 · p. 21

C.2. Signal-Clipping Control

  • A tight b=1 OPD advantage clamp suppresses late gradient spikes at both budgets, whereas b=5 leaves peaks and high-gradient updates nearly unchanged.At update 279, the tight clamp changes held-out average@16 accuracy by +0.55 and −1.08 points relative to unclipped OPD at 15K and 18K, respectively.Source: §C.2 · p. 21, Table 8 · p. 22

C.3. Intervention Dynamics and Exploration-Floor Ablation

  • Takeover rates range from 2–8% at 7K to 35% initially at 15K and 91% initially at 18K; longer-budget runs respond again to late discrepancy increases.In the 18K shadow run, the would-trigger rate rose to 25% at update 201 before the gradient spike began at update 204; removing the exploration floor changed matched accuracy by −0.05 point.Source: §C.3 · p. 22

D.1. Learning Curves and Matched-Checkpoint Results

The checkpoint analyses distinguish early learning gains from late-training performance and extend comparisons across controller arms and accuracy targets. DTDD’s advantage is strongest across shared checkpoints, while later gains are smaller and depend on the controller and target.

D.1. Learning Curves and Matched-Checkpoint Results

  • 4.18 points is the mean held-out accuracy gain over OPD for the C=128 rewind run across 12 shared checkpoints, versus 0.56 over the final five and 0.10 at the last checkpoint.The larger all-checkpoint gain and smaller late-window gain reflect faster early learning rather than a higher final plateau.Source: §D.1 · pp. 22–23, Table 9 · p. 23
  • Per-benchmark results extend the base-initialization comparisons to shared update 240 for DAPO-Math-17K and update 200 for OpenThoughts3.The section also reports matched-checkpoint results at updates 120 and 240, including the 45% restart arm.Source: §D.1 · pp. 22–23
  • 1.47–1.68 points are the final-five checkpoint gains for all three OpenThoughts3 controllers, with positive gains also at update 200.These summaries support a sustained late-training improvement beyond selected best checkpoints.Source: Table 10 · p. 23, §D.1 · p. 23
  • At the 22.29% target, C=64 and C=256 reach the level at updates 60 and 100, versus update 200 for OPD seed 0; C=128 reaches it at update 220.At 22.75%, C=256 and the 45% restart arm cross at updates 180 and 220, while C=64 and C=128 stay below through update 240.Source: §D.1 · p. 23

D.2. Controller Dynamics and Design Ablations

The appendix details how controller behavior, trigger and loss choices, and computational costs vary across the base-student experiments. It also qualifies these comparisons: controller designs differ in realized intervention characteristics, and timing measurements use different hardware allocations.

D.2. Controller Dynamics and Design Ablations

  • The length-gated control’s rollout-level takeover rises from approximately 25% to 94% as rollouts lengthen, while the 45% rate-controlled restart realizes approximately 37% after eligibility filtering.Realized coverage differs from the initial or target rate used to label each arm.Source: §D.2 · p. 24
  • Figure 11 tracks takeover rate, normalized segment divergence, teacher-written token fraction, and takeover position for C=64, 128, and 256.These measurements distinguish takeover frequency from how much of each triggered rollout the teacher writes.Source: §D.2 · p. 24, Figure 11 · p. 25
  • Table 13 compares complete controller designs, not isolated causal effects of trigger granularity or suffix objective.The unified-KD suffix control can exclude the teacher-written token because its reverse-KL surrogate uses the student’s top-K support.Source: §D.2 · pp. 24–25, Table 13 · p. 26
  • From update 60 onward, seed-to-seed standard deviation is at most 0.45 points for both methods, against a 6.3–8.2-point gap between them.The two-seed table also includes earlier evaluations at updates 20 and 40.Source: §D.2 · p. 25, Table 15 · p. 26

D.3. Offline SFT Baseline and Computational Cost

  • SFT followed by OPD reaches 21.55% at its first evaluation, update 20, and ranges from 20.6% to 22.8% through update 240.Most of its observed gain is present by the first evaluated OPD checkpoint.Source: §D.3 · p. 26, Table 16 · p. 27
  • 5.4 minutes is the later-training OPD update time on two A100 GPUs, versus 15.1 minutes for DTDD on four A100 GPUs and 19.5 minutes on four L40S GPUs.These observed timings use different resource allocations and are not hardware-normalized; the offline route also spends 26.9 hours generating teacher data and 11.7 hours on SFT before OPD.Source: §D.3 · pp. 26–27, Table 14 · p. 26, Table 17 · p. 27

E.1. ALFWorld Action-Level Intervention

The appendix extends the ALFWorld and agentic-code analyses with matched endpoints, intervention dynamics, and dose controls. The comparisons show that the apparent advantage varies with training capacity and intervention rate.

E.1. ALFWorld Action-Level Intervention

  • At update 400, ALFWorld LoRA success rises from 70.06% with OPD to 77.59% with DTDD, while full-model OPD reaches 79.50% and narrows DTDD’s advantage to about 0.4 point.The LoRA gain is driven primarily by the seen split, and update 400 is the last common checkpoint.Source: §E.1 · p. 27, Table 18 · p. 27
  • The ALFWorld controller withdraws as discrepancy contracts and reactivates around update 180; the companion dynamics figure tracks action discrepancy and intervention timing within episodes of up to 30 turns.The figures report takeover rate, action discrepancy, intervention turn, and relative episode depth.Source: Figure 12 · p. 28, Figure 13 · p. 28
  • 7.22 points is DTDD’s average lead across the last five matched ALFWorld checkpoints, while OPD varies by 10.2 points over that window.The reported variation motivates repeated runs to distinguish intervention effects from late-training variation.Source: §E.1 · p. 28

E.2. Agentic Code Dose

  • At shared update 130, takeover with τ=0.50 remains 1.03 points above OPD in aggregate because of gains on the two in-distribution splits, although both LiveCodeBench scores trail OPD.Held-out averages the two in-distribution splits; LCB averages LiveCodeBench v1 and v6.Source: §E.2 · p. 29, Table 19 · p. 29
  • Approximately 50% teacher-token dose reduces code aggregate accuracy from 33.93% to 28.13% at update 130, reversing the gain over OPD.The teacher reaches 60.63% on the same aggregate evaluation; the contrasting outcome points to excessive replacement of a warm-start student’s on-policy interaction.Source: §E.2 · p. 29, Table 19 · p. 29

F. Headroom Analysis and Boundary Cases

F.1. Headroom Inventory and Reporting Details

The headroom inventory reports matched-checkpoint gains across 23 arms and 13 settings, then describes boundary cases and a reanalysis of published TCOD results. Its comparisons are descriptive and depend on whether teacher behavior is valid and learnable in the evaluated setting.

F.1. Headroom Inventory and Reporting Details

  • The 23-arm inventory reports teacher scores, OPD endpoints, paired-checkpoint counts, and mean DTDD–OPD gains; each arm has at least three paired evaluations.The ALFWorld LoRA arm illustrates protocol sensitivity: its matched-checkpoint mean is +1.11 points, versus a +4.83-point best-observed gap.Source: §F.1 · p. 29, Table 20 · p. 30
  • The headroom analysis is descriptive: arms sharing a setting are not independent, and the 10% boundary summarizes observed signs rather than defining a fitted threshold.Six DAPO-Math arms share one headroom value but range from +0.11 to +4.42 points as dose and intervention form change.Source: §F.1 · p. 29

F.2. Boundary Cases

  • On the rebuilt WebShop environment, the teacher scores 18.69% success, while at update 180 OPD reaches 21.21% and DTDD reaches 11.62%.The environment’s 120K-product catalog differs from the catalog and observation format on which its WebShop-specific teacher was trained.Source: §F.2 · p. 31, Table 21 · p. 31
  • For the 0.5B ALFWorld student paired with a roughly 14× larger teacher, OPD and DTDD average 0.79% and 0.96% across six evaluations, but update-120 scores are 3.65% and 1.80%.The teacher supplies valid behavior, but takeover does not produce reliable student-only success under this training configuration.Source: §F.2 · p. 31, Table 21 · p. 31

F.3. Reanalysis of TCOD Results

  • In published TCOD results, all 18 1.7B takeover cells improve by 9.2–24.7 points, while 4B gains average 0.3–1.5 points across benchmark-level curriculum configurations, with four negative cells.The 4B OPD baselines finish within 6.9–13.4% of their teachers; the comparison uses reported scores, not new TCOD runs.Source: §F.3 · p. 31

Related papers

Loading 2610.19302v1…