PaperlayerView exact revision on arXiv

Source-linked AI summary

Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation

Bing Wang, Shaotian Yan, Chen Shen, kaiyuan liu, Sinan Fan, Ximing Li, Rui Miao, Xiaosong Yuan, Zhanming Shen, Jieping Ye

arXiv:2605.19433v1cs.CLcs.AI

TL;DR

Reasoning distillation transfers teacher capabilities to smaller students, but off-policy and on-policy methods face opposing exposure biases: student inference contexts differ from teacher trajectories, while flawed student contexts can hinder teacher guidance. MOTAB monitors on-policy generation and backtracks to a safe state for teacher intervention; experiments report an approximately 3% average improvement in complex reasoning performance.

  • Problem

    Off-policy training on teacher trajectories mismatches student inference contexts, while on-policy student contexts can hinder positive teacher supervision.

    Source: §1 · p. 2

  • Method

    MOTAB monitors student-generated trajectories against an adaptive safety boundary and backtracks to a last safe state for teacher intervention.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Approximately 3% average improvement in complex reasoning performance, with dual exposure biases effectively mitigated.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MOTAB’s trajectory monitoring and backtracking framework is reported to enhance reasoning capabilities while mitigating both exposure biases.

    Source: §6 · p. 10

    The stated unbiased objective requires student-context coverage alongside teacher supervision conditioned on a valid context.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable success in complex reasoning tasks via long chain-of-thought (CoT), yet their immense computational overhead hinders real-world deployment. LLM reasoning distillation addresses this by transferring reasoning capabilities from formidable teacher models to compact student models. However, existing distillation paradigms face a fundamental dilemma. Typical off-policy distillation strictly utilizes teacher-generated golden trajectories, suffering from an exposure bias due to the mismatch between training distributions and student-generated inference contexts, which leads to error cascades in long CoT reasoning. To address this, on-policy distillation allows students to explore their own trajectories, but we demonstrate that it inherently introduces a reciprocal reversed exposure bias: the teacher model also struggles to provide positive guidance when conditioned on student-generated sub-optimal contexts. To resolve this dual exposure biases problem, we propose Monitoring Trajectories and Backtracking when it strays (MOTAB), a new LLM reasoning distillation pipeline. Specifically, MOTAB dynamically monitors the student's on-policy generation against an adaptive safety boundary. When the generation strays and exceeds this threshold, MOTAB backtracks to the last safe state and leverages teacher intervention to correct the course. This approach inherently tolerates minor student errors to mitigate exposure bias, while preventing sub-optimal contexts to circumvent reversed exposure bias. Extensive experiments on the LIMO-v2 and AceReason datasets demonstrate that MOTAB effectively alleviates the dual exposure biases, yielding a roughly 3% average performance improvement in reasoning tasks.

1 Introduction

Reasoning distillation aims to transfer capable teachers’ reasoning to smaller, faster students, but off-policy and on-policy training each introduce a distinct exposure bias. MOTAB monitors student trajectories and backtracks when they stray; experiments report improved reasoning performance and reduced dual exposure biases.

1 Introduction

  • Off-policy training on teacher-generated golden trajectories mismatches student-generated inference contexts, risking error cascades in long reasoning chains.On-policy distillation exposes students to their own trajectories, but teachers can struggle to provide positive guidance on sub-optimal student contexts.Source: §1 · p. 2
  • MOTAB monitors student-generated trajectories against an adaptive safety boundary and backtracks to the last safe state for teacher intervention.It tolerates some student errors while seeking a sufficiently accurate context for teacher guidance.Source: §1 · p. 2
  • Approximately 3% average improvement in complex reasoning performance, with dual exposure biases effectively mitigated.Experiments use Qwen3-32B and gpt-oss-120b as teachers, LIMO-v2 and AceReason data, and three student models.Source: §1 · p. 2
  • The paper frames off-policy and on-policy biases as a fundamental dilemma in reasoning distillation.Source: §1 · p. 2

2 Investigation on Dual Exposure Biases

The section defines two competing exposure biases and investigates them empirically, then states conditions for unbiased distillation and explains why speculative knowledge distillation does not satisfy them together.

2.1 Formal Definition of Dual Exposure Biases

  • Off-policy contexts mismatch student inference contexts, while flawed student-generated contexts can make teacher supervision uninformative, creating dual exposure biases.The section frames the dilemma as difficulty simultaneously aligning training with inference-time contexts and maintaining valid, high-quality supervision.Source: §2.1 · p. 3

2.2 Empirical Analysis of Dual Exposure Biases

  • Both context-distribution divergences increase with sequence length: the student’s divergence across its own versus teacher contexts, and the teacher’s across student-generated versus teacher contexts.Figure 1 compares teacher and student next-token log probabilities under off-policy and on-policy contexts; Figure 2 compares KL divergence across context lengths.Source: Figure 2 · p. 3, Figure 1 · p. 3, §2.2 · p. 4

2.3 What is the Ideal Unbiased State for Reasoning Distillation?

  • The ideal unbiased state requires training coverage of the student’s intrinsic context distribution and teacher supervision conditioned on a valid context.The section says these conditions remain challenging to satisfy together because the valid context differs from a sampled student context.Source: §2.3 · p. 4

2.4 Can Speculative KD Resolve Dual Exposure Biases?

  • Speculative KD mixes teacher and student contexts, but its forced corrections leave a coverage gap and can preserve invalid teacher contexts, motivating MOTAB.The stated gaps are failure to expose sub-optimal student trajectories and teacher out-of-distribution collapse after forced token replacements.Source: §2.4 · pp. 4–5

3 The Proposed MOTAB

MOTAB monitors student-generated reasoning, tolerates sub-optimal paths within an adaptive boundary, and backtracks for teacher correction when the boundary is exceeded. Its stitched training trajectories aim to cover the student's generation distribution while keeping teacher supervision on valid contexts.

3.1 On-policy Trajectory Monitoring via Adaptive Boundary

  • MOTAB monitors student steps against a teacher-guided adaptive boundary, widening tolerance when teacher entropy indicates multiple acceptable reasoning paths.Steps that meet the boundary are preserved, including logically sound but stylistically different explorations.Source: §3 · p. 5, §3.1 · pp. 5–6

3.2 State Backtracking via Credit Assignment

  • When a step breaches the boundary, credit assignment traces the logical drift and backtracks to the last safe state before teacher intervention.This avoids asking the teacher to correct from a flawed context, which the paper says can trigger reversed exposure bias.Source: §3.2 · p. 6

3.3 Off-policy Trajectory Stitching

  • MOTAB stitches the student's flawed exploration to a teacher-generated correction using a [REV] token, teaching the student to recognize and correct its drift.The student is trained on its sub-optimal state to predict [REV] and then produce the correct steps.Source: §3.3 · p. 6

3.4 Theoretical Alignment to The Ideal Objective

  • The theoretical analysis presents MOTAB's objective as satisfying ideal coverage and validity conditions, unlike SKD's narrower state distribution.The student-generated prefix supplies coverage, while the teacher correction is sampled from a pristine state to preserve valid supervision.Source: §3.4 · pp. 6–7

4 Experimental Evaluation

Across the reported settings, MOTAB outperforms the baselines and is evaluated across datasets, student architectures, and downstream domains. Ablations and sensitivity tests examine its components and boundary, while trajectory probabilities are used to assess the dual exposure biases.

4.1 Main Results

  • 66.64% average accuracy with Qwen3-32B as teacher, over 7% above strongest baseline SKD; MOTAB also outperforms baselines with gpt-oss-120b.The comparison spans teacher models and metrics; the evaluation uses LIMO-v2 and AceReason data and three student models.Source: §4.1 · p. 7, §4 · p. 7
  • MOTAB shows superior trends on AceReason, transfers to Qwen2.5-7B and Llama3.2-3B, and achieves state-of-the-art cross-domain results on IFEval and GPQA.The authors report the cross-domain results regardless of teacher model.Source: §4.1 · p. 8, Table 3 · p. 8, Table 6 · p. 9
  • MOTAB significantly outperforms off-policy baselines and remains comparable to on-policy methods, which the authors describe as more time- and memory-intensive.This comparison uses Qwen3-8B as teacher and Qwen3-1.7B as student because on-policy methods require substantial memory and vocabulary alignment.Source: §4.1 · p. 8, Table 4 · p. 8

4.2 Ablation and Sensitivity Analysis

  • Removing any core module degrades performance; removing state backtracking causes the largest reported drop, from 34.17% to 30.83% on AIME24 Pass@1.Omitting sub-optimal trajectory stitching or replacing the adaptive boundary with a static threshold also harms accuracy.Source: §4.2 · p. 8, Table 2 · p. 8
  • AIME25 Pass@1 follows an inverted-U as γ0 increases, peaking at γ0 = 0.3, while stricter boundaries monotonically raise backtracking frequency.The paper attributes low thresholds to missed severe drifts and high thresholds to interrupting acceptable exploration.Source: Table 5 · p. 8, §4.2 · p. 9

4.3 Evaluating Dual Exposure Biases

  • Trajectory log probabilities show stable teacher confidence with backtracking; at γ0 = 0.3 a distributional gap remains, while at 0.5 student and teacher distributions nearly align.The authors characterize the remaining gap at 0.3 as preserving student exploration while maintaining valid teacher supervision.Source: §4.3 · p. 9

5 Related Works

Prior LLM reasoning distillation work is described as largely off-policy, with on-policy approaches emerging to address exposure bias. The paper frames MOTAB as addressing the dilemma between these paradigms.

5 Related Works

  • Prior off-policy reasoning distillation typically samples problems, generates multiple teacher responses, and filters them using criteria such as length, correctness, and student adaptability.Source: §5 · p. 9
  • On-policy distillation lets students generate trajectories while receiving dense teacher feedback, including token-level divergences or probabilities.The passage also notes reinforcement-learning methods using LLM-generated rollouts.Source: §5 · p. 9
  • The paper positions MOTAB as addressing the dual-bias dilemma, noting that mixture and speculative distillation schemes do not satisfy the ideal unbiased condition.Source: §5 · pp. 9–10

6 Conclusion

The paper frames reasoning distillation as a dilemma: off-policy training risks inference error cascades, while on-policy training can leave teachers unable to guide sub-optimal student contexts. Its theoretical analysis argues that naive mixing cannot remove both biases, whereas MOTAB uses adaptive monitoring and backtracking to address them.

6 Conclusion

  • MOTAB uses an adaptive safety boundary to let students explore their own errors, then backtracks for teacher intervention before contexts degrade irreparably.The conclusion presents this as a way to mitigate exposure bias while circumventing reversed exposure bias.Source: §6 · p. 10

A Theoretical Analysis

  • The theoretical analysis sets out questions about why off-policy and on-policy distillation fail, why their mixture fails, and how MOTAB guarantees coverage and valid supervision.It identifies the dual exposure biases as the central dilemma.Source: A Theoretical Analysis · p. 14

A.1 RQ1: Exposure Bias in Off-Policy Distillation

  • Off-policy distillation’s inference-time sequence divergence can grow as O(L^2) in the worst case, with early student errors moving later prefixes off the teacher’s data manifold.The analysis states that compounding sequence error grows quadratically as O(L^2ϵ).Source: §A.1 · p. 14

A.2 RQ2: Reversed Exposure Bias in On-Policy Distillation

  • On sub-optimal student prefixes, the teacher’s conditional distribution can collapse toward uniform, making its supervision suppressive rather than constructive.The analysis says this can force the student to unlearn deterministic logical reasoning; strict on-policy supervision also requires shared tokenizers and synchronous teacher evaluation.Source: §A.2 · p. 15

A.3 RQ3: The Failure of Naive Distribution Mixing

  • Naive mixing cannot resolve both biases: emphasizing student contexts risks entropy collapse in teacher supervision, while emphasizing teacher contexts preserves inference error cascades.The analysis characterizes this as a zero-sum trade-off for any mixture coefficient.Source: §A.3 · pp. 15–16

A.4 RQ4: Theoretical Guarantees of MOTAB

The analysis argues that MOTAB’s truncated prefix distribution covers the student’s authentic distribution up to bounded divergence, while backtracking preserves valid teacher supervision. Its algorithm monitors student steps, accepts those within the adaptive boundary, and stitches a teacher correction after a breach.

A.4 RQ4: Theoretical Guarantees of MOTAB

  • MOTAB’s truncated prefix distribution covers the authentic student distribution up to bounded divergence, while its adaptive boundary triggers only for sufficiently severe logical drift.The boundary is conservative when multiple valid reasoning paths are available, so recoverable stylistic variation can remain covered.Source: §A.4 · p. 16
  • After a boundary breach, MOTAB backtracks to a verified safe state and has the teacher generate a correction from that pristine context, preserving valid supervision.The student learns a mapping from its flawed exploration to the corrective sequence through a [REV] transition.Source: §A.4 · p. 17

B Algorithmic Pipeline of MOTAB

  • The pipeline accepts student steps within the boundary; on a critical drift, it locates a safe bifurcation state and stitches the teacher’s corrective suffix to the exploration using [REV].The teacher evaluates transition likelihood and context entropy during generation.Source: B Algorithmic Pipeline of MOTAB · p. 17

C Computational Complexity Analysis

MOTAB shifts teacher work toward parallel verification and selective correction, and its measured synthesis time is comparable to SKD in the reported setup. The supplied setup passages describe the training data sources and trajectory counts.

C Computational Complexity Analysis

  • MOTAB tolerates sub-optimal exploration to reduce teacher interventions, shifting workload from autoregressive generation to parallel verification and using cached values for zero-pass backtracking.The analysis also states that temporal-difference backtracking enables credit assignment at zero additional inference cost.Source: C Computational Complexity Analysis · pp. 18–19
  • MOTAB and SKD each require approximately 3 to 4 hours to generate 1k reasoning trajectories on average on 8×A800 GPUs.The comparison covers three hyperparameter configurations controlling teacher intervention rates.Source: C Computational Complexity Analysis · p. 19

D.1 Training Datasets

  • The training data come from 800 LIMO-v2 mathematical problems with five responses each and a sampled 10,000 AceReason problems with one response each.The section introduces these as the two primary training sources.Source: §D.1 · p. 19

D.2 Evaluation Benchmarks

The experimental setup evaluates student models on mathematical and cross-domain benchmarks, compares them with fine-tuning and distillation baselines, and specifies implementation protocols. The supplied final passage only introduces a section of additional results and analysis.

D.2 Evaluation Benchmarks

  • The evaluation suite includes AIME2024, AIME2025, and MATH500 for mathematics, plus IFEval and GPQA for cross-domain capabilities.AIME and GPQA use eight samples per problem with pass@k reporting; MATH500 reports accuracy by difficulty level.Source: §D.2 · pp. 19–20

D.3 Model Architectures

  • The experiments use Qwen3-32B and gpt-oss-120b teachers, with Qwen3-4B-Instruct, Qwen2.5-7B-Instruct, and Llama3.2-3B-Instruct students.The student models span 3B to 7B parameters.Source: §D.3 · p. 20

D.4 Baseline Methods

  • The baseline set includes SFT, RFT, ImitKD, SKD, top-k OPD, and DASD.The descriptions distinguish fine-tuning, imitation-based, speculative, and on-policy distillation approaches.Source: §D.4 · pp. 20–21

D.5 Implementation Details

  • MOTAB data generation uses bounded reasoning-step and trajectory lengths, estimates teacher entropy from renormalized top-20 log probabilities, and creates corrective suffixes from a safe context.The resulting SFT trajectory joins student exploration with “However,” and teacher correction; student training uses six epochs of full-parameter SFT.Source: §D.5 · p. 21

E More Experimental Results

E.1 Experiments on More Student Models

Across three student models and diverse evaluation tasks, MOTAB outperforms off-policy and on-policy baselines, with results also showing gains across model scales and architectures.

E.1 Experiments on More Student Models

  • 22.08% AIME24 Pass@1 on Qwen3-1.7B, up from 9.58% with SFT, and 58.21% MATH500 Level 5 accuracy illustrate MOTAB’s gains over baselines.The evaluation also reports consistent superiority across Qwen3-1.7B, Llama3.2-3B-Instruct, and Qwen2.5-7B-Instruct.Source: §E.1 · pp. 21–22, Table 8 · p. 22
  • MOTAB also improves results on IFEval and GPQA, extending its reported enhancements beyond mathematical reasoning.The authors describe these gains across general instruction following and advanced scientific knowledge tasks.Source: §E.1 · p. 22
  • For Llama3.2-3B, MOTAB performs well in cross-architecture distillation from a Qwen teacher despite stylistic and distribution discrepancies.The passage contrasts MOTAB with SKD, whose optimal manual threshold varies by student architecture and evaluation metric.Source: §E.1 · p. 22

E.2 Analysis of Backtracking Dynamics

MOTAB’s interventions can rewind several steps to address errors rooted in earlier reasoning, while its safety threshold trades sharper detection for reduced exploratory coverage. Case studies illustrate teacher-guided correction after a logical breakdown.

E.2 Analysis of Backtracking Dynamics

  • Unsafe points occur throughout trajectories, and backtracking often rewinds multiple steps, consistent with errors rooted in earlier premises rather than only the immediately preceding step.Absolute unsafe points cluster earlier, whereas relative unsafe points are distributed more uniformly; depths greater than one occur across threshold settings.Source: §E.2 · pp. 22–23, Figure 5 · p. 23

E.3 Dynamics of Teacher Confidence and Safety Margins

  • A higher base threshold makes teacher value scores converge toward 1.0 faster and widens safety-margin separation, but can truncate acceptable exploration; γ0 = 0.3 balances detection and coverage.The safety-margin comparison is between trajectories that trigger backtracking and those that do not.Source: §E.3 · p. 23, Figure 6 · p. 24

E.4 Qualitative Case Studies

  • In the arithmetic case, a score of 0.42 below the 0.49 boundary triggers backtracking and teacher correction after the student spirals over a non-integer human count.A second case traces a premature boundary-condition leap back to the vertex calculation and adds a teacher-guided case analysis.Source: §E.4 · p. 24

F Limitations

MOTAB increases computational overhead during offline data synthesis, but that cost does not affect the fine-tuned student’s inference efficiency or deployment costs.

F Limitations

  • MOTAB’s offline trajectory synthesis is more computationally intensive than traditional off-policy methods.The process requires continuous step-wise monitoring, teacher conditional-likelihood calculations, predictive-entropy estimates, and frequent synchronous student–teacher interactions.Source: F Limitations · pp. 24–25
  • The stated overhead arises from synchronously monitoring student generation and computing teacher likelihoods and predictive entropy for the adaptive boundary.The passage contrasts this data-synthesis workload with traditional off-policy methods.Source: F Limitations · pp. 24–25
  • The added computational cost is confined to offline training preparation; fine-tuned student inference efficiency and deployment costs remain unaffected.The authors characterize this as a trade-off against performance gains and bias-mitigated distillation data.Source: F Limitations · p. 25
Loading 2605.19433v1…