PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing

Andrey Goncharov, Daniil Vyazhev, Simon Epanov, Alexey Zaytsev

arXiv:2610.20914v1cs.LG

TL;DR

When online teacher access is unavailable, offline distillation trains smaller models on pre-generated teacher outputs. This paper combines recursive entropy-based trace sampling with single-token answer mixing, reporting 57.15 ± 1.01 versus 51.61 on average for naive distillation.

  • Problem

    Access to an online teacher is not always possible, motivating offline distillation from pre-generated teacher outputs.

    Source: §1 · p. 1

  • Method

    The method recursively samples reasoning traces by student-proxy entropy gain and supplements them with single-token ground-truth answers.

    Source: §3 · p. 2

  • Results

    57.15 ± 1.01 vs 51.61 on average: the method outperforms naive distillation.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Adding a small portion of direct answers to reasoning chains provides a major uplift, with entropy gain sampling contributing additional gains.

    Source: §5 · p. 4

    The study uses a single dataset, so how well the results translate to other settings is unknown.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach. Frequently, researchers do not have full access to a teacher model and its internal representations, which limits the resulting performance. We propose a recursive offline distillation method combining two components: entropy-based sampling for selective distillation and answer mixing. Specifically, for three open models (3-4B), at each epoch, we sample reasoning chains weighted by entropy difference with an auxiliary medium model (70-72B) and, next, supplement thinking traces with single-token ground truth answers. Our approach consistently outperforms naive distillation with comparable data budgets ($57.15 \pm 1.01$ vs 51.61 on average) and offers a much steeper learning curve. Additionally, we present a comparative analysis of methods to generate and normalize distillation traces.

1 Introduction

Large models can be costly to adapt or deploy, while access to an online teacher is not always possible, motivating capable small models and offline distillation. The paper proposes a recursive offline pipeline using entropy-based sampling and answer mixing, and reports better average performance than naive distillation.

1 Introduction

  • Large models can achieve exceptional results, but adapting or running them requires computational resources unavailable to many practitioners.The introduction motivates capable smaller models that can be fine-tuned and deployed in resource-constrained environments.Source: §1 · p. 1
  • Offline distillation trains a student on a teacher’s pre-generated text output when access to an online teacher is unavailable.The paper presents its recursive pipeline as an offline distillation approach.Source: §1 · p. 1
  • 57.15 ± 1.01 vs 51.61 on average: the proposed method outperforms naive distillation.The pipeline is designed for resource-constrained environments.Source: §1 · p. 1
  • The pipeline combines a family of entropy-based samplers with a lightweight data mix.The paper also analyzes augmentation strategies for synthetic reasoning chains and studies trace generation, normalization, and sampling sensitivity.Source: §1 · p. 1

2 Related Work

Related work spans training objectives, data selection, and curriculum learning for distillation, alongside methods for synthesizing reasoning traces. The paper positions its approach as recursively selecting pre-generated traces using student-proxy entropy differences.

2 Related Work

  • Recent distillation work explores training objectives, data selection, and curriculum learning.Reported curriculum benefits vary with model capability, task complexity, and the difficulty metric.Source: §2 · pp. 1–2
  • ELAD selects uncertain student explanations for teacher revision, while complexity-aware fine-tuning selects reasoning examples by single-token entropy once.The paper contrasts these approaches with recursively sampling pre-generated traces using student-proxy entropy differences.Source: §2 · p. 2
  • Trace-synthesis approaches include explaining known solutions and learning from corrections; the paper compares direct generation and augmentations under constrained trace lengths.Source: §2 · p. 2

3 Method

The method recursively selects pre-generated reasoning traces using the student-proxy entropy difference and supplements them with single-token ground-truth answers. It motivates the entropy heuristic through uncertainty assumptions and uses a weighted training-data mix each epoch.

3 Method

  • The entropy gain heuristic is HS(x)−HP(x), intended to identify useful knowledge gaps by comparing student and proxy single-token entropies.The paper hypothesizes that the larger proxy has lower total uncertainty; its selected 70–72B size reflects resource constraints and open-model availability.Source: §3 · p. 2
  • The uncertainty rationale assumes H(x) = A(x) + E(x), with shared aleatoric uncertainty and model-specific epistemic components, and bounds entropy differences when epistemic residuals are close.Source: §3 · p. 2
  • Each epoch supplements 1,024 reasoning chains with 256 single-token answer examples to preserve non-thinking answer capabilities and improve performance.Source: §3 · p. 2
  • Each epoch samples 1,024 reasoning traces weighted by positive entropy gain, selects 256 single-token answers from that sample, then trains on the traces followed by answers.The described pipeline uses LoRA for training; the pseudocode specifies the student, proxy, questions, ground-truth answers, and precomputed traces as inputs.Source: §3 · p. 2

4 Results

Across three student models on MMLU-Pro, the proposed pipeline outperforms naive distillation, while trace-generation and sampling studies inform its choices. The experiments also report early gains in entropy correctness discrimination and strong performance from answer mixing despite its small token cost.

4.1 Experimental Setup

  • Experiments use MMLU Pro, with 14-domain questions and a random 80/20 train/test split; students are Qwen2.5 3B, Phi-4-Mini, and LLaMA 3.2 3B, with 70B/72B proxy models.Experiments ran on a single A100/H100 GPU with 80GB of VRAM.Source: §4.1 · pp. 2–3

4.1.1 Our method and results

  • The 256 added single-token answers increase training tokens by less than 0.01%, or 9% when system-prompt tokens are counted.The evaluation runs for 200 epochs, covering 205k questions, at 2,048- and 4,096-token thinking budgets, against methods with similar data budgets.Source: §4.1.1 · p. 3

4.2 Main Results

  • The proposed method achieves the best average result across three models for both reasoning budgets, with gains of up to 10% over naive distillation.Greedy sampling is subpar and less stable; one-time greedy student-entropy distillation shows severe degradation after its initial uplift.Source: §4.2 · p. 3, Table 1 · p. 3, Figure 2 · p. 3
  • The method improves single-token entropy correctness discrimination during the first half of training, but ROC AUC later declines, loosely tracking the method’s plateau.The passage reports an early increase followed by decline, without giving ROC AUC values.Source: §4.2 · p. 3

4.3 Synthetic Trace Generation

  • Trace comparisons favor head truncation, medium-to-short chains and thinking budgets, and augmented traces; the pipeline uses 8,192-character head truncation with Corrected answer generation.Longer chains and larger thinking budgets decrease performance, while consistent data leakage in generation degrades it.Source: §4.3 · p. 4, Figure 3 · p. 4

4.4 Sensitivity Study

  • Entropy gain performs better on average than the alternative sampling methods, while random sampling also performs exceptionally well.The authors adopt entropy gain as the primary sampler and identify the data mix as a major contributor to gains.Source: §4.4 · p. 4, Table 2 · p. 4

5 Conclusion and Discussion

The paper reports that recursive offline distillation with entropy sampling and a lightweight answer mix improves mean MMLU-Pro accuracy over direct distillation. Its reported findings also favor corrected reasoning chains and shorter traces, while identifying uncertainty estimation and the data-mix effect as open questions.

Conclusion

  • 57.15 ± 1.01 mean accuracy across three students on MMLU-Pro, versus 51.61 for direct distillation.Source: §5 · p. 4
  • Adding a small portion of direct answers to reasoning chains provides a major uplift, and the entropy gain sampler contributes additional gains.Corrected reasoning chains, shorter traces, and smaller reasoning budgets perform better in the reported experiments.Source: §5 · p. 4
  • More rigorous uncertainty estimation and the nature of the data-mix effect remain future work.Source: §5 · p. 4

Limitations

The evaluation is limited to small models and a single dataset, leaving generalization to larger LLMs and other settings unknown. Several design choices and resource constraints also remain untested.

Limitations

  • The method is tested only with small models, leaving its generalization to larger LLMs unknown.Finding a suitable proxy for larger students may also be challenging.Source: Limitations · p. 5
  • The study uses a single dataset, so translation to other settings is unknown.Source: Limitations · p. 5
  • Alternative uncertainty estimations, including ensemble agreement, remain unexplored.Source: Limitations · p. 5
  • The impact of using proxies of multiple sizes on performance was not tested.Source: Limitations · p. 5
  • Different data-mix compositions were not tested because of limited resources.Source: Limitations · p. 5
  • Only one schedule for Discounted Entropy Gain was tested, leaving alternatives unexplored.Source: Limitations · p. 5
  • Some low-resource environments may lack enough VRAM to fit a suitable proxy.Source: Limitations · p. 5

A Experimental Hyperparameters

Answer mixing improves answer quality beyond formatting, while the distillation setup combines reasoning traces, single-token answers, and several trace-generation variants. The section also defines single-token entropy as a measure of uncertainty over the first answer token.

C Reasoning distillation approach

  • Students are taught to separate reasoning from final answers with <think> and </think> tokens, whose embedding rows are initialized from the mean of existing rows.The approach adds these special tokens to originally non-reasoning models.Source: C Reasoning distillation approach · p. 7
  • The mixed epoch uses 1,024 reasoning examples and 256 single-token answers to teach reasoning mode while preserving non-thinking answer capability.Reasoning evaluation prompts with <think>, while single-token entropy estimation omits that prefix.Source: C Reasoning distillation approach · p. 7

D Accuracy gains beyond answer formatting

  • At a 2,048-token budget, no-mix students already produce valid extracted answers nearly perfectly, yet answer mixing improves model quality rather than merely formatting outputs.The comparison also reports a similar pattern for two models at 4,096 tokens; Phi4-mini has a comparable number of errors between mix and non-mix runs.Source: D Accuracy gains beyond answer… · p. 8

E Single-token entropy

  • Single-token entropy measures uncertainty over the first answer token, using normalized entropy to put student and proxy values on a common [0, 1] scale.The calculation starts from softmax probabilities over the model’s full vocabulary.Source: E Single-token entropy · p. 8

F.1 Direct reasoning trace.

  • The direct-trace baseline keeps both correct and incorrect teacher responses; other variants retain only correct traces, replace wrong traces with explained answers, or append teacher corrections.For corrected answers, the teacher sees the original reasoning and both options, then continues the trace with a self-correction.Source: §F.1 · p. 8, §F.2 · p. 8, §F.3 · p. 8

F.7 Illustrative examples

  • The toy examples contrast retaining an incorrect direct trace with substituting an explained answer or appending a correction that reaches the correct option.The corrected target retains the original reasoning before its continuation.Source: §F.7 · p. 9

F.8 Per-model results

The per-model figures compare training-trace truncation and trace-generation methods across evaluation budgets. The following discussion distinguishes training truncation length from the student’s reasoning cap at evaluation.

F.8 Per-model results

  • Figures 5–7 plot test accuracy across epochs, comparing head-truncation limits and generation methods at stated evaluation caps against base-model chain-of-thought accuracy.The generation-method panels fix head truncation at 8,192 characters; the baseline values use base models prompted for chain-of-thought.Source: §F.8 · p. 9, Figure 5 · p. 10, Figure 6 · p. 10

G Synthetic trace truncation and reasoning caps

  • Training truncation limits the teacher trace in characters, while the evaluation reasoning cap limits student-generated reasoning in tokens; they are varied independently.At the evaluation cap, generation is allowed to continue toward the answer after a closing thinking token is forced.Source: G Synthetic trace truncation and… · p. 9

G.1 Which part of the trace is retained?

  • Head truncation keeps a prefix, middle truncation keeps both ends, and tail truncation keeps a suffix; corrected-trace middle truncation reserves up to 1,600 characters for the correction.At a 24,000-character limit, middle truncation keeps the first 18,000 and last 6,000 characters of overlong traces; Figure 8 compares policies at an 8,192-token evaluation cap.Source: §G.1 · pp. 9–10

G.2 Training truncation length versus evaluation cap

The section separates training truncation length from evaluation reasoning budget when comparing direct traces. Figures 9 and 10 present accuracy against evaluation caps for different training lengths.

G.2 Training truncation length versus evaluation cap

  • Direct traces are compared at training truncation lengths of 3,072, 8,192, and 24,000 characters.The comparison covers head truncation.Source: §G.2 · p. 10
  • Evaluation reasoning caps are 1,024, 2,048, 4,096, and 8,192 tokens for the direct-trace comparisons.The evaluation budget is treated separately from the training truncation length.Source: §G.2 · p. 10
  • Figures 9 and 10 separate training length from evaluation budget by plotting accuracy versus reasoning cap for each training truncation length.Source: §G.2 · p. 10

H Sensitivity Study Detailed Results

The sensitivity study compares weighted and random samplers across the three models and tracks cumulative question coverage. Its figures also show accuracy as evaluation reasoning caps vary across training truncation lengths.

H Sensitivity Study Detailed Results

I Cross-model diagnostics of the uncertainty proxy

Before distillation, the proxies are substantially more accurate than the three students on single-token answers, while proxy entropy rankings agree only moderately. The patterns are consistent with a shared difficulty baseline alongside student-specific uncertainty.

I Cross-model diagnostics of the uncertainty proxy

  • Before distillation, both uncertainty proxies are substantially more accurate than all three base students on single-token answers.Accuracy here measures single-token answers, not the reasoning accuracy reported in the main results; Table 5 covers the full MMLU-Pro dataset.Source: I Cross-model diagnostics of the… · p. 13, Table 5 · p. 13
  • Proxy entropy rankings show moderate agreement and weaker linear agreement; correctness agreement is higher between proxies than between their mean and any student.Table 6 compares entropy values and correctness with Pearson r and entropy rankings with Spearman ρ.Source: I Cross-model diagnostics of the… · p. 13, Table 6 · p. 13
  • The largest correctness gap occurs for LLaMA 3B, which also has the largest uplift over direct distillation; Phi4-mini’s closer entropy rankings may help explain its smaller uplift.These patterns are consistent with a shared difficulty baseline alongside student-specific uncertainty and motivate larger-model entropy as a knowledge-gap estimate under the stated assumption.Source: I Cross-model diagnostics of the… · p. 13

Related papers

Loading 2610.20914v1…