PaperlayerView exact revision on arXiv

Source-linked AI summary

Divergence controls entropy in distillation

Nicolas Zucchet, Scott W. Linderman

arXiv:2610.03529v1cs.LGcs.CL

TL;DR

Distillation is widely used in language-model training, but its properties are not yet well understood. The paper studies how sampling and divergence choices shape student entropy, finding that divergence acts as an entropy regularizer with different effects across training settings.

  • Problem

    Distillation minimizes divergence between teacher and student distributions, but its properties in large language models remain poorly understood.

    Source: §1 · p. 1

  • Method

    The paper separates the sampling distribution from the token-level divergence and studies their effects on student entropy.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Forward KL inflates student entropy above the teacher by the residual divergence on average, while reverse KL deflates it only while the student can mimic the teacher.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The divergence can be chosen separately from sampling and treated as an entropy knob, including to compensate for entropy loss from privileged-information conditioning in self-distillation.

    Source: §1 · p. 1, §5 · p. 9

    Reverse KL's entropy deflation depends on the student being able to mimic the teacher; when the task is too hard or the teacher too diffuse, entropy can instead inflate.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Distillation has become a core primitive of large language model training, but its properties are not yet well understood. We take an entropic perspective, studying how the entropy of the student depends on the data and the divergence that define the distillation objective. We prove that forward KL inflates the entropy of the student above that of the teacher. Since cross-entropy training is a special case, this yields an identity that we verify quantitatively in pretraining and supervised finetuning. Other divergences come with no such guarantee: reverse KL deflates entropy until the gap between student and teacher gets too large, and interpolating between the two changes entropy smoothly early in training but abruptly at convergence. The lower entropy of on-policy distillation comes from token-level reverse KL, not from on-policy sampling. The divergence therefore acts as an implicit entropy regularizer, whose role is clearest in self-distillation: as conditioning on privileged information deflates entropy, the divergence hyperparameters that work best are those that compensate for it.

1 INTRODUCTION

Distillation is widely used in large language model training, but its properties remain poorly understood; this work studies how data and divergence choices shape student entropy. It finds distinct entropy effects across divergences and training settings.

1 INTRODUCTION

  • Distillation is used across language-model pretraining and posttraining, yet its properties in this setting remain poorly understood.The paper studies student entropy as a measure related to output diversity and reinforcement-learning exploration.Source: §1 · p. 1
  • The paper separates the choice of data generation from the divergence so their effects on student entropy can be studied independently.Its token-level framing keeps sampling and divergence choices independent.Source: §1 · p. 1, Figure 1 · p. 2
  • Forward KL inflates student entropy above the teacher by the residual divergence, while reverse KL deflates it until teacher imitation becomes too difficult.Interpolating between them changes entropy smoothly early in training but abruptly at convergence.Source: §1 · p. 1
  • After pretraining and supervised finetuning, model entropy matches the cross-entropy loss minimized in training; larger models therefore have lower entropy.When distilling from a larger teacher, divergence drives entropy more than on- versus off-policy sampling, and divergence can counter entropy loss from privileged-information conditioning in self-distillation.Source: §1 · p. 1

2 DECOUPLING SAMPLING AND DIVERGENCE CHOICE IN DISTILLATION

The section separates the completion-sampling distribution from the token-level divergence used to compare student and teacher. It defines the training objective and entropy measure, and the accompanying toy-model figure illustrates how divergence affects entropy.

2 DECOUPLING SAMPLING AND DIVERGENCE CHOICE IN DISTILLATION

  • Sequence-level derivations couple the sampling distribution to the divergence, making their effects difficult to disentangle and potentially yielding suboptimal configurations.Token-level objectives separate these choices; sequence-level reverse KL forces on-policy data, while forward KL forces off-policy data.Source: §2 · p. 2
  • When completions are sampled from the student, distillation is on-policy; when sampled from the teacher, it is off-policy.The teacher may be explicit or implicit, including internet text used for next-token cross-entropy training.Source: §2 · p. 2
  • The objective minimizes expected divergence between student and teacher next-token distributions over prompts and completions sampled from p.The token-level formulation permits choosing p independently of the divergence ℓ.Source: §2 · p. 2
  • The analysis uses average model entropy, with H denoting Shannon entropy over the vocabulary.The distribution whose entropy is measured need not generate the contexts; when p = π = πθ, it is the student's normalized sequence-level entropy.Source: §2 · p. 2
  • In the toy model, forward KL inflates entropy at convergence, reverse KL usually deflates it, and generalized JS changes smoothly during training but drops abruptly at convergence.Reverse KL can instead inflate entropy when the teacher is too hard to model or too diffuse.Source: Figure 2 · p. 3

3 PROPERTIES OF DIVERGENCES AND THEIR EFFECT ON ENTROPY

The section uses a toy language-modeling setup to analyze how divergence choice affects student entropy, then examines forward and reverse KL, interpolated divergences, and top-k restrictions. Its findings depend on how closely the student can match the teacher and, for top-k objectives, how selected support and tail mass are handled.

3.1 A TOY MODEL OF LANGUAGE MODELING

  • The toy model uses random context embeddings to produce next-token distributions, with context count and teacher perplexity controlling task difficulty.It omits sequence modeling because gradients are not propagated through sampling; the authors argue that little is lost for studying next-token prediction.Source: §3.1 · pp. 3–4

3.2 DIVERGENCE CHOICE ONLY MATTERS WHEN STUDENT AND TEACHER DIFFER

  • Divergence choice affects training similarly near a teacher the student can represent and already approximates; differences arise when their distributions differ substantially.The near-optimality result holds for any f-divergence when the sampling distribution stays constant during training.Source: §3.2 · p. 4

3.3 FORWARD KL PREDICTABLY INFLATES ENTROPY

  • Forward KL gives the student higher average entropy than the teacher at optimality, and for softmax-head cross-entropy training the student’s average entropy equals its training loss.The cross-entropy identity requires a stationary point of average forward KL, but does not require the student and teacher distributions to be close.Source: §3.3 · pp. 4–5

3.4 REVERSE KL DEFLATES ENTROPY UNTIL THE GAP GETS TOO LARGE

  • Reverse KL usually lowers student entropy, but the effect breaks on difficult tasks, where a diffuse teacher or many contexts can make entropy inflate.As the teacher becomes more diffuse, the mode-seeking term weakens and the entropy-increasing term can dominate; the analysis does not predict the transition point exactly.Source: §3.4 · p. 5

3.5 IN BETWEEN FORWARD AND REVERSE KL, ENTROPY CAN SHARPLY CHANGE

  • Generalized Jensen–Shannon interpolation lowers entropy smoothly with increasing λ early in training but produces an abrupt drop at convergence.The transition boundary changes smoothly with λ, and larger λ requires harder tasks for entropy inflation.Source: §3.5 · pp. 5–6

3.6 TOP-k RESTRICTIONS SHIFT ENTROPY DEPENDING ON HOW THE TAIL IS HANDLED

  • Top-k restrictions shift entropy according to both the distribution selecting tokens and how tail mass is handled; with student-selected tokens, tail bucketing raises entropy as k decreases, unlike early renormalization.The reported top-k pattern is mostly independent of generalized Jensen–Shannon weight, though some interactions occur.Source: §3.6 · p. 6, Figure 3 · p. 6

4 ENTROPY IN DIFFERENT DISTILLATION SCENARIOS

Across pretraining, distillation, and self-distillation, student entropy depends on both the divergence and the data generation process. Cross-entropy tracks entropy on training data, while self-distillation may require entropy-inflating settings to counter feedback-induced entropy loss.

4.1 CROSS-ENTROPY PREDICTS ENTROPY IN PRETRAINING AND SUPERVISED FINETUNING

  • On trained data, student entropy matches cross-entropy after pretraining and supervised finetuning; scaling lowers both, while finetuning breaks the identity on pretraining data.The identity also holds on finetuning data after pretraining, without a guarantee.Source: Figure 4 · p. 7, §4.1 · p. 7

4.2 ON-POLICY DISTILLATION’S LOWER ENTROPY IS DUE TO TOKEN-LEVEL REVERSE KL

  • Forward KL raises entropy and reverse KL lowers it across teachers and sampling regimes; sampling has a smaller effect, and teacher size barely matters.The lower entropy of on-policy distillation is mainly due to token-level reverse KL.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 SELF-DISTILLATION DEFLATES ENTROPY, DIVERGENCE CAN COMPENSATE FOR IT

  • In self-distillation, reverse KL and a live teacher can collapse entropy as reduced diversity feeds back; an EMA teacher, even with decay 0.95, prevents collapse.Conditioning on privileged information lowers teacher entropy, which can compound when the teacher derives from the student.Source: §4.3 · p. 8, Figure 6 · p. 9
  • With a tail bucket, entropy increases as fewer top-k tokens are included; renormalizing leaves entropy roughly independent of k.Hyperparameter choices can change performance by a few percentage points when entropy does not collapse.Source: §4.3 · p. 9, Figure 6 · p. 9
  • Separating sampling from divergence matters: deriving self-distillation from sequence-level reverse KL forces a divergence that collapses entropy and damages performance.Source: §4.3 · p. 9

5 CONCLUSION

The conclusion frames divergence as an entropy regularizer distinct from sampling: forward KL inflates entropy, while reverse KL deflates it when the student can mimic the teacher. In self-distillation, the best hyperparameters counter feedback-induced entropy loss.

5 CONCLUSION

  • Forward KL inflates student entropy above the teacher’s by the residual divergence, while reverse KL deflates it only as long as the student can mimic the teacher.In self-distillation, effective hyperparameters counter the entropy drop induced by conditioning on privileged information.Source: §5 · p. 9
  • The toy language-modeling setup is designed to reproduce key large-scale phenomena while permitting precise mathematical statements.The authors hope it will also help reason about distillation beyond entropy.Source: §5 · p. 9

NOTATION

  • The notation distinguishes the data-generating process p, student and teacher next-token distributions, and average per-token entropy.Source: NOTATION · p. 15

A THE DISTILLATION FRAMEWORK

A.1.1 THE DIFFERENT OBJECTIVES

  • Sequence-level objectives compare distributions over full sequences, whereas token-level objectives compare next-token distributions in sampled contexts; implementations typically use a surrogate token-level objective.The surrogate differs when the sampling distribution depends on the student.Source: §A.1.1 · p. 16

A.1.2 HOW THE DIFFERENT OBJECTIVES AND THEIR GRADIENTS COMPARE

  • Sequence-level reverse KL matches token-level reverse KL with student sampling, and sequence-level forward KL matches token-level forward KL with teacher sampling.Sequence-level derivations therefore tie the divergence to the data-generating process, unlike token-level objectives.Source: §A.1.2 · pp. 16–17

A.1.3 WHAT THE STOP-GRADIENTS USED IN PRACTICE IGNORE

  • The practical surrogate drops the visitation gradient, which assigns tokens credit for their effects on future costs; this truncation is generally distinct from the true gradient on-policy.The full visitation gradient is a high-variance, expensive score-function estimate over rollouts.Source: §A.1.3 · pp. 17–18

A.2 PRACTICAL EVALUATION OF THE TOKEN-LEVEL DIVERGENCE

The appendix separates practical ways to estimate token-level divergences from choices of divergence family and shows how these choices affect entropy and gradients. Near an optimum, forward and reverse KL have matching first-order gradients for fixed sampling data, while the phase-transition boundary shifts smoothly across divergence weights.

A.2 PRACTICAL EVALUATION OF THE TOKEN-LEVEL DIVERGENCE

  • Token-level divergence evaluation can be expensive over the full vocabulary, motivating sampled-token estimation or restriction to a top-k subset.A sampled-token estimate needs only one teacher probability per context and is the only option for an implicit teacher such as pretraining cross-entropy.Source: §A.2 · p. 18, §A.2.1 · p. 18

A.2.1 SAMPLED-TOKEN EVALUATION

  • Sampling a single token reduces divergence-evaluation cost, but student-sampled reverse KL needs a score-function correction because the sampling distribution depends on the student.The sampled-token method is also used in on-policy implementations; its local-loss gradient is distinct from truncating the sequence-level gradient over time.Source: §A.2.1 · pp. 18–19

A.2.2 TOP-k RESTRICTIONS: RENORMALIZATION AND TAIL BUCKET

  • Top-k renormalization discards tail mass and gives no gradient signal about total mass on the selected set, whereas a tail bucket matches the teacher’s tail mass.The choice between these treatments directly affects student entropy.Source: §A.2.2 · p. 19

A.3 THE DIVERGENCE FAMILIES WE CONSIDER

  • The generalized Jensen–Shannon family interpolates between forward and reverse KL, with its endpoints defined as their normalized limits so the full interval is trainable.The family compares both models to their mixture.Source: §A.3 · pp. 19–20

B.1 DIVERGENCE CHOICE DOES NOT MATTER AROUND OPTIMALITY

  • Near optimality, forward and reverse KL have the same first-order gradients for any fixed data-generating process, so remaining run differences must come from the training data.This equivalence requires the log-ratio perturbation to be uniformly small on every token, a stronger condition than small average KL.Source: §B.1 · p. 21

B.2 WHERE THE ENTROPY GAP CHANGES SIGN

  • The phase-transition boundary evolves smoothly across generalized Jensen–Shannon weights, even though sweeping individual parameters produces sharp entropy-gap transitions.The comparison varies context count and teacher perplexity across five divergence weights.Source: §B.2 · p. 21

B.3 THE FORWARD KL INFLATES ENTROPY

At a stationary point, forward KL guarantees average student entropy exceeds teacher entropy by the residual divergence. Reverse KL yields no general entropy guarantee, only a flatness statement.

B.3 THE FORWARD KL INFLATES ENTROPY

  • Figure 7 shows forward KL inflating entropy across configurations, while increasing the interpolation weight shifts the boundary so the student must face a harder task to inflate entropy.The heatmaps vary teacher perplexity and context count and average each configuration over five seeds.Source: §B.3 · p. 22, Figure 7 · p. 22
  • At a forward-KL stationary point, average student entropy exceeds average teacher entropy by exactly the residual teacher-to-student divergence, independently of the data-generating process.When the teacher is implicit, the right-hand side is the cross-entropy loss.Source: §B.3 · p. 23
  • The forward-KL entropy identity holds pointwise when the student family permits a separate temperature direction for each context; each context’s entropy excess then equals its local misfit.This condition is weaker than requiring the model to realize the teacher distribution.Source: §B.3 · p. 23

B.4 REVERSE KL: A FLATNESS STATEMENT

  • For reverse KL, student confidence relative to the teacher depends on whether excess surprisal exceeds the reverse-KL misfit, so stationarity does not determine the entropy-gap sign.The temperature-direction argument yields only a flatness statement.Source: §B.4 · pp. 23–24

B.5 REVERSE KL INFLATES ENTROPY IN THE DIFFUSE HIGH-LOAD LIMIT

In the diffuse high-load limit, forward KL, reverse KL, and generalized Jensen–Shannon distillation share the same quadratic objective and solution. In this regime, reverse KL also inflates student entropy relative to the teacher.

B.5 REVERSE KL INFLATES ENTROPY IN THE DIFFUSE HIGH-LOAD LIMIT

  • For a diffuse teacher, forward KL, reverse KL, and the generalized Jensen–Shannon family reduce to the same leading-order quadratic objective and share its solution.The student’s optimal logits are a least-squares projection onto the embedding column space.Source: §B.5 · pp. 24–25
  • For sufficiently small β, the student inflates entropy relative to the teacher under the shared diffuse-limit solution.This conclusion follows by comparing the entropy expansions for the teacher and the student at the objective’s optimum.Source: §B.5 · p. 25
  • In the diffuse high-load limit, the entropy gap increases with context count and decreases as teacher perplexity increases.The gap depends on the student through the ratio h/n under the stated large-context and large-vocabulary approximation.Source: §B.5 · p. 25
  • For either KL direction, residual divergence equals entropy inflation in the diffuse limit; for forward KL, the identity holds for any context count and teacher.Source: §B.5 · p. 25

B.6 ADDITIONAL RESULTS ON TOP-k

The top-k appendix compares support and tail-handling approximations, while the model-family extension tests the entropy–cross-entropy identity across additional pretraining stages and models.

B.6 ADDITIONAL RESULTS ON TOP-k

  • Figure 8 repeats the top-k approximation comparison across five generalized Jensen–Shannon weights; renormalized top-1 distributions are excluded because their divergence is zero and no learning occurs.The toy-model top-k experiments use k ∈ {1, 4, 16, 64}.Source: §B.6 · p. 25, Figure 8 · p. 26, §B.7 · p. 27

B.7 SETUP OF THE TOY MODEL EXPERIMENTS

  • The toy simulations use a 64-token vocabulary and a softmax linear student trained over fixed random context embeddings.Training uses all contexts for 40,000 steps, with results averaged over five seeds.Source: §B.7 · p. 27

C.1 THE IDENTITY ACROSS MODEL FAMILIES AND PRETRAINING STAGES

  • Figure 9 reports that mean entropy equals mean cross-entropy across OLMo 2 model sizes and stages and across eight Pythia models.The comparison extends beyond the final OLMo 2 pretraining checkpoints to an earlier stage.Source: Figure 9 · p. 28, §C.1 · p. 28

C.2 ADDITIONAL ON- VS. OFF-POLICY DISTILLATION RESULTS

Forward KL raises student entropy while reverse KL lowers it, whether entropy is measured on fixed teacher completions or the student’s own completions. Accuracy patterns differ by objective, but the reported settings are not tuned to compare each objective’s best attainable accuracy.

C.2 ADDITIONAL ON- VS. OFF-POLICY DISTILLATION RESULTS

  • Forward KL inflates entropy and reverse KL deflates it on both frozen teacher completions and the student’s own completions; off-policy reverse KL deflates it most.On the student’s own completions, on-policy forward KL inflates entropy most.Source: §C.2 · p. 28
  • Forward KL, especially on-policy, degrades accuracy versus the base student, while reverse KL slightly improves it.The in-training accuracy evaluation is noisier than the endpoint evaluation; Figure 10 plots entropy on frozen teacher completions and student completions, plus accuracy against distillation step.Source: §C.2 · p. 28, Figure 10 · p. 29
  • The lower in-training accuracy of forward KL may largely reflect its higher entropy: at temperature 1, sampling a wrong token is more likely along a long solution.The authors expect accuracy to vary smoothly with entropy in this regime, and note that a lower endpoint sampling temperature yields all twelve runs above the base student.Source: §C.2 · pp. 28–29
  • At endpoint evaluation, all twelve runs score 43.00%–46.27% on MATH500 versus 42.87% for the base student, but fixed hyperparameters do not establish each objective’s best attainable accuracy.The authors did not tune the settings for accuracy and caution against reading the results as a comparison of best accuracy.Source: §C.2 · p. 29

C.3 SETUP OF THE ON- VS. OFF-POLICY DISTILLATION EXPERIMENTS

The experiment independently varies prefix source and token-level divergence across twelve distillation runs, then measures student entropy and accuracy. Its prompt, corpus, training, and evaluation choices define the scope of those comparisons.

C.3 SETUP OF THE ON- VS. OFF-POLICY DISTILLATION EXPERIMENTS

  • The twelve-run grid crosses on- versus off-policy prefixes with forward versus reverse KL for three teachers, distilling Qwen3 instruct models into Qwen3-0.6B-Base.The teachers are Qwen3 instruct models with thinking disabled at 1.7B, 4B, and 8B parameters; the shared base student is Qwen3-0.6B-Base (Yang et al., 2025).Source: §C.3 · p. 29
  • Both models use a plain-text prompt frame because the base student degrades from 42.87% to 14.93% accuracy inside the chat template, while instruct teachers rarely stop outside it.Teacher completions are cut after the first boxed answer and appended with the student’s end-of-text token.Source: §C.3 · pp. 29–30
  • Teacher completions are sampled at temperature 1 without top-p or top-k truncation; completions without a boxed answer within 4096 tokens are dropped, retaining about two thirds.Training lasts 800 steps of 32 prompts, just below one pass over the smallest corpus; Qwen3-1.7B was added because the 4B and 8B teachers were statistically indistinguishable on AMC23 (66.2% versus 66.6% accuracy).Source: §C.3 · p. 31
  • Entropy is measured on student completions and cached teacher completions; pooled-token entropy can shift with completion lengths, and per-sequence averages can disagree in sign.Endpoint accuracy uses temperature 0.6, top-p 0.95, an 8192-token budget, and three samples per problem; teachers are compared on MATH500 because held-out DeepScaleR sets differ by teacher.Source: §C.3 · p. 31
  • Pure bfloat16 training freezes normalization gains, so these runs cannot test claims about the scale of the logits.The normalization gains remain bit-identical even as weight matrices move.Source: §C.3 · p. 31

C.4 ACCURACY VS. ENTROPY FOR SELF-DISTILLATION ON THE CHEMISTRY DATASET

Figure 11 adds accuracy to the Chemistry self-distillation results by plotting student accuracy against entropy across checkpoints. The visual identifies checkpoint progression but the supplied passages do not report specific plotted outcomes.

C.4 ACCURACY VS. ENTROPY FOR SELF-DISTILLATION ON THE CHEMISTRY DATASET

  • The corresponding accuracy is reported alongside Figure 6’s student-entropy-over-time results for completeness.Figure 6 reports entropy over time on the Chemistry dataset.Source: §C.4 · p. 31
  • Figure 11 plots student accuracy as a function of entropy for the Chemistry self-distillation experiment.Each dot represents a checkpoint, with larger dots indicating later training.Source: Figure 11 · p. 31, §C.4 · p. 31

C.4 ACCURACY VS. ENTROPY FOR THE CHEMISTRY DATASET

  • The caption identifies JS as Jensen–Shannon and EMA as exponential moving average.These abbreviations are used in the figure’s self-distillation experiment.Source: Figure 11 · p. 31

C.5 ADDITIONAL SELF-DISTILLATION EXPERIMENTS ON GSM8K

The GSM8K experiments reproduce the qualitative self-distillation patterns and extend them with learning-rate and longer-training sweeps. Entropy deflation tracks both reduced learning-rate tolerance and lower entropy at convergence, although the sharp transition interval remains unresolved.

C.5 ADDITIONAL SELF-DISTILLATION EXPERIMENTS ON GSM8K

  • On GSM8K, we repeat the Chemistry experimental setup using Qwen3-0.6B, a testbed cheap enough to sweep learning rate and train ten times longer; Table 2 lists both setups’ hyperparameters.The replicated setup uses 10^-6 without warmup; we use 10^-5 with warmup, the configuration H¨ubotter et al. report for their other setups.Source: §C.5 · p. 31, Table 2 · p. 34
  • On GSM8K, entropy decreases monotonically with generalized Jensen–Shannon weight λ; a teacher that follows the student too closely collapses, and restricting divergence to fewer tokens raises entropy.Figure 12 varies λ, teacher, and token coverage, with an optional tail bucket.Source: §C.5 · p. 32, Figure 12 · p. 32
  • More entropy-deflating configurations tolerate smaller learning rates: reverse KL loses to the base model at 10^-5, while arms nearer forward KL improve up to 3·10^-5.A slowly moving or frozen teacher tolerates rates at which a live teacher has already collapsed.Source: §C.5 · p. 32, Figure 13 · p. 33
  • Within 500 steps, only λ = 0 ends above initial entropy; every λ > 0 arm settles below it, with the largest gap between λ = 0 and λ = 0.25.The λ spacing is too coarse to confirm a phase transition in that interval, and the runs use run-specific on-policy data rather than identical data as in the toy setup.Source: §C.5 · p. 32, Figure 14 · p. 33
Loading 2610.03529v1…