PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty

arXiv:2609.36734v1cs.CL

TL;DR

LLM distillation assumes reliable teacher supervision, but high-entropy and hallucinated predictions can erode correct student priors. CaRE-KD adapts token-level divergence and rejects epistemically unreliable supervision, reporting consistent gains across eight teacher–student pairs and eleven benchmarks.

  • Problem

    Standard KD assumes uniformly reliable teacher supervision, but uncertain LLM predictions can lead Forward KL to allocate probability to unreliable long-tail predictions and degrade correct student priors.

    Source: §1 · p. 1

  • Method

    CaRE-KD switches between Forward and Reverse KL using token-level confidence and uses Revival to suppress updates when the teacher is epistemically less reliable than the student.

    Source: §1 · p. 2, §2 · p. 3, §2.2 · p. 4

  • Results

    CaRE-KD reports consistent improvements across eight teacher–student pairs and eleven benchmarks spanning instruction following, chat alignment, code generation, and mathematical reasoning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support adapting both how much to imitate the teacher and when to trust it, with students reported as more accurate and better calibrated than those trained with static divergences.

    Source: §6 · p. 11

    BALD-based rejection requires multiple stochastic passes, adding about 40% overhead at N = 3 for instruction following and up to about 170% on code, which may be undesirable in compute-constrained settings.

    Source: §5 · p. 11

Abstract

from arXiv · show

Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implicitly assuming the teacher to be a reliable oracle. In large language models (LLMs), this assumption often fails: teacher predictions can exhibit high entropy and hallucinations, causing standard KD to degrade well-calibrated student priors. We propose CaRE-KD, a confidence-gated distillation framework that replaces static objectives with uncertainty-adaptive optimization. CaRE-KD has two components: a token-level loss (CaRE-Divergence) that adaptively switches between Forward and Reverse KL divergence based on teacher--student confidence, and a batch-level epistemic rejection mechanism (Revival) that suppresses updates when the teacher is more uncertain than the student. We provide a gradient-level analysis showing how this dual-granularity design induces a conditional calibration mechanism that prior static divergences cannot reproduce. Empirically, across eight teacher--student pairs and eleven benchmarks spanning instruction following, chat alignment, code generation, and mathematical reasoning, CaRE-KD delivers consistent gains over strong baselines (Skewed-KL, $\alpha$--$\beta$ divergence). Highlights include up to $+3.2$ average ROUGE-L on instruction-following tasks, $+2.1$ pass@1 on MBPP, $+1.7$ accuracy on GSM8k, and $+1.8$ accuracy on CollegeMath over the strongest baseline, with consistent gains in LLM-as-a-judge factuality (up to $+2.5$ per task over Skewed-RKL). Revival further acts as a principled, loss-agnostic plug-in that systematically strengthens existing distillation objectives by filtering epistemically unreliable teacher supervision.

1 Introduction and Prior Art

Standard distillation can make students imitate unreliable long-tail predictions, even when their own priors are sharper. CaRE-KD adapts distillation to confidence and reports gains across diverse tasks and benchmarks.

1 Introduction and Prior Art

  • Forward KL can force students to cover unreliable long-tail predictions, degrading confident and correct priors when LLM teachers are uncertain.Prior alternatives such as Reverse KL, Skewed KL, and α–β divergence impose a fixed geometric preference despite token-level variation in teacher reliability.Source: §1 · p. 1
  • Teacher supervision helps by a median +4% when confident but only +0.5% when uncertain, while standard KD can overwrite sharper student priors.The paper names this recurrent failure mode the Fidelity Trap.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • CaRE-KD combines token-level switching between Forward and Reverse KL with batch-level rejection when the teacher is epistemically less reliable than the student.The rejection mechanism is based on BALD and is intended to prevent unlearning of correct priors.Source: §1 · p. 2
  • Across eight teacher–student pairs and eleven benchmarks, CaRE-KD reports consistent gains, including +2.1 pass@1 on MBPP and +1.7 accuracy on GSM8k.The paper also reports factuality gains up to +2.5 per task over Skewed-RKL and says Revival improves several existing divergence objectives.Source: §1 · p. 2
  • Prior approaches include static divergence reweighting, proxy-based uncertainty modeling, and dataset-level filtering; the paper frames robust handling of noisy supervision as unresolved.CaRE-KD instead modulates optimization during training without auxiliary models or data pruning.Source: §1 · p. 3

2 Methodology

CaRE-KD uses confidence at two levels: it adapts the token-level divergence and rejects batches when the teacher is epistemically less reliable than the student.

2 Methodology

  • CaRE-KD pairs token-level geometric adaptation using single-pass confidence with sequence-level rejection when the teacher is epistemically less reliable than the student.These are named CaRE-Divergence and Revival, respectively.Source: §2 · p. 3

2.1 Token-Level Dynamics: CaRE-Divergence

  • CaRE-Divergence combines Forward KL and Reverse KL through a per-token gate driven by the teacher–student confidence gap.When the teacher is confident it approaches mean-seeking Forward KL; when uncertain it approaches mode-seeking Reverse KL, suppressing tail noise.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 Batch-Level Rejection: Revival

  • Single-pass entropy cannot distinguish intrinsic ambiguity from hallucination-driven epistemic uncertainty, motivating Revival's sequence-level BALD rejection.BALD is computed via Monte Carlo dropout and measures predictive disagreement across stochastic parameter realizations.Source: §2.2 · p. 4
  • Revival rejects when teacher BALD is high and student BALD is low relative to separate running quantiles, targeting unreliable supervision rather than all hard tokens.Relative percentiles make the mask scale-invariant under training-time drift; the threshold may also be scheduled across training.Source: §2.2 · p. 4
  • The final batch objective suppresses updates when Revival rejects and otherwise applies the gated objective.The rejection condition therefore controls whether the student is updated on that batch.Source: §2.2 · p. 4

3 Theoretical Results

The analysis characterizes CaRE-Divergence's confidence-dependent gradients, distinguishes its token-wise behavior from globally fixed divergences, and states conditions for entropy proxies and conditional calibration.

3 Theoretical Results

  • The gradient analysis explains CaRE-Divergence's difference from Skewed KL and α–β divergence, while explicitly presenting the results as mechanisms rather than standalone optimization guarantees.The paper also formalizes when entropy-based confidence proxies BALD.Source: §3 · p. 4

3.1 Gradient analysis of CaRE-Divergence

  • With a soft gate, the selection signal pushes student confidence upward when the teacher is comparatively uncertain and Forward KL exceeds Reverse KL.This confidence-dependent term is absent from fixed-skew or fixed-β objectives; hard gating retains branch selection but not this selection signal.Source: §3.1 · p. 4

3.2 Strict separation from SRKL and α–β divergence

  • No global Skewed Reverse KL λ or α–β β reproduces CaRE-Divergence's token-varying branch choices, although its student-dominant limit matches Skewed Reverse KL.The stated comparison concerns global parameters versus token-specific choices.Source: §3.2 · p. 5

3.3 Entropy-confidence proxies BALD in the sharp regime

  • Under the Sharpness Hypothesis and the stated approximation, maximizing single-pass confidence approximately minimizes BALD, making entropy a computationally efficient inner-loop proxy.The equivalence is conditional on those assumptions.Source: §3.3 · p. 5

3.4 Conditional bi-directional calibration

  • Within a fixed branch regime where the student distribution converges to the teacher, student entropy converges to teacher entropy, correcting either initial overconfidence or underconfidence.The result does not require strict monotonicity.Source: §3.4 · p. 5

4 Experimental Setup and Results

The section evaluates CaRE-Divergence across instruction following, chat alignment, code generation, and mathematical reasoning, then examines Revival and hyperparameter choices. Results include gains over Forward KL on instruction-following tasks, improvements on specialized benchmarks, and a qualified account of factuality and Revival effects.

4 Experimental Setup and Results

  • Evaluation covers instruction following and chat alignment, code generation, and mathematical reasoning, with training on Dolly-15k/UltraChat200k, WizardCoder, and MetaMathQA.The reported metrics are ROUGE-L and LLM-judge factuality for instruction/chat tasks, pass@1 for code, and exact-match accuracy for math.Source: §4 · pp. 5–6

4.1 Results on instruction following tasks

  • Under SGO training, CaRE-Divergence achieves the best average ROUGE-L on three of five teacher–student pairs and best factuality on three, with significant gains over the second-best baseline (t = 3.93, p = 0.03).Under teacher forcing, it achieves 19.5/19.7 (R/LLM) on GPT2-base, versus 17.7/17.7 for SRKL and 16.1/15.6 for FKL.Source: Table 1 · p. 6, §4.1 · p. 7

4.2 LLM-as-a-judge factuality

  • CaRE-Divergence improves average judged factuality on most pairs, including +1.5 over SRKL on GPT2-base and +2.2 on GPT2-large.The authors do not claim Revival specifically selects low-factuality regions: the reported conditional tests are not significant, while student end-to-end factuality improves under the LLM judge.Source: §4.2 · p. 7

4.3 Revival as a loss-agnostic enhancer

  • With Revival versus no Revival, CaRE-Divergence shows the most consistent positive response, averaging +1.42 ROUGE-L on OPT-125M (+3.62 on Super-Natural Instructions); its average improvement across model pairs is significant (t = 3.98, p < 0.001).Source: §4.3 · p. 7, Table 2 · p. 8

4.5 Hyperparameter sensitivity and ablations

  • Across GPT and OPT ablations, 50–70% skip rates and increasing Revival schedules perform consistently well; hard gating peaks higher, while soft gating is more robust.BALD estimation improves in stability with more Monte Carlo samples, though N = 3 performs strongly with an increasing schedule; temperature peaks around T = 3.0, and lower soft-gate margin performs best.Source: §4.5 · pp. 8–9, Figure 3 · p. 9

4.4 Results on specialized domains

  • +2.1 pp on MBPP, +1.7 pp on GSM8k, and +1.8 pp on CollegeMath over the strongest baseline; CaRE-KD is also best on MBPP, GSM8K, and CollegeMath and tied-best on HumanEval.On MBPP, CaRE-Divergence reaches 62.4 pass@1, +2.1 versus SRKL; it reaches 73.9% on GSM8K, +1.7 versus SRKL, and 46.1% on CollegeMath, +1.8 versus Distillm-2/ABKD.Source: §4.4 · pp. 7–8, Table 3 · p. 9

5 Discussion

CaRE-KD keeps student uncertainty lower than the teacher’s and shows that token-level adaptation is the main driver of this separation, with Revival amplifying it. BALD-based rejection adds runtime cost, especially outside instruction following, and may be undesirable in compute-constrained or demanding training settings.

Decoupling student confidence from teacher uncertainty

  • On Self-Instruct, mean BALD is 0.52 for the teacher and 0.077 for the CaRE-KD student; KS distances exceed 0.5 on Vicuna and Dolly.Across the four instruction-following benchmarks, teachers show heavy high-uncertainty tails while CaRE-KD students remain concentrated at low BALD.Source: §5 · p. 9, Figure 4 · p. 10
  • On Dolly, uncertainty-distribution KS drift is 0.4% with Skewed-RKL, 2.4% with CaRE-Divergence alone, and 7% when Revival is added.The token-level loss is the main driver; Revival amplifies the drift, and the rejection rate rises through training as filtering becomes stricter.Source: §5 · pp. 9–10

Computational analysis

  • On Dolly-15k with OpenLLaMA-3B, N = 3 raises one-epoch runtime from 5520s to 7860s, while a 70% skip target recovers roughly 12% of BALD overhead.Overhead at N = 3 is approximately +106% for chat alignment, +170% for code, and +148% for math, so the roughly 40% instruction-following figure does not generalize across domains.Source: §5 · pp. 10–11

Limitations

  • BALD-based rejection may be undesirable with compute constraints, very large teachers, long sequences, or online updates; a single-pass entropy proxy remains an option with a small loss of precision.The entropy fallback relies on the Sharpness Hypothesis, best supported for cross-entropy-trained LLMs when individual Monte Carlo samples are sharp; reported results use full MC-dropout BALD instead.Source: §5 · p. 11

6 Conclusion

CaRE-KD adapts distillation to teacher reliability at token and sequence levels, and the paper concludes that this approach yields more accurate, better-calibrated students than static divergences. Its related-work discussion contrasts this adaptive design with fixed objectives, while the appendix describes BALD-based epistemic uncertainty estimation.

6 Conclusion

  • CaRE-KD consistently produces more accurate and better-calibrated students than static divergence objectives across instruction following, code generation, and mathematical reasoning.The conclusion frames effective distillation as adapting both how much to imitate the teacher and when to trust it.Source: §6 · p. 11

A Related Work

  • Prior divergences impose global geometric preferences, whereas CaRE-Divergence switches between Forward KL for confident teachers and Reverse KL for uncertain teachers, with Revival selectively rejecting unreliable supervision.The related-work discussion says Revival incorporates aleatoric and epistemic uncertainty and rejects supervision when the student is already confident.Source: A Related Work · pp. 16–17

B.1 Estimation of Epistemic Uncertainty (BALD)

  • BALD estimates epistemic uncertainty as mutual information between predictions and model parameters, using stochastic dropout passes and token-wise uncertainty pooled into a sequence-level rejection signal.The appendix describes Monte Carlo dropout passes, a posterior predictive mean, and mean-pooling across tokens.Source: §B.1 · p. 17

B.2 Verification that the uncertainty estimator is non-degenerate

The appendix verifies that forced-dropout BALD is non-degenerate on two teachers and explains why a coarse relative uncertainty signal can suffice. It also records numerical safeguards, gradient conditions, and a worst-case guarantee for Revival's rejected batches.

B.2 Verification that the uncertainty estimator is non-degenerate

  • 500 Dolly evaluation sequences show non-zero BALD and appreciable top-1 probability changes across N = 3 dropout passes for both tested teachers.The reported comparison is flat between N = 3 and N = 7 because Revival uses relative teacher-versus-student running quantiles rather than calibrated absolute BALD values.Source: §B.2 · p. 18, Table 4 · p. 18

B.3 Numerical stability of the divergence terms

  • The divergence implementations avoid inf or NaN values by using full-support tempered-softmax teacher probabilities, clamped log-probabilities, and uniform mass for the α–β objective.The passage reports that no inf or NaN values occurred in any run.Source: §B.3 · p. 18

C.1 Gradient of CaRE-Divergence

  • CaRE-Divergence's gradient has a learning term and a gate-induced selection term only when the gate is differentiable and not detached.Detaching the gate makes its gradient zero, removing the selection term.Source: §C.1 · pp. 18–19

C.2 Gradient of Skewed Reverse KL

  • The appendix derives the student-logit gradient for Skewed Reverse KL with a mixture target m = λp + (1 − λ)q.It notes that an additive constant cancels in the centering term of the equivalent vector form.Source: §C.2 · p. 19

C.3 Worst-case robustness under Revival

  • When Revival rejects a batch, its objective and gradient are zero, leaving student parameters and KL divergence to any reference distribution unchanged.The proof therefore establishes no degradation from noisy teacher supervision in the rejected regime.Source: §C.3 · p. 19

C.4 Proof of Proposition 3.2 (Limiting equivalence to SRKL)

These appendix sections distinguish token-varying CaRE-Divergence from a globally fixed Skewed Reverse KL preference and state conditional results for confidence proxies and student calibration. The accompanying dataset and benchmark descriptions specify the training and evaluation scope.

C.4 Proof of Proposition 3.2 (Limiting equivalence to SRKL)

  • CaRE-Divergence's token-varying Forward/Reverse KL branches cannot be reproduced by a global Skewed Reverse KL gradient unless the branch choice is constant across tokens.As λ approaches 1, Skewed Reverse KL converges to standard Reverse KL; CaRE-Divergence matches that geometry per token when g = 0.Source: §C.4 · p. 20

C.5 Proof of Theorem 3.4 (Conditional equivalence of confidence proxies)

  • Under the Sharpness Hypothesis and a single-pass approximation to the posterior predictive mean, normalized-entropy confidence approximately minimizes BALD and can serve as its computationally efficient inner-loop proxy.The argument relies on the stated assumptions; it does not claim unconditional equivalence.Source: §C.5 · p. 20

C.6 Proof of Theorem 3.5 (Conditional calibration of overconfident students)

  • In a fixed branch regime, CaRE-Divergence moves an initially overconfident student toward the teacher's entropy level.This conditional calibration result assumes convergence to the branch minimizer under the stated support conditions.Source: §C.6 · pp. 20–21

C.7 Proof of Theorem 3.5 (Conditional sharpening of underconfident students)

  • In a fixed branch regime, CaRE-Divergence moves an initially underconfident student toward the teacher's entropy level.This conditional sharpening result assumes convergence to the branch minimizer under the stated support conditions.Source: §C.7 · p. 21

D.1 Training Datasets

  • Training spans general instruction following, chat alignment, code generation, and mathematical reasoning, using Dolly-15k, UltraChat-200k, WizardCoder, and MetaMathQA.Dolly-15k is the primary controlled instruction-following dataset; WizardCoder and MetaMathQA train code- and math-specialized students, respectively.Source: §D.1 · p. 21

D.2 Evaluation Benchmarks

  • Evaluation covers held-out instruction-following, alignment, code-generation, and mathematical-reasoning benchmarks, including Dolly Evaluation, Super-Natural Instructions, MBPP, GSM8k, and CollegeMath.The benchmark descriptions include diverse instructions, preference-based alignment, unit-test code validation, and grade-school or college-level math problems.Source: §D.2 · p. 22

D.3 Training Details

The appendix documents training settings, uncertainty and rejection schedules, evaluation procedures, and hardware. It also reports a non-SGO comparison in which CaRE-Divergence is competitive with the strongest baseline across all five model pairs.

D.3 Training Details

  • Instruction-following students use learning rate 5 × 10^-5, LoRA rank 16, and model-size-dependent batch sizes and training lengths; other domain-specific students train for two or three epochs.Instruction models below 1B parameters use batch size 32 for up to 20 epochs; larger models use batch size 8 for 10 epochs.Source: §D.3 · p. 22
  • BALD uses N = 3 stochastic dropout passes by default, while Revival's target skip percentage can follow constant, increasing, or decreasing schedules.Moderate sampling temperature 3.0 is the default; increasing selectivity gives broad early supervision followed by stricter rejection.Source: §D.3 · pp. 22–23

D.4 Evaluation Protocol

  • Instruction-following generations use temperature 0.8, top-p 0.95, and a 512-token limit, while code and math use greedy decoding with a 1024-token limit.Instruction-following results are averaged over five random seeds.Source: §D.4 · p. 23

D.5 Prompt Structure

  • In student-generated-output training, the teacher scores student samples and the confidence gate uses those same on-policy tokens; otherwise, both loss and gate use teacher-forced tokens.Evaluation generations use vLLM, and experiments run on one NVIDIA A100 80GB GPU.Source: §D.5 · p. 23, §D.6 · p. 23

D.7 LLM-as-a-judge Factuality Evaluation Prompt

  • Factuality scores come from a fixed GPT-5-Mini rubric that rates candidate answers against references from 0 to 100.The evaluation uses temperature 0 and averages three independent judge calls per example to limit stochastic variance.Source: §D.7 · p. 23
  • Without student-generated outputs, CaRE-Divergence achieves the highest average ROUGE-L on three of five model pairs and matches SRKL on the other two within standard error.Improvements over FKL/RKL are statistically significant in a paired t-test (t = 12.84, p < 0.005).Source: Table 5 · p. 24

E.1 Comparison of CaRE-KD against baselines

CaRE-KD outperforms strong distillation baselines across teacher–student configurations and remains robust across tested design settings. Its validation-loss and exact-match comparisons also favor the confidence-gated objective over static divergences.

E.1 Comparison of CaRE-KD against baselines

  • CaRE-KD consistently achieves the strongest overall performance across all three teacher–student configurations against standard and divergence-based baselines.Source: §E.1 · p. 24
  • 19.64 average ROUGE-L lets CaRE-KD outperform MiniLLM (19.30) and Distillm (19.27) for GPT-2 family distillation.It also leads on Dolly (25.55 vs. 24.86 for Distillm) and Self-Instruct (12.88 vs. 12.73 for GKD).Source: §E.1 · p. 24, Table 6 · p. 25
  • 19.25 average ROUGE-L is CaRE-KD's highest average for OPT distillation, with scores of 12.03 on Self-Instruct and 23.93 on Super-Natural Instructions.Standard KD scores 15.04 vs. 14.43 for SFT, which the passage attributes to noise propagation from uncertain teacher predictions.Source: §E.1 · p. 24, Table 7 · p. 25
  • 26.23 average ROUGE-L makes CaRE-KD competitive with Distillm (26.41) on OpenLLaMA, while leading on Self-Instruct (20.93 vs. 20.84) and Super-Natural Instructions (37.34 vs. 36.64).Source: §E.1 · pp. 24–25, Table 8 · p. 26
  • 1.73 validation loss for CaRE-Divergence compares with 2.21 for Forward KL and 2.69 for Skewed-RKL in OpenLLaMA-3B.Reverse KL losses exceed 4.8 in the student-generated-output regime, where the passage reports high variance and unstable convergence.Source: §E.1 · p. 25, Figure 6 · p. 26
  • Skip rates of 50–70% perform strongest, increasing schedules dominate other schedules, and moderate sampling temperatures around T = 3.0 yield optimal performance.Performance remains stable across a broad high-skip plateau; increasing the MC sample count from 3 to 7 modestly improves stability, and soft gating is slightly stronger on average while hard gating reaches the highest peaks.Source: §E.1 · pp. 26–27
  • 3.9–4.1 exact-match scores for CaRE-Divergence in GPT-2-Large exceed Reverse KL (0.6–1.9) and α–β divergence (3.1).Source: §E.1 · p. 25, Figure 7 · p. 27

F.1 Decoupling student confidence from teacher uncertainty

Without selective rejection, distillation can carry the teacher’s epistemic uncertainty into the student. Revival separates student confidence from teacher uncertainty by rejecting unreliable supervision, while static divergence objectives continue to align on unreliable tokens.

F.1 Decoupling student confidence from teacher uncertainty

  • Students distilled without selective rejection track the teacher’s heavy-tailed uncertainty, with Smirnov distances exceeding 0.5 on Vicuna and Dolly.Source: §F.1 · pp. 28, 30
  • Static mode-seeking objectives reduce prediction variance but still align on unreliable teacher tokens, leaving students with non-trivial inherited epistemic uncertainty.Source: §F.1 · p. 30
  • Revival rejects supervision when the teacher is unreliable, and the resulting student uncertainty distribution separates more decisively from the teacher’s.The paper describes the design as disentangling how to learn (optimization geometry) from when to learn (epistemic reliability).Source: §F.1 · p. 30

F.2 Evolution of student confidence over distillation iterations

Revival activations rise during training under constant and increasing schedules, reflecting growing student confidence relative to the teacher. The increasing schedule most strongly supports this progression, and the qualitative trend persists across MC sample counts.

F.2 Evolution of student confidence over distillation iterations

  • MRevival rises under constant and increasing schedules, stabilizing near the target skip percentage as student uncertainty decreases.MRevival = 1 denotes greater teacher than student epistemic uncertainty, so its increasing share reflects growth in relative student confidence.Source: §F.2 · p. 30
  • The increasing schedule most strongly increases relative student confidence, while decreasing schedules re-expose students to high-uncertainty teacher supervision and suppress that growth.The passage connects the schedule ordering to inferior performance for decreasing schedules reported in Section 4.5.Source: §F.2 · p. 30
  • Increasing MC dropout samples slightly smooths MRevival trajectories, but the qualitative schedule ordering remains unchanged across architectures.The overall pattern is a progressive rise in student confidence during distillation under constant and increasing schedules.Source: §F.2 · pp. 30–31

G.1 Calibration (ECE) and selective prediction

On Dolly’s single-reference evaluation, ECE largely tracks confidence and sharpness rather than establishing which method is best calibrated. Separately, the study finds no significant difference in question hardness or response diversity between examples where Revival fires and the complementary region.

G.1 Calibration (ECE) and selective prediction

  • ECE rises with average confidence from Forward KL (0.67, ECE 0.22) to default CaRE-KD (0.79, ECE 0.34), while token accuracy is nearly identical across objectives (0.43–0.47).The authors caution that single-reference ECE is largely a sharpness proxy and do not claim CaRE-KD is best calibrated by ground-truth ECE.Source: §G.1 · p. 31, Table 9 · p. 31
  • The reported calibration property is student entropy converging toward the teacher’s, not toward a ground-truth-optimal value; risk–coverage curves are monotone for every objective.On Dolly, CaRE-KD has 7% post-distillation KS drift versus 0.4% for Skewed-RKL.Source: §G.1 · p. 31

G.2 Revival does not preferentially reject hard examples or reduce diversity

  • Revival firing and complementary regions show no significant difference in LLM-judged question hardness or response diversity on OpenLLaMA-Dolly.Source: §G.2 · p. 32, Table 10 · p. 32
Loading 2609.36734v1…