NormKD: Normalized Logits for Knowledge Distillation

1. Introduction

• Knowledge distillation transfers knowledge from heavyweight teacher models to lighter students, while temperature softens logits to expose dark knowledge.Source: §1 · p. 1
• A single temperature inadequately softens different samples because their logit distributions vary, producing under-softened or over-softened labels.Confident samples may lack dark knowledge when the temperature is too small, while less confident samples lose useful variety when it is too large.Source: Figure 1 · p. 1, §1 · p. 2
• Multiple temperatures improve vanilla KD performance, supporting the conclusion that one temperature cannot adequately distill knowledge from varied samples.Source: §1 · p. 2
• NormKD customizes each sample’s temperature from its logit distribution and normalizes teacher and student outputs into similar distributions.The method uses a temperature related to the sample’s standard variance to make dark-knowledge transfer more equal.Source: §1 · p. 2
• NormKD is a simple logit-based method with almost no extra computation or storage cost and can complement other logit-based methods.Source: §1 · p. 2

2. Related Work

• Knowledge distillation trains a lighter student under a cumbersome pretrained teacher, commonly aligning final predictions with KL-Divergence and softened logits.Source: §2 · p. 2
• KD methods are categorized as logit-based, feature-based, or relation-based according to whether they match logits, intermediate features, or inter-sample relationships.Source: §2 · p. 2
• Feature-based methods often achieve state-of-the-art performance but commonly require extra computation and storage for intermediate-layer alignment or structures.Source: §2 · p. 2
• Prior logit-based work reweights samples, reuses teacher classifiers, or decouples KL-Divergence, with DKD reaching performance comparable to feature-based methods.Source: §2 · p. 3
• Logit-based methods still have potential despite lacking the location information transferred by feature-based methods.Source: §2 · p. 3
• NormKD advances logit-based KD by customizing temperature for each sample according to its logit distribution.Source: §2 · p. 3

3.1. Vanilla Knowledge Distillation

• Vanilla KD aligns teacher and student predictions using KL divergence, with temperature T softening logits while cross-entropy trains against the true labels.The total loss combines cross entropy and temperature-scaled KL divergence; T is fixed for a student–teacher pair.Source: §3.1 · p. 3

3.3. Normalized Knowledge Distillation

• NormKD adds a normalization operation before the softmax layers and uses sample-specific student and teacher temperatures derived from their respective logit variances.Its pseudo-code computes both standard deviations, rescales them with T_norm, and applies the resulting temperatures in distillation.Source: Figure 2 · p. 3, §3.3 · p. 5

3.2. Distill With Multiple Temperature

• Multiple-temperature KD averages predictions produced with several temperatures before computing the KL-divergence loss.The compensation weight T_mul is selected as the maximum temperature in the chosen set.Source: §3.2 · pp. 3–4
• With an appropriate temperature set, multiple-temperature KD improves CIFAR-100 performance, but accuracy declines when too many temperatures are used.The best result with 14 temperatures surpasses some effective KD methods, whereas single temperatures 1, 2, 4, and 8 show no obvious progress.Source: §3.2 · p. 4, Table 1 · p. 4
• Choosing multiple temperatures manually is tedious because the useful temperatures, their number, and the averaging compensation must be tuned for each model.This makes the approach difficult to apply generally despite its experimental gains.Source: §3.2 · p. 4

3. Method

• A fixed temperature softens samples unequally because their logits have different distributions, leaving some outputs over-softened and others under-softened.Multiple temperatures improve performance, supporting the conclusion that one temperature cannot adequately distill knowledge from all samples.Source: §3.3 · p. 4

3.3. Normalized Knowledge Distillation

• NormKD treats each sample’s logits as approximately normally distributed and uses their standard variance to rescale the temperature.The mean has no effect on the normalized softmax, so the method only needs to calculate and divide by the standard variance, then apply T_norm.Source: §3.3 · pp. 4–5
• NormKD customizes each sample’s temperature from its logit distribution, normalizing teacher and student outputs before softmax to equalize softening.It scales temperature by the sample’s standard variance and adds only the computation needed to calculate that variance.Source: §3.3 · pp. 4–5

4.1. Datasets and baselines.

• CIFAR-100 and ImageNet experiments use standard classification benchmarks with varied teacher–student architectures and established KD baselines.Source: §4 · p. 6, §4.1 · p. 6

4.2. Main Results

• NormKD improves vanilla KD across CIFAR-100 architectures and outperforms DKD and ReviewKD on several model pairs.Source: §4.2 · p. 6
• DKD+NormKD improves DKD by more than 1.0% on ResNet32×4/ResNet8×4 and approaches or exceeds ReviewKD depending on architecture.Source: §4.2 · p. 6
• On ImageNet, DKD+NormKD improves DKD by around 0.3% and 0.7% and exceeds ReviewKD on both evaluated model pairs.Source: §4.2 · p. 6

4.4. Ablation

• Tnorm=2.0 is effective in most CIFAR-100 cases, while Tnorm=1.0 is best for the evaluated ResNet-50/MobileNet-V1 ImageNet pair.Source: §4.4 · pp. 6–7, Table 6 · p. 7, Table 7 · p. 7
• Using Vmax or Vmax−Vmin as temperatures improves vanilla KD but performs worse than NormKD.Source: §4.4 · p. 7, Table 8 · p. 7
• NormKD has training time close to vanilla KD and introduces no extra parameters, adding only a normalization operation before softmax.Source: §4.4 · pp. 7–8, Table 9 · p. 8

4.5. Visualization

• t-SNE shows more separated representations with NormKD applied to vanilla KD or DKD, while normalized-logit correlations become more similar despite larger raw-logit differences.Source: §4.5 · p. 8

• NormKD assigns each sample a temperature based on its logit distribution, using the sample’s standard variance to transfer knowledge more equally.Source: §5 · p. 8
• NormKD barely adds cost, complements vanilla KD, and combines easily with other logit-based methods.Source: §5 · p. 8
• When combined with DKD, NormKD can improve logit-based distillation beyond feature-based methods in more cases.Source: §5 · p. 8
• The conclusion links NormKD’s gains to the untapped potential of logit-based knowledge distillation.Source: §5 · p. 8

• NormKD assumes that logit outputs follow a normal distribution during its normalization operation.Source: §6 · p. 8
• Real logit distributions may be insufficiently ideal for this assumption, making the normalization approach inappropriate in some cases.Source: §6 · p. 9
• Future work should investigate methods that describe logit distributions more correctly and transfer knowledge from each sample more equally.Source: §6 · p. 9