PaperlayerView exact revision on arXiv

Source-linked AI summary

Regularizing Class-wise Predictions via Self-knowledge Distillation

Sukmin Yun, Jongjin Park, Kimin Lee, Jinwoo Shin

arXiv:2003.13964v2cs.LGcs.CVstat.ML

TL;DR

Large deep networks can overfit, motivating regularization for better generalization. This paper introduces CS-KD, which matches predictions across same-label samples within one network; across image-classification datasets, it consistently lowers top-1 error versus prior output-regularization methods, with larger overall gains in top-5 error and expected calibration error.

  • Problem

    Large deep networks can overfit and generalize poorly, motivating regularization strategies for image classification.

    Source: §1 · p. 1

  • Method

    CS-KD regularizes a single network by matching predictive distributions across different samples with the same label, thereby regularizing its dark knowledge.

    Source: §1 · p. 1, §2.1 · p. 3

  • Results

    Across varied image-classification datasets, CS-KD consistently lowers top-1 error versus prior output-regularization methods, with larger overall gains in top-5 error and expected calibration error.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method's class-wise consistency objective is reported to mitigate overconfident predictions and reduce intra-class variation.

    Source: §1 · p. 2

Abstract

from arXiv · show

Deep neural networks with millions of parameters may suffer from poor generalization due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label during training. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network (i.e., a self-knowledge distillation) by forcing it to produce more meaningful and consistent predictions in a class-wise manner. Consequently, it mitigates overconfident predictions and reduces intra-class variations. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve not only the generalization ability but also the calibration performance of modern convolutional neural networks.

1. Introduction

Because large deep neural networks can overfit, the paper introduces class-wise self-knowledge distillation (CS-KD), which matches predictive distributions across different samples of the same class. The method aims to make predictions more consistent while reducing overconfidence and intra-class variation.

1. Introduction

  • CS-KD matches predictive distributions between different same-label samples, regularizing one network’s dark knowledge rather than transferring it from a teacher to a student [21].The paper describes this single-network approach as self-knowledge distillation.Source: §1 · p. 1
  • CS-KD aims to make same-class samples share similar wrong predictions, preventing overconfident predictions and reducing intra-class variation with one principle.The introduction contrasts this with cross-entropy’s lack of predictive-distribution consistency across samples and with separate entropy-regularization [13] [36] [43] and margin-based methods [5].Source: §1 · p. 2
  • Across image-classification benchmarks, CS-KD consistently lowers top-1 error versus prior output regularizers and recent self-distillation, with overall larger gains in top-5 error and expected calibration error.Experiments use ResNet and DenseNet models on datasets including CIFAR-100, TinyImageNet, CUB-200-2011, Stanford Dogs, MIT67, and ImageNet.Source: §1 · p. 2
  • 30.71% from 37.09% top-1 error with Mixup, and 34.47% from 39.32% with KD, after adding CS-KD.These improvements are reported on CUB-200-2011 using ResNet-18 with Mixup and ResNet-10 with KD, respectively.Source: §1 · p. 2
  • Rather than requiring perturbed inputs to be similar, CS-KD enforces consistency between different samples of the same class; the authors knew of no prior work studying this class-wise regularization.The authors suggest the method may be useful in face recognition [11] and image retrieval [45].Source: §1 · p. 2

2. Class-wise self-knowledge distillation

CS-KD regularizes a classifier by matching predictions for different samples with the same label, using the model’s own predictions to encourage consistent class-wise outputs.

2.1. Class-wise regularization

  • CS-KD matches predictions for different same-class samples using a single network, distilling its dark knowledge rather than matching one sample’s predictions across two networks.The method compares a sample with a randomly sampled input sharing its label; the original KD instead matches predictions from two networks [22].Source: §2.1 · pp. 2–3
  • The class-wise KL regularizer compares predictions with a fixed copy of the model parameters, with gradients stopped through that copy to avoid model collapse.Training adds this regularizer to standard cross-entropy with weight λ_cls, and multiplies the loss by temperature T squared following original KD [22].Source: §2.1 · p. 3

2.2. Effects of class-wise regularization

  • CS-KD uses other samples’ predictions as soft labels to mitigate overconfidence, while directly reducing differences between logits within a class to reduce intra-class variation.The paper contrasts these model-derived soft labels with the artificial soft labels produced by label smoothing [43].Source: §2.2 · p. 3
  • On commonly misclassified CIFAR-100 samples, CS-KD produces less-confident predictions and raises ground-truth-class scores, while also enhancing scores for classes correlated with the ground truth.The paper reports that meaningful predictions of this kind can improve classification accuracy and calibration in its experiments.Source: §2.2 · p. 3

3. Experiments

Across conventional, fine-grained, and large-scale image-classification experiments, CS-KD improves classification results over cross-entropy and other regularizers, reduces intra-class feature variation, and improves calibration. Combining it with Mixup, knowledge distillation, or sample-wise consistency regularization also yields benefits in the tested settings.

3.1. Experimental setup

  • The experiments span conventional, fine-grained, and ImageNet classification, using ResNet and DenseNet architectures trained from scratch; evaluation includes top-1/top-5 error, ECE, and R@1.The setup selects temperature T and loss weight λcls by validation top-1 error; R@1 evaluates intra-class feature variation.Source: §3.1 · pp. 3–4

3.2. Classification accuracy

  • On CUB-200-2011, CS-KD lowers top-1 error from 46.00% to 33.28%, and on CIFAR-100 lowers top-5 error from 6.91% to 5.69% versus cross-entropy.The paper reports that CS-KD consistently outperforms the compared output-regularization baselines on top-1 error.Source: Table 1 · p. 4, §3.2 · p. 5, Table 6 · p. 7
  • On CUB-200-2011, CS-KD reaches 33.28% top-1 error versus 41.17% for DDGSD and 40.76% for BYOT; on ImageNet, it improves top-1 error by 0.4% across tested architectures.The ImageNet architectures were ResNet-50, ResNet-101, and ResNeXt-101-32x4d.Source: Table 2 · p. 4, §3.2 · p. 5, Table 5 · p. 5
  • On CUB-200-2011 with ResNet-18, CS-KD lowers Mixup top-1 error from 37.09% to 30.71%; with ResNet-10, combining CS-KD with KD lowers it from 39.32% to 34.47%.CS-KD applies its class-wise loss to mixed inputs and labels with Mixup, and learns dark knowledge from both a teacher and itself with KD.Source: Table 3 · p. 5, §3.2 · pp. 5–6, Table 4 · p. 5

3.3. Ablation study

  • On TinyImageNet, CS-KD achieves 47.15% R@1 versus 44.66% for AdaCos, 44.69% for Virtual-softmax, and 30.59% for cross-entropy, while t-SNE shows reduced intra-class variation.The paper describes higher R@1 as indicating smaller intra-class variation in the feature embedding.Source: §3.3 · p. 6, Figure 3 · p. 6
  • On a mixed fine-grained dataset, CS-KD captures the hierarchy nearly perfectly and reaches 99.3% hierarchical accuracy for the bird class group, while also improving fine-grained top-1 error.The dataset combines 387 fine-grained labels from bird, dog, and indoor categories; the authors attribute the lower error to dark knowledge from more labels.Source: §3.3 · p. 6, Figure 4 · p. 6

3.4. Calibration effects

  • On CIFAR-100, CS-KD lowers ECE from 15.45% for cross-entropy to 5.17%, and its reliability diagram is closest to perfect calibration among the compared baselines.AdaCos and Maximum-entropy have ECE values of 73.76% and 56.41%, respectively, worse than cross-entropy.Source: Figure 5 · p. 7, §3.4 · p. 7, Table 6 · p. 7
  • On CIFAR-100, adding sample-wise consistency regularization lowers CS-KD’s ECE from 5.17% to 4.69%, while CS-KD-E also outperforms baselines on top-1 and top-5 error.CS-KD-E combines class-wise regularization with consistency between a training sample and its augmented version.Source: §3.4 · pp. 7–8, Table 6 · p. 7

4. Related work

The section situates the method among predictive-distribution regularizers, self-distillation, and margin-based losses, emphasizing its focus on intra-class variation and compatibility with prior approaches.

Regularization techniques

  • Predictive-distribution regularizers include label-smoothing [43], Mixup, low-entropy penalties [36], and consistency regularizers [2] [7] [37] [44]; this method can be combined with prior methods.Müller et al. found label-smoothing improved generalization and calibration in image classification and machine translation.Source: §4 · p. 8

Knowledge distillation

  • Unlike prior teacher–student and self-distillation approaches, this method uses soft targets to reduce intra-class variations and can also be combined with prior distillation methods.Data-distortion guided self-distillation transfers knowledge between augmented versions of the same data, while Be Your Own Teacher ensembles predictions from multiple branches.Source: §4 · p. 8

Margin-based softmax losses

  • Margin-based methods seek stronger recognition by enlarging inter-class margins and reducing intra-class variation, using metric losses, center and range losses, and angular-margin softmax variants [51] [29] [28] [47] [14] [11] [5].Virtual-softmax [5] encourages larger margins by injecting an additional virtual negative class.Source: §4 · p. 8

5. Conclusion

The paper introduces a KL-divergence regularizer that aligns predictive distributions across same-label samples, regularizing dark knowledge to encourage meaningful predictions. The authors report benefits for generalization and calibration and suggest several broader applications.

5. Conclusion

  • The proposed regularizer minimizes KL divergence between predictive distributions of different samples with the same label.It is intended to regularize dark knowledge—the model’s knowledge on wrong predictions—and encourage more meaningful predictions.Source: §5 · p. 8
  • The authors demonstrate that the method can benefit neural-network generalization and calibration.Source: §5 · p. 8
  • The authors propose exploring the technique in deep reinforcement learning [17], transfer learning [1], face verification [11], and out-of-distribution sample detection.Source: §5 · p. 8

Supplementary Material: · A. Effects of hyper-parameters

On CIFAR-100 with PreAct ResNet-18, the method is fairly robust to T and λcls, except at small T and large λcls values.

A. Effects of hyper-parameters

  • The study varied T over {0.1, 0.5, 1, 4, 10, 20} on PreAct ResNet-18 using CIFAR-100.Other settings were kept the same as in Section 3.1.Source: A. Effects of hyper-parameters · p. 11
  • The study varied λcls over {0.1, 0.5, 1, 2, 3, 4, 10, 20} on PreAct ResNet-18 using CIFAR-100.Other settings were kept the same as in Section 3.1.Source: A. Effects of hyper-parameters · p. 11
  • The method is fairly robust across the tested T and λcls values, with exceptions at T ≤0.5 and λcls ≥10.The supplied passages do not provide the table’s cell values.Source: Table 7 · p. 11, A. Effects of hyper-parameters · p. 11
  • Table 7 reports top-1 error rates for PreAct ResNet-18 on CIFAR-100 across the tested hyper-parameters.Its caption does not expose the individual error-rate values.Source: Table 7 · p. 11

B. Qualitative analysis of CS-KD

Qualitative analyses show that CS-KD moderates overconfident predictions on misclassified samples and raises ground-truth-class scores across four datasets, with TinyImageNet examples also showing higher scores for classes correlated with the ground truth.

B. Qualitative analysis of CS-KD

  • On commonly misclassified TinyImageNet samples, CS-KD moderates overconfident softmax predictions and raises scores for classes correlated with the ground-truth class.The comparison uses PreAct ResNet-18 trained with cross-entropy or CS-KD.Source: B. Qualitative analysis of CS-KD · p. 11, Figure 6 · p. 11
  • Across TinyImageNet, CUB-200-2011, Stanford Dogs, and MIT67, CS-KD yields less-confident predicted-class scores and higher ground-truth-class scores on misclassified samples than cross-entropy.The authors interpret these patterns as more meaningful predictions that are more related to the ground-truth class.Source: B. Qualitative analysis of CS-KD · p. 11, Figure 7 · p. 12, Figure 8 · p. 12

Related papers

Loading 2003.13964v2…