Source-linked AI summary
Regularizing Class-wise Predictions via Self-knowledge Distillation
Sukmin Yun, Jongjin Park, Kimin Lee, Jinwoo Shin
TL;DR
Large neural networks can overfit and generalize poorly, motivating regularizers that improve predictive distributions. The paper introduces CS-KD, which distills predictions between same-class samples within one network, and reports improved generalization and calibration across image-classification tasks.
Problem
Large deep neural networks may suffer from overfitting and poor generalization, motivating methods that regularize their predictive distributions.
Method
CS-KD minimizes the KL divergence between predictive distributions of different same-label samples, regularizing a single network’s dark knowledge.
Results
CS-KD consistently improves classification error rates over prior output regularization methods and yields larger gains overall for top-5 error rates and expected calibration errors.
Takeaways & Limitations
The method encourages meaningful, consistent, and less overconfident predictions while reducing intra-class variations, supporting neural-network generalization and calibration.
Abstract
from arXiv · showhide
Deep neural networks with millions of parameters may suffer from poor generalization due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label during training. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network (i.e., a self-knowledge distillation) by forcing it to produce more meaningful and consistent predictions in a class-wise manner. Consequently, it mitigates overconfident predictions and reduces intra-class variations. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve not only the generalization ability but also the calibration performance of modern convolutional neural networks.
1. Introduction
The paper addresses overfitting and poor generalization in large deep neural networks with class-wise self-knowledge distillation. CS-KD regularizes dark knowledge between same-class samples, improving prediction consistency, generalization, and calibration.
- Motivation: Large deep neural networks may overfit and generalize poorly as model and dataset sizes increase.The introduction situates CS-KD among existing regularization strategies such as dropout, batch normalization, and data augmentation.
- Contribution: CS-KD distills predictive distributions between different samples sharing the same label within a single network.Unlike teacher-student distillation, it regularizes the network’s own dark knowledge during training.
- Contribution: The method encourages similar wrong predictions for same-class samples while targeting less overconfident predictions and lower intra-class variation.These goals are combined through one class-wise regularization principle rather than separate entropy- and margin-based methods.
- Experiments: CS-KD consistently lowers top-1 error rates than prior output regularization methods across image-classification datasets.The reported evaluation covers CIFAR-100, TinyImageNet, CUB-200-2011, Stanford Dogs, MIT67, and ImageNet.
- Experiments: The gains tend to be larger for top-5 error rates and expected calibration errors, supporting improvements in generalization and calibration.The paper also identifies potential applications in face recognition and image retrieval.
2. Class-wise self-knowledge distillation
CS-KD matches predictive distributions for randomly paired samples from the same class using a KL-divergence regularizer alongside cross-entropy. It uses a fixed parameter copy to prevent collapse and aims to produce meaningful, less overconfident predictions with reduced intra-class variation.
- Class-wise regularization: CS-KD samples an input x and another input x′ with the same ground-truth label, then regularizes their predictive distributions.The method is introduced for fully supervised classification with softmax posterior predictive distributions.
- Class-wise regularization: The class-wise loss matches the two predictive distributions by minimizing their Kullback-Leibler divergence.The reference distribution uses a fixed copy eθ of the network parameters.
- Class-wise regularization: CS-KD performs self-distillation by comparing different samples through one network, unlike conventional knowledge distillation, which compares networks on one sample.The fixed parameter copy prevents the model-collapse issue by blocking gradient propagation through eθ.
- Class-wise regularization: The total loss combines standard cross-entropy with the class-wise regularization term weighted by λ_cls.The regularizer also scales with the square of the temperature T^2, following the original knowledge-distillation formulation.
- Design rationale: CS-KD uses other samples’ model predictions as soft labels to prevent overconfidence and reduce intra-class variations.The paper contrasts these model-derived soft labels with the artificial soft labels generated by label smoothing.
- Effects: On misclassified CIFAR-100 samples, CS-KD relaxes overconfident predictions and increases scores for the ground-truth class.The observed prediction changes are associated with improved classification accuracy and calibration effects in the experiments.
3. Experiments
Experiments across conventional, fine-grained, mixed-domain, and ImageNet classification evaluate CS-KD against output and self-distillation baselines. CS-KD consistently improves error rates, calibration, hierarchical classification, and feature-related behavior, while remaining compatible with Mixup and KD.
- Experimental setup: Experiments use CIFAR-100, TinyImageNet, CUB-200-2011, Stanford Dogs, MIT67, and ImageNet with ResNet and DenseNet architectures.Networks are trained from scratch with SGD; CS-KD uses temperature T from {1, 4} and loss weight λcls from {1, 2, 3, 4}.
- Classification accuracy: CS-KD consistently outperforms Virtual-softmax, AdaCos, Maximum-entropy, and Label-smoothing on top-1 error rates across image classification tasks.On CUB-200-2011, CS-KD reduces cross-entropy top-1 error from 46.00% to 33.28%.
- Classification accuracy: 33.28% top-1 error on CUB-200-2011 compares with 41.17% for DDGSD and 40.76% for BYOT, showing lower error for CS-KD among tested self-distillation methods.The paper associates CS-KD’s superiority with its effect of reducing intra-class variations.
- Classification accuracy: 0.4% top-1 error-rate improvement occurs consistently across tested ImageNet architectures, comparable to adding 51 layers to ResNet-101.The evaluation includes ResNet-50, ResNet-101, and ResNeXt-101-32x4d.
- Ablation study and calibration effects: CS-KD reaches 99.3% hierarchical accuracy in the bird hierarchy and reduces CIFAR-100 ECE from 15.45% to 5.17%.CS-KD also improves fine-grained label classification, while CS-KD-E further reduces CIFAR-100 ECE from 5.17% to 4.69%.
4. Related work
Prior work regularizes predictive distributions through softened labels, entropy penalties, margins, augmentation consistency, and teacher–student distillation. CS-KD instead targets intra-class predictive consistency within one network.
- Regularization techniques: Label-smoothing mixes one-hot labels with the uniform distribution, while Mixup interpolates training samples and their labels.
- Regularization techniques: Entropy regularization penalizes low-entropy predictive distributions and has improved exploration and supervised-learning outcomes.
- Regularization techniques: Consistency regularizers align predictions from corrupted and original samples for semi-supervised learning.
- Knowledge distillation: Knowledge distillation transfers soft targets from a teacher to a student, while self-distillation transfers knowledge across augmentations or network branches.
- Knowledge distillation: Unlike prior distillation methods, CS-KD uses soft targets within one network and specifically reduces intra-class variations; it can also be jointly used with them.
- Margin-based softmax losses: Metric-based and margin-based methods improve recognition by increasing inter-class separation or reducing intra-class feature distances.
5. Conclusion
The paper proposes a KL-divergence regularizer that distills predictive distributions between same-label samples in a single network. It reports benefits for neural-network generalization and calibration, with broader applications suggested.
- The proposed regularization term penalizes predictive-distribution differences between different samples sharing a label by minimizing KL divergence.
- CS-KD regularizes dark knowledge itself, meaning the knowledge encoded in wrong predictions, to encourage more meaningful predictions.
- The method is reported as useful for neural-network generalization and calibration.
- The authors suggest possible applications in deep-reinforcement-learning exploration, transfer learning, face verification, and out-of-distribution detection.
Supplementary Material:
The supplementary material section is represented only by the paper title passage and contains no substantive supplementary findings.
- The supplied supplementary-material passage contains the paper title rather than supplementary analysis or results.
A. Effects of hyper-parameters
The hyper-parameter study evaluates temperature T and class-wise loss weight λcls for PreAct ResNet-18 on CIFAR-100. Table 7 reports top-1 error rates across the tested settings.
- Experimental control: Except for the parameters under study, all experimental settings remain the same as in Section 3.1.
- Observed robustness: The authors report that CS-KD is fairly robust to T and λcls, except in some extreme cases, including small T ≤
- Reported metric: Table 7 reports top-1 error rates (%) for PreAct ResNet-18 on CIFAR-100 across the tested T and λcls values.
B. Qualitative analysis of CS-KD
CS-KD moderates overconfident predictions on commonly misclassified samples while increasing softmax scores for classes correlated with the ground-truth class. Log-probability comparisons examine this behavior across TinyImageNet, CUB-200-2011, Stanford Dogs, and MIT67.
- CS-KD moderates overconfident predictions on commonly misclassified TinyImageNet samples.The comparison uses softmax scores P(y|x) from PreAct ResNet-18 trained with cross-entropy and CS-KD.
- CS-KD enhances prediction values for classes correlated with the ground-truth class on commonly misclassified samples.
- Log-probabilities of predicted and ground-truth labels are compared for TinyImageNet using PreAct ResNet-18.Figure 7 reports histograms for cross-entropy and CS-KD networks.
- The same log-probability analysis is reported for CUB-200-2011, Stanford Dogs, and MIT67 using ResNet-18.Figures 8, 9, and 10 compare cross-entropy and CS-KD networks on these datasets.
- The predicted-label histogram represents the top-1 softmax score, while the ground-truth-label histogram evaluates misclassified samples.