Source-linked AI summary
On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, Sarah Michalak
TL;DR
The paper examines whether mixup improves the calibration and predictive uncertainty of deep neural networks, an important issue for systems used in consequential decisions. It evaluates mixup across architectures and datasets, finding better calibration and less overconfident predictions, with label smoothing contributing significantly. The study does not address adversarially perturbed inputs.
Problem
The paper addresses limited understanding of mixup’s effect on calibration and predictive uncertainty, which matter when DNN predictions guide consequential decisions.
Method
The study evaluates mixup training across multiple deep architectures and datasets, using interpolated inputs and soft labels generated from random image pairs.
Results
Mixup-trained DNNs are significantly better calibrated and less prone to overconfident predictions on out-of-distribution and random-noise data.
Takeaways & Limitations
Mixup is a highly effective approach when predictive uncertainty is a significant concern, and its label smoothing plays a significant role in improving calibration.
Takeaways & Limitations
The study does not consider calibration and uncertainty on adversarially perturbed inputs.
Abstract
from arXiv · showhide
Mixup~\cite{zhang2017mixup} is a recently proposed method for training deep neural networks where additional samples are generated during training by convexly combining random pairs of images and their associated labels. While simple to implement, it has been shown to be a surprisingly effective method of data augmentation for image classification: DNNs trained with mixup show noticeable gains in classification performance on a number of image classification benchmarks. In this work, we discuss a hitherto untouched aspect of mixup training -- the calibration and predictive uncertainty of models trained with mixup. We find that DNNs trained with mixup are significantly better calibrated -- i.e., the predicted softmax scores are much better indicators of the actual likelihood of a correct prediction -- than DNNs trained in the regular fashion. We conduct experiments on a number of image classification architectures and datasets -- including large-scale datasets like ImageNet -- and find this to be the case. Additionally, we find that merely mixing features does not result in the same calibration benefit and that the label smoothing in mixup training plays a significant role in improving calibration. Finally, we also observe that mixup-trained DNNs are less prone to over-confident predictions on out-of-distribution and random-noise data. We conclude that the typical overconfidence seen in neural networks, even on in-distribution data is likely a consequence of training with hard labels, suggesting that mixup be employed for classification tasks where predictive uncertainty is a significant concern.
1 Introduction: Overconfidence and Uncertainty in Deep Learning
Deep neural networks are often overconfident, making calibration and predictive uncertainty important for high-risk decision-making. This work finds that mixup substantially improves calibration and reduces overconfident predictions, while leaving adversarially perturbed inputs for future work.
- Motivation: In high-risk decision pipelines, DNNs must indicate when they are likely to be wrong so decisions can be routed appropriately.The motivation includes medical diagnosis, autonomous vehicle control, and legal-sector applications.
- Overconfidence and calibration: Modern DNNs can be poorly calibrated and overconfident, with accuracy lower than their predictive scores indicate.Calibration is defined by predictive scores indicating the actual likelihood of correctness.
- Label entropy: One-hot labels assign all probability mass to one class, motivating investigation of label smoothing as a way to temper overconfidence.The paper treats these labels as zero-entropy training signals that admit no uncertainty about the input.
- Findings: Mixup-trained DNNs are significantly better calibrated than conventionally trained DNNs, with predicted softmax scores better indicating correctness likelihood.Figure 1 illustrates this improvement on CIFAR-100 using VGG-16 across training epochs.
- Findings: Mixing features alone does not provide the same calibration benefit, while mixup-trained models are less prone to overconfident predictions on out-of-distribution and random-noise data.The findings identify label smoothing in mixup as significant for improving calibration.
- Scope: The study does not consider calibration or uncertainty for adversarially perturbed inputs.The authors leave this setting for future exploration.
2 An Overview of Mixup Training
Mixup applies vicinal risk minimization by generating virtual samples from random pairs of training examples. It trains on both interpolated inputs and soft labels, with a Beta-distributed mixing ratio controlling interpolation and label smoothing.
- Vicinal risk minimization: Mixup generates vicinal points so the classifier trains on samples in the vicinity of each training example.This replaces the empirical Dirac delta distribution centered on each training pair with an empirical vicinal distribution.
- Interpolation: Virtual inputs and labels are formed by convexly interpolating two randomly sampled input points and their one-hot labels.The same interpolation ratio is used for both the inputs and associated labels.
- Training objective: Training minimizes empirical vicinal risk over a dataset of generated virtual feature-target pairs.The virtual pairs are sampled according to a vicinity distribution ν.
- Soft-label training: Cross-entropy is computed on soft labels rather than hard labels, encouraging classification-region strength to vary linearly between samples.Thus, mixup augments features while also smoothing the training targets.
- Mixing strength: The interpolation ratio λ is drawn from Beta(α, α), where α controls both input interpolation strength and associated label smoothing.α = 0 recovers one-hot training, whereas high α values average inputs and labels and can cause under-fitting.
3 Experiments
The experiments evaluate mixup’s calibration across image datasets, architectures, and metrics, including comparisons with label smoothing, entropy regularization, and no-mixup training. Results show improved calibration and reduced overconfidence, particularly with well-tuned mixing parameters.
- Experimental setup: Experiments cover image datasets, multiple architectures, calibration metrics, and comparisons with label-smoothing and entropy-regularized methods.Image experiments include STL-10, CIFAR-10, CIFAR-100, Fashion-MNIST, VGG-16, ResNet34, and ResNet-18.
- Calibration results: Mixup produces calibration scatterplots closer to x = y, whereas no-mixup models tend to occupy the overconfident region.This pattern is reported for both STL-10 and CIFAR-100.
- Hyperparameter effects: Best ECE is achieved for α in the [0.2,0.4] range, while larger α can worsen ECE because models become under-confident.The paper also reports that a well-tuned α can improve calibration with little loss in classification performance.
- Comparison methods: Label smoothing, entropy regularization, and mixup generally improve calibration and temper overconfidence, with mixup generally performing best in comparison.The reported comparisons use the best-performing models by validation accuracy.
- Large-scale experiments: The ImageNet experiment tests calibration on the 1000-class dataset containing over 1.2 million images using ResNet architectures.Mixup is presented as adding little training-time overhead for large-scale datasets.
4 Experiments on Natural Language Data
The paper extends mixup calibration experiments to natural language classification, adapting the strategy because direct input mixing can produce nonsensical text. Experiments use a sentence-level CNN across three NLP datasets.
- Evaluation: Figure 4 reports accuracy, calibration, and overconfidence across the NLP datasets.The supplied figure description identifies these three evaluation dimensions but does not provide numerical outcomes.
- Method: NLP experiments adapt mixup because directly mixing text inputs generally produces semantically nonsensical inputs.The approach modifies the mixup strategy rather than applying pixel-style input mixing directly to text.
- Experimental setup: The evaluation uses MR, TREC, and IMDB sentence-classification datasets with a Sentence-level CNN initialized using pretrained GloVe embeddings.The embeddings are modified during training on each dataset.
5 Effect of Soft Labels on Calibration
These experiments separate mixup’s data-augmentation effect from its soft-label effect. They find that increased label entropy, rather than feature mixing alone, is important for the observed calibration benefit.
- Experimental question: The experiments test whether mixup’s calibration improvement comes from data augmentation alone or from increased training-label entropy.They compare full mixup with feature mixing that retains hard labels and examine α-dependent label entropies.
- Label entropy: The α parameter controls the entropy distribution of mixup training labels sampled through Beta(α, α).The no-mixup base case corresponds to α = 0, with label entropy concentrated at zero.
- Feature versus full mixup: Merely mixing features does not provide the calibration benefit observed with full mixup.Feature mixing supplies data augmentation without the label-smoothing effect.
- Mechanism: Non-zero probability mass across multiple classes tempers overconfidence by preventing the largest pre-softmax logit from becoming much larger than the others.The paper connects this mechanism to both mixup label smoothing and related entropy-based regularization.
- Related variant: The paper reports additional calibration effects for hidden-layer representation mixing in supplementary material.This approach is described as a related extension involving convex combinations of hidden representations.
6 Effect of Extended Training on Mixup Calibration
Extended training does not erode mixup’s calibration: unlike the baseline, mixup avoids over-fitting and maintains low calibration error through 1000 epochs.
- 1000 epochs: mixup models maintain low calibration error, while baseline training approaches zero loss and 100% accuracy.The baseline over-fits, whereas mixup retains substantially higher training loss throughout extended training.
7 Testing on Out-of-Distribution and Random Data
Mixup produces less confident predictions on unseen-category and random-noise inputs than baseline models, while also outperforming the comparison methods in these uncertainty tests.
- Mixup DNNs are noticeably less confident than non-mixup models on both out-of-distribution and random-noise data.Their score distribution is nearly perfectly separable from the baseline in the random-noise case.
- Mixup outperforms temperature scaling and MC-dropout in the reported out-of-distribution and random-data comparisons.Temperature scaling is more conservative on real out-of-sample data but more overconfident on random noise; MC-dropout performs worse in both cases.
- AUROC results identify mixup as the best model for detecting both out-of-distribution and random-noise data.It significantly outperforms the other models as a random-noise detector.
- Temperature scaling reduces discrimination between in-distribution and out-of-distribution data despite producing well-calibrated in-distribution models.The scaling process lowers confidence on both data types, reducing detection reliability.
8 Conclusion and Future Work
The paper concludes that mixup improves calibration and predictive uncertainty for both in-sample and out-of-sample data, with potential compatibility with other calibration and rejection methods.
- Mixup-trained networks provide more reliable estimates on in-sample and out-of-sample data, being under-confident on the latter.The proposed explanations include regularization from data augmentation and entropic regularization from label smoothing.
- Mixup can be combined with temperature scaling, dropout-based model perturbations, ensemble models, and rejection classifiers.These combinations can operate during training or inference, depending on the method.
- Mixup is presented as effective for training DNNs when predictive uncertainty is a significant concern.This conclusion combines its classification-performance boost with its well-calibrated behavior.
A Additional Experiments on Mixup Calibration
Additional ResNet-18 experiments on CIFAR-10 and CIFAR-100 show that mixup’s interpolation parameter affects both classification and calibration, with lower values favoring calibration.
- CIFAR-10 and CIFAR-100 experiments use the ResNet-18 architecture to evaluate mixup calibration alongside classification performance.These experiments extend prior mixup evaluations by reporting calibration results.
- Lower α produces slightly better classification and significantly better expected calibration error than higher α settings.The comparison includes the α = 1.0 mixup setting used in previous literature.
- Expected calibration error can be high for both over-confident and under-confident models.Thus, calibration error alone does not identify the direction of miscalibration.
B Prediction Confidence of Mixup
Mixup models are less concentrated in the very-high-confidence region. Manifold mixup does not consistently improve calibration over regular mixup and can be less practical despite longer training.
- Prediction confidence: Mixup models are less peaked in the very-high-confidence region across various image datasets.The comparison concerns the distribution of winning softmax scores.
- Manifold mixup: Manifold mixup generally has lower accuracy and worse calibration errors than regular mixup at matched training duration.This comparison uses the same number of epochs as the regular mixup experiments.
- Manifold mixup: After 2000 epochs, manifold mixup accuracy improves significantly, while ECE improves in only a few cases without a consistent trend.The extended-training setup follows the procedure used in prior manifold mixup work.
- Manifold mixup: Regular mixup may be more practical because manifold mixup is more complicated, uses more hyperparameters, and takes longer to train.The paper presents this as a practical consideration for improving calibration.
D Leaving the Convex Hull
The study probes model behavior outside mixup’s training-data convex hull by adding random perturbations and varying their magnitude. Mixup remains more robust as inputs become noisier, while baseline confidence can behave pathologically.
- Leaving the convex hull: Mixup synthesizes training examples within the convex hull of the training data, motivating tests that progressively move inputs beyond that hull.The perturbations are applied in random directions.
- Experimental setup: Inputs are perturbed as X′ = X+µd̂ using random directions and varying perturbation magnitudes, then evaluated with pre-trained models.The experiment compares mixup, baseline, and temperature-scaled VGG-16 models trained on STL-10.
- Prediction behavior: Mixup accuracy degrades more slowly under increasing perturbations, and its confidence decays more gradually in line with accuracy.Prediction entropy shows similar behavior to confidence.
- Prediction behavior: The baseline quickly loses and then regains confidence farther from the training data, a pathological behavior that can defeat threshold-based confidence models.Temperature scaling preserves predictions while changing softmax scores, so its accuracy matches the baseline.