On Calibration of Mixup Training for Deep Neural Networks

1 Introduction

• Calibration matters because applications such as medical diagnosis and language recognition rely on both predictions and their assigned probabilities.The consequences of acting on high-probability versus moderate-probability predictions can differ substantially.Source: §1 · p. 1
• Overparameterized DNNs can overfit and assign extreme, overconfident probabilities, compromising calibration despite strong benchmark accuracy.The paper notes that proper scoring-rule optimization can favor discrimination without guaranteeing optimal calibration.Source: §1 · p. 2
• Data augmentation can regularize DNNs, but general-purpose choices such as Mixup lack a principled way to predict application-specific performance.The authors note that augmentation selection and performance therefore depend largely on cross-validation.Source: §1 · p. 2
• Prior work attributes Mixup’s calibration improvements to smoother decision regions, but the authors argue smoothness alone does not ensure calibrated probabilities.Their critique also applies to other regularization strategies, including label smoothing and newer data-augmentation techniques.Source: §1 · pp. 2–3
• The paper presents empirical evidence that Mixup can degrade calibration and proposes a loss combining discriminative learning with a measure of data uncertainty.The loss is motivated by Bayes decision theory, and the authors report state-of-the-art accuracy with consistent calibration improvements.Source: §1 · p. 3

2 Related Work

• Post-hoc calibration work includes Temperature Scaling and Bayesian Neural Networks for calibration when uncertainty is incorporated.The passage also describes deep ensembles as performing well for uncertainty quantification and calibration.Source: §2 · p. 3
• Self-supervised learning and pretrained models have been reported to improve robustness, uncertainty, and calibration.Other cited work evaluates calibration and uncertainty under dataset shift and robustness to common perturbations.Source: §2 · p. 3
• Data-augmentation research includes Mixup evaluations, On-Manifold Adversarial Data Augmentation, hidden-layer augmentation, and Augmix.The passage associates these approaches with evaluations or improvements in calibration, robustness, or uncertainty quantification.Source: §2 · p. 3

3 Background

• Under equal classification losses, the Bayes Decision Rule selects the class with maximum posterior probability; using a model posterior makes decision quality depend on its gap from the data-generating distribution.The rule is optimal when supplied with the data-generating distribution.Source: §3 · p. 4
• Calibration and discrimination together describe data uncertainty, while forcing probabilities toward 0 or 1 is questionable when separability or model capacity is unknown.The proposed-loss rationale is to target the underlying distribution and avoid extreme probabilities under those uncertain conditions.Source: §3 · pp. 4–5
• Mixup estimates empirical risk using linear interpolations of two samples and their labels, with the interpolation coefficient drawn from a beta distribution.The section introduces this formulation after setting up observed i.i.d. input-label pairs and categorical posterior modeling.Source: §3 · pp. 4–5
• Mixup’s linear-soft confidence transitions reduce overconfidence but do not guarantee improved calibration because the method does not account for the proportion of samples present.The authors identify sample proportions as central to proper calibration.Source: §3 · p. 5
• Mixup can worsen calibration in many cases, and the authors state that calibration could be expected only when the class distributions have a linear relation.They report this pattern in experiments and note that Mixup does not necessarily improve calibration.Source: §3 · p. 5

4 Proposed Loss: Auto-Regularized-Confidence

• ARC matches prediction confidences to batch accuracy, using accuracy as a simple statistical summary of data uncertainty.It partitions sample confidences into bins and matches each bin's confidence to its accuracy; variants match either average or individual confidences.Source: §4 · pp. 5–6
• Combining ARC with CE avoids a local minimum in which the network becomes a prior classifier that assigns prior confidences to samples.The combined loss weights CE and ARC with β, selected using a validation set.Source: §4 · p. 6
• ARC can be evaluated on Mixup images or the original inputs; the authors report that datasets generally benefit more when it is evaluated on Mixup images.The proposed rationale is to use CE on mixed images for discrimination while ARC incorporates uncertainty through accuracy; experiments also consider ARC on the original inputs.Source: §4 · pp. 5–6
• When training accuracy is near 100%, ARC provides the same learning signal as CE, limiting its usefulness on datasets with overfit training error.The authors experiment with computing ARC on an uncalibrated validation split while using the training data for CE.Source: §4 · p. 7

5 Experiments

• Mixup worsens calibration except on CIFAR100, while ARC+Mixup corrects Mixup's degradation and is best on CIFAR10 and CIFAR100; on SVHN it corrects the degradation but performs similarly to other approaches.Tables report average results across models and results for the best model per task.Source: §5 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8
• On Birds and Cars, ARC substantially corrects Mixup's miscalibration, though ECE remains worse than the baseline; Cars gains over 3% average accuracy.The results also show near state-of-the-art Cars accuracy with a DenseNet, while the best reported model is only two points higher using more complex architectures.Source: §5 · p. 8
• On CIFAR10, the authors report state-of-the-art accuracy of about 97% and ECE values of 1.03 top and 1.62 average, compared with 2.00 ECE reported in.They connect the differing calibration results with the use of deeper models than in.Source: §5 · pp. 7–8
• The proposed method performs better than MMCE, which the authors found unstable in some cases and prone to training saturation or accuracy degradation.The evaluation covers CIFAR100, CIFAR10, SVHN, Caltech Birds, and Stanford Cars, using several DNN architectures; ImageNet was not evaluated due to computational restrictions.Source: §5 · pp. 7, 9
• ARC V1 was the best hyperparameter configuration most often; only Birds and some CIFAR100 models favored ARC on Mixup images over ARC on original inputs.The latter configuration still improved calibration, and the authors suggest separately accounting for underlying data structure may provide better-calibrated uncertainties.Source: §5 · p. 9
• Applying ARC to the baseline model yields less calibration improvement than combining ARC with Mixup; on a validation set, the network may minimize ARC by increasing accuracy instead of relaxing confidence.The authors report this behavior when considering ARC on the baseline model.Source: §5 · p. 9

• Mixup does not ensure calibrated class distributions, so the proposed approach auto-regularizes Mixup training to incorporate uncertainty in learned representations.Source: §6 · p. 9
• The authors report state-of-the-art results in accuracy and calibration for their approach to Mixup training.Source: §6 · p. 9
• The authors propose extending their analysis to other data-augmentation techniques and exploring new loss functions for regularizing overconfidence.Source: §6 · p. 9

A Mixup training

• Empirical risk minimization uses the empirical distribution because the underlying data distribution is inaccessible, but that distribution lacks support across much of the input space.Source: A Mixup training · p. 12
• Vicinal risk minimization addresses this support limitation by replacing point masses at observed samples with a vicinity distribution that explores nearby input space.Source: A Mixup training · p. 12
• Mixup constructs its vicinity distribution from linear interpolations of two inputs and their labels, with γ drawn from a beta distribution.Source: A Mixup training · p. 12
• Mixup smooths predictions, but calibration is only expected if class relationships are linear; its interpolation does not account for class proportions in the input distribution.Source: A Mixup training · p. 13

B Measuring Calibration

• Brier score and NLL are proper scoring rules, whereas ECE and MCE are measures of calibration.Source: B Measuring Calibration · p. 13
• ECE and MCE compare accuracy with average confidence within equally sized confidence bins.Source: B Measuring Calibration · p. 13
• NLL conflates discrimination and calibration, so it can remain penalized for non-separable data even when a model is perfectly calibrated and discriminative.Source: B Measuring Calibration · pp. 13–14
• NLL penalizes a single extreme wrong-confidence prediction with an infinite score, while ECE is insensitive to such an error in one sample.Source: B Measuring Calibration · p. 14

C Further Loss Analysis

• ARC contributes little while a randomly initialized model has near-prior accuracy, then adjusts confidence when cross-entropy optimization stalls.Source: C Further Loss Analysis · p. 14
• For samples at 0.6–0.7 confidence, ARC moves confidence toward batch accuracy; above-range accuracy can raise confidence without changing class assignments, while below-range accuracy can lower confidence and alter decisions.Source: C Further Loss Analysis · pp. 14–15
• ARC V1 matches average confidence to accuracy, whereas ARC V2 matches individual sample confidence to accuracy.Source: C Further Loss Analysis · p. 15
• ARC V2 is intended to avoid zero-loss solutions with high confidence variance, and computing the loss over separate bins can reduce this effect.Source: V2. ARC · p. 15

D Additional Experimental Details

• The evaluation covers CIFAR100, CIFAR10, SVHN, Caltech-Birds, and Stanford-Cars, but not ImageNet because of computational restrictions.Birds and Cars use larger, more realistic images and require padding preprocessing.Source: D Additional Experimental Details · p. 15
• The evaluated architectures include Residual, Wide Residual, and DenseNet models, each tested with and without Dropout; Birds and Cars use ImageNet-pretrained models.A Dropout layer is added to the ResNet and pretrained models.Source: D Additional Experimental Details · p. 15
• ARC hyperparameters were searched on validation sets using ResNet-18 models with and without Dropout, across more than 1,000 trained networks.The authors say computational restrictions constrained this broad experimental search.Source: D Additional Experimental Details · p. 16
• ARC V1 with M = 1 is identified as a reliable hyperparameter choice after testing bin counts, loss variants, evaluation images, and β values.The search included evaluating ARC on mixed images or separate images and selecting β for good accuracy with low calibration error.Source: D Additional Experimental Details · p. 16
• On CIFAR100, Mixup improves calibration for every evaluated model except ResNet-18, so its hyperparameter does not extrapolate uniformly across models.The authors used a deeper model for further hyperparameter selection, but computational limits prevented an exhaustive search.Source: D Additional Experimental Details · pp. 16–17

D.1 Additional Results

• The additional average and best-model results extend the reported calibration improvements achieved by A+M.Table 3 reports average results and Table 4 reports the best-performing model per task and technique.Source: §D.1 · p. 17, Table 3 · p. 17, Table 4 · p. 18
• Applying ARC loss only to an uncalibrated validation set can increase its accuracy without cross-entropy, rather than relaxing confidences.The authors present this as evidence of DNN overfitting and unpredictable behavior in probabilistic machine learning.Source: §D.1 · p. 17, Table 5 · p. 18
• Table 3 reports average-model ACC, MCE, BS ×100, and NLL, while Table 4 reports those metrics for the best model per task and technique.The passages describe these metrics but do not provide their values.Source: Table 3 · p. 17, Table 4 · p. 18