Source-linked AI summary
Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem
Matthias Hein, Maksym Andriushchenko, Julian Bitterwolf
TL;DR
The paper asks how classifiers can avoid high-confidence predictions far from training data, a need that is especially important in safety-critical applications. It shows that ReLU-type networks have this failure mode and proposes robust confidence training for bounded image domains. The resulting method substantially lowers confidence on distant and unrelated inputs while preserving performance on the original task.
Problem
Classifiers should recognize when they do not know, but ReLU-type networks produce almost always high-confidence predictions far from training data.
Method
The paper proposes confidence-enhancing data augmentation and adversarial confidence-enhancing training, which enforce uniform predictions on noise and worst-case neighborhoods of noise.
Results
ACET consistently produces low confidence on adversarial noise and improves out-of-distribution confidence measures, while the methods leave test errors nearly unchanged.
Takeaways & Limitations
CEDA and especially ACET provide better confidence estimates for image data and can be used with little implementation overhead.
Takeaways & Limitations
The theorem’s arbitrarily distant high-confidence result applies to unbounded domains, whereas images lie in the bounded domain [0, 1]^d.
Abstract
from arXiv · showhide
Classifiers used in the wild, in particular for safety-critical systems, should not only have good generalization properties but also should know when they don't know, in particular make low confidence predictions far away from the training data. We show that ReLU type neural networks which yield a piecewise linear classifier function fail in this regard as they produce almost always high confidence predictions far away from the training data. For bounded domains like images we propose a new robust optimization technique similar to adversarial training which enforces low confidence predictions far away from the training data. We show that this technique is surprisingly effective in reducing the confidence of predictions far away from the training data while maintaining high confidence predictions and test error on the original classification task compared to standard training.
1. Introduction
The paper addresses neural-network overconfidence outside the training distribution, especially where safety-critical systems need classifiers to recognize unfamiliar inputs. It reviews existing calibration and out-of-distribution approaches before introducing a ReLU-specific analysis and mitigation strategy.
- Safety-critical classifiers should make low-confidence predictions far from training data, allowing redundant sensors or human experts to be consulted.
- Reported failures include high-confidence predictions on fooling images, out-of-distribution images, and medical diagnosis inputs, alongside overconfidence on the original task.
- Existing methods address calibration or out-of-distribution detection through softmax adjustment, ensembles, dropout uncertainty, or generative models.
- The paper studies ReLU-type networks because their piecewise affine classifier functions can produce arbitrarily high confidence far from training data.
2. ReLU networks produce piecewise affine functions
ReLU networks with piecewise affine activations and linear output layers can be represented as continuous piecewise affine classifiers. Their input space is partitioned into polytopes, with an affine classifier on each region.
- Piecewise affine functions partition R^d into finitely many polytopes and are affine when restricted to each polytope.
- ReLU and leaky ReLU networks with linear output layers yield continuous piecewise affine classifiers.
- The representation covers fully connected, convolutional, residual, skip-connection, average-pooling, and max-pooling layers because these operations fit the required linear or piecewise affine structure.
- For fully connected networks, layer outputs are recursively defined from weights, offsets, and ReLU activations, followed by a linear output layer.
- Activation-pattern matrices permit each hidden-layer map to be written as an affine function on a linear region.
- Each linear region containing x is characterized by intersecting constraints from the network’s hidden units, and the classifier restricts affinely to that region.
3. Why ReLU networks produce high confidence predictions far away from the training data
The paper proves that, under a weak condition, ReLU classifiers can become arbitrarily confident along almost every direction in an unbounded domain. It contrasts this with RBF behavior and explains why bounded image domains require a separate training intervention.
- Why ReLU networks produce high confidence predictions far away from the training data: For almost any direction and any ϵ > 0, a ReLU network can produce a sufficiently distant input with confidence at least 1 − ϵ for some class.The result assumes that the linear part of every region does not contain identical rows.
- Why ReLU networks produce high confidence predictions far away from the training data: Along any direction, sufficiently large positive scalings eventually remain in one linear region, enabling analysis of asymptotic predictions.
- Why ReLU networks produce high confidence predictions far away from the training data: The required nonidentical-row condition is weak, yet how to enforce it during network training remains completely open.
- Why ReLU networks produce high confidence predictions far away from the training data: The theorem implies infinitely many inputs with arbitrarily high confidence, while temperature rescaling and reject options cannot detect these cases.
- Why ReLU networks produce high confidence predictions far away from the training data: RBF networks instead produce approximately uniform confidence far from training data, satisfying the paper’s minimal requirement for unfamiliar regions.
- Why ReLU networks produce high confidence predictions far away from the training data: The result directly applies to unbounded domains such as R^d, but not to images restricted to [0, 1]^d.
4. Adversarial Confidence Enhanced Training
The paper combines confidence-enhancing augmentation with robust optimization to enforce low-confidence predictions on inputs sampled from an out-distribution while preserving the original classification objective.
- Out-distribution construction: The approach assumes an out-distribution p_out whose support has zero or nearly zero intersection probability with the in-distribution p_in.Uniform or noise distributions on bounded image domains are given as examples.
- Confidence-enhancing data augmentation: Confidence-enhancing data augmentation (CEDA) adds samples from p_out and enforces a uniform label distribution on them alongside cross-entropy training on original data.The method can be optimized with SGD by adding approximately λB out-distribution samples to a batch of size B.
- Motivation for robust optimization: CEDA may require too many samples to cover the whole out-distribution and can still produce high-confidence predictions near noise images.These shortcomings motivate a robust-optimization method analogous to adversarial training.
- Adversarial confidence enhancement: The proposed robust method approximately solves, for each z sampled from p_out, an optimization problem that identifies a nearby perturbation producing the worst confidence-related loss.The implementation uses projected gradient descent, potentially with multiple restarts, and backpropagates through the worst-loss perturbation.
- Adversarial confidence enhancement: Using p = ∞ and ϵ = 0.3, the method generates informative adversarial modifications of noise images that reduce high-confidence predictions more rapidly and substantially.The generated images are reported to lack structure resembling the in-distribution images.
5. Experiments
Across MNIST, SVHN, CIFAR-10, and CIFAR-100, ACET reduces confidence on noise, adversarial noise, out-of-distribution data, and adversarial examples while preserving test performance.
- Experimental setup: The experiments train Plain, CEDA, and ACET models on MNIST, SVHN, CIFAR-10, and CIFAR-100, then evaluate confidence on multiple out-of-distribution and adversarial inputs.Evaluation includes other image datasets, noise, adversarial noise, and adversarial samples derived from the training-domain test sets.
- Classification performance: There is almost no difference in test error among Plain, CEDA, and ACET, indicating that the confidence improvements do not impair the original classification performance.Table 1 reports test error alongside mean maximum confidence, AUROC, and false-positive rate.
- Out-of-distribution confidence: ACET consistently produces low confidence on adversarial noise and achieves high AUROC, whereas Plain fails on adversarial noise and CEDA mostly does so.CEDA generally lowers confidence on ordinary noise but is less effective against adversarially modified noise.
- Out-of-distribution confidence: ACET improves out-of-distribution confidence and AUROC on most datasets, with larger gains on visually dissimilar tasks than on related datasets.Reducing confidence is harder for related evaluations such as MNIST on EMNIST and CIFAR-10 on LSUN, where image structure is more similar.
- Adversarial examples: ACET lowers confidence on adversarial examples and improves AUROC on every dataset compared with Plain and CEDA, despite training only on adversarial noise.Figure 4 illustrates this effect for MNIST, where ACET alone yields a significant fraction of very low-confidence adversarial samples.
- Far-away inputs: ACET changes confidence far from the training data: reaching 99.9% confidence on scaled uniform noise requires a significantly larger upscaling factor than for Plain models.However, the experiment also finds that ACET cannot completely prevent high-confidence predictions on unbounded scaled inputs.
6. Conclusion
The paper concludes that high-confidence predictions far from training data are inherent to ReLU architectures, while CEDA and especially ACET improve confidence estimates for image data.
- ReLU networks retain arbitrarily high confidence far from training data despite temperature rescaling.
- CEDA and particularly ACET provide much better confidence estimates for image data.
- CEDA and ACET can be applied to any model with little implementation overhead.
- Future architectures should provably produce uniform confidence across classes far from training data.
A. Proofs
The proofs establish that ReLU classifiers eventually remain in one linear region along almost every direction, where one class can dominate the softmax arbitrarily strongly.
- Along any direction, sufficiently large positive scalings remain within a single linear region of the ReLU classifier.
- When the region’s linear output component has no identical rows, the largest class-specific slope is uniquely attained almost everywhere.
- Scaling inputs within that asymptotic region makes the softmax confidence of the dominant class arbitrarily close to one.
- The RBF-network proof bounds confidence near the minimum-distance training point by a value close to uniform confidence.
- The proof concludes after applying the distance conditions and the inequality controlling the remaining error term.
B. Additional α-scaling experiments
When α-scaling is projected into the bounded image domain, plain models can still become overconfident, whereas ACET substantially reduces such predictions across datasets.
- ACET models have a significantly smaller fraction of overconfident examples than plain models on all datasets.Overconfidence is defined here as maximum confidence higher than 95%.
- Plain models can produce overconfident predictions even after α-scaling is projected into the image domain.
C. The effect of Adversarial Confidence Enhanced Training
The experiments compare plain and ACET MNIST models on in-distribution and out-of-distribution images, showing that ACET preserves justified confidence while reducing overconfidence on unrelated images.
- Evaluated on MNIST: Both plain and ACET models assign lowest confidence to difficult MNIST images that are discontinuous, rotated, or ambiguous.
- Evaluated on EMNIST: Both models’ highest-confidence EMNIST predictions are justified when letters resemble digits.Letters such as ’o’ and ’i’ can look like digits ’0’ and ’1’, so uniform confidence is not expected for every EMNIST image.
- Evaluated on Grayscale CIFAR-10: ACET achieves near-uniform confidence on grayscale CIFAR-10, with mean maximum confidence near 10% and individual scores up to 40.41%.
- Evaluated on Grayscale CIFAR-10: The plain model assigns up to 99.60% confidence to grayscale CIFAR-10 images unrelated to digits.
- ROC curves: ROC curves evaluate separation of in-distribution and out-of-distribution images using maximum class confidence, with TPR on the vertical axis and FPR on the horizontal axis.FPR@95%TPR is the FPR at 0.95 TPR; AUROC 1 is perfect separation, while values below 0.5 indicate reversed ordering.
- MNIST ROC curves: For MNIST, ACET reaches ideal separation on noise and performs very well on adversarial noise, while plain training can be worse than random on adversarial noise.
- Cross-model comparison: CEDA and ACET significantly outperform plain training across all reported metrics, with ACET clearly exceeding CEDA on adversarial noise and adversarial samples.
- Cross-dataset comparison: ACET outperforms the other methods on noise and adversarial noise, although plain training is slightly better on LSUN in the reported timing comparison.
D.4. ROC curves for the models trained on CIFAR-100
On CIFAR-100, the qualitative ROC results match CIFAR-10, but confidence-based separation of in- and out-distribution examples works worse. The authors attribute this to CIFAR-100’s larger number of classes and higher test error.
- CIFAR-100 shows qualitatively the same ROC results as CIFAR-10.
- Confidence-based separation of in- and out-distribution examples generally works worse on CIFAR-100.
- CIFAR-100’s larger number of classes and higher test error may cause in- and out-distribution confidences to overlap more.
- The ROC curves summarize how confidence values distinguish in-distribution from evaluation-dataset examples.
E.1. Histograms of confidence values for models trained on MNIST
For MNIST, CEDA and ACET preserve confidence on clean test images while lowering confidence on several out-of-distribution and noise inputs. ACET generally produces the strongest reductions, including on adversarial inputs, though some adversarial samples remain highly confident.
- On clean MNIST test images, CEDA and ACET do not significantly change confidence values.
- For FMNIST, gray CIFAR-10, and noise inputs, CEDA lowers maximum confidence and ACET lowers it further.
- For EMNIST, confidence reductions are weaker because some handwritten letters resemble digits.
- For adversarial noise, both CEDA and ACET reduce confidence successfully, with most predictions near 10% confidence.
- On SVHN house-number and LSUN classroom examples, both CEDA and ACET assign lower confidences than the plain model.
- ACET significantly lowers confidence on adversarial noise and adversarial samples, whereas plain and CEDA models can remain highly confident.