Towards neural networks that provably know when they don't know

Alexander Meinke; Matthias Hein

Source-linked AI summary

Towards neural networks that provably know when they don't know

Alexander Meinke, Matthias Hein

arXiv:1909.12180v2Sep 26, 2019cs.LGcs.CVstat.ML

TL;DR

ReLU networks can become arbitrarily overconfident far from training data, while existing OOD methods lack mathematical guarantees. The paper integrates a generative model with classification to provide provably low confidence at distant points and certified neighborhood bounds, while retaining OOD performance and test accuracy.

Problem
ReLU networks can produce arbitrarily overconfident predictions far from training data, while existing OOD approaches do not guarantee low confidence in that regime.
Method
The method integrates a generative model with a classifier and derives certificates for low confidence far from training data and within neighborhoods around out-of-distribution points.
Results
CCU guarantees close-to-uniform predictions far from training data and low confidence over whole volumes, while retaining test accuracy and OOD performance.
Takeaways & Limitations
The approach makes ReLU networks provably know when they do not know and provides worst-case confidence guarantees beyond empirical OOD evaluation.
Takeaways & Limitations
The approach requires samples from an out-distribution, and current VAEs and GANs do not provide the controlled behavior far from data needed for guarantees.

Abstract

from arXiv · show

It has recently been shown that ReLU networks produce arbitrarily over-confident predictions far away from the training data. Thus, ReLU networks do not know when they don't know. However, this is a highly important property in safety critical applications. In the context of out-of-distribution detection (OOD) there have been a number of proposals to mitigate this problem but none of them are able to make any mathematical guarantees. In this paper we propose a new approach to OOD which overcomes both problems. Our approach can be used with ReLU networks and provides provably low confidence predictions far away from the training data as well as the first certificates for low confidence predictions in a neighborhood of an out-distribution point. In the experiments we show that state-of-the-art methods fail in this worst-case setting whereas our model can guarantee its performance while retaining state-of-the-art OOD performance.

1 INTRODUCTION

Safety-critical deployment requires models that behave reliably both near and far from training data, yet neural networks can remain overconfident on out-of-distribution inputs. The paper introduces CCU to retain useful in-distribution confidence while producing low confidence far from training data.

The central motivation is to make deep-learning behavior more dependable as such models enter applications where safety is a concern.The paper places OOD uncertainty alongside adversarial robustness as a safety-relevant reliability problem.
ReLU networks can produce arbitrarily high-confidence predictions farther from their training data, conflicting with the desired behavior of admitting uncertainty.High-confidence behavior has been reported for fooling images, OOD images, and medical diagnosis, while overconfidence also occurs on the original task.
OOD detection methods include confidence postprocessing and training modifications, but prior approaches lacked guarantees that neural networks would remain low-confidence far from training data.The introduction distinguishes calibration-oriented techniques from OOD-specific methods and notes the absence of robustness guarantees for this setting.
CCU retains high-confidence predictions near training data but produces close-to-uniform confidence far away, unlike a normal neural network that remains overconfident.The toy illustration shows similar decision boundaries for both models while their confidence behavior differs substantially away from the data.
The proposed approach proves low confidence far from training data and supplies upper bounds over neighborhoods around out-of-distribution points.These guarantees address both asymptotic behavior and worst-case local confidence, while the method is compatible with ReLU networks.

2 A GENERIC MODEL FOR CLASSIFIERS WITH CERTIFIED LOW CONFIDENCE

The proposed probabilistic classifier combines in- and out-distribution modeling to calibrate confidence, enabling low confidence away from training data while preserving classification decisions. Using controllable Gaussian-mixture densities and joint maximum-likelihood estimation, CCU provides guarantees that standard alternatives lack.

2 A GENERIC MODEL FOR CLASSIFIERS WITH CERTIFIED LOW CONFIDENCE: The model assumes access to out-distribution samples and targets uniform class confidence on them, while retaining higher confidence for inputs associated with the task distribution.Possible out-distributions include maximally uninformative uniform noise or a broad image proxy such as 80 million tiny images.
2 A GENERIC MODEL FOR CLASSIFIERS WITH CERTIFIED LOW CONFIDENCE: The full predictive distribution mixes in- and out-distribution conditionals according to their estimated densities and priors, producing near-uniform confidence when out-density dominates.For inputs with much greater in-distribution density, the calibrated prediction approximates the in-distribution classifier prediction.
2 A GENERIC MODEL FOR CLASSIFIERS WITH CERTIFIED LOW CONFIDENCE: Gaussian mixture models are used for in- and out-distribution densities because their estimates can be controlled far from training data, unlike more powerful generative models without direct such control.The metric can be adapted to the data distribution through a positive-definite matrix and the densities remain normalized in Rd.
2 A GENERIC MODEL FOR CLASSIFIERS WITH CERTIFIED LOW CONFIDENCE: CCU is named for bounds that provide guarantees on classifier confidence far away from training data, unlike prior empirical OOD approaches.With standard softmax, the objective recovers uniform-confidence training on out-distribution data as a special case, but CCU additionally calibrates through estimated densities.
2 A GENERIC MODEL FOR CLASSIFIERS WITH CERTIFIED LOW CONFIDENCE: CCU combines a classifier with jointly estimated in- and out-distribution densities, so density ratios calibrate confidence while classifier rankings remain unchanged.The model estimates the classifier and Gaussian-mixture parameters through maximum likelihood, with the out-distribution prior represented by λ.

3 PROVABLE GUARANTEES FOR CLOSE TO UNIFORM PREDICTIONS FAR

The paper provides two guarantees: confidence becomes close to uniform sufficiently far from training data, and remains bounded across certified neighborhoods around out-distribution points.

Far-from-training-data guarantee: Theorem 3.1 gives an explicit distance beyond which predictions are close to uniform, including for modified ReLU classifiers.The result applies to a general mixture model that includes the paper’s GMM as a special case.
Model construction: The model combines a classifier conditional with estimated in- and out-distribution densities, allowing the classifier confidence to be calibrated and certified far from training data.The displayed formulation uses separate in- and out-distribution density terms and a regularization parameter λ.
Far-from-training-data guarantee: The confidence bound applies to any multi-class classifier that outputs a probability distribution over labels.The theorem quantifies when confidence becomes close to uniform using the GMM parameters.
Certified neighborhood guarantee: A second guarantee upper-bounds confidence throughout a ball around a given input, rather than only at one OOD point.The authors characterize this as stronger than standard pointwise OOD evaluation and analogous to adversarial robustness guarantees.
Worst-case implication: In the adversarial-noise experiment, CCU remains provably low-confidence inside certified balls, unlike competing OOD methods under confidence maximization.The comparison targets worst-case points generated within regions certified by Corollary 3.1.

4 EXPERIMENTS

Experiments evaluate worst-case certified regions and standard OOD detection across multiple image datasets, finding that CCU preserves low confidence in certified neighborhoods while achieving strong overall OOD performance.

Certified robustness against adversarial noise: CCU is the only evaluated approach reported to guarantee low confidence in certified neighborhoods, while other OOD methods produce undesired high-confidence predictions there.The worst-case evaluation uses adversarial noise points inside balls certified by Corollary 3.1.
Experimental setup: The benchmark covers MNIST, FashionMNIST, SVHN, CIFAR10, and CIFAR100, using certified adversarial-noise regions and standard OOD datasets.Methods are compared using confidence or score, with AUC used for binary in-distribution versus out-distribution detection.
Methods: ACET uses adversarial training around out-distribution seeds, while CCU constructs its certified regions around uniform noise and controls the metric through a covariance-based transformation.CCU uses GMMs with 100 centroids for both in- and out-distributions.
Experimental setup: Hyperparameters are calibrated on a subset of 80M tiny images and evaluated on unseen distributions to avoid dataset-specific classifiers.This protocol is intended to keep the comparison generic and realistic.
OOD performance: OE and CCU have the best reported standard OOD performance, while MCD performs worse than the base model and DE is not much better than baseline on CIFAR10 and CIFAR100.The standard OOD metric is AUC for discriminating in- and out-distribution inputs.

5 CONCLUSION

The paper modifies ReLU networks with CCU to obtain provably low confidence far from training data and across volumes, while retaining strong OOD performance.

5 CONCLUSION: OE and CCU have the best OOD performance in Table 2, measured by AUC for in- versus out-distribution detection.The table reports AUC in percent, with higher values indicating better performance.
5 CONCLUSION: CCU produces close to uniform predictions far away from the training data, addressing arbitrarily confident ReLU predictions without changing the network architecture alone.The conclusion describes CCU as an architectural modification that explicitly integrates a generative model.
5 CONCLUSION: CCU is the only evaluated OOD method reported to guarantee low confidence over a whole volume rather than only pointwise.The paper states that other OOD methods fail in the corresponding worst-case setting.
5 CONCLUSION: The paper reports that CCU achieves these guarantees without loss in test accuracy or OOD performance.The conclusion also identifies stronger generative models as a future direction because comparable guarantees are not currently available for VAEs and GANs.

A APPENDIX - PROOF OF THEOREM 3.1

The theorem proof defines class-conditional and marginal generalized Gaussian mixture models, then bounds their probability ratio to establish low confidence sufficiently far from training data.

A APPENDIX - PROOF OF THEOREM 3.1: The proof models in-distribution and out-distribution densities with generalized Gaussian mixtures having positive weights and scale parameters and metric-valued centers.The construction uses separate mixture components for the in-distribution and out-distribution.
A APPENDIX - PROOF OF THEOREM 3.1: The proof obtains the bound by upper-bounding the in-distribution density relative to the out-distribution density using nearest mixture centers and Gaussian-mixture properties.Triangle-inequality bounds and a quadratic inequality are used to derive the required solution.
A APPENDIX - PROOF OF THEOREM 3.1: The final step uses finiteness of the mixture counts and bounded distances to show the relevant constants remain controlled independently of the selected mixture components.The proof explicitly notes positivity of one constant and boundedness of the remaining terms.

B APPENDIX - PROOF OF COROLLARY 3.1

The corollary extends the low-confidence guarantee from sufficiently distant points to every point within a ball around an out-distribution reference point.

B APPENDIX - PROOF OF COROLLARY 3.1: The proof starts from the global class-confidence bound and an upper bound on the out-distribution density, then separately bounds numerator and denominator terms inside a radius-R ball.This establishes a local certificate around an out-distribution point.
B APPENDIX - PROOF OF COROLLARY 3.1: The numerator bound uses the reverse triangle inequality to lower-bound distances from points in the ball to in-distribution mixture centers.The bound is expressed through the reference point, the ball radius, and each mixture-center distance.
B APPENDIX - PROOF OF COROLLARY 3.1: A corresponding denominator bound completes the local confidence certificate once both bounds are combined.The proof states that the conclusion follows immediately after establishing the two bounds.

C APPENDIX - EXPERIMENTAL DETAILS

The experiments use fixed optimizer, learning-rate, batch-size, epoch, and weight-decay settings, with balanced in- and out-distribution batches for several methods.

C APPENDIX - EXPERIMENTAL DETAILS: MNIST uses ADAM with learning rate 1e-3, while other datasets use SGD with learning rate 0.1 and the GMM uses 1e-5.All learning rates are reduced tenfold after epochs 50, 75, and 90.
C APPENDIX - EXPERIMENTAL DETAILS: Training uses batch size 128, 100 epochs, and weight decay 5e-4 unless otherwise specified.These settings are stated as the general experimental defaults.
C APPENDIX - EXPERIMENTAL DETAILS: For ACET, OE, and CCU with 80 million tiny images, training concatenates equal in-distribution and out-distribution batches into batches of size 256.Only a fraction of the tiny-image collection is seen during 100 epochs, according to the implementation description.

C.1 DATA AUGMENTATION

The data augmentation scheme uses random crops with dataset-specific padding and additional horizontal flips for CIFAR.

C.1 DATA AUGMENTATION: Random crops use 2-pixel padding for MNIST and FMNIST, versus 4-pixel padding for SVHN, CIFAR10, and CIFAR100.MNIST and FMNIST generate 60000 augmented samples, while SVHN and CIFAR generate 50000.
C.1 DATA AUGMENTATION: SVHN padding copies boundary values, CIFAR padding reflects boundary pixels, and CIFAR additionally applies random horizontal flips.

C.2 ATTACK DETAILS

The attack procedure uses adaptive-step PGD in transformed space, while certified radii are obtained by numerically inverting a monotone bound.

C.2 ATTACK DETAILS: Adaptive PGD starts from 50 random points in the ellipsoid, increasing successful step sizes by 1.1 and halving them after loss increases.The attack rotates back to the original space after each gradient step while maintaining the ellipsoid constraint in transformed space.
C.2 ATTACK DETAILS: Figure 3 summarizes certified radii in transformed space as histograms across datasets.
C.2 ATTACK DETAILS: Certified radii are computed by bisection because the certification bound increases monotonically with radius.The resulting radius is the maximal radius certified by Corollary 3.1, provided the bound at radius zero is sufficiently low.

E APPENDIX - ANALYSIS OF THE CERTIFIED BALLS AROUND UNIFORM

Certified balls around uniform-noise images remain separated from dataset images, while the confidence threshold affects test-set coverage differently across datasets.

E APPENDIX - ANALYSIS OF THE CERTIFIED BALLS AROUND UNIFORM: CCU’s certified confidence is below 1.1/M throughout each ball, making predictions close to uniform across classes.
E APPENDIX - ANALYSIS OF THE CERTIFIED BALLS AROUND UNIFORM: None of the certified balls around 200 uniform-noise images contained a training or test image for any dataset.Although the balls were large, this observation supports the proposed threat model.
E APPENDIX - ANALYSIS OF THE CERTIFIED BALLS AROUND UNIFORM: 5.3% of CIFAR10 and 1.3% of CIFAR100 test samples fall below the 1.1/M confidence threshold, versus less than 0.1% for SVHN and none for MNIST or FMNIST.
E APPENDIX - ANALYSIS OF THE CERTIFIED BALLS AROUND UNIFORM: CCU achieves 100% empirical AUC on all 200 certified balls, while worst-case certified lower bounds are 99.92% for SVHN, 94.71% for CIFAR10, and 98.70% for CIFAR100.The stated certified lower bound remains better than other models’ empirical performance on CIFAR10 and CIFAR100, and is only marginally below perfect performance on SVHN.
E APPENDIX - ANALYSIS OF THE CERTIFIED BALLS AROUND UNIFORM: ACET requires an out-distribution model for adversarial-noise training, and the appendix evaluates a uniform-noise variant alongside the original tiny-image setup.

G APPENDIX - PRECISION AND RECALL

The appendix reports precision-recall evaluation alongside AUC, finding qualitatively similar results to the main comparison and identifying OE and CCU as strongest performers in the supplied table caption.

G APPENDIX - PRECISION AND RECALL: AUPR is reported for all models and datasets as an additional in-versus-out-distribution detection metric.Precision at a threshold is defined as true positives divided by true positives plus false positives.
G APPENDIX - PRECISION AND RECALL: The AUPR results qualitatively do not differ from those reported in Table 2.
G APPENDIX - PRECISION AND RECALL: Table 6 states that OE and CCU have the best OOD performance among the compared methods.The table reports AUPR percentages for in-versus-out-distribution detection across models and datasets.