PaperlayerView exact revision on arXiv

Source-linked AI summary

A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

Kimin Lee, Kibok Lee, Honglak Lee, Jinwoo Shin

arXiv:1807.03888v2stat.MLcs.CRcs.LG

TL;DR

Softmax classifiers can be overconfident on abnormal samples, while prior detectors typically do not cover both out-of-distribution and adversarial cases. This paper uses Mahalanobis-distance confidence scores from Gaussian class-conditional feature models and reports strong performance across detection and class-incremental learning tasks.

  • Problem

    Most prior detectors do not evaluate both out-of-distribution and adversarial samples, leaving no known universal detector that works well on both tasks.

  • Method

    The method fits tied-covariance class-conditional Gaussian distributions to pretrained features under Gaussian discriminant analysis and scores samples by Mahalanobis distance.

  • Results

    The method achieves state-of-the-art performance for both out-of-distribution and adversarial detection, improves OOD TNR from 45.6% to 90.9% on ResNet, and outperforms incremental-learning baselines.

  • Takeaways & Limitations

    The approach can support class-incremental learning by incorporating detected new classes without retraining the deep models.

  • Takeaways & Limitations

    The approach assumes that each class's pretrained features follow a multivariate Gaussian distribution with tied covariance.

Abstract

from arXiv · show

Detecting test samples drawn sufficiently far away from the training distribution statistically or adversarially is a fundamental requirement for deploying a good classifier in many real-world machine learning applications. However, deep neural networks with the softmax classifier are known to produce highly overconfident posterior distributions even for such abnormal samples. In this paper, we propose a simple yet effective method for detecting any abnormal samples, which is applicable to any pre-trained softmax neural classifier. We obtain the class conditional Gaussian distributions with respect to (low- and upper-level) features of the deep models under Gaussian discriminant analysis, which result in a confidence score based on the Mahalanobis distance. While most prior methods have been evaluated for detecting either out-of-distribution or adversarial samples, but not both, the proposed method achieves the state-of-the-art performances for both cases in our experiments. Moreover, we found that our proposed method is more robust in harsh cases, e.g., when the training dataset has noisy labels or small number of samples. Finally, we show that the proposed method enjoys broader usage by applying it to class-incremental learning: whenever out-of-distribution samples are detected, our classification rule can incorporate new classes well without further training deep models.

1 Introduction

The paper introduces a unified, training-free detector for out-of-distribution and adversarial samples using feature-space probability density under class-conditional Gaussian modeling. It reports strong performance across both detection settings, robustness under difficult training conditions, and an extension to class-incremental learning.

  • Motivation: Predictive uncertainty remains challenging despite DNNs’ high accuracy, motivating reliable abnormal-sample detection for real-world deployment and applications such as active learning.The paper links predictive uncertainty to detecting samples far from the training distribution, statistically or adversarially.
  • Problem: Prior work typically evaluates either out-of-distribution or adversarial samples, and the paper identifies no known universal detector performing well on both tasks.
  • Contribution: The proposed method applies to any pre-trained softmax neural classifier without retraining and detects abnormal samples using feature-space probability density from a generative, distance-based classifier.The method assumes that pre-trained features can be modeled by class-conditional Gaussian distributions.
  • Experimental results: The method outperforms ODIN in all tested OOD-detection cases across convolutional networks and datasets including CIFAR, SVHN, ImageNet, and LSUN.Experiments use DenseNet and ResNet image classifiers.
  • Robustness: The method is more robust to hyperparameter choices and extreme conditions including noisy random labels or small training datasets, while requiring only in-distribution data for tuning.The paper contrasts this with ODIN hyperparameter tuning using validation sets of OOD samples, which may be unavailable beforehand.
  • Class-incremental learning: For class-incremental learning, the method accommodates new classes by computing their class means and updating the tied covariance without retraining the deep models.The approach exploits the expectation that new-class samples come from an out-of-training distribution.

2 Mahalanobis distance-based score from generative classifier

The method induces tied-covariance class-conditional Gaussian distributions from a pretrained softmax classifier and uses Mahalanobis distance as a confidence score for abnormal-sample detection. It further improves detection through input preprocessing and feature-level score ensembles, while supporting class-incremental learning.

  • Generative classifier: The method converts a pretrained softmax classifier into a generative classifier by modeling each class’s features with a Gaussian distribution sharing covariance Σ.The Gaussian parameters are estimated from training samples using empirical class means and covariance under maximum likelihood.
  • Mahalanobis distance-based confidence score: The confidence score M(x) is based on the Mahalanobis distance from a test sample to its closest class-conditional Gaussian distribution.The score corresponds to the log of the test sample’s probability density and uses DNN representation space rather than softmax posterior space.
  • Experimental support: The induced Mahalanobis classifier achieves the pretrained softmax classifier’s accuracy despite being a generative classifier.The method uses the posterior distribution of the generative classifier with a uniform class prior.
  • Input pre-processing: Input preprocessing adds controlled noise designed to increase the proposed confidence score, improving separation between in- and out-of-distribution samples.The perturbation is generated differently from adversarial attacks, which increase abnormality rather than the proposed confidence score.
  • Feature ensemble: Feature ensembles combine confidence scores from final and lower-level DNN features using weights learned by logistic regression on validation samples.Weighted averaging can prevent overall performance degradation when some layers provide ineffective confidence scores.
  • Class-incremental learning: The Mahalanobis confidence score also supports class-incremental learning by updating a classifier when samples from a new class become available.This setting involves progressively updating a classifier pretrained on base classes while addressing catastrophic forgetting under limited memory.

3 Experimental results

Experiments on multiple vision datasets show that the proposed method consistently improves detection of out-of-distribution and adversarial samples over baseline and competing methods. It also performs strongly under robustness tests and class-incremental learning, where its Mahalanobis classifier outperforms alternatives as new classes increase.

  • Experimental setup: The method is evaluated with DenseNet and ResNet on CIFAR, SVHN, ImageNet, and LSUN vision datasets.OOD experiments train on CIFAR-10, CIFAR-100, or SVHN, treating the remaining datasets, TinyImageNet, and LSUN as OOD.
  • Out-of-distribution detection: The method outperforms the baseline and ODIN across all tested OOD dataset pairs.Using DenseNet, TNR for detecting LSUN improves from 41.2% with ODIN to 91.4% when 95% of CIFAR-100 samples are correctly detected.
  • Robustness: Under hyperparameter tuning using only in-distribution and FGSM adversarial samples, the method consistently outperforms the baseline and ODIN.ODIN performs poorly in some cases, including DenseNet trained on SVHN, whereas the proposed method remains robust.
  • Adversarial detection: The proposed method outperforms tested adversarial-detection methods in most normal-versus-adversarial pairs.For CW detection with ResNet trained on CIFAR-10, AUROC improves from 82.2% for LID to 95.8% for the proposed method.
  • Class-incremental learning: The Mahalanobis distance-based classifier significantly outperforms other classifiers in class-incremental learning as the number of new classes increases.Experiments use CIFAR-100 and downsampled ImageNet, comparing AUC over learned classes and base-new class accuracy after adding new classes.

4 Conclusion

The paper proposes a unified method for detecting abnormal test samples, including both out-of-distribution and adversarial samples. It combines an LDA-based generative classifier with a confidence score and calibration techniques, performing strongly across multiple tasks.

  • The method detects both out-of-distribution and adversarial test samples.
  • Its core idea is to induce a generative classifier under an LDA assumption and define a new confidence score from it.
  • Input preprocessing and feature ensembling calibrate the method, which performs strongly across multiple detection tasks.

Supplementary Material: · A Preliminaries for Gaussian discriminant analysis · B Experimental setup

The supplementary material introduces discriminative and generative classifiers, formalizes Gaussian discriminant analysis and its shared-covariance LDA special case, and explains its equivalence to softmax classification. It also states that the experimental setup details all experiments from Section 3.

  • A Preliminaries for Gaussian discriminant analysis: Discriminative classifiers directly model P(y|x), while generative classifiers specify P(x|y) and P(y) to define P(x,y) and the posterior indirectly.The softmax classifier is presented as a representative discriminative model.
  • A Preliminaries for Gaussian discriminant analysis: The preliminaries place the proposed analysis within the broader distinction between discriminative and generative classification.The generative formulation specifies the joint distribution as P(x,y) = P(y)P(x|y).
  • A Preliminaries for Gaussian discriminant analysis: Gaussian discriminant analysis assumes class-conditional multivariate Gaussian distributions and class priors represented by β_c.The Gaussian parameters are the class mean μ_c and covariance Σ_c, while β_c is the unnormalized class prior.
  • A Preliminaries for Gaussian discriminant analysis: The paper focuses on LDA, a GDA special case in which all classes share one covariance matrix, Σ_c = Σ.Under the shared-covariance assumption, the quadratic term cancels from the posterior representation.
  • A Preliminaries for Gaussian discriminant analysis: With shared covariance, the generative classifier’s posterior has a form equivalent to the softmax classifier.This equivalence follows by treating the corresponding Gaussian-discriminant expressions as the softmax weights and biases.
  • A Preliminaries for Gaussian discriminant analysis: The equivalence implies that inputs may be fitted to Gaussian distributions during training of a softmax classifier.The supplementary derivation connects the Gaussian-discriminant parameters to the softmax classifier’s weights and biases.
  • B Experimental setup: The supplementary experimental-setup section provides detailed explanations of all experiments described in Section 3.No additional experimental protocol or quantitative result is supplied in the provided passage.

B.1 Experimental setups in detecting out-of-distribution

The experiments evaluate DenseNet and ResNet classifiers on CIFAR and SVHN, using realistic TinyImageNet and LSUN images as out-of-distribution samples. Comparisons include maximum softmax confidence and ODIN, with detection assessed by threshold-dependent and threshold-independent metrics.

  • Datasets and models: DenseNet and ResNet classify CIFAR-10/100 and SVHN using softmax classifiers trained with cross-entropy and SGD with Nesterov momentum.DenseNet has 100 layers and growth rate k = 12; ResNet has 34 layers.
  • Datasets and models: CIFAR and SVHN test sets provide in-distribution positives, while downsampled TinyImageNet and LSUN images provide realistic out-of-distribution negatives.TinyImageNet contains 10,000 test images across 200 classes, and LSUN contains 10,000 test images across 10 scenes.
  • Compared methods: The evaluation compares the proposed method with maximum softmax posterior confidence and ODIN.ODIN uses temperature scaling and image perturbations, selecting εodin from eleven values and T from {1, 10, 100, 1000}.
  • Proposed-method setup: The method extracts confidence scores from every DenseNet dense block or ResNet residual block and applies average pooling to reduce feature-map dimensions.Its hyperparameters include the noise magnitude and layer indexes used for feature ensembling.
  • Evaluation metrics: Performance is measured by TNR at 95% TPR, AUROC, AUPR, and detection accuracy, with AUROC, AUPR, and detection accuracy being threshold-independent.TNR is computed as TN/(FP+TN) when TPR equals 95%.

B.2 Experimental setups in detecting adversarial samples · C More experimental results

The experiments evaluate adversarial-sample detection across four attack methods, compare detection baselines, and specify training, evaluation, and hyper-parameter procedures. Additional results examine robustness under varied training data and class-incremental learning.

  • B.2 Experimental setups in detecting adversarial samples: The study evaluates adversarial-sample detection using FGSM, BIM, DeepFool, and Carlini-Wagner attacks.FGSM perturbs inputs along the loss gradient, BIM iterates FGSM with smaller steps, DeepFool seeks nearby adversarial examples geometrically, and CW uses optimization.
  • B.2 Experimental setups in detecting adversarial samples: CW uses an L2 constraint and is slower in practice than FGSM and BIM.Adversarial samples were generated with Facebook’s library, and Table 4 reports attack perturbation and classification accuracy statistics.
  • B.2 Experimental setups in detecting adversarial samples: The comparison includes LID and kernel-density detectors, with features extracted at the ends of DenseNet or ResNet blocks.LID uses nearest-neighbor distances, while kernel density uses a Gaussian kernel with bandwidth σ.
  • B.2 Experimental setups in detecting adversarial samples: Logistic-regression detectors train on 10% of the original test samples, while the remaining samples are reserved for evaluation.Training examples include adversarial, normal, and noisy samples; noisy examples add random noise to normal examples.
  • B.2 Experimental setups in detecting adversarial samples: Nested cross-validation tunes detector hyper-parameters, including KD bandwidth, LID neighbor count, and input noise.The reported search spaces include k values from 10 to 90, bandwidths from 0.1 to 1, and noise magnitudes from 0 to 0.2.
  • B.2 Experimental setups in detecting adversarial samples: The robustness evaluation varies the number of training samples and the proportion of randomly labeled training data on CIFAR-10.Figure 5 compares AUROC (%) under these training-data conditions using ResNet.
  • B.2 Experimental setups in detecting adversarial samples: Class-incremental experiments on CIFAR-100 and CIFAR-10 report AUC versus learned classes and base-new class accuracy after adding the final new classes.These results are summarized in Figure 6.
  • C More experimental results: Section C provides more experimental results.The supplied passage identifies the section but does not specify the additional results.

C.1 Robustness of our method in detecting adversarial samples · C.2 Class-incremental learning

The method remains robust for adversarial-sample detection across varying training-set sizes and randomly assigned labels, outperforming LID in all CIFAR-10 experiments. In class-incremental learning, its Mahalanobis distance-based classifier achieves higher AUC than softmax and Euclidean-distance baselines as new classes increase.

  • C.1 Robustness of our method in detecting adversarial samples: On CIFAR-10, robustness was evaluated by varying the number of training data used to train ResNet.
  • C.1 Robustness of our method in detecting adversarial samples: The CIFAR-10 robustness experiments also assigned random labels to the training data.
  • C.1 Robustness of our method in detecting adversarial samples: Our method outperformed LID for all CIFAR-10 experiments.
  • C.2 Class-incremental learning: Class-incremental learning was evaluated by pre-training on CIFAR-100 and using CIFAR-10 as new classes.
  • C.2 Class-incremental learning: The proposed Mahalanobis distance-based classifier outperformed other methods by a significant margin as the number of new classes increased.
  • C.2 Class-incremental learning: 47.7% AUC was achieved by the proposed method after all new classes were added, versus 41.0% for softmax and 43.0% for Euclidean distance.

C.3 Experimental results on joint confidence loss · C.4 Comparison with ODIN

The Mahalanobis detector typically outperforms ODIN under standard cross-entropy training and improves further when used with confidence-loss-trained models. The comparison evaluates in- versus out-of-distribution discrimination under validation tuning and under tuning with in-distribution plus FGSM samples.

  • C.3 Experimental results on joint confidence loss: The proposed detector typically outperforms ODIN when both use softmax classifiers trained with their respective standard and confidence losses.ODIN’s confidence loss jointly trains a generator and classifier to calibrate posterior distributions.
  • C.3 Experimental results on joint confidence loss: The Mahalanobis detector provides further improvement when applied to a model trained with confidence loss.The method is presented as improving any pre-trained softmax classifier.
  • C.3 Experimental results on joint confidence loss: Figure 7 compares the baseline, ODIN, and Mahalanobis detectors under various training losses.The cited figure summarizes detector performance across training-loss conditions.
  • C.4 Comparison with ODIN: Table 5 evaluates distinguishing in-distribution from out-of-distribution image-classification test data.Hyperparameters are tuned using validation sets containing in- and out-of-distribution samples, and values are reported as percentages.
  • C.4 Comparison with ODIN: Table 5 marks the best image-classification detection results in bold.The table reports all values as percentages.
  • C.4 Comparison with ODIN: Table 6 compares in- versus out-of-distribution discrimination when ODIN and the proposed method use only in-distribution and adversarial FGSM samples for hyperparameter tuning.All values are percentages, with boldface indicating relatively better results.

C.5 LID for detecting out-of-distribution samples

Figures 8 and 9 compare ODIN, LID, and the Mahalanobis detector across in- and out-of-distribution pairs, with the proposed method outperforming all tested methods.

  • C.5 LID for detecting out-of-distribution samples: The proposed method outperforms ODIN and LID across the evaluated in- and out-of-distribution pairs.The comparison is reported for image classification using ResNet and DenseNet.
  • C.5 LID for detecting out-of-distribution samples: Figures 8 and 9 evaluate distinguishing in- and out-of-distribution test-set data with ResNet and DenseNet, respectively.Figure 8 covers ResNet, while Figure 9 covers DenseNet.

D Evaluation on ImageNet dataset · E Adaptive attacks against Mahalanobis distance-based detector · F Hybrid inference of generative and discriminative classifiers

On ImageNet, the Mahalanobis detector outperformed tested alternatives for adversarial detection, while generative classification remained reasonably accurate but below softmax. Adaptive white-box garbage images could fool the detector, motivating hybrid inference that combines generative and discriminative classifiers while preserving the softmax decision boundary.

  • D Evaluation on ImageNet dataset: ImageNet experiments used 1,000 classes, 1.28 million training images, 50,000 validation images, and pre-trained ResNets.The evaluation focused on ImageNet 2012 using publicly available pre-trained ResNet models.
  • D Evaluation on ImageNet dataset: The generative classifier achieved reasonable ImageNet top-1 accuracy, but the softmax classifier outperformed it in every reported case.The accuracy gap decreased as training accuracy increased.
  • D Evaluation on ImageNet dataset: The Mahalanobis distance-based detector outperformed all tested methods, including LID, for detecting FGSM- and BIM-generated adversarial samples on ImageNet.The detector used confidence scores extracted from every residual block of an 18-layer ResNet.
  • E Adaptive attacks against Mahalanobis distance-based detector: The adaptive-attack evaluation used a white-box setting in which the classifier and Mahalanobis detector parameters were accessible.Access to detector means and covariances requires information about the training data.
  • E Adaptive attacks against Mahalanobis distance-based detector: Minimizing Mahalanobis distance generated random-noise-like garbage images that successfully fooled the pre-trained classifier.Experiments used a 34-layer ResNet on CIFAR-10, attacking either the penultimate layer or every residual-block endpoint.
  • F Hybrid inference of generative and discriminative classifiers: Because the generative classifier can degrade classification performance, the method introduces hybrid inference combining softmax and sample-based generative posteriors.The generative posterior is derived under Gaussian discriminant analysis assumptions using Bayes rule.
  • F Hybrid inference of generative and discriminative classifiers: The hybrid construction matches generative-classifier parameters to softmax weights and biases, producing a generative classifier with the same decision boundary as softmax.The construction uses the empirical covariance to induce the corresponding generative parameters.
  • F Hybrid inference of generative and discriminative classifiers: Table 8 reports classification test-set accuracy for DenseNet and ResNet on CIFAR-10, CIFAR-100, and SVHN.The supplied passage identifies the evaluated architectures and datasets but does not provide the table values.
Loading 1807.03888v2…