Source-linked AI summary
Modeling the Distribution of Normal Data in Pre-Trained Deep Features for Anomaly Detection
Oliver Rippel, Patrick Mertens, Dorit Merhof
TL;DR
Image anomaly detection must model normality despite rare, ambiguously defined anomalies and limited labeled anomaly data. The paper transfers ImageNet-trained deep features, fits an MVG using normal data, and scores deviations with Mahalanobis distance, achieving state-of-the-art performance on MVTec AD while analyzing feature components and operating-point selection.
Problem
Anomaly detection commonly relies on normal data because anomalies are scarce and their appearance is ambiguous, while feature learning from scratch has limited use of large-scale pretraining.
Method
The method fits a multivariate Gaussian to ImageNet-pretrained classifier features from normal data and uses Mahalanobis distance as the anomaly score.
Results
The approach achieves 96.7% ± 1.0% AUROC for EfficientNet-B4 level 7 and outperforms prior state-of-the-art performance on MVTec AD.
Takeaways & Limitations
Low-variance principal components are highly discriminative, supporting the use of transferred deep features and enabling reduced-complexity modeling while retaining anomaly-detection performance.
Takeaways & Limitations
Evaluation on larger anomaly-detection datasets is still required, and architectural changes may improve the match between desired and achieved false-positive rates.
Abstract
from arXiv · showhide
Anomaly Detection (AD) in images is a fundamental computer vision problem and refers to identifying images and image substructures that deviate significantly from the norm. Popular AD algorithms commonly try to learn a model of normality from scratch using task specific datasets, but are limited to semi-supervised approaches employing mostly normal data due to the inaccessibility of anomalies on a large scale combined with the ambiguous nature of anomaly appearance. We follow an alternative approach and demonstrate that deep feature representations learned by discriminative models on large natural image datasets are well suited to describe normality and detect even subtle anomalies in a transfer learning setting. Our model of normality is established by fitting a multivariate Gaussian (MVG) to deep feature representations of classification networks trained on ImageNet using normal data only. By subsequently applying the Mahalanobis distance as the anomaly score we outperform the current state of the art on the public MVTec AD dataset, achieving an AUROC value of $95.8 \pm 1.2$ (mean $\pm$ SEM) over all 15 classes. We further investigate why the learned representations are discriminative to the AD task using Principal Component Analysis. We find that the principal components containing little variance in normal data are the ones crucial for discriminating between normal and anomalous instances. This gives a possible explanation to the often sub-par performance of AD approaches trained from scratch using normal data only. By selectively fitting a MVG to these most relevant components only, we are able to further reduce model complexity while retaining AD performance. We also investigate setting the working point by selecting acceptable False Positive Rate thresholds based on the MVG assumption. Code available at https://github.com/ORippler/gaussian-ad-mvtec
I. INTRODUCTION
Image anomaly detection must identify rare, ambiguously defined deviations despite datasets dominated by normal examples. This motivates transferring representations learned from large-scale natural-image data instead of learning features entirely from scratch.
- Image anomaly detection identifies irregularities that differ significantly from the norm across applications including industrial quality control and medical image analysis.
- Anomalies are rare events with ambiguous appearance, making their types difficult to define in advance.
- These properties produce heavily imbalanced datasets containing few anomalies for model verification and testing.
- Semi-supervised methods therefore commonly model normality from normal data only, while relatively little work explores ImageNet pretraining for anomaly detection.
- The paper transfers ImageNet-trained deep features, fits an MVG to normal representations, and uses Mahalanobis distance for anomaly scoring.
- PCA shows that low-variance components are highly discriminative, suggesting why features learned from normal data alone may be difficult to obtain.
B. Transfer Learning AD with Deep Feature Representations
Transfer-learning anomaly detection uses representations learned from large-scale ImageNet training, but prior approaches differ in feature granularity, datasets, anomaly difficulty, and evaluation scope.
- ImageNet-trained deep representations have been used less extensively for anomaly detection than semi-supervised feature-learning methods.
- Patch-wise segmentation methods limit receptive fields and feature complexity while assuming anomalies fit within individual patches.
- Earlier image-level work fitted an MVG and used Mahalanobis distance on deep features but evaluated only a small in-house dataset.
- That earlier agricultural setting involved anomalies with large appearance differences and lacked benchmarking against other anomaly-detection approaches.
- Lee et al. modeled in-distribution data with an MVG for OOD detection, a task distinct from detecting anomalies within an application domain.
III. MODELING NORMAL DATA DISTRIBUTION IN DEEP FEATURE REPRESENTATIONS
The paper models normal deep-feature distributions with multivariate Gaussians and scores deviations using Mahalanobis distance, whose probabilistic interpretation supports anomaly detection.
- The method models each normal class’s feature probability density with a multivariate Gaussian using normal data only and without fine-tuning the pretrained model.
- The MVG uses a D-dimensional mean vector and symmetric D-by-D covariance matrix that must be positive definite.
- Mahalanobis distance measures the distance between a feature point and the fitted Gaussian distribution.
- Mahalanobis distance determines an observation’s Gaussian probability density and measures its uncertainty.
- For Gaussian samples, the squared Mahalanobis distance follows a chi-squared distribution with D degrees of freedom.
A. Covariance Estimation
Covariance estimation is challenging when feature dimensionality approaches the number of normal samples. The method addresses this instability with shrinkage toward a scaled identity matrix.
- The covariance matrix is estimated from observed feature vectors using their sample covariance and empirical mean.
- The sample covariance is well-conditioned only when feature dimension D is much lower than sample count n.
- When D exceeds n, the covariance estimate becomes singular and cannot be inverted for Mahalanobis scoring.
- Shrinkage combines the empirical covariance with a scaled identity matrix to produce a better-conditioned estimate.
- The shrinkage intensity ρ controls the balance between the empirical covariance and the well-conditioned identity estimate.
- Ledoit–Wolf selection chooses ρ by minimizing expected squared error relative to the true covariance.
B. Setting the Working Point
Assuming multivariate Gaussian features, the detector’s working point can be selected from the desired false positive rate using the chi-square CDF and its inverse.
- Under the MVG assumption, a Mahalanobis distance threshold corresponds to the probability of observing a normal sample.This probability matches the expected true negative rate at that threshold, while its complement is the allowed false positive rate.
- For Gaussian data, the probability that a sample’s Mahalanobis score is below t is given by the chi-square distribution CDF evaluated at t^2.
- The anomaly-detection threshold can be obtained from the inverse CDF for any desired false positive rate.
IV. EXPERIMENTS AND RESULTS
The experiments evaluate pre-trained classifier features for image anomaly detection, comparing distributions, architectures, feature levels, and model complexity. Multivariate Gaussian scoring with Mahalanobis distances performs strongly, especially for deeper EfficientNet features, while performance can saturate or decline in the deepest or most complex settings.
- Method: The method extracts features from every level of ImageNet-pre-trained EfficientNet or ResNet models, pools spatial maps, and scores them with Mahalanobis distance.Scores can be summed across levels, and the approach uses average pooling for spatial feature maps.
- Experimental design: The experiments compare multivariate Gaussian, fixed-variance Gaussian, and feature-independent Gaussian assumptions using corresponding anomaly scores.The fixed-variance case reduces to L2 distance to the training mean, while the feature-independent case uses standardized Euclidean distance.
- Experimental design: The evaluation uses 5 folds per MVTec AD category and reports mean ± SEM AUROC across categories and folds.The first feature-level evaluation performs no feature reduction and also reports scores summed over all levels.
- Distribution and feature-level results: 96.7% ± 1.0% AUROC is achieved by the MVG at EfficientNet-B4 level 7, where the MVG is reported as best suited for AD.Deeper feature representations are reported as more suitable for transfer-learning anomaly detection, although multivariate performance saturates or declines at levels 8 and 9.
- Architecture comparison: 90.4% ± 3.6% AUROC at the best level and 88.2% ± 4.0% for the sum predictor are reported for ResNet-34, below the EfficientNet results.The text attributes the difference to EfficientNet’s architecture and Swish activation, noting that standardized Euclidean scoring can fail with ReLU-clipped features.
- Score aggregation: 94.8% ± 1.6% average AUROC is achieved by equally weighting feature-level scores, without a learned linear weighting.The comparison is made against an OOD approach that uses gradient-ascent input preprocessing and pursues a somewhat different objective.
- Model complexity: Performance is worse for EfficientNet-B0–B3, saturates for more complex variants, and degrades for EfficientNet-B7.The authors suggest that the most complex variants may overfit ImageNet and learn features that generalize less well to new domains or use cases.
A. Why Pre-Trained Deep Features Work so Well
The paper explains pre-trained features’ anomaly-discriminative power by showing that low-variance principal components are especially informative and can support dimensionality reduction without losing performance.
- PCA retaining only high-variance components reduces anomaly-detection performance across feature levels.Removing components accounting for 1% of variance reduces performance across all evaluated levels.
- NPCA retains low-variance components, preserves performance, and can improve performance in sum mode.NPCA 0.01% reduces level-7 dimensionality from 272 to 15.6 features on average.
- Low-variance feature combinations are ultimately those that can discriminate between normal and anomalous images.These combinations do not occur in normal data and therefore cannot be learned effectively from normal examples alone.
- The findings support a possible explanation for poorer performance by approaches that learn features from scratch using only normal data.Such approaches may fail to learn feature combinations that vary little in normal data but distinguish anomalies.
B. Choosing a Working Point Solely on FPR
The MVG assumption provides a theoretical way to select an acceptable false-positive-rate threshold, but reliable working-point selection depends on augmentation, model complexity, and compression.
- The MVG assumption enables selecting a working point by choosing an acceptable FPR threshold.This framework is unavailable directly for the sum mode, where feature-level Mahalanobis distances are added.
- Augmentations were essential for enabling working-point selection; without them, the procedure could fail catastrophically.For EfficientNet-B0 without compression, an example reached 99.8% FPR and 99.9% TPR at 3σ.
- PCA decreases test-set FPR, whereas NPCA increases it, making PCA potentially useful for robust FPR estimates at reduced AD performance.The reported inverse behavior is observed across the evaluated compression modes.
- Even with artificially enlarged datasets, sensible training-based FPR setting was possible only for EfficientNet-B0.The authors attribute this boundary to dimensionality-driven overfitting in more complex models and call for evaluation on larger AD datasets.
C. Comparison with State of the Art on MVTec AD
On MVTec AD, MVG estimation on pre-trained deep features substantially outperforms prior non-fully-supervised methods, while remaining close to fully supervised performance. The evaluation compares multiple baselines, architectures, feature-level strategies, and compression settings.
- The evaluation compares the proposed method with a convolutional autoencoder, a fully supervised classifier, an oc-SVM, and reported state-of-the-art methods.The oc-SVM uses pre-trained EfficientNet representations, while the classifier serves as an upper bound.
- The study also evaluates sum-mode aggregation across all EfficientNet-B4 feature levels to assess general applicability and complexity reduction.The authors report comparable performance while omitting feature-level selection.
- MVG estimation on pre-trained deep features achieves 10% higher average AUROC than the next-best model, SPADE, on MVTec AD.The comparison includes state-of-the-art methods and reports AUROC with SEM; SPADE also uses pre-trained feature spaces.
- The proposed approach is mostly comparable to fully supervised fine-tuning but is sometimes slightly worse, especially for texture categories.Fine-tuning achieves better performance for some object categories, including pill and screw.
V. DISCUSSION
The discussion attributes the method’s performance to ImageNet-pretrained features and the MVG assumption, while identifying semantic distance and multimodal normality as important scope boundaries.
- ImageNet-pretrained deep features support general-purpose image anomaly detection, including settings semantically different from the training dataset.
- The MVG assumption is crucial for attaining state-of-the-art anomaly-detection performance with pre-trained deep features.
- PCA shows that low-variance principal components in normal data are necessary for discriminating normal from anomalous images.
- Out-of-the-box performance is expected to decrease as the semantic distance from natural images increases, such as in medical images.
- The presented approach assumes a unimodal normal-data distribution, while multimodal anomaly-detection settings remain a target for future extension.
- The method can select an acceptable false-positive rate, but it cannot provide PAC-style true-positive-rate guarantees for security-critical use cases.
APPENDIX
The appendix includes an EfficientNet-B0 baseline-network table, a ResNet feature-level AUROC table, an augmentation pipeline figure, representative-anomaly figure, and model-comparison table.
- Table VI presents the EfficientNet-B0 baseline network used in the study.
- Table VII reports feature-level AUROC scores with Mahalanobis distance for ResNet architectures.
- Figure 2 depicts the augmentation pipeline applied to enhance the MVTec AD dataset.
- Table VIII reports AUROC scores for all models across MVTec AD categories and highlights the highest score in each row.
- Figure 3 shows representative anomalies across all MVTec AD categories with segmentation-ground-truth boundaries.