Source-linked AI summary
Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distance
Taylor Denouden, Rick Salay, Krzysztof Czarnecki, Vahdat Abdelzad, Buu Phan, Sachin Vernekar
TL;DR
The paper addresses cases where reconstruction autoencoders assign low error to OOD images, limiting reconstruction error as a novelty measure. It adds Mahalanobis distance from the training distribution in latent space, and reports that the hybrid score often improves OOD detection over reconstruction error alone. The approach is evaluated with single-class MNIST autoencoders, while broader datasets and multi-class inlier modeling remain future work.
Problem
Reconstruction autoencoders can give low reconstruction error to OOD samples far from encoded training data, making reconstruction error alone insufficient for reliable detection.
Method
The method combines reconstruction error with Mahalanobis distance from the mean latent embedding of the encoded training data.
Results
The hybrid approach often improves OOD detection performance over using reconstruction error alone.
Takeaways & Limitations
Latent distance can improve separation between inlier and OOD data while retaining the reconstruction-autoencoder framework.
Takeaways & Limitations
The study identifies extending the method to more complex naturalistic image datasets and modeling multiple inlier classes as future work.
Abstract
from arXiv · showhide
There is an increasingly apparent need for validating the classifications made by deep learning systems in safety-critical applications like autonomous vehicle systems. A number of recent papers have proposed methods for detecting anomalous image data that appear different from known inlier data samples, including reconstruction-based autoencoders. Autoencoders optimize the compression of input data to a latent space of a dimensionality smaller than the original input and attempt to accurately reconstruct the input using that compressed representation. Since the latent vector is optimized to capture the salient features from the inlier class only, it is commonly assumed that images of objects from outside of the training class cannot effectively be compressed and reconstructed. Some thus consider reconstruction error as a kind of novelty measure. Here we suggest that reconstruction-based approaches fail to capture particular anomalies that lie far from known inlier samples in latent space but near the latent dimension manifold defined by the parameters of the model. We propose incorporating the Mahalanobis distance in latent space to better capture these out-of-distribution samples and our results show that this method often improves performance over the baseline approach.
1 Introduction
Safety-critical perception systems need to detect inputs from classes outside their training data. This work focuses on reconstruction autoencoders as an unsupervised, low-assumption approach to improving OOD detection.
- Safety-critical applications such as autonomous vehicles require perception systems that can recognize out-of-distribution inputs.
- The paper focuses on reconstruction autoencoder-based OOD detection methods.
- Reconstruction autoencoders can operate unsupervised while making few assumptions about the inlier data-generating distribution.
- The authors argue that reconstruction error alone is insufficient as a novelty metric and propose adding latent-space distance.
2 Related work
Prior work improves OOD detection with latent-space distances or latent-embedding probabilities, but these approaches involve computational costs or modeling choices. Mahalanobis-based classification methods provide a related precedent while requiring labeled data and multiple inlier classes.
- KNN distance in autoencoder latent space improved OOD detection but required inference-time iteration over training data.
- The KNN approach has high computational and memory costs that can make it impractical for many real-world applications.
- A VAE-based method incorporates the probability of latent embeddings relative to a specified prior distribution.
- Classification-based Mahalanobis detection achieved state-of-the-art results on multiple OOD benchmarks but requires labeled training data.
- The paper's unsupervised method supports OOD detection when a single object class is treated as the inlier class.
3 Background
Autoencoders compress inputs through a lower-dimensional bottleneck and use reconstruction error as a novelty measure. Mahalanobis distance instead measures standardized deviation from a multivariate mean while accounting for feature correlations.
- Autoencoders learn compressed representations through bottleneck layers with fewer dimensions than the original input.
- The standard assumption is that OOD images reconstruct poorly, making reconstruction error a novelty measure.
- Mahalanobis distance measures how many standard deviations a test sample lies from a multivariate Gaussian mean vector.
- Mahalanobis distance is scale-invariant and accounts for correlations between data dimensions.
- Unlike Mahalanobis distance, Euclidean distance assumes data dimensions share a common measurement scale.
4 Problem and approach
Autoencoders can assign low reconstruction error to visibly distinct OOD samples, making threshold-based detection unreliable. The proposed explanation is that such samples lie near the latent manifold but far from encoded training data, motivating an additional latent-distance measure.
- Autoencoders sometimes reconstruct OOD samples with less error than many inlier samples.
- This overlap prevents a novelty threshold from correctly classifying those OOD samples while preserving correct classification of all inliers.
- MNIST examples show OOD digits with low reconstruction error despite being visibly distinct from the digit-0 inlier class.
- The failure occurs when OOD samples lie near the bottleneck's latent manifold but far from encoded training samples.
- Increasing bottleneck dimensionality can increase model power and potentially allow identity-function reconstruction, so the authors propose adding latent-space distance instead.
5 Experiments and results
The experiments train single-class MNIST autoencoders across bottleneck sizes and evaluate a novelty score combining reconstruction loss with latent-space Mahalanobis distance. This hybrid generally outperforms reconstruction error alone, with performance varying by bottleneck size.
- Ten autoencoder architectures used bottleneck sizes from 2 to 798, each trained on images from one MNIST digit class treated as inlier.
- The novelty score combines Mahalanobis distance from encoded training data with reconstruction loss, using the autoencoder's encoding and decoding functions.
- Validation-set normalization scales the two components by reciprocal standard deviations so neither dominates the unified novelty score.
- In most cases, incorporating latent distance improves OOD detection over reconstruction error alone.Figure 3 compares encoded-data separation and FPR (95% TPR); the best hybrid bottleneck varies between 8 and 64 for each digit.
6 Conclusion and future work
The paper concludes that reconstruction error alone can miss OOD images that reconstruct accurately despite being far from training data in latent space. It combines reconstruction error with latent Mahalanobis distance, while identifying broader datasets, combination tuning, and multi-class inliers as future directions.
- The proposed hybrid novelty score addresses OOD samples that are far from training data in latent space but reconstructed with minimal error.
- Using latent Mahalanobis distance alongside reconstruction error often improves performance over reconstruction error alone.
- Future work includes evaluation on more complex naturalistic image datasets and generalization to multiple inlier classes in one autoencoder.
- Table 1 compares baseline reconstruction-error detection with the hybrid approach using AUROC, AUPR (in), AUPR (out), and FPR (95% TPR).
Model architecture
The autoencoder uses two convolutional layers with ReLU and max pooling, followed by flattening and a regularized dense bottleneck whose size is varied.
- The architecture has two 3x3 convolutional layers with ReLU activation, each followed by max pooling.
- The first convolution uses 32 filters and the second uses 2 filters before flattening and a dense layer.
- The dense layer uses L1 regularization multiplied by 0.00001, and its size varies to test differently sized bottlenecks.
Training procedure
Models are trained with Adadelta and binary cross-entropy on a 50,000-image MNIST training split, using early stopping within a 500-epoch limit.
- Each model is trained with the Adadelta optimizer and binary cross-entropy loss.The loss was selected for normalized image values in [0, 1] and faster convergence to a better solution than mean squared error.
- The MNIST training set is split into 50,000 training samples and 10,000 validation samples.
- Training runs for at most 500 epochs with early stopping.
Complete results
The results section compares baseline reconstruction-error OOD detection with a hybrid approach across inlier classes 0 through 9. Each table identifies the best values in bold.
- Tables compare the baseline reconstruction-error approach with the hybrid approach for inlier classes 0, 1, and 2.
- Across these tables, bold formatting marks the best values.
- Tables compare the baseline reconstruction-error approach with the hybrid approach for inlier classes 3, 4, and 5.
- Tables compare the baseline reconstruction-error approach with the hybrid approach for inlier classes 6, 7, 8, and 9.