Source-linked AI summary
SSD: A Unified Framework for Self-Supervised Outlier Detection
Vikash Sehwag, Mung Chiang, Prateek Mittal
TL;DR
The paper asks whether effective OOD detection can be designed from only unlabeled in-distribution data, given poor unsupervised detectors and label-dependent supervised methods. SSD uses self-supervised representation learning with Mahalanobis detection, and reports strong performance together with gains from few-shot OOD samples and label incorporation.
Problem
Effective OOD detection commonly relies on fine-grained labels, while existing unlabeled-data detectors often perform poorly on complex data such as images.
Method
SSD learns self-supervised feature representations and applies Mahalanobis-distance detection in feature space, with extensions for few-shot OOD samples and available labels.
Results
SSD outperforms most unsupervised detectors, matches or sometimes exceeds supervised detectors, and its performance improves with more unlabeled data.
Takeaways & Limitations
SSD provides an effective and flexible OOD-detection framework that can exploit unlabeled data, a few targeted OOD samples, or training labels.
Takeaways & Limitations
SSD’s hyperparameter behavior remains a consideration, including temperature sensitivity in the contrastive loss and feature-layer-dependent cluster selection.
Abstract
from arXiv · showhide
We ask the following question: what training information is required to design an effective outlier/out-of-distribution (OOD) detector, i.e., detecting samples that lie far away from the training distribution? Since unlabeled data is easily accessible for many applications, the most compelling approach is to develop detectors based on only unlabeled in-distribution data. However, we observe that most existing detectors based on unlabeled data perform poorly, often equivalent to a random prediction. In contrast, existing state-of-the-art OOD detectors achieve impressive performance but require access to fine-grained data labels for supervised training. We propose SSD, an outlier detector based on only unlabeled in-distribution data. We use self-supervised representation learning followed by a Mahalanobis distance based detection in the feature space. We demonstrate that SSD outperforms most existing detectors based on unlabeled data by a large margin. Additionally, SSD even achieves performance on par, and sometimes even better, with supervised training based detectors. Finally, we expand our detection framework with two key extensions. First, we formulate few-shot OOD detection, in which the detector has access to only one to five samples from each class of the targeted OOD dataset. Second, we extend our framework to incorporate training data labels, if available. We find that our novel detection framework based on SSD displays enhanced performance with these extensions, and achieves state-of-the-art performance. Our code is publicly available at https://github.com/inspire-group/SSD.
1 INTRODUCTION
SSD addresses the challenge of effective OOD detection when only unlabeled in-distribution data are available. It combines self-supervised representations with feature-space modeling and extends the framework to few-shot OOD data and available labels.
- Deep neural networks can fail on inputs far from their training distribution, motivating OOD detection before downstream processing.
- Unlabeled-data detectors often perform poorly on complex modalities such as images, while supervised detectors assume costly fine-grained labels.
- SSD learns feature representations through self-supervised training and models in-distribution features without requiring class labels.
- Few-shot OOD detection uses one to five targeted outlier samples, which can provide an additional performance boost.
- When labels are available, incorporating them directly into the contrastive loss yields a tuning-parameter-free detector with state-of-the-art performance.
- SSD outperforms most existing unsupervised detectors by a large margin and performs on par with, or sometimes better than, supervised detectors across four datasets.
2 RELATED WORK
Prior OOD and anomaly-detection work spans unsupervised reconstruction, classification, and probabilistic methods, alongside supervised detectors that model in-distribution features. Recent approaches also use OOD samples or combine self-supervised objectives with labels, while SSD is extended to few-shot OOD data and available training labels.
- OOD detection with unsupervised detectors: Unsupervised OOD detectors include reconstruction-error methods, classification-based methods such as Deep-SVDD, and probabilistic detectors.Reconstruction approaches use auto-encoders or variational auto-encoders.
- OOD detection with supervised learning: Supervised OOD detectors have been especially successful for complex inputs such as images and language by modeling in-distribution features at the output or in feature space.
- Access to OOD data at training time: Some detectors require OOD samples for hyperparameter tuning, motivating few-shot OOD detection with only a few available OOD examples.The SSD extension is designed to utilize these few samples for further performance gains.
- In conjunction with supervised training: Several state-of-the-art systems combine supervised cross-entropy with leave-one-out ensembles, contrastive self-supervision, or rotation-based self-supervision.SSD is extended to incorporate data labels when available and is reported to outperform existing state-of-the-art methods.
- Anomaly detection: Anomaly detection addresses semantically related anomalies in applications including intrusion, spam, disease, image, and video detection.The related literature includes many methods focused on single-class modeling.
3 SSD: SELF-SUPERVISED OUTLIER/OUT-OF-DISTRIBUTION DETECTION
SSD detects OOD samples using contrastive self-supervised representations and cluster-conditioned Mahalanobis scoring, without requiring labels. Its extensions incorporate few-shot OOD samples or labels when available.
- Unsupervised detection: SSD first learns features through contrastive self-supervised training, then detects OOD samples using hidden features without data labels.The contrastive objective pulls augmented views together and pushes representations of other images apart.
- Unsupervised detection: SSD partitions in-distribution features into m clusters, models each cluster independently, and scores test inputs by their nearest cluster distance.The framework uses k-means clustering and computes the minimum distance across clusters.
- Unsupervised detection: Mahalanobis distance scales principal directions by covariance, avoiding Euclidean distance’s bias toward high-variance components that provide less discrimination.For CIFAR-10 versus CIFAR-100, the figure reports AUROC along individual principal eigenvectors and shows better performance with Mahalanobis scaling.
- Few-shot OOD detection: Few-shot SSD uses one or five samples from each targeted OOD class and estimates OOD statistics with shrunk covariance and data augmentation.These techniques address covariance estimation when the number of OOD samples is smaller than the feature dimension.
- Using data labels: With labels, SSD uses supervised contrastive training to select positives and negatives more effectively, producing a parameter-free detector with state-of-the-art performance.The labeled formulation uses images with identical labels as positives and retains the cluster-conditioned Mahalanobis framework.
4 EXPERIMENTAL RESULTS
SSD uses NT-Xent self-supervised training and Mahalanobis-based feature detection, with experiments examining representation choices, training parameters, unlabeled data, and supervised or few-shot extensions. Across these settings, SSD improves over standard unsupervised detectors, sometimes exceeds supervised representations, and its extensions achieve further gains.
- Common setup: SSD trains image representations with NT-Xent and applies Mahalanobis distance in feature space, using AUROC, FPR, and AUPR for evaluation.The main experiments use ResNet-50, with additional architecture ablations and a supervised baseline matched for training budget and detection method.
- Performance of SSD: SSD improves average AUROC by up to 55 compared to standard density, reconstruction, and one-class outlier detectors.Those baselines include PixelCNN++, autoencoders, variational autoencoders, and Deep-SVDD.
- Ablation studies: CIFAR-100 is hardest to detect among four OOD datasets, with training epochs and batch size playing key roles in its detection performance.Architecture has little effect on AUROC for most OOD datasets, while increasing the training-dataset size helps across all four datasets.
- Comparison with supervised representations: Self-supervised representations achieve better performance than supervised representations for 56% of tasks across sixteen in-distribution and OOD dataset pairs.The comparison uses identical training budgets and Mahalanobis distance in the feature space.
- Success in anomaly detection: In anomaly detection on CIFAR-10, SSD outperforms previous works and achieves competitive performance with Tack et al. (2020).This setting adds feature-space ℓ2 regularization with scaling coefficient 0.01 and reduces NT-Xent temperature from 0.5 to 0.1.
- Success when using data labels (SSD+): Using training labels improves average AUROC by 0.8 over the previous state-of-the-art detector, while five-shot OOD detection adds a further gain of 1.4.For CIFAR-100 in-distribution and CIFAR-10 OOD, labels raise Mahalanobis AUROC from 55.5 to 72.1, and a two-layer MLP raises it to 78.3.
5 DISCUSSION AND CONCLUSION
The discussion examines SSD’s training choices, feature-learning behavior, and use of unlabeled data, while reporting extensions and evaluation context.
- SSD avoids additional tuning parameters when combining self-supervised and supervised losses, and uses standard self-supervised parameters with a single-cluster feature model.
- Table 4 compares label-incorporating SSD+ with supervised state-of-the-art detectors.
- SSD’s performance depends on contrastive-learning temperature, with lower temperatures degrading outlier-detection performance.
- Self-supervised representations begin less discriminative than supervised ones but quickly catch up after half the training epochs.
- Adding unlabeled data improves SSD performance, increasing AUROC from 94.7 to 99.4 on CIFAR-100 OOD detection in the STL-10 experiment.
A.1 TRAINING AND EVALUATION SETUP FOR DEEP NEURAL NETWORKS.
The setup describes neural-network architectures, self-supervised training and evaluation metrics, dataset preprocessing, and the datasets used for experiments.
- Major experiments use ResNet-50, ablations use ResNet-18, and additional results use ResNet-34 and ResNet-101.
- NT-Xent training uses batch size 512, reflecting its need for more negatives than supervised cross-entropy training.
- Evaluation reports FPR at TPR=95%, AUROC, and AUPR as outlier-detection metrics.
- OOD images are bilinearly scaled when resolutions differ, and overlapping OOD classes are removed.
- Uniform-noise images sample each pixel uniformly from [0, 1], while Gaussian-noise images use mean 0.5 and standard deviation 0.25.
B.1 LIMITATIONS OF OUTLIER DETECTORS BASED ON SUPERVISED TRAINING
This section examines how supervised-label assumptions, extra unlabeled data, feature-extractor choice, and clustering affect outlier detection.
- Supervised detectors are evaluated under coarse-label settings created by grouping CIFAR-10 classes into CIFAR-2 and CIFAR-5.
- SSD can achieve similar performance to an existing supervised detector without fine-grained labels.
- Using extra unlabeled training data can further improve SSD performance.
- The optimal cluster count depends on the residual-network layer used for feature extraction.
B.3 ABLATION STUDY FOR FEW-SHOT OOD DETECTION
The few-shot OOD ablation measures how the number of image transformations affects SSD_k performance and selects a computationally balanced setting.
- The final experiments use ten transformations per sample to balance detection performance and computational cost.
B.4 PERFORMANCE OF SSD IMPROVES WITH AMOUNT OF UNLABELED DATA
On STL-10, SSD benefits substantially from adding unlabeled training data. The comparison uses self-supervised and supervised networks trained with 5,000 labeled images, then adds 10,000 unlabeled images.
- SSD achieves large performance gains when 10,000 additional unlabeled STL-10 images are added to training.The additional images come from the dataset’s available 100,000 unlabeled images.
- The experiment compares SSD-5k, a self-supervised network, with Sup-5k, an equivalent supervised network.Both networks are trained with 5,000 STL-10 training images before adding unlabeled data.
B.5 RESULTS WITH DIFFERENT PERFORMANCE METRICS
The paper reports detailed SSD component experiments using multiple performance metrics across ImageNet, CIFAR-10, CIFAR-100, and STL-10.
- Detailed experimental results evaluate each component of the SSD framework with three performance metrics.
- Table 6 reports multiple SSD performance metrics on ImageNet.
- Table 7 reports multiple SSD performance metrics across CIFAR-10, CIFAR-100, and STL-10.