Source-linked AI summary
Contrastive Training for Improved Out-of-Distribution Detection
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, Taylan Cemgil, S. M. Ali Eslami, Olaf Ronneberger
TL;DR
Reliable OOD detection matters because classifiers can fail on unfamiliar inputs, especially in safety-critical settings. The paper uses contrastive training to learn richer representations and introduces CLP to quantify inlier–outlier similarity. It reports consistent improvements across benchmarks, particularly for near-OOD classes, without requiring explicit OOD examples during training or tuning.
Problem
OOD detection is needed because accurate classifiers can make arbitrarily bad predictions on unfamiliar inputs, while near-OOD evaluation remains difficult because similarity between inlier and outlier distributions varies.
Method
The approach jointly trains contrastive and supervised objectives in a SimCLR-based architecture using in-distribution data, then scores class-wise Gaussian densities over learned activations.
Results
The method consistently improves OOD detection across benchmarks, especially in near-OOD settings, and outperforms competitive methods without requiring outlier-distribution data during training or tuning.
Takeaways & Limitations
Contrastive training can improve OOD detection beyond purely supervised representations, while CLP provides a similarity-based measure of task difficulty.
Takeaways & Limitations
The additional training objective cannot guarantee that all features necessary for near-OOD classes are learned.
Abstract
from arXiv · showhide
Reliable detection of out-of-distribution (OOD) inputs is increasingly understood to be a precondition for deployment of machine learning systems. This paper proposes and investigates the use of contrastive training to boost OOD detection performance. Unlike leading methods for OOD detection, our approach does not require access to examples labeled explicitly as OOD, which can be difficult to collect in practice. We show in extensive experiments that contrastive training significantly helps OOD detection performance on a number of common benchmarks. By introducing and employing the Confusion Log Probability (CLP) score, which quantifies the difficulty of the OOD detection task by capturing the similarity of inlier and outlier datasets, we show that our method especially improves performance in the `near OOD' classes -- a particularly challenging setting for previous methods.
1 Introduction
OOD detection is needed because accurate classifiers can fail on unfamiliar inputs, especially in safety-critical deployment. The paper uses contrastive training and CLP to improve detection, with particular attention to difficult near-OOD cases.
- Motivation: Accurate neural networks can make arbitrarily bad predictions on unfamiliar inputs, complicating real-world deployment.OOD detection can instead route unfamiliar inputs for separate processing, such as human review.
- Contribution: Contrastive training encourages task-agnostic semantic features by pulling transformed views of an image together and pushing other images apart.The approach learns a richer representation from in-distribution data without explicit OOD examples.
- Motivation: Supervised training may omit features unnecessary for inlier classification but necessary for detecting OOD inputs.The toy example shows contrastive training preserving sensitivity to an additional input dimension required for detection.
- Motivation: Near-OOD evaluation uses meaningfully similar inlier and outlier distributions, whereas far-OOD evaluation uses unrelated distributions.The paper argues near-OOD cases are more prevalent in practice and proposes quantifying distributional similarity.
- Contribution: The method consistently improves OOD detection across benchmarks and outperforms competitive methods such as Outlier Exposure without requiring outlier-distribution data during training or tuning.This avoids a data requirement that can be difficult to satisfy in practice.
- Contribution: Confusion Log Probability (CLP) measures similarity between inlier and outlier datasets, showing especially strong gains in near-OOD settings.Figure 1 organizes CIFAR-100 outlier classes by similarity to CIFAR-10 inliers using class-wise CLP.
2 Related Work
Prior OOD detection work has developed feature scores, alternative training objectives, uncertainty estimates, and generative or hybrid models. These approaches differ in how they enrich representations or estimate novelty.
- Representations from classification networks: Many OOD methods score intermediate representations from discriminatively trained classifiers, using techniques such as MSP, ODIN, or class-conditional Gaussian estimates.Their effectiveness depends strongly on whether the learned feature space retains properties relevant to OOD detection.
- Alternative training strategies: Alternative training strategies improve OOD scores through confidence losses, auxiliary objectives, margin losses, outlier exposure, or multi-head architectures.These methods adapt training rather than relying only on improved scoring functions.
- Bayesian approaches: Bayesian approaches estimate model uncertainty by learning distributions over network weights or using Monte Carlo dropout sampling.The cited methods operate under a Bayesian uncertainty-estimation paradigm.
- Generative and hybrid models: Generative and hybrid approaches compute likelihood-based OOD scores, but direct image-space generative modeling has not achieved state-of-the-art results even with classification networks.The related work includes ensembles and likelihood-ratio scoring as variants of this strategy.
3 Proposed Method
The proposed method jointly learns a richer representation with contrastive and supervised objectives using a SimCLR-based multitask architecture. OOD scores are then obtained from class-wise Gaussian density models over penultimate-layer activations.
- Representation learning: Contrastive training learns representations capable of distinguishing individual training samples while incorporating identity-preserving transformations.This addresses features that supervised classification alone may not retain for OOD detection.
- Architecture: The architecture uses an encoder fθ and two heads: gφ for class predictions and hν for a lower-dimensional contrastive embedding.The feature space is learned using in-distribution training samples only.
- Architecture: Two randomly selected transformations of each training image are encoded to produce paired representations for contrastive learning.Examples include crop-resize and color distortions.
- Objective: The contrastive objective increases cosine similarity for views of the same image and decreases similarity for views of different images.A projection head maps representations into a lower-dimensional embedding space where this relational classification task is optimized.
- Training: Training first optimizes Lcon, then optimizes the combined loss Lcon + λLclass for fewer epochs.The second stage adds supervised class information after contrastive representation learning.
- Density estimation: OOD detection fits a class-wise multivariate Gaussian to penultimate activations and assigns each input the highest class-conditional density.The method relies on a richer representation rather than detector ensembling or input preprocessing.
4 Confusion Log Probability (CLP) as a Measure of Dataset Distance
The paper introduces Confusion Log Probability (CLP) to measure OOD task difficulty from classifier confusion between inlier and outlier datasets. CLP captures visual similarity, distinguishes near from far OOD, and varies across classes and datasets.
- Current single-pair metrics such as AUROC cannot resolve performance differences between near and far OOD, motivating a spectrum-sensitive difficulty measure.
- CLP addresses limitations of class-count and image-space distance measures by grounding dataset distance in visual class confusion across the near-to-far spectrum.
- CLP measures OOD detection difficulty using the probability that a classifier confuses outliers with inliers.
- The CLP calculation trains an ensemble on the joint inlier–outlier dataset and estimates class confusions on held-out test data.
- Low CLP indicates far OOD, whereas high CLP indicates near OOD; the measure is asymmetric because it evaluates confusion with inlier classes.
- Class-wise CLP evaluates individual OOD classes, while dataset CLP aggregates all samples in the test dataset.
5 Experiments
Experiments evaluate contrastive training across challenging near-to-far OOD dataset pairs using AUROC and sample-level OOD rank. Contrastive training improves detection broadly, with the strongest gains in difficult near-OOD regimes and when combined with label smoothing.
- Experiments: The benchmark uses CIFAR-10, CIFAR-100, and SVHN in-distribution/outlier pairs intended to represent challenging near-to-far OOD settings.
- Experiments: AUROC provides threshold-independent, calibration-free detection evaluation, while OOD rank measures how an individual score compares with inlier test scores.
- Results: 92.9 AUROC is obtained for CIFAR-10 inliers and CIFAR-100 outliers, more than two points above the next-best method without labeled outlier training data.
- Results: +18 AUROC points is the largest contrastive-training gap in the high-CLP regime, while the advantage remains across the full CLP spectrum.
- Results: 90.2 AUROC is achieved on average across all three dataset pairs, outperforming the previous state-of-the-art method without outlier data or labels during training or tuning.
- Ablation: Combining contrastive training with label smoothing produces significantly better results than either technique alone and reduces variation in OOD ranks between runs.
- Failure mode analysis: For most of the baseline’s worst CIFAR-100 mistakes, contrastive training successfully identifies the samples as outliers.
6 Discussion and Conclusion
The paper concludes that contrastive training improves OOD detection across settings without requiring outlier-distribution data, while acknowledging limits in near-OOD feature discovery and evaluating richer representations rather than scoring methods.
- Contrastive training improves OOD detection across varied settings without requiring access to outlier-distribution data during training or tuning.The approach uses joint training: contrastive loss separates representations, while supervised loss clusters them by class.
- The method cannot guarantee discovery of every feature needed for near-OOD classes, including pickup truck and bus versus existing automobile and truck classes.These class pairs are also among the most challenging for fully supervised training according to CLP.
- The work studies improvements from learning richer representations, while standard Gaussian density modeling remains the scoring method.The authors expect improved density estimation to complement contrastive training.
- The setup can use completely unlabelled images from arbitrary distributions, avoiding expensive manual removal of outlier images in applications such as medical imaging.Routine imaging can provide large collections of unlabelled images.
- The paper frames near-OOD detection as closely related to unsupervised representation learning, suggesting this perspective as a direction for further progress.
Broader Impact
The paper situates OOD detection as important for safe deployment in high-impact domains and evaluates contrastive training across near- and far-OOD regimes using standard benchmarks.
- Accurate OOD recognition is essential in medical imaging and self-driving cars to avoid catastrophic errors from unfamiliar inputs.The paper emphasizes withholding predictions or referring difficult cases for second opinions.
- The proposed approach is presented as scalable and based on representation learning, with evaluation spanning near and far OOD settings.The reported experiments use standard benchmark datasets.
- Figure 6 compares class-wise detection performance with and without contrastive training across CIFAR-100, SVHN, Places365, and Gaussian noise outliers relative to CIFAR-10.Classes are ordered by increasing similarity to inliers using CLP.
B Implementation Details
Experiments use a wide ResNet-50 trained first with contrastive learning and then with joint supervised-contrastive learning, using specified augmentation and optimization settings.
- The experiments use a ResNet-50 with a 3× width multiplier, contrastive pretraining, and subsequent joint supervised-contrastive finetuning.Pretraining lasts 1000 epochs; finetuning lasts 100 epochs for CIFAR-10 and 200 epochs for CIFAR-100.
- The finetuning supervised loss multiplier is λ = 100, with LARS optimization, momentum 0.9, and weight decay 1 × 10^-6.
- The data augmentation sequence applies random cropping, random horizontal flipping, and random color distortion.
C CLP Training Details
CLP is estimated from an ensemble trained across five datasets, then used to quantify dataset and class similarity through confusion probabilities and hierarchical clustering.
- CLP estimation uses an ensemble of five ResNet-34 models trained on the union of CIFAR-10, SVHN, CIFAR-100, Places365, and Gaussian noise.The combined dataset contains 486 classes.
- For Places365 relative to CIFAR-10, ensemble softmax outputs are averaged into a 486-class vector, and probability assigned to CIFAR-10 classes is logged as CLP.This estimates the probability of confusing a Places365 example with a CIFAR-10 example.
- The combined confusion matrix shows greater mutual confusion between CIFAR-10 and CIFAR-100 than with Places365, SVHN, or Gaussian noise.The latter datasets are described as far OOD, and the datasets have almost 100% classification accuracy.
- A 486-way classifier supplies expected class probabilities for pairwise class confusion analysis across the five datasets.
- Pairwise distances derived from confusion probabilities are clustered hierarchically, producing class relationships that match visual similarity.SVHN classes separate from CIFAR classes, while CIFAR-100 orange and apple cluster near house numbers.
- The agreement between clustering and visual similarity supports CLP as a metric for quantifying OOD-task difficulty.
E Additional Results
The paper reports additional OOD datasets and metrics, including Gaussian noise and Places365, while averaging proposed-method and baseline results over five runs.
- AUROC evaluates OOD detection across all thresholds by plotting true positive rate against false positive rate.
- AUPR summarizes the area under the precision-recall curve for OOD detection.
- FPR@95%TPR measures the probability that an OOD example is correctly identified when the true positive rate is 95%.
- Additional evaluations include Gaussian noise and Places365 as out-of-distribution datasets.Gaussian noise uses 32 × 32 random images; Places365 contains scene photographs resized to 32 × 32.
- Table 3 averages additional OOD detection results over five independent runs for the proposed method and a baseline without label smoothing or contrastive training.
F Additional Ablations
The ablations examine supervised-loss weighting, model capacity, and contrastive-loss temperature. Performance is sensitive to loss weighting and capacity, while higher temperature performs better across the tested values.
- Supervised loss multiplier: Increasing the supervised-loss multiplier particularly improves near-OOD detection for CIFAR-100 inliers and CIFAR-10 outliers, with marginal gains beyond λ = 100.The authors conjecture that large ratios are needed because supervised and contrastive losses differ in scale.
- Model capacity: Reducing ResNet-50 width to 1 significantly lowers performance on both near- and far-OOD pairs, whereas width 4 provides no improvement.The authors hypothesize that contrastive training requires higher-capacity models to capture richer task-agnostic representations.
- Contrastive-loss temperature: Higher contrastive-loss temperatures improve OOD detection across the tested settings, with optimal performance at τ = 1.The tested temperatures are 0.01, 0.1, 0.5, 1, and 2.