Source-linked AI summary
Non-Parametric Outlier Synthesis
Leitian Tao, Xuefeng Du, Xiaojin Zhu, Yixuan Li
TL;DR
OOD detection lacks unknown-data supervision, while prior outlier synthesis can impose restrictive parametric assumptions. NPOS synthesizes artificial outliers non-parametrically around boundary ID embeddings and interprets the procedure as rejection sampling. It achieves superior OOD detection performance, including a reduction in FPR95 from 41.87% to 5.76% against a cited baseline.
Problem
OOD detection needs training signals for unknown data, while prior outlier synthesis can rely on restrictive parametric assumptions about ID embeddings.
Method
NPOS uses non-parametric nearest-neighbor density estimation to identify boundary ID embeddings, samples Gaussian-kernel outliers around them, and applies rejection sampling.
Results
NPOS achieves superior OOD detection performance, reducing FPR95 from 41.87% to 5.76% against the cited baseline.
Takeaways & Limitations
NPOS provides a flexible outlier-synthesis approach without distributional assumptions and is reported to scale to large datasets.
Takeaways & Limitations
A salient implementation challenge is computational efficiency when drawing samples uniformly from the relevant region.
Abstract
from arXiv · showhide
Out-of-distribution (OOD) detection is indispensable for safely deploying machine learning models in the wild. One of the key challenges is that models lack supervision signals from unknown data, and as a result, can produce overconfident predictions on OOD data. Recent work on outlier synthesis modeled the feature space as parametric Gaussian distribution, a strong and restrictive assumption that might not hold in reality. In this paper, we propose a novel framework, Non-Parametric Outlier Synthesis (NPOS), which generates artificial OOD training data and facilitates learning a reliable decision boundary between ID and OOD data. Importantly, our proposed synthesis approach does not make any distributional assumption on the ID embeddings, thereby offering strong flexibility and generality. We show that our synthesis approach can be mathematically interpreted as a rejection sampling framework. Extensive experiments show that NPOS can achieve superior OOD detection performance, outperforming the competitive rivals by a significant margin. Code is publicly available at https://github.com/deeplearning-wisc/npos.
1 INTRODUCTION
OOD detection is vital for reliable deployment, but training lacks unknown-data supervision and prior outlier synthesis can rely on restrictive distributional assumptions. NPOS addresses this by synthesizing outliers non-parametrically around boundary ID embeddings and achieves strong OOD detection results.
- OOD detection is necessary because deployed models encounter unknown samples beyond the training distribution.
- Prior outlier synthesis provides training OOD data but can require assumptions about the feature-space distribution.
- NPOS identifies boundary ID embeddings using non-parametric nearest-neighbor density estimation and synthesizes nearby artificial outliers with Gaussian kernels.
- Its uncertainty loss performs level-set estimation to separate ID embeddings from synthesized outliers and learn a compact decision boundary.
- 41.87% to 5.76%: NPOS reduces FPR95 against the cited pre-trained-model baseline, a direct 36.11% improvement.
- NPOS is presented as a rejection-sampling framework that offers stronger performance and generality than parametric VOS.
2 PRELIMINARIES
The preliminaries formalize supervised in-distribution classification and distinguish it from deployment in which unknown classes may appear. OOD detection is framed around identifying inputs outside the training distribution.
- The closed-world setup assumes training and test classes coincide, with labeled ID data drawn independently from a joint distribution.
- Empirical risk minimization defines the classifier by minimizing a loss over the labeled training set.
- OOD detection addresses deployment settings where samples from unknown classes outside the training label set may appear.
- The ID lower level set is estimated from the training dataset, with an example false-rate constraint of 0.05.
3 METHOD
NPOS jointly optimizes ID classification and OOD detection by synthesizing outliers without assuming a parametric ID embedding distribution. It uses boundary-focused feature-space sampling and end-to-end losses to learn compact ID representations and an ID/OOD decision boundary.
- The framework jointly optimizes ID classification and OOD detection, integrating distributional uncertainty into training.Rclosed classifies ID samples, while Ropen distinguishes ID from OOD and helps prevent overconfident predictions on unknown data.
- NPOS formalizes OOD detection as β-level-set estimation, with the empirical boundary defined by ˆPin(x) = β.The OOD and ID conditional distributions partition the embedding space, making the empirical level set the Bayes decision boundary under the stated setup.
- Rejection sampling draws feature-space candidates around ID embeddings and accepts them according to their OOD likelihood ratio relative to ˆPin.The procedure samples an index uniformly, draws v from N(h(xi), σ2I), and accepts it with probability Q(h(x)|OOD)MˆPin.
- Because direct sampling from ˆPin is inefficient, NPOS selects high-k-NN-distance ID embeddings as boundary samples and synthesizes outliers around them.Candidate outliers are filtered by retaining those with the highest k-NN distance, encouraging sufficient separation from ID data.
- The binary uncertainty loss trains on ID and synthesized outlier embeddings to estimate the level set, while the ID loss aligns samples with class prototypes.The two losses are trained end to end: compact, distinguishable ID representations support synthesis, and the uncertainty loss supports a compact ID/OOD boundary.
- At test time, the detector uses a prototype-based score threshold, classifying inputs with sufficiently high scores as ID and others as OOD.The threshold is chosen so that a high fraction of ID data, such as 95%, lies above it.
4 EXPERIMENTS
NPOS is evaluated across standard and large-scale OOD settings, with ablations testing its training design, synthesis hyperparameters, representations, and architecture. It consistently improves detection, including large gains over competitive baselines and applicability beyond pretrained initialization.
- Main results: NPOS reduces FPR95 from 41.87% to 5.76% against Fort et al. under the same CLIP-B/16 setup.The authors report a direct 36.11% improvement.
- Main results: NPOS outperforms VOS by 13.40% in FPR95, including after VOS is given the same classification loss and representation space.The comparison isolates the effect of non-parametric versus parametric outlier synthesis.
- Large-scale evaluation: On ImageNet-1k, NPOS reduces FPR95 from 67.31% to 37.93% under the same ViT-B/16 image encoder.The result is presented as evidence that NPOS scales to large datasets with complex visual diversity.
- Ablations: Replacing the proposed ID classification loss with cross-entropy yields 17.94% average FPR95 and 95.75% AUROC on ImageNet-100.The authors use the worsened detection performance to emphasize the role of compact ID representations.
- Ablations: NPOS remains effective with RN50x4, achieving 7.94 FPR95 versus 5.76 for CLIP-B/16, while larger capacity strengthens performance.The comparison uses ImageNet-100 as the ID dataset.
- Ablations: OOD performance improves with mild Ropen weighting, remains insensitive to k from 100 to 500, and is stable under moderate sampling variance.Excessive variance can cause synthesized outliers to overlap severely with ID data, degrading performance.
- Additional results: NPOS also works when trained from scratch, using EMA-updated class-conditional image embeddings instead of pretrained text embeddings.The from-scratch evaluation covers CIFAR-10, CIFAR-100, and ImageNet-100.
- Additional results: NPOS consistently improves over published baselines across six OOD datasets and beats VOS by 27.41% in FPR95 on CIFAR-100.The evaluation includes TEXTURES, SVHN, PLACES365, LSUN-RESIZE, LSUN-C, and ISUN.
5 RELATED WORK
Related work approaches OOD detection through scoring functions, training-time regularization, and large-scale or pretrained-model settings. NPOS is positioned as a distinct training and fine-tuning approach within this literature.
- Scoring-based detection: One line of prior work detects OOD inputs using confidence-, energy-, and distance-based scoring functions.The cited methods include both confidence-based and energy-based scores, as well as distance-based approaches.
- Training-time regularization: Another line of work uses training-time regularization to reduce confidence or increase energy on OOD data.These methods regularize model outputs or representations during training.
- Large-scale OOD detection: Large-scale OOD research addresses expanded semantic label spaces and leverages large pretrained or vision-language models for detection.The paper distinguishes NPOS from these prior directions as a new training or fine-tuning approach.
6 CONCLUSION
NPOS jointly addresses ID classification and OOD uncertainty estimation without imposing distributional assumptions on embeddings. It uses non-parametric outlier synthesis, formally interpreted as rejection sampling, and achieves competitive results across multiple settings.
- NPOS combines ID classification and OOD uncertainty estimation in one coherent framework.
- The framework synthesizes outliers without imposing distributional assumptions on ID embeddings.
- NPOS formally interprets non-parametric outlier synthesis as a rejection sampling framework.
- NPOS achieves competitive performance on challenging real-world OOD detection tasks with vision-language models and models trained from scratch.
REPRODUCIBILITY STATEMENT
The reproducibility statement identifies the resources and documentation supporting replication of the work. It covers datasets, baselines, methodology, hyperparameters, ablations, and public code, datasets, and checkpoints.
- The datasets used in the study are publicly available and documented in the paper and appendix.
- Descriptions and hyperparameters for OOD detection baselines are provided in the paper and appendix.
- The methodology, pseudo algorithm, hyperparameters, and ablation studies are documented across the main text and appendices.
- Code, datasets, and model checkpoints are publicly available.
A DETAILS OF DATASETS
The paper evaluates OOD detection using diverse datasets and a non-parametric training procedure. Its algorithm maintains class-conditional feature queues, selects boundary embeddings, synthesizes and filters outliers, and optimizes ID and OOD objectives.
- Datasets: ImageNet-100 is created by randomly sampling 100 classes from ImageNet-1k.
- Datasets: OOD evaluation uses datasets whose categories are disjoint from the corresponding ID classes, including iNaturalist, SUN, Places, and Texture subsets.
- Algorithm: The framework constructs and periodically updates a class-conditional queue of training-sample feature embeddings.
- Algorithm: NPOS selects high-distance boundary embeddings, synthesizes nearby outliers, accepts those with large k-NN distances, and optimizes level-set and ID embedding losses.
- Algorithm: At test time, NPOS computes an OOD score and performs detection by threshold comparison.
D EXPERIMENTAL DETAILS AND RESULTS ON TRAINING FROM SCRATCH
Experiments training from scratch evaluate NPOS on CIFAR-10, CIFAR-100, and ImageNet-100. NPOS substantially outperforms baselines and retains strong performance without relying on pretrained models.
- Experimental setup: NPOS is evaluated on CIFAR-10, CIFAR-100, and ImageNet-100 using training-from-scratch configurations.The section summarizes configurations in Table 7.
- CIFAR results: 10.16% average FPR95 on CIFAR-10 outperforms the best baseline VOS at 27.88%.
- CIFAR results: NPOS’s strong performance also holds on CIFAR-100.
- ImageNet-100 results: 18.96% lower FPR95 than the best baseline KNN is reported for ImageNet-100.
- Generalization: NPOS achieves strong OOD detection without necessarily relying on pretrained models, supporting generality across training from scratch and fine-tuning.
E ADDITIONAL EXPERIMENTS ON MODEL CALIBRATION AND DATA-SHIFT
NPOS preserves distribution-shift robustness relative to a vanilla cross-entropy classifier and maintains comparable, sometimes better, calibration while improving OOD uncertainty estimation.
- Data-shift robustness: NPOS does not substantially change distributional robustness compared with a vanilla classifier trained only with cross-entropy loss.The evaluation reports mean classification accuracy and standard deviation across shifted test data, with clean in-distribution accuracy shown separately.
- Model calibration: NPOS maintains overall comparable calibration performance, and in some cases improves it, while achieving much stronger OOD uncertainty estimation.Calibration is measured with Expected Calibration Error (ECE, in %) across datasets using five random seeds.
F ADDITIONAL ABLATIONS ON HYPERPARAMETERS AND DESIGNS
Ablations show that NPOS is generally robust to several hyperparameter choices, while performance depends on when regularization is introduced and how density estimation is implemented.
- Boundary samples: NPOS is not sensitive to the number of boundary samples varied from 100 to 400 per class.The ablation varies m across seven settings: 100, 150, 200, 250, 300, 350, and 400.
- Class-conditional queue: OOD detection performance is not sensitive to class-conditional ID queue sizes from 1000 to 3000, although larger queues can improve density estimation accuracy.The queue-size ablation evaluates five settings and recommends a sufficiently large queue for more accurate non-parametric density estimation.
- Candidate outliers: A reasonably large number of candidate outliers helps provide a meaningful selection pool when sampling around each ID boundary sample.The ablation varies the number of candidates per boundary sample from 600 to 1400 using the Gaussian kernel.
- Regularization timing: Adding Ropen(g) at the beginning of training performs slightly worse, whereas adding regularization in the middle of training yields more desirable performance.The authors suggest early representations may not yet be well formed.
- Density estimation design: NPOS synthesizes and rejects outliers using class-conditional k-NN density estimation rather than an alternative class-agnostic implementation.The class-conditional procedure identifies boundary ID samples and rejects synthesized outliers with lower k-NN distance.
G ADDITIONAL RESULTS ON THE MEAN AND STANDARD DEVIATIONS
Across five independent training runs on ImageNet-100 with pre-trained ViT-B/16, NPOS is relatively stable and significantly outperforms VOS.
- Repeated-run results: Across five runs, NPOS is relatively stable and outperforms VOS by a significant margin.The comparison reports means and standard deviations for both methods.
- Evaluation setting: The repeated-run evaluation uses ImageNet-100 with pre-trained ViT-B/16 and five independent training repetitions.The implementation uses Python 3.8.5, PyTorch 1.11.0, and eight NVIDIA GeForce RTX 2080Ti GPUs.