PaperlayerView exact revision on arXiv

Source-linked AI summary

VOS: Learning What You Don't Know by Virtual Outlier Synthesis

Xuefeng Du, Zhaoning Wang, Mu Cai, Yixuan Li

arXiv:2202.01197v4cs.LGcs.CV

TL;DR

OOD detectors lack supervision about unknowns and can become overconfident outside the training distribution. VOS synthesizes virtual outliers in feature space and uses them to shape uncertainty, achieving state-of-the-art object-detection results while preserving ID accuracy.

  • Problem

    OOD detectors lack explicit knowledge of unknowns during training, producing overconfident predictions that undermine reliability in unfamiliar or safety-critical settings.

  • Method

    VOS samples virtual outliers from low-likelihood regions of estimated class-conditional feature distributions and contrastively regularizes uncertainty between ID data and synthesized outliers.

  • Results

    VOS establishes state-of-the-art object-detection results, reducing FPR95 by up to 9.36% versus the best method while preserving ID-task accuracy.

  • Takeaways & Limitations

    VOS provides an unknown-aware OOD framework effective for both object detection and image classification without requiring real outlier data.

  • Takeaways & Limitations

    VOS assumes object-instance feature representations follow class-conditional multivariate Gaussian distributions.

Abstract

from arXiv · show

Out-of-distribution (OOD) detection has received much attention lately due to its importance in the safe deployment of neural networks. One of the key challenges is that models lack supervision signals from unknown data, and as a result, can produce overconfident predictions on OOD data. Previous approaches rely on real outlier datasets for model regularization, which can be costly and sometimes infeasible to obtain in practice. In this paper, we present VOS, a novel framework for OOD detection by adaptively synthesizing virtual outliers that can meaningfully regularize the model's decision boundary during training. Specifically, VOS samples virtual outliers from the low-likelihood region of the class-conditional distribution estimated in the feature space. Alongside, we introduce a novel unknown-aware training objective, which contrastively shapes the uncertainty space between the ID data and synthesized outlier data. VOS achieves competitive performance on both object detection and image classification models, reducing the FPR95 by up to 9.36% compared to the previous best method on object detectors. Code is available at https://github.com/deeplearning-wisc/vos.

1 INTRODUCTION

Neural networks trained only on in-distribution data can make overconfident predictions on unknown categories because their decision boundaries lack explicit knowledge of unknowns. VOS addresses this by synthesizing low-likelihood feature-space outliers and contrastively regularizing uncertainty while preserving the ID task.

  • Motivation: OOD inputs can receive high posterior probabilities because neural networks are optimized primarily on in-distribution data, leaving decision boundaries poorly suited to unknowns.This vulnerability is illustrated for both classification-style settings and object detection, where unknown objects should not be predicted.
  • VOS framework: VOS estimates class-conditional feature-space distributions and samples virtual outliers from low-likelihood ID regions to regularize the model’s decision boundary.The framework is designed to address both outlier synthesis and effective model regularization with synthesized outliers.
  • VOS framework: VOS jointly performs the ID task and OOD uncertainty regularization using a contrastive objective that shapes uncertainty between ID data and synthesized outliers.At inference, the uncertainty branch assigns larger probabilistic scores to ID data and smaller scores to OOD data.
  • Scope: VOS applies to both object detection and image classification, addressing the limitation of image-level OOD detection when only certain image regions are out-of-distribution.The paper positions object-detection OOD detection as timely and underexplored.
  • Results: 9.36%: VOS reduces FPR95 by up to 9.36% compared to the best method while preserving accuracy on the ID task.The paper also reports extensive ablations and evaluations on common OOD benchmarks and an underexplored object-detection task.

2 PROBLEM SETUP

The paper formulates OOD detection for object detectors at the object-region level, distinguishing in-distribution from out-of-distribution instances rather than treating entire images as anomalies. It defines the data, model outputs, and binary prediction objective for identifying whether a detected object lies within the known label support.

  • Motivation: The object-level formulation addresses ambiguity in whole-image anomaly detection because natural images contain numerous objects and components.Identifying anomalous regions can support safer handling of images.
  • Problem formulation: Object detection training uses in-distribution samples D = {(xi, bi, yi)}N drawn from an unknown joint distribution P.Inputs lie in X = Rd, bounding boxes in R4, and labels represent K-way classification.
  • Problem formulation: Neural networks with parameters θ model both bounding-box regression pθ(b|x) and classification pθ(y|x, b).These outputs define the object detector used before OOD status is predicted.
  • Object-level OOD objective: OOD detection is framed as binary classification between in-distribution and out-of-distribution objects.The target is pθ(g|x∗, b∗), where g = 1 denotes in-distribution and g = 0 denotes out-of-distribution.

3 METHOD

VOS synthesizes virtual outliers in a low-dimensional feature space and uses them in an unknown-aware objective to shape the model’s uncertainty boundary. During inference, an uncertainty score and threshold distinguish predicted in-distribution from out-of-distribution objects.

  • 3.1 Virtual Outlier Synthesis: VOS estimates class-conditional multivariate Gaussian distributions over penultimate-layer feature representations and samples virtual outliers from their low-likelihood regions.The feature dimension is smaller than the input dimension, and a small ϵ places samples near class boundaries.
  • 3.1 Virtual Outlier Synthesis: Online class-conditional queues maintain embeddings for efficient Gaussian parameter estimation during training.Each iteration enqueues embeddings into their corresponding class queues and dequeues the same number of instances.
  • 3.2 Unknown-Aware Learning: The unknown-aware objective trains the model to assign low OOD scores to ID data and high OOD scores to synthesized outliers.It uses an energy-based level-set formulation with ID data assigned negative energy and synthesized outliers positive energy.
  • 3.2 Unknown-Aware Learning: VOS uses a nonlinear MLP energy surface and a completely hyperparameter-free uncertainty regularization loss.The MLP enables a flexible energy surface, contrasting with prior squared-hinge regularization requiring two margin hyperparameters.
  • 3.2 Unknown-Aware Learning: The object-detection training objective combines standard classification and localization losses with uncertainty regularization weighted by β.For classification, the localization term is omitted.
  • 3.3 OOD Detection: At inference, VOS thresholds the uncertainty score from its logistic regression branch to distinguish ID and OOD predictions.The threshold γ is typically selected so that a high fraction of ID data, such as 95%, is correctly classified.

4 EXPERIMENTAL RESULTS

Experiments evaluate VOS for OOD detection in object detection and image classification using ID-only training data and multiple architectures and datasets. VOS consistently improves OOD detection, preserves or improves ID performance, and benefits from feature-space synthesis and its uncertainty loss.

  • Experimental scope: Experiments assess VOS on real-world object detection and image classification tasks.Object detection uses PASCAL VOC and BDD-100k as ID data, while image classification uses CIFAR-10.
  • Experimental setup: Evaluations use ID training data and OOD images from MS-COCO and OpenImages, with manual checks excluding ID categories.The benchmark data is open-sourced for future evaluation.
  • Object detection results: VOS outperforms competitive OOD detection baselines trained without auxiliary outlier datasets, highlighting the benefit of synthesized outliers for model regularization.Compared methods include Maximum Softmax Probability, ODIN, energy score, Mahalanobis distance, Generalized ODIN, CSI, and Gram matrices.
  • Object detection results: 12.76% and 13.40%: VOS improves OOD detection FPR95 over GAN-based synthesis on BDD-100k and PASCAL VOC, respectively, with COCO as OOD data.VOS also preserves high ID mAP, whereas CSI decreases BDD-100k mAP by 0.7%.
  • Ablation studies: VOS outperforms pixel-space and feature-space synthesis alternatives; GAN synthesis is unstable, mixup harms detection, and Gaussian noise is promising but relatively simple.The comparison covers GAN, mixup, noise, and negative RPN proposals, with virtual-outlier classification outputs encouraged toward a uniform distribution.
  • Architecture and layer analysis: RegNet provides better ID accuracy and OOD detection than the alternative architecture, while synthesizing outliers at the penultimate layer is most effective.The authors attribute the layer result to deeper features being more discriminative.

5 RELATED WORK

Related work spans post hoc and regularization-based OOD detection, including natural or GAN-synthesized outliers, while object-detection OOD detection remains underexplored. VOS instead synthesizes virtual outliers and introduces an unknown-aware framework applicable to both detectors and classifiers.

  • OOD Detection for Classification: Classification OOD detection broadly comprises post hoc methods and regularization-based approaches, including OpenMax and maximum softmax probability baselines.OpenMax uses extreme value theory, while maximum softmax probability provides a simple baseline.
  • OOD Detection for Classification: Regularization methods use natural outlier images or GAN-synthesized images, whereas VOS automatically synthesizes virtual outliers for greater flexibility and generality.The passage notes that real outlier data is often infeasible to obtain.
  • OOD Detection for Object Detection: OOD detection for object detection is underexplored; VOS differs from prior energy-based labeling by targeting OOD detection with an unknown-aware objective and test-time detection score.VOS’s framework applies to both object detectors and classification models.
  • Open-World Object Detection: Open-world object detection includes out-of-domain generalization, zero-shot detection, and incremental detection, which often address forgetting or use auxiliary information rather than VOS’s focus.The cited auxiliary information includes class attributes for detecting unseen data.

6 CONCLUSION

VOS is an unknown-aware OOD-detection framework that adaptively synthesizes virtual outliers from low-likelihood regions of class-conditional distributions, improving the ID–OOD decision boundary while preserving ID-task performance.

  • 6 CONCLUSION: VOS adaptively synthesizes virtual outliers during training instead of requiring real outlier data.It samples them from the low-likelihood region of class-conditional distributions.
  • 6 CONCLUSION: The synthesized outliers meaningfully improve the decision boundary between ID and OOD data while preserving ID-task performance.This leads to superior OOD detection performance.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility through public datasets, documented baselines and methodology, specified training details, and released code and data.

  • Datasets: Publicly available datasets are described in Sections 4.1 and 4.2 and Appendix A.
  • Baselines: OOD detection baseline descriptions and hyperparameters are provided in Appendix E.
  • Model training: Object-detection training uses the public Detectron2 codebase, with hyperparameters in Section 4.1 and ablations in Appendix C.
  • Methodology: The method is documented in Section 3, and its pseudocode is detailed in Algorithm 1.
  • Open Source: The authors will release the codebase and dataset, with code available at the VOS GitHub repository.

ETHICS STATEMENT

The project aims to improve machine-learning reliability and safety, with potential benefits for safety-critical applications such as autonomous driving. The authors report no human-subject involvement, legal-compliance violations, or anticipated harmful consequences.

  • ETHICS STATEMENT: The project targets more reliable and safer machine-learning models, particularly for safety-critical applications such as autonomous driving.The authors also hope that conducting the study and releasing its code will increase research and societal awareness of the problem.
  • ETHICS STATEMENT: The study involves no human subjects or legal-compliance violations, and the authors anticipate no potentially harmful consequences.

Supplementary Material · A EXPERIMENTAL DETAILS · B SOFTWARE AND HARDWARE

The supplementary material specifies the OOD evaluation setup, training schedules, uncertainty-regularization settings, and software/hardware environment. Evaluation tasks are summarized in Table 5, using OOD datasets with labels disjoint from the corresponding ID datasets.

  • A EXPERIMENTAL DETAILS: OOD test datasets are selected from MS-COCO and OpenImages with labels disjoint from their respective ID datasets.The dataset construction uses separate labels for OOD and ID data.
  • A EXPERIMENTAL DETAILS: 18,000 iterations are used to train the PASCAL model, while 90,000 iterations are used for the BDD-100k model.The two models therefore use different total training durations.
  • A EXPERIMENTAL DETAILS: The uncertainty regularizer is added starting from 2/3 of training, with β set to 0.1.The regularizer and its weight are specified as training hyperparameters.
  • A EXPERIMENTAL DETAILS: Detailed ablations on the hyperparameters are provided in Appendix C.The experimental-details passage directs readers to Appendix C for further hyperparameter analysis.
  • A EXPERIMENTAL DETAILS: OOD detection evaluation tasks are summarized in Table 5.The table is identified as the summary of the OOD detection evaluation tasks.
  • B SOFTWARE AND HARDWARE: All experiments use Python 3.8.5 and PyTorch 1.7.0.These versions define the software environment used for the experiments.
  • B SOFTWARE AND HARDWARE: Experiments run on NVIDIA GeForce RTX 2080Ti GPUs.The hardware environment consists of these GPUs.

C EFFECT OF HYPERPARAMETERS … I DISCUSSION ON THE DETECTED, REJECTED AND IGNORED OOD OBJECTS

The supplementary analyses examine VOS hyperparameters, visualizations, baseline evaluation, layer choice, learnable weighting, virtual-outlier geometry, and OOD detection outcomes. VOS benefits from mid-training regularization, penultimate-layer synthesis, near-boundary outliers, and improves detected and rejected OOD boxes relative to vanilla Faster-RCNN.

  • C EFFECT OF HYPERPARAMETERS: Uncertainty regularization works best when introduced mid-training; VOS uses Z = 12,000 during an 18,000-iteration PASCAL-VOC training schedule.Adding it too early produces insufficiently discriminative features for Gaussian distribution estimation.
  • D ADDITIONAL VISUALIZATION RESULTS: VOS visualizations show fewer false positives than vanilla Faster-RCNN on MS-COCO and OpenImages OOD images for Pascal VOC and BDD-100k models.Blue boxes are ID detections, while green boxes are OOD objects detected by VOS.
  • E BASELINES: Baseline comparisons apply MSP, ODIN, Generalized ODIN, Mahalanobis distance, CSI, energy score, and gram matrices to detector classification branches.ODIN uses T = 1000, while contrastive-learning loss weights are both 0.1.
  • F VIRTUAL OUTLIER SYNTHESIS USING EARLIER LAYER: Synthesizing virtual outliers in the penultimate layer outperforms synthesis in the earlier 1,024-dimensional layer for OOD detection.The main results use the penultimate layer, while Table 10 compares the two layer choices with COCO as OOD data.
  • G VISUALIZATION OF THE LEARNABLE WEIGHT COEFFICIENT w IN GENERALIZED ENERGY SCORE: Learned weight coefficients follow the number of training objects per class, supporting learnable weights over a constant all-ones vector.The visualization uses BDD-100k, RegNetX-4.0GF, and averages coefficients over three runs.
  • H VISUALIZATION OF THE VIRTUAL OUTLIERS: VOS virtual outliers occupy the near-boundary region of in-distribution feature clusters, helping learn a compact ID–OOD decision boundary.They are synthesized in low-dimensional feature space and therefore cannot be visualized directly in pixel space.
  • I DISCUSSION ON THE DETECTED, REJECTED AND IGNORED OOD OBJECTS: +25% detected OOD boxes and +63% rejected OOD samples are achieved by VOS versus vanilla Faster-RCNN under the same default thresholds on BDD100K.The comparison uses a ResNet backbone; VOS focuses on preventing confidently classifying detected OOD objects as in-distribution.
Loading 2202.01197v4…