PaperlayerView exact revision on arXiv

Source-linked AI summary

MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space

Rui Huang, Yixuan Li

arXiv:2105.01879v1cs.CVcs.LG

TL;DR

OOD detection has limited evidence for large-scale image classification despite its importance for real-world deployment. The paper groups related semantic classes and introduces MOS, achieving state-of-the-art detection with lower FPR95 and faster inference than the previous best method. Experiments on ImageNet models and four diverse OOD datasets show improved scaling to large label spaces.

  • Problem

    OOD detection remains largely unexplored for large-scale image classification because existing solutions are mainly driven by small, low-resolution datasets with few classes.

  • Method

    The framework decomposes the semantic space into groups, applies group softmax with an others category, and uses the Minimum Others Score (MOS) for OOD detection.

  • Results

    14.33% lower average FPR95 and 6x speedup in inference are reported versus the previous best method across four diverse OOD datasets.

  • Takeaways & Limitations

    MOS effectively scales OOD detection to large label spaces while preserving competitive in-distribution classification accuracy.

Abstract

from arXiv · show

Detecting out-of-distribution (OOD) inputs is a central challenge for safely deploying machine learning models in the real world. Existing solutions are mainly driven by small datasets, with low resolution and very few class labels (e.g., CIFAR). As a result, OOD detection for large-scale image classification tasks remains largely unexplored. In this paper, we bridge this critical gap by proposing a group-based OOD detection framework, along with a novel OOD scoring function termed MOS. Our key idea is to decompose the large semantic space into smaller groups with similar concepts, which allows simplifying the decision boundaries between in- vs. out-of-distribution data for effective OOD detection. Our method scales substantially better for high-dimensional class space than previous approaches. We evaluate models trained on ImageNet against four carefully curated OOD datasets, spanning diverse semantics. MOS establishes state-of-the-art performance, reducing the average FPR95 by 14.33% while achieving 6x speedup in inference compared to the previous best method.

1. Introduction

OOD detection remains underexplored for large-scale, high-resolution image classification, where increasing semantic classes expand uncertainty. The paper proposes grouping related concepts and using MOS to simplify in-versus-out decision boundaries.

  • Motivation: Existing OOD solutions mainly use small, low-resolution datasets, leaving large-scale image classification insufficiently studied.The paper highlights the mismatch with deployed systems such as autonomous vehicles, which process higher-resolution images and many more categories.
  • Motivation: 76.94% FPR95 is reached at 1,000 ImageNet-1k classes versus 17.34% at 50 classes for a common baseline.This analysis demonstrates rapidly worsening OOD detection as the semantic label space expands.
  • Approach: The proposed framework decomposes the semantic space into smaller groups with similar concepts to simplify in-versus-out decision boundaries.Coarser group membership is treated as easier to estimate than fine-grained class membership.
  • Approach: MOS uses group softmax and an others category in every group to score whether an input is outside each group.OOD inputs receive high others scores across groups, while in-distribution inputs receive a lower score in their relevant group.
  • Results: 14.33% lower average FPR95 and 6x faster inference are reported versus the previous best method.The method is presented as scaling substantially better for large label spaces while preserving competitive in-distribution classification accuracy.

2. Preliminary and Analysis

The preliminary analysis defines OOD detection as distinguishing samples from a different distribution and examines how label-space size affects baseline performance. MSP performance deteriorates rapidly as the number of ImageNet-1k classes grows.

  • Preliminaries: OOD detection evaluates a classifier trained on in-distribution samples against samples drawn from a different distribution.The detector is formulated as a binary classifier using a scoring function and threshold.
  • Preliminaries: The detector threshold γ is chosen so that a high fraction, such as 95%, of in-distribution data is correctly classified.The scoring function S(x) determines the binary OOD decision relative to this threshold.
  • Baseline: MSP scores an input using the maximum softmax probability from a flat-softmax classifier.The analysis revisits MSP as a common baseline for studying label-space effects.
  • Analysis: 17.34% to 76.94% FPR95 occurs as ImageNet-1k in-distribution classes increase from 50 to 1,000.The same degradation is presented in Figure 2 as performance declining with increasing class count.

3. Method

The method decomposes a large semantic space into groups, adding group-level “others” categories to simplify OOD boundaries and derive MOS. It supports taxonomy, feature-clustering, or random grouping strategies for large-scale classification.

  • Method Overview: Grouping decomposes the label space into smaller semantic groups, simplifying OOD decision boundaries and reducing uncertainty.An input is classified as OOD when it belongs to none of the groups.
  • Group-based Learning: Group-wise softmax computes class probabilities within each group, using logits and softmax probabilities for the group’s categories.
  • Category “Others”: Each group adds an “others” category that models whether an input lies outside that group and creates virtual group-level outlier data without external data.Examples belonging to other groups are assigned the others label during training.
  • OOD Detection with MOS: MOS uses the lowest others score across groups because OOD inputs score highly on others in every group, while in-distribution inputs score lowly in their belonging group.The score sign is negated so higher values indicate in-distribution data.
  • OOD Detection with MOS: For animal-group ImageNet validation images, the minimum others score is significantly lower for in-distribution data than for OOD data.This separation supports distinguishing the two distributions.
  • Grouping Strategies: Grouping can use ImageNet taxonomy, feature clustering, or random assignment, with random grouping serving as a lower-bound comparison.Taxonomy uses eight ImageNet super-classes; feature clustering applies K-Means to class-average embeddings.

4. Experiments

The experiments evaluate MOS on ImageNet-1k using diverse, non-overlapping OOD datasets and compare it with established baselines and design variants. MOS improves OOD detection, remains more stable as the semantic class space grows, and benefits from stronger grouping and feature representations.

  • Evaluation Setup: The evaluation uses ImageNet-1k models and four diverse OOD datasets spanning fine-grained, scene, and textural images.The OOD benchmarks are curated to avoid concept overlap with ImageNet-1k.
  • MOS vs. Existing Methods: 31.06% lower FPR95 than Energy is achieved by MOS under the same BiT-S-R101x1 backbone while retaining 75.16% in-distribution accuracy versus 75.20% for flat softmax.The comparison covers MSP, ODIN, Mahalanobis, and Energy baselines.
  • MOS vs. Existing Methods: 14.33% lower FPR95 and 6x faster inference are achieved by MOS compared with KL matching.KL matching requires divergence calculations to all in-distribution class centers, causing runtime to grow linearly with the number of categories.
  • Increasing Numbers of Classes: As classes increase from 50 to 1,000, MSP FPR95 on iNaturalist rises from 21.02% to 63.36%, while MOS degrades by only 4.76%.MOS is also less sensitive than MSP when the total number of training images is fixed.
  • Grouping Strategies: Feature clustering outperforms MSP by 18.2% average FPR95 while maintaining classification accuracy within 0.16% of taxonomy-based grouping.Random grouping exceeds MSP on three of four OOD datasets but reduces in-distribution accuracy by 0.98% relative to taxonomy-based grouping.
  • Model Capacity: Larger pre-trained feature extractors improve both OOD detection and classification, while MOS consistently outperforms MSP across extractor capacities and fine-tuning settings.The extractor ablation fine-tunes only the final fully connected layer; the fine-tuning ablation varies the number of residual blocks.

5. Related Work

Prior OOD research includes softmax-based scores, auxiliary-outlier training, generative modeling, and large-scale classification approaches. Related work also uses hierarchical or grouped labels for efficient and improved visual recognition.

  • OOD Detection with Pre-trained Models: Common OOD baselines include MSP, ODIN, deep ensembles, Mahalanobis distance, generalized ODIN, energy scores, and dynamic inference.These methods primarily estimate uncertainty or confidence from models trained on in-distribution data.
  • OOD Detection with Model Fine-tuning: Auxiliary-outlier methods regularize predictions using realistic or synthetic outlier images and objectives such as uniform probabilities, background classes, or higher energies.The auxiliary data may be generated by GANs or drawn from realistic image datasets.
  • Generative Modeling Based OOD Detection: Generative OOD detectors use input density, but deep generative models can assign high likelihood to OOD data, motivating likelihood-based corrective strategies.Proposed mitigations include improved metrics and likelihood ratios.
  • OOD Detection for Large-scale Classification: Large-scale OOD studies include one-vs-rest training with background regularization and KL matching as an OOD scoring function.The one-vs-rest approach requires auxiliary data, while KL matching is identified as a baseline for large-scale image classification.
  • Learning with Hierarchical Labels: Hierarchical and group-based labels have been used for efficient inference, improved classification accuracy, stronger detection, and learned label-tree structures.Group-based learning is established across multiple image-classification settings.

6. Conclusion

The paper presents MOS as a group-based framework for scaling OOD detection to real-world classification with large label spaces. It contributes diverse evaluation datasets and reports improved performance over existing approaches.

  • Conclusion: MOS combines group-based OOD detection with a novel scoring function to scale detection to large label spaces.The framework is intended for real-world image-classification settings.
  • Conclusion: Four diverse OOD evaluation datasets support large-scale benchmarking and future evaluation of OOD detection methods.The paper aims to broaden OOD detection research beyond small benchmarks.
  • Conclusion: Extensive experiments report significantly improved OOD detection performance for the group-based framework compared with existing approaches.The conclusion frames the improvement specifically within the large-scale setting.

A. Selected Categories in OOD Datasets

The paper constructs four non-overlapping OOD benchmarks from iNaturalist, SUN, Places365, and Textures. Their selected categories span fine-grained biological classes, scenes, and textural patterns.

  • Dataset Selection: The four benchmarks are curated across diverse domains and selected to avoid concept overlap with ImageNet-1k.The datasets are iNaturalist, SUN, Places365, and Textures.

B.1. MOS with Increasing Numbers of Classes (A More Challenging Setting)

With a fixed total of 35,000 training images, increasing the number of ImageNet-1k classes creates a more challenging OOD setting because each class has fewer images. MOS is substantially less sensitive to this class-space increase than MSP, while taxonomy-based grouping performs best near eight groups.

  • Experimental setting: 35,000 total training images are fixed while the number of ImageNet-1k classes increases from 50 to 1,000, reducing images per class.This setting is more challenging than fixing the number of images per class.
  • Class-space sensitivity: 56.74%: MSP FPR95 degradation on iNaturalist as classes increase from 50 to 1,000, compared with 42.34% in the previous setting.
  • Class-space sensitivity: MOS is much less sensitive than MSP to increasing in-distribution classes; on Places, MSP drops from 14.45% to 88.02% FPR95, while MOS degrades by only 38.73%.
  • Grouping strategy: Taxonomy grouping varies across levels by merging or splitting super-classes along the WordNet label hierarchy, with Level 0 representing eight super-classes.Feature clustering and random grouping use matching group counts for fair comparison.
  • Grouping strategy: Taxonomy-based grouping is almost optimal at eight groups, and increasing or decreasing the group count does not improve performance.It outperforms feature clustering and random grouping when K is small and mildly large, while feature clustering surpasses it at sufficiently large K.
  • AUROC comparison: MOS and MSP show large AUROC gaps across all four OOD datasets, including an iNaturalist FPR reduction from 63.69% to 9.28% at 95% TPR.

D. AUPR Results

AUPR results complement AUROC and FPR95, particularly for class-imbalanced OOD detection. MOS again achieves state-of-the-art AUPR performance.

  • AUPR complements AUROC and FPR95 as an informative metric when OOD detection involves class imbalance.
  • MOS demonstrates state-of-the-art performance in terms of AUPR.

E. Others Scores for All In-distribution Groups and OOD Datasets

Average others scores separate in-distribution from OOD images across the eight groups. In-distribution images have a low score in at least one group, whereas OOD images have high scores in all groups.

  • Average others scores are computed separately for eight ImageNet-1k in-distribution groups and four curated OOD datasets.In-distribution scores average validation images; OOD scores average sampled curated-dataset images.
  • In-distribution images have low others scores in at least one group, while OOD images have high others scores across all eight groups.This score pattern provides visual justification for MOS's in- versus out-of-distribution discrimination.
Loading 2105.01879v1…