PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?

Yifei Ming, Yiyou Sun, Ousmane Dia, Yixuan Li

arXiv:2203.04450v3cs.CVcs.LG

TL;DR

OOD detection requires representations that separate unknown inputs from known classes, whereas off-the-shelf contrastive objectives may prioritize ID classification. CIDER learns hyperspherical embeddings with compactness and dispersion losses, and reports superior OOD detection performance while linking embedding properties to detection quality.

  • Problem

    Existing contrastive objectives can support ID classification but are not optimally designed for separating OOD inputs, motivating better representation learning for OOD detection.

  • Method

    CIDER jointly optimizes dispersion and compactness losses over hyperspherical embeddings to separate class prototypes and pull samples toward their prototypes.

  • Results

    CIDER outperforms competitive OOD detection methods across common benchmarks and improves ID-OOD separability by 42.36% relative to SupCon across five OOD datasets.

  • Takeaways & Limitations

    Hyperspherical embedding quality, especially inter-class dispersion and compactness, is related to OOD detection performance and can support multiple distance-based scores.

Abstract

from arXiv · show

Out-of-distribution (OOD) detection is a critical task for reliable machine learning. Recent advances in representation learning give rise to distance-based OOD detection, where testing samples are detected as OOD if they are relatively far away from the centroids or prototypes of in-distribution (ID) classes. However, prior methods directly take off-the-shelf contrastive losses that suffice for classifying ID samples, but are not optimally designed when test inputs contain OOD samples. In this work, we propose CIDER, a novel representation learning framework that exploits hyperspherical embeddings for OOD detection. CIDER jointly optimizes two losses to promote strong ID-OOD separability: a dispersion loss that promotes large angular distances among different class prototypes, and a compactness loss that encourages samples to be close to their class prototypes. We analyze and establish the unexplored relationship between OOD detection performance and the embedding properties in the hyperspherical space, and demonstrate the importance of dispersion and compactness. CIDER establishes superior performance, outperforming the latest rival by 19.36% in FPR95. Code is available at https://github.com/deeplearning-wisc/cider.

1 INTRODUCTION

OOD detection aims to identify unknown inputs reliably, but existing contrastive objectives can produce embeddings that classify ID samples without sufficiently separating OOD data. CIDER addresses this gap by jointly shaping hyperspherical embeddings for compactness and dispersion, improving OOD detection performance.

  • OOD detection determines whether a test input belongs to the known distribution or comes from an unknown distribution.
  • Distance-based methods use feature embeddings and assume OOD samples lie relatively far from ID data clusters.
  • Existing contrastive objectives can suffice for ID classification while remaining sub-optimal for OOD detection.
  • CIDER jointly optimizes dispersion and compactness losses to create hyperspherical embeddings with separated class prototypes and samples close to their prototypes.
  • 42.36% relative improvement in ID-OOD separability is achieved by CIDER compared to SupCon on CIFAR-10.
  • 13.33% error reduction in FPR95 is achieved by CIDER compared to the latest rival on CIFAR-100.
  • CIDER establishes the relationship between hyperspherical embedding quality and OOD detection and remains effective under varied settings, including ImageNet.

2 PRELIMINARIES

OOD detection is formulated as deciding whether a test sample belongs to the ID distribution or an OOD distribution. CIDER’s framework uses normalized hyperspherical embeddings and combines compactness with dispersion to shape the representation space.

  • OOD detection is treated as binary classification between samples from the ID distribution and samples from an OOD distribution.
  • An OOD score classifies samples as ID when their score exceeds a threshold selected to retain a high fraction of correctly classified ID data.
  • Hyperspherical embeddings lie on the surface of a unit hypersphere, whose points have constant Euclidean norm.
  • CIDER’s compactness and dispersion terms encourage samples to remain close to class prototypes while different prototypes become more angularly separated.

3 METHOD

CIDER learns hyperspherical embeddings by modeling classes with von Mises–Fisher distributions and jointly optimizing compactness and dispersion. The framework uses these complementary properties to align samples with their class prototypes while separating class prototypes angularly for OOD detection.

  • Framework overview: CIDER uses an encoder and projection head to produce normalized hyperspherical feature embeddings for its training objective.The encoder maps augmented inputs to high-dimensional features, while the projection head produces lower-dimensional representations that are normalized before applying the loss.
  • Hyperspherical modeling: The von Mises–Fisher model represents each class with a unit-norm prototype, while κ controls concentration around its mean direction.Larger κ produces stronger concentration; κ = 0 gives a uniform distribution on the hypersphere.
  • Compactness loss: The compactness loss encourages each sample embedding to align closely with its class prototype and assigns higher probability to the correct class than to incorrect classes.The loss follows from the negative log-likelihood of the class-conditional vMF model and uses the sample’s class index and a temperature parameter.
  • Dispersion loss: The dispersion loss maximizes angular distances among different class prototypes, because OOD samples may lie between ID clusters.Larger angular separation between ID prototypes can improve the separability of OOD samples that fall between those clusters.
  • Combined objective: CIDER combines dispersion and compactness as LCIDER = Ldis + λcLcomp to shape embeddings for both ID classification and OOD detection.The coefficient λc controls the relative importance of the two loss terms, which are described as complementary.
  • Prototype estimation and update: CIDER updates class prototypes with an exponential-moving-average rule rather than repeatedly computing means over all training samples.The update uses factor α to combine the previous prototype with the normalized embedding of samples from the class.

4 EXPERIMENTS

Experiments evaluate CIDER with distance-based OOD scores on CIFAR and ImageNet benchmarks, then analyze how its loss components shape hyperspherical embeddings. Across these evaluations, CIDER improves OOD detection and produces more compact, dispersed, and separable representations.

  • Experimental setup: CIFAR-100 experiments use ResNet-34, with CIFAR-100 as ID and SVHN, Places365, Textures, LSUN, and iSUN as OOD datasets.The evaluation reports FPR95, AUROC, and ID classification accuracy.
  • Main results: CIDER improves OOD detection over competitive methods, reducing FPR95 by 20.3% versus SSD+ and 13.3% versus KNN+.It also reduces average FPR95 by 20.6% compared with CSI.
  • OOD score ablation: CIDER improves performance with both non-parametric KNN and parametric Mahalanobis scores.CIDER with KNN reduces FPR95 by 13.33% versus SupCon+KNN, while CIDER+Maha reduces FPR95 by 22.77% versus SSD+.
  • Loss ablation: The compactness loss alone supports ID classification, while inter-class dispersion from the dispersion loss is key to OOD detection.The loss components work synergistically to improve hyperspherical embeddings for both tasks.
  • Embedding analysis: CIDER embeddings are more compact and distribute classes more uniformly than CE embeddings in UMAP visualizations.The visualization links improved compactness and dispersion with greater OOD separability.
  • Embedding analysis: CIDER improves inter-class dispersion by 12.03 degrees over SSD+ and yields a relative 42.36% improvement in ID-OOD separability over SupCon across five OOD datasets.Higher separability indicates that OOD samples are easier to detect.
  • Additional analysis: CIDER remains competitive on large-scale ImageNet-100 evaluations and across ablations involving training and model settings.The ImageNet-100 study fine-tunes a pre-trained ResNet-34, while additional experiments vary batch size and architecture.

5 RELATED WORKS

Related work spans OOD detection, contrastive representation learning, and deep metric learning. Prior representation-learning approaches often adapt general-purpose contrastive losses for OOD detection, while CIDER focuses on hyperspherical embeddings and prototype dispersion.

  • Contrastive representation learning: Contrastive representation learning aligns positive samples while dispersing negative samples across supervised, semi-supervised, and unsupervised settings.
  • Representation learning for OOD detection: Prior OOD studies apply off-the-shelf losses such as SupCon and SimCLR, whereas CIDER focuses on hyperspherical embeddings.
  • Deep metric learning: Deep metric learning develops embeddings for verification, re-identification, and retrieval, but prior work does not focus on embeddings designed for OOD detection.

6 CONCLUSION AND OUTLOOK

CIDER uses hyperspherical embeddings and jointly optimizes compactness and dispersion to promote ID-OOD separability. The framework also introduces embedding-quality measurements and reports strong performance across OOD benchmarks.

  • CIDER jointly optimizes dispersion and compactness losses to promote strong ID-OOD separability.
  • CIDER achieves superior performance on common OOD benchmarks, including large-scale OOD detection tasks.
  • The work introduces measurements linking hyperspherical embedding quality with OOD detection performance and evaluates CIDER through extensive ablations.

A ALGORITHM DETAILS AND DISCUSSIONS

CIDER trains normalized hyperspherical embeddings with compactness and dispersion objectives, while prototypes are updated using exponential moving averages. The framework distinguishes prototype dispersion from instance-level contrastive uniformity and from cross-entropy training.

  • CIDER training framework: CIDER jointly optimizes compactness for sample-prototype alignment and dispersion for larger angular distances among class prototypes.
  • Prototype updates: Class prototypes are updated by exponential moving average rather than receiving direct gradients from the dispersion loss.
  • Relation to contrastive losses: CIDER promotes prototype-to-prototype dispersion, unlike uniformity losses that promote instance-to-instance uniformity from randomly sampled pairs.
  • Relation to cross-entropy: CIDER is designed for hyperspherical embeddings, whereas cross-entropy does not explicitly optimize intra-class compactness and inter-class dispersion in feature space.
  • Embedding measurements: Embedding quality is measured through inter-class dispersion and intra-class compactness using class prototypes or representative samples.

B EXPERIMENTAL DETAILS

The experiments use specified hardware, projection-head architectures, training schedules, and KNN-based OOD scores. Settings vary across CIFAR-10, CIFAR-100, model scale, and batch size.

  • Software and hardware: Experiments run on RTX-2080Ti GPUs for small or medium settings and A100 GPUs for large batches or larger encoders.
  • Architecture: CIDER uses a projection head with output dimension 128, with a two-layer nonlinear head for CIFAR-10 and CIFAR-100.
  • Training: CIFAR-10 models train for 100 epochs, while CIFAR-100 models train for 200 epochs under scheduled learning-rate decay.
  • OOD detection score: The default OOD score is non-parametric KNN, using K = 100 for CIFAR-10 and K = 300 for CIFAR-100.

C ADDITIONAL ABLATION STUDIES

Additional ablations show that CIDER remains effective across batch sizes, prototype-update choices, hyperparameters, and architectures. EMA prototypes provide stronger stability, while moderate loss-weight changes preserve performance.

  • Batch size: CIDER outperforms SupCon across different batch sizes, aided by instance-to-prototype compactness and class-conditional prototypes that reduce batch-size dependence.The prototype updates capture class-average statistics and provide a memory-efficient alternative to instance-to-instance distance measurement.
  • Loss weights: CIDER performance is relatively stable for λc from 0.5 to 2, with best performance around λc ∈[1, 2].This reduces the need for extensive tuning of the loss scale.
  • Prototype updates: For CIFAR-100, α = 0.5 achieves average FPR95 of 46.89% under KNN, while larger α values improve CIFAR-10 performance.Detection remains competitive across a wide range of prototype moving-average discount factors.
  • Optimization: A relatively high initial learning rate is preferable, whereas a small temperature τ produces better OOD detection performance.Too small an initial learning rate, such as 0.1, degrades performance.
  • Architecture: CIDER improves FPR95 across various test sets with ResNet-50, following the trend observed with ResNet-34.The architecture ablation attributes the improvement to the learned representation.
  • Prototype update rules: CIDER with EMA improves average FPR95 by 5.08% and has smaller standard deviation than learnable prototypes, indicating stronger training stability.The comparison averages three independent CIFAR-10 runs.
  • Stability: Across three independent seeds, CIDER yields consistently strong OOD detection performance on CIFAR-10 and CIFAR-100.The reported stability uses ResNet-18 for CIFAR-10 and ResNet-34 for CIFAR-100.

D RESULTS ON LARGE-SCALE DATASETS

CIDER is evaluated when fine-tuning pre-trained models on ImageNet-100 and remains competitive with SupCon. Using the same KNN detection score, it outperforms SupCon across all tested OOD datasets.

  • Fine-tuning setup: CIDER and SupCon are compared by fine-tuning ImageNet-pre-trained ResNet-34 on ImageNet-100 for 10 epochs.The final residual block and nonlinear projection head are updated while earlier residual blocks are frozen.
  • Architecture context: CIDER’s architecture ablation on ResNet-50 reports results based on that network.The supplied architecture caption identifies the evaluation backbone but does not state a numerical comparison.
  • Results: With the same KNN detection score, CIDER consistently outperforms SupCon across all OOD test datasets.The comparison is reported for ImageNet-100 ID data.

E RESULTS ON CIFAR-10

The CIFAR-10 evaluation uses matched encoder and embedding settings for contrastive methods while varying the training objective, with Mahalanobis scoring used for several methods including CIDER.

  • Evaluation setup: For contrastive-loss methods, CIFAR-10 uses the same network encoder and embedding dimension while varying only the training objective.The evaluation uses Mahalanobis scoring for SSD+, CE+SimCLR, SupCon, and CIDER.

F ID CLASSIFICATION ACCURACY

ID classification accuracy is reported separately for CIFAR-10 and CIFAR-100, with contrastive methods evaluated using linear probes on normalized features. The supplied passages identify the relevant tables but provide no accuracy values.

  • Evaluation protocol: For contrastive methods including KNN+, SSD+, and CIDER, ID classification uses linear probes on normalized features.This follows the common practice used for contrastive representation learning.
  • CIFAR-10 results: CIDER is described as achieving strong OOD detection performance and ID classification accuracy on CIFAR-10.The supplied table caption does not provide the corresponding numerical values.
  • CIFAR-10: Table 8 reports ID classification accuracy on CIFAR-10.The supplied table passage provides the table’s topic but no numerical values.
  • CIFAR-100: Table 9 reports ID classification accuracy on CIFAR-100.The supplied table passage provides the table’s topic but no numerical values.
Loading 2203.04450v3…