PaperlayerView exact revision on arXiv

Source-linked AI summary

ViM: Out-Of-Distribution with Virtual-logit Matching

Haoqi Wang, Zhizhong Li, Litong Feng, Wayne Zhang

arXiv:2203.10807v1cs.CV

TL;DR

Existing OOD detectors often rely on a single information source, although OOD examples can be easier to identify in feature space than logit space, or vice versa. ViM combines feature-space residual information with class-dependent logits through a virtual OOD logit, and extensive evaluations show stable performance across model architectures and OOD datasets, including 90.91% average AUROC with BiT versus 86.62% for the best baseline.

  • Problem

    Existing OOD detection methods often rely on a single input source, although different OOD examples can be distinguishable in feature space but not logit space, or vice versa.

  • Method

    ViM constructs a virtual OOD logit from a feature's residual against the principal subspace, matches it to the original logits by rescaling, and uses its softmax probability as the OOD score.

  • Results

    90.91% average AUROC with BiT across four OOD datasets surpassed the best baseline's 86.62%, while ViM remained stable across evaluated model architectures and dataset types.

  • Takeaways & Limitations

    ViM provides a lightweight OOD detector that fuses class-agnostic feature information with class-dependent logit information without extra OOD data or retraining.

  • Takeaways & Limitations

    ViM shows smaller gains on datasets with small residuals and is limited by the feature quality of the original network because it does not require training.

Abstract

from arXiv · show

Most of the existing Out-Of-Distribution (OOD) detection algorithms depend on single input source: the feature, the logit, or the softmax probability. However, the immense diversity of the OOD examples makes such methods fragile. There are OOD samples that are easy to identify in the feature space while hard to distinguish in the logit space and vice versa. Motivated by this observation, we propose a novel OOD scoring method named Virtual-logit Matching (ViM), which combines the class-agnostic score from feature space and the In-Distribution (ID) class-dependent logits. Specifically, an additional logit representing the virtual OOD class is generated from the residual of the feature against the principal space, and then matched with the original logits by a constant scaling. The probability of this virtual logit after softmax is the indicator of OOD-ness. To facilitate the evaluation of large-scale OOD detection in academia, we create a new OOD dataset for ImageNet-1K, which is human-annotated and is 8.8x the size of existing datasets. We conducted extensive experiments, including CNNs and vision transformers, to demonstrate the effectiveness of the proposed ViM score. In particular, using the BiT-S model, our method gets an average AUROC 90.91% on four difficult OOD benchmarks, which is 4% ahead of the best baseline. Code and dataset are available at https://github.com/haoqiwang/vim.

1. Introduction

OOD detection addresses confident misclassification of inputs outside a model’s training distribution. ViM combines feature-space and logit information, alongside the human-annotated OpenImage-O benchmark, to improve broad evaluation.

  • Motivation: OOD detection determines whether an input belongs to the training distribution, which matters when closed-world classifiers encounter open-world inputs in production.Such failures can deteriorate customer experience through confident misclassification.
  • Motivation: Existing scores use probability, logit, or feature information, but single-source methods can lose complementary information.Feature-only scores disregard class-dependent classification weights, while logit- and probability-based scores may miss feature-space variation.
  • Method: ViM creates a virtual OOD-class logit from the feature residual against a principal subspace and combines it with the original logits through scaling.The softmax probability of the virtual class serves as the OOD score.
  • Results: 90.91% average AUROC was achieved by ViM with BiT across four OOD datasets, versus 86.62% for the best baseline.The evaluation spans OpenImage-O, ImageNet-O, Texture, and iNaturalist, with model architectures including CNNs and vision transformers.
  • Dataset: OpenImage-O is a diverse, complex-scene OOD dataset created to facilitate large-scale ImageNet-1K detection research.The dataset addresses concerns about biased or noisy datasets selected from predefined public-dataset tags.

2. Related Work

Prior OOD work either designs scores from model outputs or changes network training using regularization and additional OOD data. ViM instead combines feature-based and logit-based information without retraining or extra OOD data.

  • OOD/ID Score Design: Output-based OOD scores derive from softmax probabilities, logits, or features and can be computed from existing models without retraining.Examples include MSP, ODIN, MaxLogit, energy, Mahalanobis, and residual-based scores.
  • OOD/ID Score Design: ViM combines the strengths of feature-based and logit-based scores through a virtual-logit mechanism.This addresses the loss of class-dependent information in feature-only approaches.
  • Network/Loss Design: Training-based methods redesign losses or add regularization to shape ID and OOD representations.These approaches require retraining the network.
  • OOD Data Exposure: OOD-exposure methods use auxiliary, generated, or external unlabeled data to improve detection.ViM differs by not requiring additional OOD data.

3. Motivation: The Missing Info in Logits

Feature-, logit-, and probability-based detectors capture different information, creating complementary strengths across OOD datasets. This motivates ViM’s fusion of class-agnostic feature residuals with class-dependent logits.

  • Missing Information: Feature-based scores can outperform logit- and probability-based scores when OOD examples differ in feature space but remain difficult to distinguish through logits.On ImageNet-O and Texture, feature-based methods show stronger detection than output-based alternatives.
  • Missing Information: Logits are transformed from features through the final fully connected layer, so projection can discard deviations that remain visible in the full feature space.The feature decomposes into components in the classifier’s column space and null space; the null-space component does not affect classification.
  • Feature-Space Scores: NuSA uses the feature angle relative to the classifier weight space, while Residual measures deviation from the principal space.NuSA achieves 95.50% AUROC on Texture, and Residual improves over NuSA on both Figure 2 datasets.
  • Fusion Motivation: Feature-only scores remain class-agnostic, ignoring similarity between a sample’s features and individual ID classes.The paper therefore hypothesizes that unifying feature-space and logit information can improve detection across broader OOD types.

4. Virtual-logit Matching

ViM unifies class-agnostic residual information from feature space with class-dependent logits by constructing a scaled virtual OOD logit and applying softmax. Its score uses the virtual-logit probability, with residual magnitude increasing OOD-ness and larger original logits decreasing it.

  • Pipeline: ViM extracts the feature residual against a principal subspace, rescales its norm into a virtual logit, and returns that logit’s softmax probability as the OOD score.The pipeline operates at the feature, logit, and probability levels.
  • Principal Subspace and Residual: The principal subspace is defined from training features, and the residual is the projection onto its orthogonal complement.The residual is represented using the complement of the selected principal subspace.
  • Virtual-logit Matching: The per-model constant α matches the virtual-logit scale to the mean maximum original logit on training examples.This prevents a small residual from being buried by the original logits after softmax.
  • The ViM Score: The virtual logit is appended to the original logits, and its softmax probability defines the ViM score.An input is classified as OOD when the score exceeds threshold τ.
  • Interpretation: ViM combines class-agnostic feature information with class-dependent logits, while adding computational overhead comparable to the final fully connected layer.The score can also be viewed as augmenting the energy method with residual information.

5. OpenImage-O Dataset

OpenImage-O is a large, manually annotated ImageNet-1K OOD dataset designed to provide diverse, image-level evaluation. Its construction addresses noise from class-level selection and the limited coverage that can make benchmarks easier to improve.

  • Dataset Overview: OpenImage-O contains 17,632 manually annotated images selected from the OpenImage-V3 test set for ImageNet-1K OOD evaluation.The source test set contains 125,436 Flickr images collected without predefined class names or tags.
  • Motivation: Human image-level annotation helps avoid noisy OOD benchmarks in which selected images remain indistinguishable from in-distribution samples.The paper contrasts this with selection based solely on dataset class labels.
  • Benchmark Coverage: The BiT benchmark shows AUROC gaps of 5.61 for OpenImage-O, 6.06 for iNaturalist, 10.52 for Texture, and 14.39 for ImageNet-O.Larger gaps indicate greater ease of improvement for a dataset.
  • Construction Process: OpenImage-O uses human decisions against an ImageNet-1K model’s top-10 predicted categories, with independent labels from at least two annotators and consensus filtering.Labelers could mark uncertain cases as Difficult, and random inspection further supported annotation quality.

6. Experiment

Experiments on ImageNet-1K evaluate ViM across diverse OOD datasets, model architectures, baselines, and hyperparameter settings. ViM performs strongly and robustly, while its limitations and computational trade-offs depend on dataset residuals and model features.

  • Experimental Setup: ViM is evaluated on ImageNet-1K using four OOD datasets and both CNN-based and transformer-based models.The OOD datasets are OpenImage-O, Texture, iNaturalist, and ImageNet-O; models include BiT and ViT among others.
  • Results on BiT: 90.91% average AUROC is achieved by ViM with BiT, surpassing the second-place method by 4.29%.ViM also attains the lowest average FPR95 and leads on AUROC and FPR95 for OpenImage-O, Texture, and ImageNet-O.
  • Limitations: ViM shows smaller gains on iNaturalist, whose samples have the smallest average residual norm among the four OOD datasets.The authors also state that avoiding retraining limits ViM by the feature quality of the original network.
  • Information Source: ViM combines complementary information sources: logit-based methods struggle on Texture and ImageNet-O, whereas feature-only Residual struggles on iNaturalist and OpenImage-O.The proposed score remains competent across dataset types by combining class-dependent and class-agnostic information.
  • Results on ViT: ViM and Mahalanobis have close AUROCs on ViT, but ViM is lightweight and fast because Mahalanobis computes class-wise distances.The comparison covers four OOD datasets and reports that ReAct, Energy, MaxLogit, and ODIN are the next-best methods.
  • More Model Architectures: ViM remains robust across RepVGG, ResNet50-d, Swin, and DeiT, with average AUROC and FPR95 reported across four OOD datasets.These experiments use pretrained weights and assess robustness to model architecture changes.
  • The Effect of Hyperparameter: ViM is robust to principal-space dimension selection, while its matching parameter assigns more weight to features or logits when perturbed.The authors report that performance changes little over a wide range of D values and that the unperturbed matching parameter is best overall.

7. Conclusion

The paper presents ViM, an OOD detection score combining feature-space and logit information, and reports robustness across CNN and transformer architectures. It also introduces OpenImage-O, a large-scale ImageNet-1K OOD dataset.

  • ViM combines feature-space and logit information to provide class-agnostic and class-dependent OOD evidence.
  • The experiments show ViM’s effectiveness and robustness across CNN-based and transformer-based models.
  • OpenImage-O is a high-quality, large-scale OOD dataset created for ImageNet-1K evaluation.

B. Detailed Results of Four Models (Sec. 6.3)

The detailed evaluation covers multiple models and shows that OpenImage-O has broader ImageNet-1K class coverage than other OOD datasets. The experiments include a quality-control inspection process for the dataset.

  • The benchmark includes detailed AUROC and FPR95 results for RepVGG, ResNet-50d, Swin Transformer, and DeiT.
  • Dataset construction uses model-based similarity retrieval and human labeling to distinguish OpenImage samples from similar ImageNet-1K categories.
  • OpenImage-O covers ImageNet-1K classes more broadly than the other three OOD datasets.

D. Details on Grouping (Sec. 6.5)

The grouping strategy is evaluated using officially released MOS settings and has limited effectiveness for the ViT model. ViM changes little when grouping is added.

  • MOS uses fine-tuning with the officially specified grouping strategy, whereas MaxGroup and ViM+Group use pretrained BiT weights in the reported comparison.
  • Grouping Results on ViT: The grouping strategy is less effective for ViT than for the settings examined elsewhere.
  • Grouping Results on ViT: For ViT, grouping produces only a small AUROC improvement for MSP while worsening FPRs.
  • Grouping Results on ViT: ViM and ViM+Group perform very similarly on ViT, with original ViM slightly better.

E. Details on Baselines (Sec. 6)

The paper compares ViM with feature-, logit-, softmax-, grouping-, and training-based baselines across models and OOD datasets. ViM generally achieves strong AUROC and FPR95 results, while grouping and model-dependent behavior vary.

  • Mahalanobis reaches 56% average AUROC on four OOD datasets when lower-level BiT features are included.
  • The model benchmark includes three CNN-based and three transformer-based architectures, with ViT and Swin Transformer pretrained on ImageNet-21K.
  • Table 6 reports AUROC and FPR95 percentages for four models across OpenImage-O, Texture, iNaturalist, and ImageNet-O.

G. Running Time of Four Methods (Sec. 6.2)

ViM is the fastest among the four most competitive OOD detection methods, whose inference times are compared across four datasets.

  • ViM is the fastest among ViM, Mahalanobis, KL Matching, and Residual on the four datasets.Their inference times are reported in Tab. 8.
Loading 2203.10807v1…