PaperlayerView exact revision on arXiv

Source-linked AI summary

Delving into Out-of-Distribution Detection with Vision-Language Representations

Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, Yixuan Li

arXiv:2211.13445v1cs.CVcs.AIcs.LG

TL;DR

OOD detection needs to recognize novel samples, but most methods rely on a single modality and leave multimodal semantics underused. The paper proposes Maximum Concept Matching, which compares visual features with textual prototypes in a zero-shot setting. MCM performs strongly across real-world tasks, including a 13.1% AUROC advantage over a pure-visual baseline on a hard semantically similar-class task.

  • Problem

    Most OOD detection methods are single-modal, limiting use of semantic information in text and challenging detection when OOD inputs resemble ID data visually.

  • Method

    MCM uses textual embeddings of the task’s ID class labels as concept prototypes and scores test images by their visual-feature matching to those prototypes.

  • Results

    MCM achieves superior performance across many real-world tasks, including a 13.1% AUROC improvement over a pure-visual baseline on a hard semantically similar-class OOD task.

  • Takeaways & Limitations

    Zero-shot vision-language representations support OOD detection without downstream ID training or prior information about OOD inputs.

  • Takeaways & Limitations

    The analysis assumes OOD inputs produce similar cosine similarities across ID concepts for the theoretical treatment of softmax scaling.

Abstract

from arXiv · show

Recognizing out-of-distribution (OOD) samples is critical for machine learning systems deployed in the open world. The vast majority of OOD detection methods are driven by a single modality (e.g., either vision or language), leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of OOD detection from a single-modal to a multi-modal regime. Particularly, we propose Maximum Concept Matching (MCM), a simple yet effective zero-shot OOD detection method based on aligning visual features with textual concepts. We contribute in-depth analysis and theoretical insights to understand the effectiveness of MCM. Extensive experiments demonstrate that MCM achieves superior performance on a wide variety of real-world tasks. MCM with vision-language features outperforms a common baseline with pure visual features on a hard OOD task with semantically similar classes by 13.1% (AUROC). Code is available at https://github.com/deeplearning-wisc/MCM.

1 Introduction

OOD detection must identify samples outside task-specific known classes, yet most methods use only one modality and can miss semantically different but visually similar inputs. MCM addresses this gap with zero-shot vision-language concept matching and performs strongly across real-world tasks.

  • Motivation: Single-modal OOD detection can miss inputs that are visually similar to in-distribution data but semantically different from known classes.The paper motivates using multimodal information because text can encode semantic information omitted by one-hot labels.
  • Method: MCM defines textual embeddings of in-distribution class labels as concept prototypes and measures each image’s distance to the closest prototype.Images closer to an ID prototype are treated as more likely in-distribution, while farther images are more likely OOD.
  • Advantages: MCM is training-free, generalizable across tasks, OOD-agnostic, and scalable to large real-world classification settings.The method uses one pretrained model without downstream fine-tuning or prior information about OOD inputs.
  • Results: 91.49% average AUROC on ImageNet-1k shows MCM outperforming methods that require training.The result is reported alongside competitive performance on challenging hard-OOD evaluations.
  • Analysis: The paper provides empirical and theoretical analysis explaining why MCM works and supports future OOD detection with multimodal features.Its analysis focuses on understanding the effectiveness of the proposed approach.

2 Preliminaries

The paper defines zero-shot OOD detection relative to the known classes of a target classification task, using a pretrained vision-language model without training on the target ID samples. The detector flags samples outside those classes and otherwise assigns them to a known class.

  • Contrastive vision-language pre-training: Vision-language pretraining aligns image and textual descriptions in a shared feature space through separate image and text encoders.CLIP is presented as an example using a self-supervised contrastive objective.
  • Zero-shot OOD detection: Zero-shot OOD detection defines known and out-of-distribution classes relative to the classification task, not the pretraining distribution.The detector must identify samples outside the task’s known classes and assign other samples to one of them.
  • Zero-shot OOD detection: The OOD detector is a binary function G(x; Yin, T, I) that maps a test image to either in or out.Its inputs include the task’s class names and pretrained text and image encoders.
  • Zero-shot OOD detection: The method uses only the given class names and a pretrained model, with no training on in-distribution samples.This absence of ID-sample training is the basis for calling the procedure zero-shot.

3 OOD Detection via Concept Matching

MCM performs zero-shot OOD detection by matching image features against text-derived concept prototypes for the ID classes. Its softmax-scaled matching score and theoretical analysis target separability between ID and OOD inputs across tasks.

  • Concept construction: MCM constructs one textual concept vector per ID label using prompts such as “this is a photo of a ⟨yi⟩.”
  • Concept matching: For each test image, MCM computes cosine similarities between its visual feature and every ID concept vector, then uses the maximum matching score.
  • Detection rule: The detector labels an input ID when its softmax-scaled MCM score reaches threshold λ; otherwise it labels the input OOD.
  • Design distinction: MCM differs from Mahalanobis-style detection by using textual, multimodal prototypes rather than prototypes formed from pure visual embeddings.
  • Theoretical insight: For CLIP-like models, OOD inputs tend to have uniform cosine similarities across ID concepts, while softmax scaling increases ID-OOD separability.
  • Scope and advantages: MCM is presented as a zero-shot detector that generalizes across tasks using one model and only the new task’s visual concept names.

4 A Comprehensive Analysis of MCM

MCM is evaluated as a zero-shot OOD detector across diverse datasets, large-scale ImageNet-1k, hard OOD settings, and comparisons with alternative CLIP-based scores. The analyses emphasize broad applicability, training-free operation, and strong detection performance.

  • Datasets and metrics: Evaluations cover CUB-200, STANFORD-CARS, FOOD-101, OXFORD-PET, and ImageNet variants, using FPR95, AUROC, and ID classification accuracy.The datasets increase evaluation variety, image resolution, and class count beyond small-scale OOD benchmarks.
  • Zero-shot evaluation: MCM achieves strong zero-shot detection across seven ID datasets with one pretrained model and no training samples or OOD information.For STANFORD-CARS as ID, MCM obtains an average FPR95 of 0.08%.
  • Scalability: On ImageNet-1k, larger CLIP models improve MCM performance, while CLIP-L outperforms MOS by 1.38% in AUROC without training.CLIP-L also reduces FPR95 by 4.57% relative to CLIP-B and reaches 73.28% zero-shot ID accuracy.
  • Hard OOD detection: MCM remains competitive on hard OOD tasks, including semantically similar ImageNet-10 versus ImageNet-20 categories, without training involved.Against Mahalanobis, MCM improves FPR95 by 73.32% in one direction and 30.12% in the reverse direction.
  • Comparisons: MCM outperforms an enhanced label-based baseline on all OOD datasets while avoiding an OOD label set and an additional caption generator.The baseline’s detection depends on caption quality, and obtaining captions can increase computational overhead.

5 Discussion: A Closer Look at MCM

The discussion examines softmax scaling, visual-versus-multimodal prototypes, scaling functions, and backbone or prompt choices. These analyses connect MCM’s performance to calibrated similarity processing and the use of textual concepts.

  • Softmax scaling: 22.6% improvement in FPR95 results when softmax scaling with temperature τ = 1 replaces direct maximum cosine similarity on ImageNet-100 versus iNaturalist.Increasing τ to 10 produces similar performance.
  • Softmax scaling: τ = 1 is provably superior to no softmax scaling under the reported bound because Kλ−1 ≈ 0.65 and τ = 1 > 0.65.The empirical estimates are λwo ≈ 0.26, δ ≈ 0.03, ŝ_y2 ≈ 0.23, and λ ≈ 0.011.
  • Vision-language versus visual features: 90.77% average AUROC for MCM versus 73.14% for Mahalanobis is reported across four ImageNet-1k OOD test datasets.Both methods use the same CLIP-B image encoder, while MCM defines prototypes from textual features and Mahalanobis from visual embeddings.
  • Scaling ablation: 11.33% and 26.34% average FPR95 improvements occur on ImageNet-20 and ImageNet-1k when MCM uses softmax scaling instead of maximum cosine similarity.The advantage is smaller on easier Food-101, Stanford-Cars, and Oxford-Pet tasks.
  • Prompt design: Prompt ensembling reduces ImageNet-1k average FPR95 from 38.17% to 35.23% while allowing textual embeddings to be precomputed and averaged.Five prompts slightly outperform 80 prompts without increasing inference-time cost.

6 Related Works

Related work spans visual, language, and vision-language OOD detection, with multimodal OOD detection remaining comparatively new and limited. MCM addresses this gap without candidate OOD labels and extends evaluation to larger realistic datasets.

  • Visual OOD detection includes generative and discriminative methods for deriving binary ID-OOD classifiers alongside visual classification.
  • MCM contributes a zero-shot OOD paradigm that incorporates textual information and supports a wide variety of tasks.
  • Language OOD detection studies shifts from changing topics, domains, and unexpected user utterances, with pretrained language models improving robustness.
  • Vision-language models use either single-stream or dual-stream architectures to learn multimodal representations.
  • Multimodal OOD detection has limited prior work, often relying on candidate OOD labels or focusing on small-scale inputs.MCM is OOD-agnostic and expands evaluation to large-scale realistic datasets.

7 Conclusion

The paper presents MCM as a multimodal, zero-shot approach to OOD detection using textual features as concept prototypes. It combines theoretical analysis with broad evaluations and reports advantages over pure visual features and prior approaches.

  • MCM uses joint vision-language representations by treating textual features as concept prototypes for OOD detection.
  • MCM is training-free, generalizable across tasks, scalable to hundreds of classes, and does not require prior information about OOD inputs.
  • The paper provides theoretical guarantees that softmax scaling improves zero-shot OOD detection.
  • Experiments cover large-scale realistic tasks, including several types of hard OOD datasets.
  • The paper reports an advantage of vision-language features over pure visual features for OOD detection.

A Theoretical Justification: Softmax Scaling for Zero-Shot OOD Detection

The theoretical analysis explains how softmax scaling affects MCM-based OOD detection. Under the stated assumptions, moderate temperature scaling yields a false-positive-rate bound relative to unscaled scoring, supported by empirical verification.

  • Definitions: MCM defines the maximum concept matching score as the maximum cosine similarity between an input image feature and textual concept vectors.The detector classifies inputs using a threshold on this score.
  • Theoretical comparison: The theory compares softmax-scaled scoring at temperature τ with unscaled scoring using their respective false positive rates.FPR(τ, λ) uses temperature τ and threshold λ, whereas FPRwo(λwo) omits softmax scaling.
  • Proof: Under Assumption A.1, a threshold transformation makes minimizing false positive rate over λ equivalent to minimizing it over λ′.
  • Assumptions: The result requires a moderately large temperature and relies on the largest and second-largest cosine similarities for an OOD input.
  • Proof: The proof establishes FPR(τ, λ) ≤ FPRwo(λwo), showing that unscaled scoring has a larger false positive rate under the stated temperature conditions.
  • Implementation: The MCM temperature is set to τ = 1 by default, and performance remains similar across τ ∈ [0.5, 100].

B.3 Datasets

The evaluation uses curated ImageNet subsets and established large-scale OOD datasets. It includes hard OOD classes selected for semantic similarity and provides dataset-generation resources for reproducibility.

  • ImageNet-derived datasets: ImageNet-10 contains ten high-resolution categories designed to mimic the class distribution of CIFAR-10.
  • ImageNet-derived datasets: ImageNet-20 contains 20 classes semantically similar to ImageNet-10 for hard OOD evaluation, including examples such as dog versus wolf.
  • Reproducibility: The authors provide scripts for generating ImageNet-10 and ImageNet-20 to support future large-scale hard OOD research.
  • ImageNet-derived datasets: ImageNet-100 is formed by randomly sampling 100 classes from ImageNet-1k.
  • Conventional OOD datasets: Conventional OOD evaluation uses non-overlapping subsets from iNaturalist, SUN, Places, and Texture for ImageNet-1k.
  • Conventional OOD datasets: The iNaturalist subset contains 110 plant classes, while the SUN subset contains 50 natural-object classes absent from ImageNet-1k.
  • Conventional OOD datasets: The Places subset contains 50 categories absent from ImageNet-1k, and Texture uses the entire dataset because its categories do not overlap.

C Spurious OOD Datasets

The paper examines spurious OOD samples containing environmental features without object features related to ID classes, using Waterbirds-based datasets.

  • Spurious OOD samples contain environmental features but no object features related to the ID classes.Examples include backgrounds such as color, texture, and scenery that co-occur with ID samples without capturing label cues.
  • The Waterbirds ID dataset combines CUB-200 bird photographs with water or land backgrounds from Places.
  • The spurious OOD dataset consists of land and water backgrounds from Places.
  • Figure 6 illustrates spurious OOD samples for Waterbirds.

D ID Classification Accuracy

This section reports multi-class classification accuracy on ImageNet-1k for the methods evaluated in Table 2.

  • Table 7 reports multi-class classification accuracy on ImageNet-1k for the methods listed in Table 2.
  • The reported metric is ID classification accuracy on ImageNet-1k, expressed as a percentage.

E Implementation of CLIP-Based Baselines

The CLIP-based baselines score images against ID and candidate OOD labels using normalized cosine similarities. Candidate labels can be generated from captions and filtered to reduce overlap with ID labels.

  • Overview of Baselines: The baseline expands the ID label set with candidate OOD labels and normalizes cosine similarities over the combined label space.
  • Overview of Baselines: The scoring function uses temperature scaling with a positive hyperparameter τ.
  • Obtaining OOD Candidate Labels: ZO-CLIP generates candidate labels with a transformer decoder trained from scratch on COCO, but its categories do not scale to ImageNet.
  • Obtaining OOD Candidate Labels: The improved pipeline uses a caption generator, syntactic parser, and filtering module to produce candidate OOD labels.ClipCap generates captions, the parser extracts noun objects, and filtering removes labels overlapping with the ID set.
  • String-based Filtering: Filtering can raise an ID input's score from approximately 0.5 to 1 by removing generated labels overlapping with ID labels.
  • Alternative Scoring Functions: MCM remains the most promising across most OOD test sets compared with entropy, variance, and scaled-difference alternatives.
Loading 2211.13445v1…