PaperlayerView exact revision on arXiv

Source-linked AI summary

LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning

Atsuyuki Miyai, Qing Yu, Go Irie, Kiyoharu Aizawa

arXiv:2306.01293v3cs.CV

TL;DR

Few-shot OOD detection seeks to identify unseen-class images using only a few labeled ID examples, but CoOp can encode ID-irrelevant content in text embeddings. LoCoOp regularizes CLIP’s local nuisance features away from ID text embeddings, and it outperforms zero-shot, fully supervised, and prompt-learning methods on ImageNet OOD benchmarks, including with one label per class.

  • Problem

    Few-shot OOD detection must identify unseen-class images from only a few labeled ID images, while CoOp can include ID-irrelevant information in class text embeddings.

  • Method

    LoCoOp treats ID-irrelevant portions of CLIP local features as OOD and learns to push them away from ID class text embeddings during prompt training.

  • Results

    LoCoOp substantially outperforms zero-shot, fully supervised, and prompt-learning methods on ImageNet OOD benchmarks, including with one label per class.

  • Takeaways & Limitations

    Local-feature OOD regularization improves separation between ID and OOD while requiring only a few labeled ID examples.

  • Takeaways & Limitations

    LoCoOp requires models with strong local visual-text alignment, and applying it to other lightweight tuning methods remains future work.

Abstract

from arXiv · show

We present a novel vision-language prompt learning approach for few-shot out-of-distribution (OOD) detection. Few-shot OOD detection aims to detect OOD images from classes that are unseen during training using only a few labeled in-distribution (ID) images. While prompt learning methods such as CoOp have shown effectiveness and efficiency in few-shot ID classification, they still face limitations in OOD detection due to the potential presence of ID-irrelevant information in text embeddings. To address this issue, we introduce a new approach called Local regularized Context Optimization (LoCoOp), which performs OOD regularization that utilizes the portions of CLIP local features as OOD features during training. CLIP's local features have a lot of ID-irrelevant nuisances (e.g., backgrounds), and by learning to push them away from the ID class text embeddings, we can remove the nuisances in the ID class text embeddings and enhance the separation between ID and OOD. Experiments on the large-scale ImageNet OOD detection benchmarks demonstrate the superiority of our LoCoOp over zero-shot, fully supervised detection methods and prompt learning methods. Notably, even in a one-shot setting -- just one label per class, LoCoOp outperforms existing zero-shot and fully supervised detection methods. The code will be available via https://github.com/AtsuMiyai/LoCoOp.

1 Introduction

LoCoOp addresses few-shot CLIP-based OOD detection by regularizing ID-irrelevant local features, improving separation between ID and OOD classes. Across ImageNet OOD benchmarks, it outperforms prior methods, including in the one-shot setting.

  • Few-shot OOD detection uses only a few labeled ID images to detect samples from unseen classes.
  • CoOp can embed backgrounds and other ID-irrelevant information into class text embeddings, causing high confidence on OOD images.
  • LoCoOp treats ID-irrelevant portions of CLIP local features as OOD and pushes them away from ID class text embeddings.
  • LoCoOp performs OOD regularization at low additional cost and is compatible with GL-MCM, which uses global and local features.With 100 OOD features per ID image, training is about 1.4× longer and memory use about 1.1× higher than CoOp.
  • LoCoOp substantially improves over zero-shot, fully supervised, and prompt-learning methods on large-scale ImageNet OOD benchmarks.The method also outperforms existing methods with only one label per class.

2 Method

LoCoOp extracts ID-irrelevant local regions from CLIP features and regularizes their class predictions during few-shot training. It combines this local objective with CoOp training and uses global-local scores for test-time OOD detection.

  • Problem setting: The task uses a few ID samples per class without access to OOD data, with ID and OOD classes disjoint.
  • CoOp: CoOp learns prompt context vectors while keeping CLIP parameters fixed, then predicts classes from global image-text similarity.
  • Local features: LoCoOp obtains local CLIP features by projecting visual feature-map regions into the textual space using fixed CLIP projections.
  • Region extraction: Regions whose ground-truth class is absent from the top-K predictions are selected as ID-irrelevant regions.
  • OOD regularization: Entropy maximization makes selected local features dissimilar to every ID text embedding.
  • Training objective: The final objective combines the CoOp loss on the whole image with a weighted OOD-regularization loss.
  • Test-time detection: GL-MCM combines global and local matching scores, while MCM uses only global image-text scores for test-time OOD detection.

3 Experiment

Experiments evaluate LoCoOp on ImageNet OOD benchmarks using few-shot protocols, comparing it with zero-shot, fully supervised, and prompt-learning methods. LoCoOp with GL-MCM achieves strong OOD detection, including in the one-shot setting, while its performance is relatively robust to non-extreme K values.

  • Experimental setup: Experiments use ImageNet-1K as ID data and evaluate 1, 2, 4, 8, and 16 shots per class across four OOD datasets.The OOD datasets include iNaturalist, SUN, Places, and TEXTURE; results are averaged over three runs.
  • Experimental setup: The comparison includes zero-shot, fully supervised, CoOp prompt-learning, and LoCoOp methods using CLIP-B/16.LoCoOp with GL-MCM is identified as the most effective method in Table 1.
  • Main results: 93.52% AUROC: LoCoOp with GL-MCM in the 16-shot setting, while other methods remain below 92%.The evaluation reports FPR95 and AUROC on the ImageNet OOD benchmarks.
  • Main results: 33.52 average FPR95 and 92.14 average AUROC: LoCoOp with GL-MCM in the one-shot setting, outperforming zero-shot and fine-tuned methods.LoCoOp uses one training sample per class, whereas fine-tuned methods use 1,000 training images per class.
  • Main results: GL-MCM boosts LoCoOp more than CoOp across all evaluated numbers of ID labeled samples.Figure 2 compares LoCoOp and CoOp with MCM and GL-MCM variants.
  • Ablation studies on K: K=0 degrades detection by treating all local regions as OOD, while LoCoOp maintains high performance after K=200 and remains effective at K=980.Performance is generally insensitive to K except at extreme values such as 0 and 1,000; K=980 uses about one OOD region per image.

4 Analysis

The analysis evaluates LoCoOp across backbones, ID accuracy, inference considerations, and ID-irrelevant region extraction. Results indicate stronger OOD detection can accompany slightly lower ID classification accuracy, while rank-based extraction performs well.

  • Comparison with CoCoOp: LoCoOp’s per-image inference time is evaluated against CoCoOp because OOD detection precedes the close-set classifier and therefore benefits from short inference time.The comparison reports 16-shot results and measurements on a single Nvidia A100 GPU.
  • ID accuracy: LoCoOp is slightly inferior to CoOp in ID accuracy, while excluding background nuisances improves OOD detection performance.The paper uses this result to argue that strong ID classification accuracy is not always aligned with robust OOD detection.
  • Effectiveness with CNN architectures: LoCoOp remains effective with the CLIP-ResNet-50 backbone, extending the method beyond the ViT experiments.The comparison uses LoCoOp with GL-MCM, denoted LoCoOpGL.
  • ID-irrelevant region extraction: Probability-based thresholding performs comparably to rank-based extraction, whereas entropy-based thresholding performs poorly because its threshold is difficult to determine.The comparison concerns 16-shot OOD detection results.
  • Visualization of extracted OOD regions: Rank-based extraction accurately identifies OOD regions in the visualization, supporting its role in LoCoOp’s regularization pipeline.The method extracts regions predicted without the ground-truth class among the top-K classes, treating them as ID-irrelevant.

5 Theoretical background

The paper motivates foreground-background separation using prior theory and visualizes LoCoOp’s extracted ID-irrelevant regions. CLIP’s local visual-text alignments support distinguishing foregrounds from backgrounds.

  • Theoretical background: Prior research theoretically supports using foreground features for optimal decision boundaries, while CLIP’s local alignments help identify foregrounds and backgrounds.The paper contrasts this capability with conventional classifiers, which had difficulty using only foreground features.
  • Visualization: LoCoOp’s visualization shows that its approach can correctly extract ID-irrelevant regions.These regions are treated as OOD features for regularization.

6 Limitations

The paper identifies three scope boundaries: application to other lightweight tuning methods, models without rich local visual-text alignment, and recognition tasks beyond classification.

  • Application to other light-weight tuning methods: Applying LoCoOp to lightweight tuning methods such as Tip-Adapter and visual prompt methods remains future work.The study focuses on text prompt learning methods for tuning CLIP.
  • Application to models without rich local visual-text alignment: LoCoOp can be challenging to apply to models lacking strong local visual-text alignment capabilities.The approach relies on capabilities such as those provided by CLIP’s image encoder.
  • Extending to other visual recognition tasks: The study focuses only on classification, leaving object detection and segmentation as future extensions.The paper notes that prompt learning is also used for these other visual recognition tasks.

7 Related Work

Related work covers conventional OOD scoring and training-time regularization, prompting for foundation models, and CLIP’s local visual features. The paper positions LoCoOp at the intersection of these lines of research.

  • Out-of-distribution detection: OOD detection methods commonly use probability-based, logit-based, or feature-based score functions, or leverage OOD data for training-time regularization.These approaches form the broader methodological context for LoCoOp.
  • Prompting for foundation models: Prompt learning adapts foundation models to downstream tasks by learning prompt representations, but existing methods mainly optimize close-set ID classification accuracy.The paper states that their OOD detection performance remains unclear.
  • Local features of CLIP: CLIP provides local visual features aligned with textual concepts, complementing the global features typically used for classification.Global features are produced by pooling the feature map, which loses spatial information.

8 Conclusion

LoCoOp is a prompt-learning approach for few-shot OOD detection that regularizes text embeddings using CLIP local features. It improves ID–OOD separation and reports advantages over several existing detection approaches, while its sensitivity to λ is limited except at λ = 1.

  • Conclusion: LoCoOp performs OOD regularization by treating portions of CLIP local features as OOD images during prompt learning.The method pushes ID-irrelevant local features away from ID class text embeddings.
  • Conclusion: This regularization removes nuisances from ID class text embeddings and leads to better separation between ID and OOD samples.
  • Conclusion: Experiments on ImageNet OOD benchmarks show advantages over zero-shot, fully supervised, and existing prompt-learning detection methods.
  • Conclusion: The approach reduces data-gathering effort but requires caution in sensitive applications because the authors note potential vulnerability to misuse.
  • Conclusion: When λ is smaller than 1, LoCoOp outperforms CoOp with both MCM and GL-MCM, while sensitivity is limited except at λ = 1.The reported setting is λ = 0.25.

A.2 Detailed results on few-shot OOD detection

Detailed evaluations across different numbers of ID samples show LoCoOp as the most effective comparison method for few-shot OOD detection.

  • A.2 Detailed results on few-shot OOD detection: LoCoOp is the most effective method among the comparison methods across detailed few-shot OOD detection results.The section reports results for 2, 4, and 8 shots across all OOD datasets.
  • A.2 Detailed results on few-shot OOD detection: The detailed evaluation varies the number of labeled ID samples used for few-shot OOD detection.The reported settings include 2, 4, and 8 shots.
  • A.2 Detailed results on few-shot OOD detection: The evaluation reports average FPR and AUROC scores for the few-shot OOD detection comparisons.

A.3 The effectiveness of LoCoOp on small-scale datasets

LoCoOp remains effective on ImageNet-100 and clarifies why OOD detection performance can diverge from ID classification accuracy. Its background-removal behavior may lower ID accuracy while improving OOD detection.

  • A.3 The effectiveness of LoCoOp on small-scale datasets: LoCoOp outperforms CoOp for OOD detection on ImageNet-100.ImageNet-100 is used as the 100-class ID subset, with the same OOD datasets as the ImageNet-1K evaluation.
  • A.3 The effectiveness of LoCoOp on small-scale datasets: The detailed evaluation is organized as a table of methods and FPR95/AUROC results across iNaturalist, SUN, Places, Texture, and average columns.
  • A.4 Relationship between the OOD detection performance and ID accuracy: Zero-shot and prompt-learning methods can outperform fully supervised methods in OOD detection despite having lower ID accuracy.The explanation is tied to their language-image similarity-based OOD scores rather than last-layer class probabilities.
  • A.4 Relationship between the OOD detection performance and ID accuracy: In a 1-shot setting, LoCoOp can learn from many OOD features and therefore outperforms CoOp in ID accuracy.
  • A.4 Relationship between the OOD detection performance and ID accuracy: In a 16-shot setting, removing ID-correlated background nuisances can reduce ID accuracy while improving OOD detection performance.The paper illustrates this trade-off with green grass serving as a potentially useful background cue for dog classification.

B Dataset details

The experiments use ImageNet-1K for ID evaluation and four disjoint OOD datasets, following few-shot settings from 1 to 16 shots. Results are averaged over three runs and evaluated on ImageNet validation data and designated OOD samples.

  • B Dataset details: ImageNet-1K provides the ID data, with 1, 2, 4, 8, and 16 shots used for few-shot training.
  • B Dataset details: Evaluation uses the ImageNet validation set containing 50,000 images across 1,000 classes.The protocol follows existing CLIP and CoOp few-shot evaluation procedures.
  • B Dataset details: The OOD evaluation uses iNaturalist, SUN, Places, and Texture datasets following prior large-scale OOD studies.
  • B Dataset details: iNaturalist evaluation uses 10,000 images from 110 classes disjoint from ImageNet-1K.
  • B Dataset details: SUN and Places each contribute 10,000 images sampled from 50 classes disjoint from ImageNet-1K.
  • B Dataset details: The entire 5,640-image Texture dataset is used for evaluation.
Loading 2306.01293v3…