PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No

Hualiang Wang, Yi Li, Huifeng Yao, Xiaomeng Li

arXiv:2308.12213v2cs.CVcs.AI

TL;DR

Zero-shot OOD detection needs to identify unknown classes using only ID class names, but CLIP lacks reliable negation semantics for hard OOD cases. CLIPN adds “no” prompts, a “no” text encoder, opposite-semantic losses, and threshold-free inference, outperforming existing methods in the reported benchmarks.

  • Problem

    Zero-shot OOD detection methods driven by CLIP have received less attention despite the need to distinguish unknown classes using only ID class names.

  • Method

    CLIPN adds learnable “no” prompts and a “no” text encoder, trains them with two opposite-semantic losses, and uses two threshold-free inference algorithms.

  • Results

    At least 2.34% higher AUROC and 11.64% lower FPR95 were reported for ViT-B-16 on ImageNet-1K zero-shot OOD detection.

  • Takeaways & Limitations

    CLIPN provides a foundation for leveraging CLIP in downstream OOD tasks through positive and negation-semantic prompts.

Abstract

from arXiv · show

Out-of-distribution (OOD) detection refers to training the model on an in-distribution (ID) dataset to classify whether the input images come from unknown classes. Considerable effort has been invested in designing various OOD detection methods based on either convolutional neural networks or transformers. However, zero-shot OOD detection methods driven by CLIP, which only require class names for ID, have received less attention. This paper presents a novel method, namely CLIP saying no (CLIPN), which empowers the logic of saying no within CLIP. Our key motivation is to equip CLIP with the capability of distinguishing OOD and ID samples using positive-semantic prompts and negation-semantic prompts. Specifically, we design a novel learnable no prompt and a no text encoder to capture negation semantics within images. Subsequently, we introduce two loss functions: the image-text binary-opposite loss and the text semantic-opposite loss, which we use to teach CLIPN to associate images with no prompts, thereby enabling it to identify unknown samples. Furthermore, we propose two threshold-free inference algorithms to perform OOD detection by utilizing negation semantics from no prompts and the text encoder. Experimental results on 9 benchmark datasets (3 ID datasets and 6 OOD datasets) for the OOD detection task demonstrate that CLIPN, based on ViT-B-16, outperforms 7 well-used algorithms by at least 2.34% and 11.64% in terms of AUROC and FPR95 for zero-shot OOD detection on ImageNet-1K. Our CLIPN can serve as a solid foundation for effectively leveraging CLIP in downstream OOD tasks. The code is available on https://github.com/xmed-lab/CLIPN.

1. Introduction

Zero-shot OOD detection seeks to identify unknown classes without ID-dataset training, but hard OOD samples can resemble ID classes and CLIP lacks reliable negation semantics. CLIPN adds learnable “no” prompts, a “no” text encoder, opposite-semantic losses, and threshold-free inference algorithms.

  • Motivation: Hard-to-distinguish OOD samples lie close to certain ID classes, weakening scores based only on ID-ness.These samples are contrasted with easier OOD cases that are distant from all ID classes.
  • Motivation: CLIP fails to accurately match images with negative prompts, implying that the original model lacks “no” logic.The motivating toy experiment compares dog and cat images using positive and negation-semantic prompt groups.
  • Method: CLIPN equips CLIP with learnable “no” prompts and a “no” text encoder to represent negation semantics.The architecture complements standard positive-semantic prompts with negative text representations.
  • Training: The image-text binary-opposite and text semantic-opposite losses teach CLIPN when to match images with “no” prompts and how to separate opposite semantics.The first aligns images with correct “no” prompts, while the second pushes standard and “no” prompts apart.
  • Inference: Competing-to-win and agreeing-to-differ are threshold-free algorithms for OOD detection using negation semantics.They use the outputs of standard and “no” text encoders during inference.
  • Results: CLIPN outperforms most existing OOD detection algorithms on both large-scale and small-scale tasks.The introduction reports this pattern across the evaluated task scales.

2. Related Work

Related work covers vision-language representation learning, zero-shot learning with CLIP, and conventional OOD detection based on scores derived from probabilities, logits, or features.

  • Vision-Language Learning: Vision-language research progressed from attention-based methods to transformer architectures and prompt strategies, including CLIP.These methods study relationships between visual and linguistic information.
  • Zero-Shot Learning: Zero-shot learning targets unseen knowledge, while CLIP provides holistic, task-agnostic representations learned from large-scale data.The passage identifies CLIP’s open-set versatility as a major contribution to zero-shot learning.
  • OOD Detection: OOD detection identifies OOD images among test data containing both ID and OOD images.This is the operational goal of the task reviewed in the related work.
  • OOD Detection: Conventional OOD methods derive scores from probabilities, logits, or features, whereas CLIPN also uses negation-semantic texts to identify OOD samples.Examples include MSP, MaxLogit, energy, Mahalanobis distance, and residual-based scores.

3. Methodology

CLIPN extends CLIP for zero-shot OOD detection by adding learnable negation prompts and a learnable “no” text encoder. It trains these components with opposite-semantic objectives and uses two threshold-free inference algorithms to combine standard and “no” text predictions.

  • Architecture: CLIPN extends CLIP with an image encoder, standard text encoder, and learnable “no” text encoder for zero-shot OOD detection.The image encoder remains frozen, while the “no” text encoder captures negative semantics from opposite-semantic text inputs.
  • Prompt Design: Learnable “no” prompt pools generate negative descriptions such as “a photo without {t}” for the “no” text encoder.During training, a negative prompt is sampled and its token embeddings are supplied to the “no” text encoder; negative prompt features can also be represented by learnable parameters.
  • Training Loss Design: Image-text binary-opposite loss distinguishes matched-yet-unrelated from reversed-matched image–“no” text pairs and trains their intended associations.The match-ness value is 1 for matched-yet-unrelated pairs and 0 for reversed-matched pairs.
  • Training Loss Design: Text semantic-opposite loss separates standard and “no” text features because their semantics should be opposite in the feature space.The total mini-batch loss is the sum of the image-text binary-opposite and text semantic-opposite losses.
  • Inference algorithm of CLIPN: Competing-to-win uses the highest ID probability and the corresponding “no” class text, while agreeing-to-differ incorporates all ID probabilities and the “no” probability.Agreeing-to-differ reformulates the C ID-class probabilities as C + 1 probabilities by creating an unknown class.

4. Experiment

The experiments evaluate CLIPN across large-scale, small-scale, and in-domain OOD detection settings, including ablations of its losses, prompts, feature separation, and computational cost. CLIPN generally achieves stronger detection performance and reduces training costs compared with prior methods or CLIP.

  • Experimental Settings: CLIPN is evaluated on large-scale ImageNet-1K, small-scale CIFAR-100, and in-domain CIFAR-10/CIFAR-100 OOD detection settings.The large-scale setting uses Texture, iNaturalist, SUN, and Places365 as OOD datasets; the small-scale setting uses CIFAR-10, ImageNet R, and LSUN.
  • Large-Scale OOD Detection: 2.34% AUROC and 11.64% FPR95 are the minimum average improvements of ViT-B-16 CLIPN over MCM on ImageNet-1K OOD detection.Using ViT-B-32, CLIPN reports at least 4.12% and 23.25% improvements in AUROC and FPR95, respectively.
  • Small-Scale OOD Detection: CLIPN achieves the best average AUROC and FPR95 performance on the small-scale CIFAR-100 OOD detection benchmark.The evaluation uses CIFAR-10, ImageNet R, and LSUN as OOD datasets with a ViT-B-32 image encoder.
  • In-Domain OOD Detection: 4.5% and 7.75% AUROC improvements over MSP are reported for CLIPN on CIFAR-10 and CIFAR-100 in-domain OOD detection, respectively.The results are averaged over five splits, and the paper notes that its MSP implementation uses a different pretrained model from ZOC’s reported baseline.
  • Ablation Study: Adding TSO to ITBO increases CLIPN-A AUROC by 2.19%, 2.32%, 1.41%, and 2.45% across four OOD datasets.With ITBO alone, the corresponding AUROC scores are 93.08%, 91.61%, 89.52%, and 89.83%.
  • Computational Cost: CLIPN reduces ViT-B-32 training time per iteration by 36.4% and GPU usage by 29.0% compared with CLIP, while inference adds 0.016G computation and 0.51M parameters.The reported training reductions are attributed to fewer backward-computation FLOPs and parameters.

5. Conclusion and Limitation

CLIPN teaches CLIP to distinguish ID and OOD samples using positive and negation semantics. The framework introduces new prompts, training losses, threshold-free inference algorithms, and identifies limitations in downstream task extension and specialized datasets.

  • Conclusion: CLIPN equips CLIP to distinguish OOD and ID samples through positive-semantic and negation-semantic prompts.It introduces a framework for OOD detection by teaching CLIP to respond with “no”.
  • Method: The framework introduces learnable “no” prompts, a no text encoder, and two losses for learning negation semantics.The losses are an image-text binary-opposite loss and a text semantic-opposite loss.
  • Inference: CLIPN proposes competing-to-win and agreeing-to-differ as threshold-free inference algorithms for OOD detection.Both algorithms use negation semantics from the no prompts and text encoder.
  • Limitation: The approach lacks clear evidence for extension to OOD segmentation or detection tasks.This is identified as one limitation of the approach.
  • Limitation: Its effectiveness for OOD classification on specialized medical and satellite-image datasets remains uncertain because CLIP’s performance there is underexplored.The limitation concerns specialized datasets beyond the evaluated setting.
Loading 2308.12213v2…