Source-linked AI summary
Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, Lei Shu
TL;DR
The paper asks whether OOD detection can classify seen samples and detect unseen samples without training data beyond seen class names. ZOC extends CLIP with a description generator that creates candidate unseen labels and compares them with seen labels, outperforming the reported baselines. The method targets near-OOD detection without validation OOD data.
Problem
Zero-shot OOD detection must classify seen samples and detect unseen samples using only seen class names, without training data or a closed-world classifier.
Method
ZOC extends CLIP with a textual description generator that dynamically supplies candidate unseen labels and computes an OOD score from similarities to seen and generated labels.
Results
ZOC outperforms all baselines by a large margin, including fully supervised methods and baselines using the same pretrained backbone.
Takeaways & Limitations
ZOC demonstrates zero-shot OOD detection without training samples or prior knowledge of unseen classes, including for the near-OOD setting.
Takeaways & Limitations
The authors report no ablation study because no component of ZOC can be removed while preserving its function.
Abstract
from arXiv · showhide
In an out-of-distribution (OOD) detection problem, samples of known classes(also called in-distribution classes) are used to train a special classifier. In testing, the classifier can (1) classify the test samples of known classes to their respective classes and also (2) detect samples that do not belong to any of the known classes (i.e., they belong to some unknown or OOD classes). This paper studies the problem of zero-shot out-of-distribution(OOD) detection, which still performs the same two tasks in testing but has no training except using the given known class names. This paper proposes a novel yet simple method (called ZOC) to solve the problem. ZOC builds on top of the recent advances in zero-shot classification through multi-modal representation learning. It first extends the pre-trained language-vision model CLIP by training a text-based image description generator on top of CLIP. In testing, it uses the extended model to generate candidate unknown class names for each test sample and computes a confidence score based on both the known class names and candidate unknown class names for zero-shot OOD detection. Experimental results on 5 benchmark datasets for OOD detection demonstrate that ZOC outperforms the baselines by a large margin.
Introduction
The paper defines zero-shot OOD detection as classifying seen samples and detecting unseen samples using only seen class names. ZOC extends CLIP with generated candidate labels and outperforms supervised and comparable-backbone baselines.
- The closed-world assumption often fails in real-world deployment, making OOD detection important for safety-critical applications such as autonomous driving and healthcare.
- Zero-shot OOD detection performs seen-class classification and OOD detection using only the names of seen classes, without training data or a closed-world classifier.
- CLIP provides zero-shot image classification through a shared image-text feature space learned from 400 million image-caption pairs.
- ZOC dynamically generates candidate OOD labels because original CLIP matches every image to one of the provided seen labels and lacks OOD detection functionality.
- ZOC extends CLIP with a textual description generator, uses its output as unseen candidate labels, and scores images using similarities to seen and generated labels.
- ZOC outperforms many state-of-the-art fully supervised OOD detection baselines and baselines using the same pretrained backbone.
Related Work
Prior OOD approaches include SVM- and deep-learning-based methods, while related pretrained-model work often assumes unseen-label information. ZOC targets near-OOD detection without validation OOD data.
- OOD, open-set detection, and open-world classification are treated as equivalent terms in this paper despite differing dataset conventions.
- Outlier-exposure methods may access actual test OOD data or large outlier datasets during training, unlike most methods discussed here.
- Related CLIP-based work assumes unseen labels as weak OOD information, whereas this paper addresses near-OOD detection without validation OOD data.
Method
ZOC extends CLIP with an image-description generator that supplies candidate unseen labels, then compares image similarity across seen and generated labels to obtain an OOD score.
- Method: ZOC extends CLIP’s closed-world zero-shot classification to OOD detection by comparing an image with seen labels and generated unseen labels.
- CLIP zero-shot classification: CLIP zero-shot classification matches an encoded image with textual class descriptions and uses the highest similarity as the predicted label.
- Training the Image Description Generator: A text generator is trained on image-caption pairs using features from CLIP’s image encoder to produce descriptions for test images.
- Training the Image Description Generator: The CLIP image encoder represents an image as transformer-processed embedding vectors, with zout serving as its whole-image semantic representation.
- Training the Image Description Generator: The decoder attends to CLIP image features and is trained with cross-entropy under teacher forcing to generate textual descriptions.
- Inference in Testing: During inference, Decodertext generates a description, from which top-k candidate unseen labels Yu are retrieved before CLIP evaluates Ys ∪ Yu.
- Inference in Testing: CLIP computes cosine similarities for seen and generated labels, applies softmax, and defines the OOD confidence score from the accumulated probability of Yu.
- Inference in Testing: In the boat example, the actual unseen label is absent from Yu, yet other generated candidates support the correct OOD decision.
Experiments
ZOC is evaluated across five OOD benchmarks using a CLIP-based architecture, multiple baselines, and AUROC measurements. It substantially outperforms the compared methods, while case studies illustrate both successful detections and failure modes.
- Model Architecture and Training Details: The model uses frozen CLIP image and text encoders plus a BERT-based textual description generator trained on MS-COCO.The decoder is trained for 25 epochs with Adam at a constant learning rate of 10^-5; CLIP encoders are not fine-tuned.
- Datasets: Experiments cover CIFAR10, CIFAR100, CIFAR+10, CIFAR+50, and TinyImagenet with varying numbers of seen and unseen classes.For example, TinyImagenet uses 20 seen and 180 unseen classes, while CIFAR10 uses 6 seen and 4 unseen classes.
- Baselines: ZOC is compared with 11 OOD detection baselines, including supervised classifiers, CLIP-based systems, DOC, OpenMax, G-OpenMax, and OSRCI.The baselines either train a closed-world classifier or use a pretrained backbone with labeled training data.
- Results and Discussion: ZOC outperforms all baselines by a large margin on AUROC, the evaluation measure used for Table 1.Each table result is averaged over five dataset splits, with standard deviations reported.
- Results and Discussion: ZOC consistently outperforms CLIP-based MSP on all datasets, supporting dynamically generated unseen labels over identical inference across samples.The comparison is designed to distinguish ZOC’s confidence-scoring technique from CLIP’s pretrained feature space alone.
- Limitations: The paper reports no ablation study because removing any algorithm component would prevent ZOC from functioning.This limits component-level attribution of the method’s performance.
- Case study and error analysis: Case studies show that generated labels can support correct seen- and unseen-class decisions, but semantically similar seen labels can cause errors.The paper notes that fine-grained labels may be absent from the MS-COCO training corpus, and plans to use a larger corpus.
- Limitations: ZOC does not model relations between unseen labels, which the authors identify as potentially important for detecting more sophisticated OOD samples.The authors leave incorporating such relations for future work.
Conclusion
The paper introduces zero-shot OOD detection using CLIP and proposes ZOC, which generates candidate unseen labels and scores images against seen and generated labels. ZOC outperforms traditional supervised models and baselines using pretrained CLIP encoders.
- ZOC extends CLIP to dynamically generate candidate unseen labels for each test image.The method builds on zero-shot classification with a pretrained language-vision model.
- ZOC computes a confidence score from the test image’s similarity to seen and generated candidate unseen labels.The score is defined in the shared feature space.
- ZOC uses no concrete training samples, relying on known or seen class label names in the zero-shot setting.
- ZOC outperforms traditional supervised models and baselines that use pretrained CLIP encoders.