PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models

Felix Vogel, Nina Shvetsova, Leonid Karlinsky, Hilde Kuehne

arXiv:2209.06103v1cs.CVcs.AIcs.CL

TL;DR

The paper asks whether web-trained vision-language models’ zero-shot performance reflects genuine attribute-based recognition or prior exposure to class labels and concepts. It analyzes class occurrence in LAION-400M and evaluates attribute-based classification with public models, finding label-based classification stronger than attribute-only recognition.

  • Problem

    Class-name prompting is called zero-shot without accounting for how often a class appeared in training, while attribute-based zero-shot recognition tests whether images can be mapped to classes without class labels.

    Source: §1 · p. 2

  • Method

    The study analyzes class-name occurrence in LAION-400M and benchmarks CLIP, OpenCLIP, and FLAVA on CUB and AWA2 using prompts with class labels, attributes, or both.

    Source: §1 · p. 2

  • Results

    Attribute-based classification performs significantly below label-based classification, and most classes in popular zero-shot benchmarks occur during pretraining.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results distinguish label-driven zero-shot performance from attribute-based recognition and show that attribute-based capability is present but weaker.

    Source: §1 · p. 2

    The study uses LAION-400M class occurrence as a reference for web-mined data and relies on publicly available models and training data for its analysis.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Vision-language models trained on large, randomly collected data had significant impact in many areas since they appeared. But as they show great performance in various fields, such as image-text-retrieval, their inner workings are still not fully understood. The current work analyses the true zero-shot capabilities of those models. We start from the analysis of the training corpus assessing to what extent (and which of) the test classes are really zero-shot and how this correlates with individual classes performance. We follow up with the analysis of the attribute-based zero-shot learning capabilities of these models, evaluating how well this classical zero-shot notion emerges from large-scale webly supervision. We leverage the recently released LAION400M data corpus as well as the publicly available pretrained models of CLIP, OpenCLIP, and FLAVA, evaluating the attribute-based zero-shot capabilities on CUB and AWA2 benchmarks. Our analysis shows that: (i) most of the classes in popular zero-shot benchmarks are observed (a lot) during pre-training; (ii) zero-shot performance mainly comes out of models' capability of recognizing class labels, whenever they are present in the text, and a significantly lower performing capability of attribute-based zeroshot learning is only observed when class labels are not used; (iii) the number of the attributes used can have a significant effect on performance, and can easily cause a significant performance decrease.

1. Introduction

The paper asks whether web-trained vision-language models perform genuinely zero-shot recognition or rely on prior class exposure and class labels. It analyzes training-data occurrence and attribute-based recognition, finding that attribute-only performance is lower than label-based classification.

1. Introduction

  • Web-trained vision-language models can classify from class-name prompts, but this conventional zero-shot setup does not establish how often classes appeared in training.Attribute-based zero-shot learning instead maps images to classes through attribute descriptions without class information.Source: §1 · pp. 1–2
  • The majority of ImageNet classes appear 100 to 100,000 times in LAION-400M, and occurrence-performance patterns vary with class ambiguity and visual distinctiveness.High-occurrence, low-performance examples are semantically ambiguous; some low-occurrence, high-performance examples are distinctive, descriptive, or special forms of more general classes.Source: Figure 1 · p. 1, §1 · p. 2
  • The study examines class occurrence in pretraining and introduces VL-Taboo to analyze attribute learning in vision-language models.It evaluates both prompts that include class labels and prompts based only on attributes.Source: §1 · p. 2
  • Attribute-based classification performs significantly below label-based classification, while attributes have less impact when the class label is given.Additional descriptions and prompt length can affect some models.Source: §1 · p. 2

2. Related work

Prior work spans web-scale vision-language models, model-evaluation benchmarks, and attribute-based zero-shot learning. This study uses publicly available models and training data to examine how class labels and attributes affect classification, a focus distinct from existing benchmarks.

2.1. Web-scale Vision Language Models

  • OpenCLIP and LAION-400M enable analysis using public model checkpoints and a reference distribution of web-mined training data.The authors also tested ViLT, whose class-only prompting recall on CUB was 4.4.Source: §2.1 · pp. 2–3, Table 2 · p. 3

2.2. Evaluation of VL models

  • Existing evaluation benchmarks test generalization, linguistic phenomena, object and attribute understanding, or reasoning, while this benchmark measures how labels and attributes affect classification.It also asks whether longer descriptions with more attributes improve recognition and how much models rely on object labels versus attributes.Source: §2.2 · p. 3

2.3. Attribute based (zero-shot) learning

  • Classical attribute-based zero-shot benchmarks use held-out classes, but their results are not directly comparable to this study because they were designed for models trained or fine-tuned on attribute data.AWA2 has 40 training and 10 test classes; CUB uses 150 training and 50 validation classes.Source: §2.3 · p. 3

3. System description

The study evaluates public vision-language models by comparing image embeddings with text-prompt embeddings. It uses prompts with class names, attributes, or both to measure classification performance.

3.1. Models

  • CLIP and OpenCLIP use jointly trained image and text encoders, whereas FLAVA combines unimodal encoders with a multimodal encoder and several training objectives.The analysis uses ViT-B/32 for CLIP and OpenCLIP and the unimodal contrastive image and text encoders for FLAVA.Source: §3.1 · p. 4

3.2. Evaluation setup

  • For each image, the evaluation compares its normalized embedding with normalized prompts for every dataset class and predicts the class with the highest cosine similarity.The reported accuracy is averaged over the dataset.Source: §3.2 · p. 4
  • Prompts begin with a class label or a dataset-level animal or bird description, then append image attributes using each attribute’s description and expression.The class-label version begins “a photo of a <class label>”; without a label, it begins “a photo of an animal” or “a photo of a bird.”Source: §3.2 · p. 4

4. Datasets

The study uses CUB and AWA2 attribute annotations to construct prompts and evaluate images. CUB provides image-level attributes, while AWA2 provides class-level attributes that the authors filter for each image.

4. Datasets

  • CUB contains 11,788 images across 200 bird species, with 312 binary attributes and certainty annotations for each image.The evaluation uses the full dataset and retains attributes rated probably or definitely for each image.Source: §4 · p. 4
  • CUB class attributes are represented by probabilities; the authors retain at most one attribute per description when its probability is above zero.They select the highest-probability attribute for each description.Source: §4 · p. 4
  • AWA2 contains 37,322 images across 50 animal classes with 85 binary class attributes, which are appended as a comma-separated list to prompts.Because AWA2 lacks image-level attributes, the authors select attributes per image using model-specific image–attribute similarities, retaining on average 20 per class.Source: §4 · pp. 4–5

5. VL-Taboo

VL-Taboo tests how attributes affect image–text classification in five prompt setups, both with and without class labels. It contrasts label-based prompting with attribute-only classification and examines qualitative retrieval examples.

5.1. Classification Performance with Class Label

  • With instance attributes added only to the correct-class prompt, OpenCLIP and FLAVA accuracy drops, while CLIP benefits only for a limited number of attributes before tending below baseline.The authors attribute the drops especially for FLAVA to the increased length of the instance prompt; the prompt-distance analysis finds the strongest instance-embedding shift for FLAVA.Source: §5.1 · p. 5, Figure 3 · p. 6
  • Adding the same image attributes to every class prompt stabilizes performance across all three models, with only slight deviations from the class-only baseline.This setup keeps the prompts the same length; the distance analysis reports that CLIP and OpenCLIP prompts with more attributes move closer to the image embedding, while FLAVA’s distance changes vary.Source: Figure 3 · p. 6, Figure 4 · p. 6, §5.1 · p. 7
  • Adding correct class attributes slightly improves performance for CLIP and OpenCLIP on AWA2 and CLIP on CUB, but the gains are not strong.On CUB, adversarial attributes initially outperform correct class attributes; the effect of correct attributes becomes apparent only with four or more attributes.Source: Figure 3 · p. 6, Figure 4 · p. 6, §5.1 · p. 7

5.2. Attribute-based Classification Performance

  • Without class labels, adding more attributes improves accuracy across all tested models, but attribute-only classification performs significantly worse than classification with class labels.The authors note a larger improvement for AWA2 than CUB and include an oracle upper bound based on ground-truth image attributes.Source: Figure 5 · p. 7, §5.2 · p. 8
  • In the adversarial comparison, all models prefer the correct class label with wrong attributes over a wrong label with correct attributes, though FLAVA shows a slight tendency to favor correct attributes as more are added.The figure compares the average percentage of cases choosing the correct label with wrong attributes over the wrong label with correct attributes.Source: Figure 6 · p. 7, §5.2 · p. 8

5.3. Qualitative results

  • In qualitative top-five retrieval, AWA2 examples show high probability for the correct class even under adversarial conditions, while CUB results are less clear.AWA2 retrievals also include relevant related classes as the second and third options.Source: §5.3 · p. 8, Figure 7 · p. 8

6. Conclusion

The conclusion describes an in-depth study of attribute-based zero-shot capabilities in web-scale-trained vision-language models, using two benchmark settings to examine class labels and attributes.

6. Conclusion

  • The study benchmarks attribute-annotated datasets in one setting that adds attributes to prompts with class labels and another that assesses attributes alone.The supplied conclusion passage introduces these settings but does not include the result following “It shows.”Source: §6 · p. 8

Supplementary

The supplementary document lists additional experiments, methodological details, and further results that complement the main paper.

Supplementary

  • Section A contains additional experiments with different prompt generation on AWA2.Source: Supplementary · p. 12
  • Section B gives more information about the images used for attribute selection, and Section C details counting class occurrence in LAION400M.Source: Supplementary · p. 12
  • Section D lists numeric evaluation results for comparison, while Section E provides more qualitative results.Source: Supplementary · p. 12

A. Prompt Generation on AWA2

The AWA2 comparison tests CUB-style prompts against content-based prompts, which insert attributes according to meaning and part of speech. With content-based prompts, accuracy decreases more, though FLAVA recovers with more attributes while CLIP and OpenCLIP have lower accuracy at higher attribute counts.

A. Prompt Generation on AWA2

  • CUB-style prompts use meaningful attribute descriptions for CUB and comma-separated attributes for AWA2, with the class label at the prompt's front.The authors use this format to make the prompt structures comparable across datasets.Source: A. Prompt Generation on AWA2 · p. 12
  • Content-based AWA2 prompts place attributes according to their part of speech and meaning, forming a more natural sentence around the antelope class label.For example, the prompt describes a group of active antelope that walks and is in fields, ground, or plains.Source: A. Prompt Generation on AWA2 · p. 12
  • Content-based prompts lower accuracy more; FLAVA later recovers with many attributes, while CLIP and OpenCLIP remain less accurate with many added attributes.Both prompt types use the same class label and image attributes; their structure differs.Source: A. Prompt Generation on AWA2 · p. 12

B. Number of Attributes

The evaluations vary the number of attributes, but images with too few annotations for a setup are excluded. Ignored-image rates are higher on AWA2 than CUB, reflecting differences in attribute availability and filtering.

B. Number of Attributes

  • Images with fewer annotated attributes than a setup requires are ignored, with evaluations using up to 7 attributes for CUB and 20 for AWA2.This exclusion rule applies across Setups (1)–(5).Source: B. Number of Attributes · p. 12
  • AWA2 has a higher ignored-image rate than CUB because it provides class-level rather than image-level attributes, which are filtered per image.After filtering, each AWA2 image retains about 20 attributes on average; the reported model cutoffs are 0.2, 0.24, and 0.245 for OpenCLIP, CLIP, and FLAVA, respectively.Source: B. Number of Attributes · p. 12
  • On AWA2, ignored-image rates depend on the model; on CUB, they are the same across models and setups because image attributes are model-independent and each setup uses the same attribute count.The rates are reported in Figure 9 and Tables 3–4.Source: B. Number of Attributes · p. 12

C. Counting Class Occurrence in Laion400M

The class-occurrence analysis searches LAION-400M image descriptions for benchmark class labels, treating their frequency as correlated with the true number of class-related training samples. Search-term variants are processed to reduce missed matches and matches to unrelated subwords.

C. Counting Class Occurrence in Laion400M

  • The analysis treats counts of LAION-400M descriptions containing a class label as correlated with the true number of training samples for that class.This is an assumption underlying the class-occurrence estimate.Source: C. Counting Class Occurrence in… · p. 12
  • Direct text searches add whitespace or punctuation boundaries and optional plural endings to avoid matches such as “cat” inside “catbird.”The search uses combinations of prefixes, labels, plural forms, and suffixes.Source: C. Counting Class Occurrence in… · pp. 12–13
  • An Aho–Corasick search tree processes LAION-400M descriptions to count samples matching each class label.The procedure creates 24 search terms per class label and pads description boundaries with whitespace.Source: C. Counting Class Occurrence in… · p. 13

D. Detailed Evaluation Results

The supplementary results provide exact accuracy values for all five setups on AWA2 and CUB, alongside tables documenting ignored-image rates for each dataset.

D. Detailed Evaluation Results

  • Tables 5 and 6 report exact accuracy values for all five setups on AWA2 and CUB, respectively.Source: D. Detailed Evaluation Results · p. 13
  • Table 3 reports AWA2 ignored-image rates across OpenCLIP, CLIP, and FLAVA for Setups (1)–(5).Source: Table 3 · p. 14
  • Table 4 reports CUB ignored-image rates, which are the same across models and setups because each setup uses the same number of model-independent image attributes.Source: Table 4 · p. 14

E. More Qualitative Results

The qualitative predictions show high top-1 accuracy without attributes, alongside semantically related classes among top-5 results. Across the illustrated attribute configurations, the models appear to behave similarly.

E. More Qualitative Results

  • Without attributes, all models have high top-1 accuracy but also tend to rank semantically related classes among their top-5 predictions.Examples include dog species for a collie, sea animals for a dolphin, and similar bird species for CUB images.Source: E. More Qualitative Results · p. 14, Figure 10 · p. 16
  • For CUB, top-5 predictions often include similar bird species, including subspecies of the same species or birds with similar appearances.The rufous hummingbird is given as an example of predictions including several subspecies of the same species.Source: E. More Qualitative Results · p. 14
  • Across the illustrated prompts in Setup (1) and Setup (3), models seem to behave similarly regardless of attribute configuration.Setup (1) compares image attributes plus class labels against class labels alone; Setup (3) compares image attributes plus class labels against class attributes plus class labels.Source: E. More Qualitative Results · p. 14, Figure 11 · p. 16, Figure 12 · p. 16

Related papers

Loading 2209.06103v1…