PaperlayerView exact revision on arXiv

Source-linked AI summary

Waffling around for Performance: Visual Classification with Random Words and Broad Concepts

Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, Zeynep Akata

arXiv:2306.07282v2cs.CVcs.LG

TL;DR

LLM-generated fine-grained descriptors can improve zero-shot vision-language classification, but their diverse, sometimes weakly visual or ambiguous content leaves the source of the gains unclear. The paper introduces WaffleCLIP, which replaces those descriptors with random words or characters, and studies high-level concepts as an alternative when LLMs are available. Across eleven benchmarks, WaffleCLIP achieves comparable or better results than fine-grained GPT-3 descriptors, while high-level concepts better exploit semantics and help address class ambiguity.

  • Problem

    Although LLM-generated descriptors can improve zero-shot classification, their diversity, limited visual relevance, and ambiguity leave the source of their gains unclear.

    Source: §1 · p. 1

  • Method

    WaffleCLIP replaces LLM-generated fine-grained descriptors with random word or character descriptors, and the paper also proposes adding automatically generated high-level concepts when LLMs are available.

    Source: §1 · pp. 1–2

  • Results

    Across eleven visual classification benchmarks, WaffleCLIP without external LLMs achieves comparable or better results than fine-grained GPT-3 descriptors.

    Source: §5 · p. 9

  • Takeaways & Limitations

    High-level concepts better exploit available semantics and can help address class ambiguity, while WaffleCLIP provides a lower-cost alternative and sanity check for methods using external queries.

    Source: §1 · p. 2, §5 · p. 9

    DCLIP is detrimental on very fine-grained benchmarks such as Stanford Cars, losing 1.46% against CLIP.

    Source: §4.5 · p. 7

Abstract

from arXiv · show

The visual classification performance of vision-language models such as CLIP has been shown to benefit from additional semantic knowledge from large language models (LLMs) such as GPT-3. In particular, averaging over LLM-generated class descriptors, e.g. "waffle, which has a round shape", can notably improve generalization performance. In this work, we critically study this behavior and propose WaffleCLIP, a framework for zero-shot visual classification which simply replaces LLM-generated descriptors with random character and word descriptors. Without querying external models, we achieve comparable performance gains on a large number of visual classification tasks. This allows WaffleCLIP to both serve as a low-cost alternative, as well as a sanity check for any future LLM-based vision-language model extensions. We conduct an extensive experimental study on the impact and shortcomings of additional semantics introduced with LLM-generated descriptors, and showcase how - if available - semantic context is better leveraged by querying LLMs for high-level concepts, which we show can be done to jointly resolve potential class name ambiguities. Code is available here: https://github.com/ExplainableML/WaffleCLIP.

1. Introduction

LLM-generated class descriptors can improve zero-shot classification, but their visual relevance and class interpretations vary, leaving the source of their gains unclear. The paper introduces random-descriptor WaffleCLIP and examines high-level concepts as a more useful way to add semantic context.

1. Introduction

  • LLM-generated descriptors can be diverse, weakly visual, or ambiguous, making the source of their performance gains unclear.Examples include potentially conflicting attributes such as “steamed” and “fried,” nonvisual attributes, and interpretations of “Peking duck” as an animal rather than food.Source: §1 · p. 1
  • WaffleCLIP replaces fine-grained LLM descriptors with random words or character lists, retaining zero-shot use without external LLM queries.The paper also positions it as a sanity check for methods that rely on external model queries.Source: §1 · pp. 1–2
  • The paper studies whether LLM-descriptor gains reflect added semantics or instead structured noise ensembling.It reports that the gains are not fully driven by additional semantics.Source: §1 · p. 2
  • When semantic context is available, coarse high-level concepts are a better way to introduce it and can jointly address class-name ambiguity.The paper proposes automatically generating these concepts with GPT-3.Source: §1 · p. 2
  • Across benchmarks, random descriptors achieve comparable zero-shot classification performance to LLM-generated descriptors at lower cost.The paper presents WaffleCLIP as a low-cost alternative and a sanity check for future methods using external queries.Source: §1 · p. 2

2. Related Work

Related work covers prompt tuning and external knowledge for improving vision-language classification, alongside other sources of class descriptions. It also distinguishes WaffleCLIP’s use of prompt noise for zero-shot classification from noise augmentation used to prevent training overfitting.

Related approaches

  • Prompt-tuning work adapts the text encoder, visual encoder, or both, while other approaches use synthetic images or LLM-generated prompts and semantic hierarchies [67] [66] [33] [54] [1] [7] [61] [32] [62] [56] [2] [46] [38] [42].Fine-grained LLM descriptors have also been used to enhance accuracy and provide descriptor-weight interpretability [36].Source: §2 · p. 2

Other description sources

  • Class descriptions also come from manual crafting or external resources such as Wikipedia, WordNet, and ImageNet-Wiki [48] [16] [45] [39] [11] [52] [49] [6].The paper contrasts these knowledge sources with random-character and random-word prompts that do not query external image-generation methods.Source: §2 · p. 2

Noise augmentation

  • WaffleCLIP uses character- and word-level prompt noise for zero-shot classification, unlike prior noise augmentation aimed at preventing overfitting during training [3] [41].The paper reports that randomized descriptors can match the gains of fine-grained LLM descriptors and argues that available semantics are better integrated through high-level context.Source: §2 · p. 2

3. Method

The method builds from class-descriptor retrieval in CLIP to WaffleCLIP’s randomized prompt variants, then proposes high-level concepts when LLM access is available. Its motivating experiment finds that shared random descriptors can match or exceed DCLIP, while concept prompts target class ambiguity.

3.1. Image classification with class descriptors

  • CLIP classifies by nearest-neighbor retrieval from image and language embeddings using a prompt built from each class name.The standard prompt is “A photo of a {c}.”Source: §3.1 · p. 2
  • DCLIP replaces single-class matching with an average over a class’s fine-grained descriptor representations, requiring external LLM-generated descriptors.The table compares CLIP, DCLIP, and shared random descriptor sets sized at 1× or 2× the average DCLIP descriptor count.Source: §3.1 · p. 3, §3.2 · p. 3, Table 1 · p. 3

3.2.1 Motivation

  • Assigning the same randomly sampled descriptor set to every class produces performance close to DCLIP, suggesting descriptor averaging is a key driver of gains.The study uses a set whose size equals the average number of DCLIP descriptors per class, then tests a set twice as large.Source: §3.2.1 · p. 3
  • 58.56% versus 58.16% in ViT-B/32 total average, and a 0.83% Food101 improvement, show shared random descriptors closely match or sometimes outperform DCLIP.Increasing the shared random descriptor set from 1× to 2× moves ViT-B/32 from 58.16% to 58.29%.Source: §3.2.1 · p. 3, Table 1 · p. 3

3.2.2 WaffleCLIP Method

  • WaffleCLIP fills each class descriptor set with class-independent random words or character lists, using class-label word and character averages to set prompt lengths.The random sequences are appended in the descriptor-prompt structure used by DCLIP.Source: §3.2.2 · p. 3

3.3. Better semantics and reduced ambiguity via high-level concepts

  • When LLMs are available, the method queries GPT-3 for a shared high-level concept across class names and adds it to the prompt, helping resolve context-dependent label ambiguity.Non-specific outputs such as “Object” or “Thing” are filtered out; the concept is omitted on three of eleven benchmarks.Source: §3.3 · pp. 3–4, Figure 2 · p. 4

4. Experiments

The experiments compare WaffleCLIP with LLM-generated descriptors and examine how descriptor averaging, semantic guidance, and randomization affect classification. They find comparable performance from randomized descriptors, complementary gains from high-level concepts, and limits to replacing class-conditioned descriptors with latent noise.

4.2. WaffleCLIP vs LLM-generated descriptors

  • 58.56% using DCLIP versus 58.57% using WaffleCLIP with ViT-B/32; randomized descriptors match or exceed LLM-generated descriptors on most studied datasets.WaffleCLIP also has comparable reported averages with ViT-L/14 and ResNet50.Source: Table 2 · p. 5, §4.2 · p. 5

4.3.1 Impact of Descriptor Averaging

  • Selecting the maximum descriptor similarity instead of averaging lowers performance, supporting descriptor ensembling rather than fine-grained semantics as the main driver of gains.In some cases, maximum selection even underperforms the underlying CLIP baseline.Source: §4.3.1 · p. 5, Table 3 · p. 6
  • 58.56% to 55.03% for ViT-B/32 when descriptor lists are interchanged; performance approaches DCLIP as descriptors are randomized more independently.The progression tests interchanged lists, shuffled words, and random sampling from the available descriptors.Source: §4.3.1 · p. 6, Table 4 · p. 7

4.3.2 Structural differences between LLM-generated and randomized descriptors

  • DCLIP flips more predictions than WaffleCLIP, but their effects are complementary: adding GPT descriptors raises ViT-B/32 performance from 58.57% to 60.21%.DCLIP’s greater flip rate holds even on examples where WaffleCLIP’s accuracy is higher.Source: §4.3.2 · p. 6, Figure 3 · p. 6

4.4. Semantic guidance with high-level concepts

  • High-level concepts improve performance across most benchmarks and backbones, with especially clear gains for ambiguous or generic labels.For CLIP with ViT-B/32, accuracy rises from 40.78% to 48.86%; swapping concepts generally reduces performance, while similarity for selected ambiguous samples increases by up to 13%.Source: §4.4 · pp. 6–7

4.6. Comparison to prompt ensembles

  • WaffleCLIP outperforms handcrafted prompt ensembling on eight of eleven benchmarks, and adding concepts to prompt ensembling raises average accuracy from 55.58% to 56.94%.Prompt ensembling still requires human input and prompt design.Source: Table 7 · p. 8, §4.6 · p. 8

4.5. Evaluation on additional (OOD) benchmarks.

  • WaffleCLIP and concept guidance also gain on additional benchmarks; on ImageNet-A, WaffleCLIP improves 29.63% to 31.52%, while DCLIP loses 1.46% against CLIP on Stanford Cars.On Stanford Cars, WaffleCLIP with concepts reaches 59.70% from 58.54% through WaffleCLIP alone.Source: §4.5 · pp. 7–8

4.7. Comparison to latent noise; 4.8.1 Dependence on descriptor counts; 4.8.2 Impact of randomization types

  • Latent-space noise does not reproduce WaffleCLIP’s gains; competitive performance begins with 4–15 descriptor pairs, and combining words with characters gives the most transferable average improvement.Increasing latent noise continuously reduces performance, while even one randomized descriptor pair consistently outperforms CLIP.Source: §4.7 · p. 9, §4.8.1 · p. 9, §4.8.2 · p. 9

5. Conclusion

WaffleCLIP replaces LLM-generated descriptors with randomized words and characters, achieving comparable or better results across eleven visual classification benchmarks without external LLMs. Joint use of randomized words and characters gives the most consistent improvements, and gains appear with few descriptor pairs.

5. Conclusion

  • Joint randomized word and character sequences provide the most consistent performance improvements across benchmarks and backbones.Source: Table 8 · p. 9
  • Four randomized descriptor pairs yield consistent competitive gains, while one pair already outperforms the CLIP reference.Source: Figure 6 · p. 9
  • Across eleven visual classification benchmarks, WaffleCLIP achieves comparable or better results than fine-grained GPT-3 descriptors.Source: §5 · p. 9

SUPPLEMENTARY MATERIAL: Waffling around for Performance: Visual Classification with Random Words and Broad Concepts

The supplementary material extends the main paper's results to additional backbones and provides GPT-generated descriptors for additional benchmarks, along with example images.

SUPPLEMENTARY MATERIAL: Waffling around for Performance: Visual Classification with Random Words and Broad Concepts

A. Additional results

Additional experiments extend the paper's findings to ViT-L/14 and ResNet50, supporting randomized descriptors' competitiveness and their complementarity with LLM-generated descriptors.

A. Additional results

  • On ViT-L/14 and ResNet50, random descriptor assignment avoids a significant performance drop and matches DCLIP.Source: A. Additional results · p. 13
  • WaffleCLIP matches LLM-descriptor approaches such as DCLIP without external LLM access; randomized and LLM-generated descriptors are complementary, and high-level concepts add gains.Source: A. Additional results · p. 13
  • On ViT-L/14 and ResNet50, moving from systematic semantic shifts to independently subsampled descriptors can recover and even beat DCLIP performance.Source: A. Additional results · p. 13

B. Exemplary GPT-3 generated descriptors for additional benchmarks

The supplement gives four example descriptors for random classes in each of three datasets added beyond those used in [36].

B. Exemplary GPT-3 generated descriptors for additional benchmarks

Flowers102

The descriptor examples characterize flowers through petal shape and number, color, and center details.

Flowers102

  • Flower descriptors cover petal count and shape, including five-petaled and star-shaped forms.Source: Flowers102 · p. 13
  • Other examples describe flower colors and centers, including a yellow center, vibrant colors, and a dark center with disk florets.Source: Flowers102 · p. 13
  • The examples also include a balloon-like shape, large bright yellow petals, and narrow leaves tapered to a point.Source: Flowers102 · p. 13

FGVCAircraft

Additional-backbone results report that WaffleCLIP matches DCLIP, with complementary gains from combining descriptors and using high-level concepts.

FGVCAircraft

  • The additional descriptor examples range from color and body shape to cabin capacity, wing design, engine type, and tail configuration.Source: FGVCAircraft · p. 13
  • The motivational experiments are extended to ViT-L/14 and ResNet50 backbones.Source: Table 9 · p. 14
  • WaffleCLIP matches DCLIP with ViT-L/14 and ResNet50, while combining randomized and GPT descriptors is complementary and adding high-level concepts provides further gains.Source: Table 10 · p. 14

Stanford Cars

The additional-backbone study reports that independently subsampled descriptors can recover DCLIP performance after an initial drop; the examples also include car-specific visual details.

Stanford Cars

  • Car descriptors mention exterior color, an Acura logo, distinctive headlights, chrome accents, BMW grille styling, and round headlights and taillights.Source: Stanford Cars · p. 14
  • The examples include a Honda Odyssey Minivan 2012 and descriptors of four doors, a hatchback, and wide round headlights.Source: Stanford Cars · p. 14
  • With ViT-L/14 and ResNet50, independently subsampled descriptors recover DCLIP performance after an initial drop from systematic semantic shifting.Source: Table 11 · p. 15
Loading 2306.07282v2…