PaperlayerView exact revision on arXiv

Source-linked AI summary

What does a platypus look like? Generating customized prompts for zero-shot image classification

Sarah Pratt, Ian Covert, Rosanne Liu, Ali Farhadi

arXiv:2209.03320v3cs.CVcs.LG

TL;DR

Open-vocabulary image classification depends on natural-language prompts, but hand-written templates are costly, generic, and require dataset knowledge. CuPL combines an LLM with an open-vocabulary model to generate category-specific descriptive prompts, improving accuracy across zero-shot benchmarks, including more than a one-percentage-point ImageNet gain. The method remains zero-shot and requires no additional training or labeled data.

  • Problem

    Hand-written prompts are costly to scale, cannot express category-specific visual distinctions, and require dataset knowledge that may not generalize.

  • Method

    CuPL uses an LLM to generate customized category descriptions as image-prompts for a zero-shot open-vocabulary classifier.

  • Results

    CuPL prompts outperform hand-written templates on 15 zero-shot benchmarks, including a greater than 1 percentage point gain on ImageNet Top-1 accuracy.

  • Takeaways & Limitations

    LLM-generated prompts provide a simple, category-customized way to improve zero-shot image classification with less prompt engineering.

  • Takeaways & Limitations

    Some dataset-dependent type specification remains necessary when category names are ambiguous, such as “boxer” or “Tornado.”

Abstract

from arXiv · show

Open-vocabulary models are a promising new paradigm for image classification. Unlike traditional classification models, open-vocabulary models classify among any arbitrary set of categories specified with natural language during inference. This natural language, called "prompts", typically consists of a set of hand-written templates (e.g., "a photo of a {}") which are completed with each of the category names. This work introduces a simple method to generate higher accuracy prompts, without relying on any explicit knowledge of the task domain and with far fewer hand-constructed sentences. To achieve this, we combine open-vocabulary models with large language models (LLMs) to create Customized Prompts via Language models (CuPL, pronounced "couple"). In particular, we leverage the knowledge contained in LLMs in order to generate many descriptive sentences that contain important discriminating characteristics of the image categories. This allows the model to place a greater importance on these regions in the image when making predictions. We find that this straightforward and general approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet. Finally, this simple baseline requires no additional training and remains completely zero-shot. Code available at https://github.com/sarahpratt/CuPL.

1. Introduction

Open-vocabulary models enable zero-shot classification through natural-language prompts, but hand-written templates are costly, generic, and dependent on dataset knowledge. CuPL addresses these limitations by using an LLM to generate category-specific descriptive prompts, improving classification across benchmarks without additional training or labeled data.

  • Motivation: Open-vocabulary models classify images by matching image embeddings with prompts for arbitrary categories, without labeled training data for each task.Categories can be selected or adjusted at inference time through natural-language prompts.
  • Limitations of standard prompting: Hand-written prompt templates require substantial human effort, especially because datasets typically need different template sets.Doubling prompts for a category requires doubling the associated manual effort.
  • Limitations of standard prompting: Generic templates cannot include category-specific visual details, limiting fine-grained discrimination between visually similar classes such as frog species.The example identifies large eyes as a distinguishing feature of tree frogs that a general template cannot express.
  • Limitations of standard prompting: High-performing hand-written templates also require prior knowledge about dataset representations, which may not generalize across datasets.ImageNet-specific examples include black-and-white photos and toy representations, unlike datasets such as FGVC Aircraft.
  • CuPL: CuPL couples an LLM with a zero-shot open-vocabulary model to generate customized prompts containing discriminating visual descriptions for each category.For example, an LLM-generated tree-frog description mentions large eyes, a feature distinguishing it from similar frog classes.
  • Results: CuPL prompts outperform hand-written templates on 15 zero-shot benchmarks, including gains greater than 1 percentage point on ImageNet Top-1 and greater than 6 percentage points on DTD.The method uses fewer hand-written prompts and requires no additional training or labeled data for either model.

2. Methods

CuPL generates category-specific image prompts with an LLM and uses them in a zero-shot open-vocabulary classifier. Its design separates prompts that query the LLM from the generated prompts matched against images, while reducing hand-engineering relative to standard CLIP prompting.

  • Method overview: CuPL first generates customized prompts for dataset categories and then uses those prompts for zero-shot image classification.These are the algorithm's two stated steps.
  • Generating customized prompts: LLM-prompts cue GPT-3 to generate descriptions of dataset categories, rather than directly serving as image-classification prompts.An example asks, “What does a platypus look like?”
  • Generating customized prompts: Image-prompts describe categories and are the LLM outputs matched with images by the downstream zero-shot classifier.CuPL uses these generated descriptions as the prompts for classification.
  • Generating customized prompts: GPT-3 generates multiple image-prompts from a small set of hand-written LLM-prompt templates, with generation diversity encouraged by high temperature.For ImageNet, five LLM-prompt templates produce ten image-prompts each, or 50 customized image-prompts per category.
  • Zero-shot classification: CLIP embeds text prompts and images, averages and normalizes prompts sharing a category, and assigns each image to the category with the highest cosine similarity.This is the standard matching procedure that CuPL modifies by replacing hand-filled templates with GPT-3 outputs.
  • Prompt settings: CuPL has base and full settings that trade off accuracy against hand-engineering, with the base setting using three hand-written sentences across all examined datasets.The full setting uses dataset-specific LLM-prompt templates while retaining fewer hand-written templates than the standard method.
  • Results: CuPL full prompts improve performance on all examined datasets while requiring five ImageNet sentences instead of the baseline's 80.The reported improvements include ImageNet Top-1, DTD, SUN397, FGVC Aircraft, and Flowers 102.

3. Experiments and Results

CuPL improves zero-shot classification across benchmarks while using fewer hand-written prompts than standard prompting. Analyses show gains persist across model scales and prompt counts, and connect textual part mentions with visual importance.

  • Results: CuPL (base) improves performance on 13 of 15 datasets using three hand-constructed sentences across all datasets.The standard setting uses nearly 175 unique image-prompt templates across these datasets.
  • Results: CuPL (full prompts) improves performance on all examined datasets, including gains above one percentage point on several benchmarks.The reported benchmarks include ImageNet Top-1, DTD, SUN397, FGVC Aircraft, and Flowers 102.
  • Model Size: Larger CLIP and LLM models generally produce higher CuPL accuracy, with consistent gains across CLIP model sizes.The second- and third-largest LLMs perform similarly.
  • Number of Prompts: CuPL outperforms the baseline with one hand-written sentence and begins outperforming it with 25 image-prompts, versus 80 baseline image-prompts.The prompt-count analysis varies LLM-prompts and image-prompts separately.
  • Importance Heatmaps: CuPL prompts shift importance toward semantically relevant image regions, such as a Schipperke’s tail and a frog’s eyes.These regions correspond to descriptions appearing in the CuPL prompts.
  • Importance of Segmented Parts: Across segmented animal parts, textual mention frequency strongly correlates with visual importance under CuPL.The analysis uses WordNet tagging and PartImageNet segmentations to compare text and image part importance.

4. Related Work

Related work improves zero-shot or multimodal classification by extracting textual knowledge, generating language-model outputs, or optimizing prompt formats. CuPL differs from prior prompt-design methods by requiring no additional training or labeled data.

  • Text-Based Knowledge: Prior methods use text descriptions, attributes, or encyclopedic sources to identify visual parts and support zero-shot classification.Examples include unstructured internet text, bird descriptions, hand-collected attributes, and Wikipedia descriptions.
  • Language-Model Methods: Recent work uses language models for caption paraphrasing, commonsense knowledge generation, text labeling, multimodal representation learning, and caption refinement.These approaches apply language-model outputs to data augmentation, classification, representation learning, or image captioning.
  • Prompt Design: Prompt formats substantially affect accuracy, motivating crowdsourced prompts, explanation-oriented prompts, and learned prompt formats.Prior prompt-learning methods use supervised or unsupervised training for zero-shot image classification.

5. Conclusion

CuPL uses LLM knowledge to generate customized prompts, improving zero-shot image classification while reducing hand-engineering and dataset-specific prior knowledge.

  • CuPL leverages LLM knowledge to generate customized prompts for desired image categories.
  • Customized prompts reduce the prior dataset knowledge required when crafting prompt templates.
  • Querying an LLM for prompt construction is simple and immediately beneficial according to the reported results.

C. Evaluation Metric

CuPL uses a small set of general LLM-prompt templates, specialized by dataset type, to generate descriptive image prompts for zero-shot classification.

  • CuPL requires hand-written templates for LLM-prompts, whereas standard CLIP requires them for image-prompts.
  • CuPL templates remain general, while standard templates can encode dataset-specific information such as toy representations.
  • The base prompt setting uses three general sentences that ask what a category looks like or what identifies it.
  • Dataset specifications adapt the prompts to categories such as textures, cars, food, aircraft, flowers, actions, satellite imagery, and birds.

D. Comparison with Definitions and Wikipedia Descriptions

The paper compares CuPL-generated descriptions with WordNet and Wikipedia descriptions, and evaluates combining CuPL with standard prompts.

  • WordNet definitions produce lower ImageNet Top-1 accuracy than CuPL or standard prompts.WordNet also requires 1000 unique hand-written definitions, compared with 175 standard image-prompt templates and 45 CuPL templates.
  • Wikipedia descriptions are collected and preprocessed by selecting article sentences, truncating them to CLIP’s input limit, and handling unmatched categories.
  • Table 4 reports ImageNet Top-1 accuracy for different methods of generating image-prompts.
  • The ensemble of CuPL and standard prompts outperforms standard prompts alone on every examined dataset.It outperforms CuPL alone on 11 datasets, while CuPL alone performs better on others.

F. Additional Qualitative Results

Additional qualitative results compare the image regions emphasized by CuPL and baseline prompts when distinguishing visually similar categories.

  • CuPL prompts make the model place more importance on semantically relevant image regions than baseline prompts.This difference is observed when distinguishing between two visually similar categories.

G. Open-Source LLMs and ChatGPT

CuPL remains effective with smaller or conversational language models, while reducing hand-engineering and outperforming simple single-sentence baselines.

  • Open-source LLMs: GPT-J-6B-generated CuPL prompts surpass human-written prompts but remain below GPT-3-generated prompts.Smaller models require strategies such as lower temperature and additional prompt generation to reduce irrelevant outputs.
  • ChatGPT: ChatGPT-generated CuPL prompts outperform hand-written prompts by 0.55%.Generating all ImageNet prompts cost less than $5 USD after filtering unsuitable responses.
  • Single-sentence baselines: CuPL outperforms a single hand-written template and a single generated image-prompt in both low-effort comparisons.The comparison uses either one hand-written LLM-prompt template generating ten image-prompts or one generated image-prompt.

I. Robustness

CuPL improves accuracy on distribution-shifted ImageNet variants while preserving CLIP’s robustness, although its per-class gains are highly uneven.

  • Robustness: CuPL improves performance on two common ImageNet distribution shifts while maintaining CLIP’s robustness.The evaluation uses CuPL ImageNet image-prompts on two ImageNet variants.
  • Per-class variation: CuPL’s per-class accuracy changes range from approximately 40 percentage-point gains to approximately 40 percentage-point losses.The overall improvement is therefore not distributed uniformly across ImageNet classes.
  • Per-class variation: Large gains and losses often occur in visually similar or co-occurring class pairs, including agaric/mushroom and desk/monitor.Figure 8 lists the classes with the greatest accuracy changes relative to standard prompts.

K.2. visual similarity analysis

CuPL’s class-level gains are concentrated around visually related categories, suggesting a changed allocation of predictions among similar or co-occurring objects.

  • Visual similarity: CuPL often increases accuracy for one visually related class while decreasing accuracy for another.Examples include mushroom, graduation cap, and monitor gaining while related classes lose accuracy.
  • Visual similarity: CuPL changes the prediction balance between similar or co-occurring classes, such as canoe and paddle.The standard method prefers canoe over paddle more strongly than CuPL.
  • Co-occurrence: A CuPL canoe caption explicitly mentions a paddle, potentially capturing object co-occurrence.The example caption describes a canoe as propelled with a paddle.
  • Evaluation: ImageNet-ReaL separates images containing one applicable ImageNet label from images containing multiple applicable labels for the accuracy analysis.Accuracy is computed for both sets using standard ImageNet labels.

L. Error Analysis

CuPL errors tend to be visually closer to the ground-truth class than baseline errors, while prompt-generation strategies produce overlapping rather than fully distinct embedding distributions.

  • Visual error metric: An image offset of 1 means an incorrect prediction is the class with the closest average visual embedding to the ground-truth class.Class embeddings are computed by averaging and normalizing image embeddings, then ranking the other 999 classes by distance.
  • Linguistic error metric: A WordNet offset of 2 means the predicted and ground-truth labels share the same parent in the WordNet tree.The linguistic error analysis ranks label similarity using WordNet.
  • Error comparison: CuPL is more likely than the baseline to make errors with image offset 1, whereas the baseline is more likely to make errors with WordNet offset 1.The authors connect this difference to CuPL’s visually descriptive captions and the baseline’s lack of visual descriptions.
  • Prompt distribution: Image-prompts generated from the same LLM-prompt show slight clustering but substantial overlap in the embedding visualization.The visualization colors prompts according to their generating LLM-prompt.

N. Temperature Analysis

The analysis varies GPT-3 temperature to control image-prompt diversity and finds that more diverse prompts achieve higher ImageNet accuracy. Higher temperatures also produce increasingly different prompts and broader embedding distributions.

  • Diversity control: Changing GPT-3 temperature changes the likelihood of selecting lower-probability tokens, making generated image-prompts more diverse.
  • Accuracy effect: More diverse prompts lead to higher ImageNet accuracy, with comparisons conducted using a single LLM-prompt to reduce computational cost.
  • Prompt examples: For the Tench category, prompts generated at the lowest temperature are nearly identical, while increasing temperature increases their differences.
  • Embedding analysis: Higher-temperature image-prompts cover a wider distribution in the two-dimensional visualization of their text embeddings.
Loading 2209.03320v3…