PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Classification via Description from Large Language Models

Sachit Menon, Carl Vondrick

arXiv:2210.07183v2cs.CVcs.LG

TL;DR

Standard VLM zero-shot classification uses only category names, limiting feature-level explanations and criteria control. This paper instead classifies through LLM-generated visual descriptors, improving recognition while supporting interpretability, novel-category adaptation, and bias mitigation.

  • Problem

    VLM zero-shot classification uses category names without leveraging descriptive visual features or providing intermediate explanations and criteria control.

  • Method

    The framework uses large language models to generate category descriptors and compares images with those descriptors instead of directly with category names.

  • Results

    The method improves recognition across datasets and distribution shifts, achieves approximately 4–5% higher top-1 ImageNet accuracy, and recognizes categories unseen during training.

  • Takeaways & Limitations

    Descriptor-based classification provides interpretable decisions while improving recognition, adapting to new concepts, and supporting descriptor edits for bias mitigation.

  • Takeaways & Limitations

    Descriptor generation can fail when language models produce nonvisual features or misinterpret ambiguous category names.

Abstract

from arXiv · show

Vision-language models (VLMs) such as CLIP have shown promising performance on a variety of recognition tasks using the standard zero-shot classification procedure -- computing similarity between the query image and the embedded words for each category. By only using the category name, they neglect to make use of the rich context of additional information that language affords. The procedure gives no intermediate understanding of why a category is chosen, and furthermore provides no mechanism for adjusting the criteria used towards this decision. We present an alternative framework for classification with VLMs, which we call classification by description. We ask VLMs to check for descriptive features rather than broad categories: to find a tiger, look for its stripes; its claws; and more. By basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used. In the process, we can get a clear idea of what features the model uses to construct its decision; it gains some level of inherent explainability. We query large language models (e.g., GPT-3) for these descriptors to obtain them in a scalable way. Extensive experiments show our framework has numerous advantages past interpretability. We show improvements in accuracy on ImageNet across distribution shifts; demonstrate the ability to adapt VLMs to recognize concepts unseen during training; and illustrate how descriptors can be edited to effectively mitigate bias compared to the baseline.

1 INTRODUCTION

The paper proposes classification by description, an interpretable alternative to VLM zero-shot classification that checks visual descriptors rather than only category names. LLMs provide these descriptors scalably, enabling cue control without additional training or substantial inference overhead.

  • Motivation: People can naturally describe category features and use verbal descriptions to aid perception, whereas leveraging such schemata for machine perception remains challenging.This gap motivates using descriptive visual features in recognition.
  • Motivation: Standard VLM zero-shot classification compares an image with embedded category words, but provides no intermediate explanation and lacks a clear way to supply human-relevant cues.The method is motivated by unreasonable mistakes and failures to use cues people can readily identify.
  • Classification by description: Classification by description uses language as an interpretable bottleneck, checking descriptors such as a hen’s beak and feathers instead of querying only its category name.The framework allows comparison with arbitrary words when particular visual features should guide recognition.
  • Descriptor generation: LLMs such as GPT-3 generate class descriptors at scale, avoiding the cost of hand-writing features for many categories.The paper treats LLMs as queryable, noisy knowledge bases containing broad world knowledge.
  • Advantages: The proposed alternative requires no additional training or substantial inference-time computational overhead while providing inherent interpretability through the descriptors used for each decision.A prediction can be linked to a feature such as tiger stripes rather than an unrelated feature such as its tail.

2 METHOD

The method replaces category-name matching with classification by description: language models generate visual descriptors, and vision-language models ground those descriptors to score categories. Additive descriptor evidence makes predictions interpretable and editable by revealing which features support or oppose each category.

  • Descriptor-based representation: Each visual category c is represented by a natural-language phrase and a dictionary D(c) of descriptive visual features rather than only its category name.Descriptors can cover colors, shapes, parts, counts, relationships, and category-specific cues.
  • Descriptor scoring: The category score averages additive descriptor evidence, rewarding categories whose descriptors strongly match the image while allowing some descriptors to be absent.The descriptor contribution φ(d, x) is the log probability that descriptor d pertains to image x.
  • Descriptor generation: A prompted large language model such as GPT-3 automatically constructs each category’s descriptor dictionary by describing features that distinguish the category in a photograph.The generated answer list becomes D(c).
  • Visual grounding: CLIP similarity grounds the generated descriptors visually, using class-conditioned text such as “{category_name} which has {descriptor}” to form φ.Class conditioning handles descriptors whose meaning depends on the category, such as relative tail length.
  • Prediction and interpretability: The model selects the category with the highest score, making its prediction explainable by construction through the descriptors activated or not activated in the image.The same mechanism indicates why competing categories were rejected.

3 EXPERIMENTS

Experiments show that classification by description improves ImageNet recognition, supports classifiers for post-training concepts, and enables descriptor-based bias correction while providing explanations. The approach nevertheless has failure modes in descriptor generation and recognition, including modality errors, word ambiguity, syntactic repetition, and unexpected CLIP activations.

  • Explainable object recognition: Over 3% average ImageNet improvement over CLIP is achieved without training on ImageNet, while explanations accompany classification decisions.The experiments compare against CLIP across multiple domains and report improved benchmark performance despite the interpretability mechanism.
  • Explainable object recognition: The model correctly identifies an airliner that CLIP misclassifies as an albatross by checking for airplane-related features and rejecting bird-like features.The descriptor-based decision is grounded in the absence or presence of visual features associated with the competing categories.
  • Novel categories: 100% recall is obtained for both Ever Given and Wordle in the top ten, versus 10% and 0% for CLIP, respectively.GPT-3 can construct descriptors for these post-February-2021 concepts because they have been visually described sufficiently online.
  • Bias correction: Significantly higher accuracy than CLIP on cultural variations of wedding photographs shows that editing descriptors can correct bias toward inclusive recognition.The linguistic representation exposes bias sources and permits manual intervention through altered clothing attributes.
  • Failure modes: GPT-3 descriptor creation fails through nonvisual modality attributes, ambiguous category meanings, and rare syntactic errors such as repeated descriptors.Ambiguity often resolves correctly through the most common meaning, but text alone cannot always determine the intended visual category.
  • Failure modes: CLIP descriptor recognition can retrieve unintended categories when descriptors match multiple categories or when language-reading abilities bias visual matching toward stock imagery.The reported examples include descriptors that activate outside the intended category and jaguar descriptions that favor stock photos over visually stocky cats.

4 RELATED WORK

The work builds on vision-language models, interpretable prototype-based classification, and zero-shot attribute-based recognition. It uses text prototypes to leverage language-model knowledge while enabling interpretable and editable visual classification.

  • Vision-language models: CLIP established vision-language models as a dominant paradigm for visual recognition, with strong zero-shot performance across benchmarks and distribution shifts.ALIGN, FLAVA, Florence, and other models have further advanced this paradigm.
  • Prototype learning: The approach relates to prototype learning, particularly methods that make visual classification decisions interpretable through comparisons with prototypical training examples.It is described as having marked differences from prior prototype-learning approaches.
  • Text prototypes: Text prototypes leverage world knowledge condensed in large language models such as GPT-3, avoiding costly learning processes while remaining interpretable and easy to edit.The passage contrasts editing text prototypes with editing visual data.
  • External text knowledge: Related research shows that external text knowledge can enhance vision-language prompts and connect large language models to vision-language models for capabilities such as image captioning.Examples include K-LITE and work enabling emergent multimodal capabilities.
  • Attribute-based classification: The work is also related to zero-shot attribute-based classification, including systems that construct nameable attributes, connect attributes to classes, or use text-derived word embeddings.These approaches apply external textual knowledge to zero-shot visual recognition.

5 CONCLUSION

The paper introduces a zero-shot classification framework that uses large language models to generate textual descriptors for visual categories and compares images against those descriptors rather than category names. Using GPT-3 and CLIP, the framework provides interpretable model decisions and shows promising performance improvements.

  • Framework: The framework generates textual descriptors for each visual category using linguistic knowledge from large language models.These descriptors serve as feature-level cues for zero-shot classification.
  • Framework: It classifies images by comparing them with category descriptors instead of directly measuring similarity to category names.This changes the decision process from broad category matching to descriptor-based comparison.
  • Results: Experiments with GPT-3 and CLIP show promising results for interpretable model decisions and improved performance.The supplied conclusion passage identifies interpretability and performance improvement as demonstrated capabilities of the framework.

6 ETHICS

GPT-3 and CLIP learn biases from Internet data, and systems combining them may reproduce, exacerbate, or compound these biases. The proposed interpretable model can help reveal biases that might otherwise remain unknown.

  • 6 ETHICS: GPT-3 and CLIP can encode biases relating to race, culture, and gender, which systems using them may reproduce, exacerbate, or compound.The passage attributes these biases to Internet training data and warns that using both models together may compound their effects.
  • 6 ETHICS: The proposed interpretable model has the potential to shed light on biases that could otherwise remain unknown.Interpretability is presented as a way to make such biases more visible.

A PROMPTING THE LANGUAGE MODEL

The method prompts the language model to produce discrete visual descriptors in a reliably parseable bulleted format rather than embedding category information together. It uses a fixed Q:/A: prompt with sampling settings and optional examples, without tuning prompts after initial descriptor generation.

  • Prompt structure: The prompt asks for useful visual features distinguishing a category in a photo and requests the answer as a separate list of descriptors.This structure is designed to avoid requiring human parsing of an embedded response.
  • Prompt structure: Adding a trailing ‘-’ typically elicits a bulleted list whose descriptors can be extracted automatically by removing hyphens.The formatting directly supports scalable descriptor collection.
  • Implementation: The implementation uses OpenAI’s Q:/A: structure with text-davinci-002, temperature 0.7, and maximum token length 100.These settings follow OpenAI API recommendations for the query and desired response.
  • Implementation: Providing one or two examples of the desired output makes list formatting more reliable.The paper illustrates this with a lemur example containing descriptors such as a long tail, large eyes, and furry bodies.
  • Limitation: The authors did not tune the prompts after initially generating descriptors for the 1000 ImageNet classes.They note that further prompting effort could improve the procedure.

B DATASET DETAILS

The evaluation spans eight datasets covering everyday objects, distribution shifts, fine-grained birds, satellite imagery, scenes, food, animals, and textures. It also examines how language-model size affects descriptor-based performance across VLM scales.

  • Datasets: Experiments use eight datasets spanning object recognition, distribution shift, fine-grained birds, satellite images, scenes, food, animals, and textures.The datasets are ImageNet, ImageNetV2, CUB, EuroSAT, Places365, Food101, Oxford Pets, and Describable Textures.
  • Language-model scaling: Smaller OPT language models can hurt performance, whereas descriptor-based gains emerge beyond a certain model size across VLM sizes.The authors hypothesize that larger language models possess more knowledge of the visual world.

C FURTHER COMPARISON

Additional comparisons find that replacing GPT-3 descriptors with WordNet or Wiktionary information generally does not improve performance. Wiktionary tends to hurt, while WordNet has mixed effects, likely because both resources emphasize definitions over visual descriptions.

  • Auxiliary information comparison: Wiktionary information tends to hurt performance relative to GPT-3 descriptors.The comparison uses auxiliary information from WordNet and Wiktionary instead of GPT-3.
  • Auxiliary information comparison: WordNet sometimes slightly helps and sometimes slightly hurts performance.The mixed effect is attributed to WordNet information concerning definitions more often than visual descriptions.

D EDITABILITY/BIAS MITIGATION EXPERIMENT DETAILS

The bias-mitigation experiment evaluates wedding recognition across four cultural traditions by collecting licensed images and editing white-wedding descriptors with cross-cultural analogs. It adds “wedding” to ImageNet and forms five subgroups from the edited descriptors.

  • Data collection: The evaluation collects 10 images for each of four wedding traditions: Western Africa, China, Japan, and Northern India.Flickr supplies the images except for Western African examples, which are collected using Google Images.
  • Evaluation setup: The experiment adds “wedding” to the 1000 ImageNet categories to evaluate recognition performance.
  • Descriptor editing: Descriptors are edited by identifying cross-cultural analogs and replacing Western-specific words, such as “tuxedo” → “dashiki” for the Western African example.The edited descriptors correspond to the original GPT-3-constructed descriptors of a white wedding.
  • Evaluation setup: The resulting evaluation contains 5 subgroups, including the four additional cultures and the original Western wedding subgroup.

E TOP DESCRIPTOR ACTIVATIONS

Figure 9 presents the top retrievals for various descriptors.

  • Figure 9 shows the top retrievals associated with various descriptors.
Loading 2210.07183v2…