PaperlayerView exact revision on arXiv

Source-linked AI summary

Why are Visually-Grounded Language Models Bad at Image Classification?

Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, Serena Yeung-Levy

arXiv:2405.18415v2cs.CVcs.AIcs.CLcs.LG

TL;DR

VLMs underperform CLIP on image classification, prompting an investigation of inference, training, and data factors. The paper finds that classification data is central to this gap and tests adding it to VLM training, reporting improved classification and ImageWikiQA performance.

  • Problem

    Public and proprietary VLMs underperform CLIP on image classification despite often using CLIP as a vision encoder.

    Source: §1 · p. 1

  • Method

    The authors test inference, training, and data hypotheses, then integrate classification-focused datasets into VLM training.

    Source: §1 · p. 2

  • Results

    Classification data is the primary factor identified in VLM classification performance, and ImageNet-fine-tuned VLMs outperform pre-trained VLMs on ImageWikiQA by 11.8%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that classification-focused training can improve both VLM object recognition and performance on more complex object-related questions.

    Source: §1 · p. 2

    Computational constraints limited evaluation to LLaVA and BLIP on four datasets, and decoding latent classification information required large-scale training.

    Source: Limitations · p. 14

Abstract

from arXiv · show

Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, we explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is data-related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMs can match the accuracy of state-of-the-art classification models. Based on these findings, we enhance a VLM by integrating classification-focused datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers to its general capabilities, resulting in an improvement of 11.8% on the newly collected ImageWikiQA dataset.

1 Introduction

Although VLMs can use CLIP vision encoders, they underperform CLIP on image classification. The paper attributes this gap primarily to data and finds that adding classification data can also improve VLM performance on more complex visual questions.

1 Introduction

  • Class presence during VLM training strongly correlates with performance on those classes, and classification-dataset training reaches state-of-the-art classifier performance.The paper tests hypotheses about inference, training, and data.Source: §1 · p. 2
  • On ImageWikiQA, ImageNet-fine-tuned VLMs outperform pre-trained VLMs by 11.8% on questions about ImageNet objects.ImageWikiQA contains complex real-world questions about those objects.Source: §1 · p. 2
  • Across ten VLMs and four benchmarks, VLMs significantly lag behind CLIP in image classification.The evaluation covers public and proprietary models.Source: §1 · pp. 1–2
  • The analysis identifies lack of alignment data, rather than information loss or the training objective, as the primary reason for VLMs’ classification underperformance.The authors report that classification information is encoded in VLM latent spaces but requires sufficient training data to decode.Source: §1 · p. 2
  • Adding classification-focused datasets to VLM training improves object recognition and overall performance.The authors describe classification capability as foundational for complex object-related reasoning.Source: §1 · p. 2

2 VLMs are Bad at Image Classification

The section evaluates VLM classification against CLIP models on standard benchmarks and finds that VLMs substantially underperform, especially on fine-grained datasets. It also describes the model selection, benchmark data, and evaluation protocols used for the comparison.

2.1 Models

  • The comparison uses two state-of-the-art classifiers, CLIP-L and EVA-G, which also serve as vision encoders for the LLaVA and BLIP series, respectively.The selected VLMs include three proprietary and seven public models.Source: §2.1 · p. 3

2.2 Data

  • The four benchmark datasets span coarse- and fine-grained object categories.The reported test-set sizes are 50,000, 6,149, 8,041, and 4,331 images, with 1,000, 102, 196, and 101 classes, respectively.Source: §2.2 · p. 3

2.3 Evaluation Protocol

  • VLMs are evaluated in open-world and closed-world settings, with success defined by whether generation includes the ground-truth label.CLIP is evaluated only in the closed-world setting using the class with the highest image-text cosine similarity.Source: §2.3 · p. 4

2.4 Results

  • On ImageNet, GPT4 achieves 60.6% closed-world accuracy versus EVA-G’s 79.2%, while LLaVANeXT-M7B achieves 32.3% open-world accuracy; the gap is larger on fine-grained datasets.All LLaVA models use CLIP-L as their vision encoder, yet have at least 20 times its parameter count and underperform it.Source: §2.4 · p. 4
  • Closed-world evaluation often outperforms open-world evaluation, but supplying all class names can create costly contexts or exceed VLM context limits.Most public VLMs support only 4K context, which cannot accommodate all 1,000 ImageNet classes.Source: §2.4 · p. 4

3 Why are VLMs Bad Image Classifiers?

VLMs retain classification information and can learn to decode it, but inference changes alone do not close their gap with CLIP. The analyses identify class exposure in training as the primary factor, while finding that caption-focused data can work similarly to classification-focused data.

3 Why are VLMs Bad Image Classifiers?

  • The analyses find that class exposure is the primary cause of poor classification performance, and that increasing multimodal data quantity linearly boosts performance.The paper states multimodal data is essential for alignment; it also reports that VLMs can match CLIP when trained with enough data.Source: §3 · p. 4, §3.3 · p. 8

3.1 Inference

  • Inference changes improve or narrow the gap but do not close it: probabilistic inference with classifier-free guidance reaches 47.6% on ImageNet versus 74.8% for CLIP-L.The probabilistic approach requires computing each class’s probability, making it computationally expensive in practice.Source: Table 2 · p. 5, §3.1 · p. 6

3.2 Training

  • 77.1% probing accuracy for LLaVA1.5-7B on ImageNet, versus 85.2% for CLIP-L, indicates classification information is largely preserved but not effectively decoded.LLaVA’s best reported ImageNet accuracy is 47.6%.Source: §3.2 · p. 6, Table 3 · p. 6
  • 85.7% ImageNet accuracy for fine-tuned LLaVA1.5-7B matches 85.2% for fine-tuned CLIP-L, showing text-generation training can eliminate the VLM–CLIP gap.The reported result holds across the VLMs and datasets evaluated.Source: Table 3 · p. 6, §3.2 · p. 7

3.3 Data

  • 82.7% accuracy on ImageNet classes with over 10,000 training occurrences contrasts with 3.0% for classes with fewer than 10; class frequency and performance correlate strongly (Spearman 0.76).The correlation disappears after ImageNet classification fine-tuning, and CLIP-L shows no corresponding correlation with LLaVA training-data frequency.Source: §3.3 · p. 7, Figure 3 · p. 7
  • Caption-focused and classification-focused fine-tuning perform similarly on Flowers102, StanfordCars, and Caltech101, with at least 48.6% accuracy gains over the non-fine-tuned model.The finding suggests that data containing the class name can suffice; classification-specific data is not required for these evaluations.Source: §3.3 · p. 8, Table 4 · p. 8

4 Improving VLM with Classification Data

The paper adds classification-focused data to VLM training and introduces ImageWikiQA to test whether better object recognition transfers to knowledge-intensive visual questions. Combined classification and instruction-tuning data improves ImageNet and ImageWikiQA performance, while classification-only fine-tuning lowers ImageWikiQA accuracy.

4.1 Motivation

  • The paper integrates classification-focused data into VLM training to improve both classification accuracy and general capabilities.The motivation is that object recognition supports more advanced visual tasks, such as answering questions about an identified object.Source: §4 · p. 8, §4.1 · p. 8

4.2 ImageWikiQA

  • Existing VQA benchmarks emphasize reasoning and knowledge grounding, while classification datasets lack advanced questions; ImageWikiQA is designed to bridge this evaluation gap.Its questions require identifying the specific pictured class to answer correctly.Source: §4.2 · p. 9
  • ImageWikiQA contains 2,000 four-option questions linking ImageNet images to knowledge about their object classes, with 25.0% random-guess accuracy.Questions are generated from Wikipedia pages, and four annotators answered 96.5% accurately when given the ground-truth class name and Wikipedia page.Source: §4.2 · p. 9

4.3 Results

  • On ImageWikiQA, GPT-4 reaches 100% accuracy when given the ground-truth class name but 61.2% with images; Claude3 and GeminiPro reach 54.3% and 49.1%, respectively.The authors interpret these results as evidence that poor classification limits more advanced capabilities.Source: §4.3 · p. 9, Table 5 · p. 9
  • 11.8% accuracy boost on ImageWikiQA follows fine-tuning LLaVA1.5-7B on ImageNet classification data combined with its original instruction-tuning data.The combined-data model reaches 84.4% on ImageNet, compared with 22.8% for the non-fine-tuned model; ImageNet-only fine-tuning yields 30.6% on ImageWikiQA.Source: §4.3 · p. 10

5 Related Work

The paper situates its work among research on VLM architectures, training, data, and components, and revisits image classification as a foundation for advanced visual capabilities.

Related work

  • VLMs combine a visual encoder, language model, and projector, and are commonly trained on image-text captioning and instruction-tuning data.The paper evaluates widely used VLMs and analyzes LLaVA and BLIP architectures.Source: §5 · p. 10
  • Prior studies examine VLM limitations across architectures, training, data, encoders, language models, and input resolution; this work investigates classification performance and integrates classification-focused data into training.Tong et al. [45] identify limitations in some CLIP vision encoders and suggest alternatives such as DINO.Source: §5 · p. 10
  • The paper revisits classification, a task advanced by datasets such as ImageNet, and argues that classification remains foundational to advanced visual intelligence.It reports that improving VLM classification capabilities improves overall performance.Source: §5 · p. 10

6 Conclusion

The paper evaluates VLMs as image classifiers, analyzes their limitations, and trains a VLM with enhanced general capabilities.

Conclusion

  • VLM performance as image classifiers is limited across various datasets.Source: §6 · p. 10
  • The paper analyzes reasons for VLM classification limitations.Source: §6 · p. 10
  • Based on its analysis, the paper trains a VLM with enhanced general capabilities.Source: §6 · p. 10

Broader Impacts and Ethics Statement

The authors caution that race- or gender-based classification data could amplify discriminatory biases and emphasize responsible use of their approach.

Broader Impacts and Ethics Statement

  • Race- or gender-based classification data could amplify discriminatory biases in VLMs.The authors emphasize ethical and responsible use of their approach.Source: Broader Impacts and Ethics Statement · p. 14
  • The approach is presented as potentially useful for real-world applications, including virtual assistants for visually impaired individuals.The passage also reports improved classification accuracy and general capabilities after adding classification data to VLM training.Source: Broader Impacts and Ethics Statement · p. 14
  • The authors call for ethical and responsible use to ensure the approach contributes positively.Source: Broader Impacts and Ethics Statement · p. 14

Reproducibility Statement

The authors release an open-source implementation and ImageWikiQA dataset to support reproducing the experiments and conducting further analyses.

Reproducibility Statement

Limitations

Computational constraints limited the evaluated models and datasets; the authors identify zero-shot decoding of latent classification information as a future direction.

Limitations

  • The authors could not evaluate all possible VLMs and datasets because of computational constraints.Source: Limitations · p. 14
  • Their analyses selected LLaVA and BLIP as representative models and covered four datasets.Source: Limitations · p. 14
  • The authors propose zero-shot decoding of latent classification information without extensive training as a promising future direction.Source: Limitations · p. 14

Compute Resources

The appendix reports implementation and evaluation details, including label-set analyses, compute costs, and feature-probing procedures. Across smaller label sets, the VLM–CLIP gap narrows in absolute terms but grows relatively.

Compute Resources

  • Fine-tuning LLaVA1.5-7B on 1.28M ImageNet examples takes 1.4 days per epoch on four L40S GPUs, or 3.5 days with 665K instruction-tuning examples added.The 13B model's cost is approximately 40% higher.Source: Compute Resources · p. 14

Summary of Appendix

  • The appendix maps sections A–D to model and data details, experimental settings and results, ImageWikiQA details, and a contribution summary.Tables 6 and 7 are identified as model-details and data-details tables, respectively.Source: Summary of Appendix · p. 14, §A.1 · p. 14

A.3 Synonymous Text Labels in ImageNet Evaluation

  • On ImageNet, accepting any listed label synonym as correct improves accuracy by only 1%–3%, leaving a notable gap to CLIP.The evaluation accounts for classes with multiple valid textual labels.Source: §A.3 · p. 15, Table 8 · p. 15

B.1 Prompt Variation Detail

  • The prompt-variation appendix lists three object-identification prompts; for fixed label order, it uses each image's default dataset label order.Figure 4 samples 100, 20, 5, or 2 candidate classes per image and shows the relative error-rate trend between LLaVA1.5-7B and CLIP-L.Source: §B.1 · p. 15, Figure 4 · p. 16

B.2 Label Set Size Details

  • In two-way ImageNet classification, VLMs have a 5.7% error rate versus CLIP's 0.2%, a 28.5x gap; for 20 classes, the rates are 18.0% versus 2.5%, a 7.2x gap.Figure 4 plots the relative error rate between LLaVA1.5-7B and CLIP-L against the number of classes.Source: Figure 4 · p. 16, §B.2 · p. 16
  • Reducing the label set narrows the absolute VLM–CLIP gap, but a gap remains even in two-way classification.The absolute gap is larger on fine-grained Flowers and Cars than on ImageNet and Caltech as label sets shrink; the passage attributes this possibly to their finer-grained classes and VLMs' weaker fine-grained performance.Source: §B.2 · p. 16

B.3 Inference Strategy Details

  • Probabilistic inference on ImageNet's 1,000 classes takes 1,000 times longer per example than direct generation, so evaluation sampled 1,000 validation examples per dataset.The appendix also reports that, on Flowers102 with LLaVA1.5-7B, probing the last or average token outperforms other token positions.Source: §B.3 · p. 16, Table 10 · p. 17

B.4 Information Loss Details

  • The authors argue that CLIP's second-to-last-layer local patch features contain at least as much information as its last-layer global feature.For probing, features are extracted from the VLM's last layer using default prompts; linear probes use a 512-example batch, 1e-3 learning rate, Adam, and 500 epochs, with best validation performance reported.Source: §B.4 · p. 17

B.5 Training Objective Detail

The analyses examine fine-tuning stability, class-frequency relationships, inference settings, and how classification information becomes expressible. They also detail ImageWikiQA and report improved performance when classification and instruction-tuning data are combined.

B.5 Training Objective Detail

  • Joint projector-and-LM LoRA fine-tuning produces loss spikes; ImageNet fails to recover despite hyperparameter changes, unlike occasional normalization on StanfordCars.Source: §B.5 · p. 18, Figure 5 · p. 18

B.6 Data Detail

  • Class-frequency correlations are similar for pre-training and instruction tuning; ImageNet Spearman/Pearson values are 0.73/0.34 and 0.76/0.33, respectively.Class frequencies are computed from tokenized training data and compared with per-class accuracy; balanced ImageNet probing yields zero correlation.Source: §B.6 · p. 19, §B.12 · p. 21, Figure 7 · p. 21

B.7 Analysis of Proprietary VLMs

  • GPT-4 achieves 60.6% accuracy without CoT and 62.2% with CoT; proprietary training and data hypotheses cannot be directly tested because those details are unavailable.Source: §B.7 · p. 19

B.8 Open-world vs. Closed-world Setting

  • Open-world prompts omit labels and closed-world prompts include them; random versus fixed label order tests whether candidate ordering affects performance.Source: §B.8 · p. 20, §B.9 · p. 20

B.10 Why Do VLMs Have Information but Cannot Decode?

  • VLMs contain classification information, but added classification data helps make it expressible.Source: §B.10 · p. 20

B.11 Why Does Fine-tuning Only on ImageNet Get Worse Performance on ImageWikiQA?

  • 49.8 versus 38.0: combining ImageNet classification with instruction-tuning data improves ImageWikiQA over the pretrained model.Source: §B.11 · p. 21

C.1 ImageWikiQA Details

  • ImageWikiQA pairs image-specific questions with ImageNet validation images; fine-tuning combines 665K instructions with 1.28M classification examples and trains only the projector.Source: §C.1 · p. 21, §C.2 · p. 21

C.3 Performance of Fine-tuned VLM on Other Datasets

The authors assess whether classification-focused fine-tuning preserves performance on other benchmarks and examine limits of task-specific alternatives. They report near-identical results across three benchmarks and distinguish image recognition from world-knowledge demands.

C.3 Performance of Fine-tuned VLM on Other Datasets

  • Fine-tuned LLaVA1.5-7B scores 58.0% on TextVQA versus 58.2% originally, improves POPE Popular and Adverse by 0.2% and 0.3%, and remains at 31.1% on MMVet.Source: §C.3 · pp. 22–23, Table 13 · p. 23

C.4 Low Accuracy on ImageWikiQA with Ground-truth Class Names

  • With ground-truth class names, LLaVA1.5-7B reaches 55.9% on ImageWikiQA, which the authors attribute to limited world knowledge for ImageNet classes.Source: §C.4 · p. 23

C.5 Other Fine-Tuning and Inference Strategies

  • KNN-based inference targets classification alone, while adding a token and linear-probing loss did not improve ImageWikiQA; the authors favor adding data while retaining general capabilities.Source: §C.5 · p. 23

D Summary of Contributions

The authors identify weak image classification as an overlooked VLM limitation, investigate its causes, and test a data-based improvement. They report that adding classification data improves both classification and broader object-related capabilities.

D Summary of Contributions

  • Public and proprietary VLMs lag CLIP on four common classification benchmarks, despite some using CLIP as their vision encoder.The authors describe this performance gap as a weakness not widely noted in previous work [2] [49].Source: D Summary of Contributions · p. 23
  • Tests of inference, training, and data hypotheses identify lack of alignment data as the major issue, rather than lost visual information or the text-generation objective.Source: D Summary of Contributions · pp. 23–24
  • Adding classification data improves VLM classification and general capabilities, supporting object recognition as foundational for complex questions about objects.The authors note that PaliGemma [6] also reports benefits from longer pre-training with more data.Source: D Summary of Contributions · p. 24
Loading 2405.18415v2…