PaperlayerView exact revision on arXiv

Source-linked AI summary

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, Saining Xie

arXiv:2401.06209v2cs.CV

TL;DR

The paper investigates systematic visual shortcomings in MLLMs that may stem from CLIP-based visual representations rather than language reasoning alone. It identifies CLIP-blind pairs, builds the MMVP benchmark, and finds widespread failures on straightforward visual questions, while Mixture-of-Features improves visual grounding. The findings indicate that scaling CLIP-based models alone is insufficient and that visual representation learning remains an open challenge.

  • Problem

    MLLMs still show systematic and elementary visual shortcomings, raising whether their errors originate in visual modality, language understanding, or alignment.

  • Method

    The paper identifies CLIP-blind pairs, constructs the MMVP benchmark from them, analyzes nine visual patterns, and integrates DINOv2 features with CLIP through Mixture-of-Features.

  • Results

    MLLMs struggle with straightforward visual questions, seven of nine visual patterns resist large-scale CLIP-based models, and Interleaved-MoF enhances visual grounding while maintaining instruction following.

  • Takeaways & Limitations

    CLIP vision encoders can bottleneck multimodal systems, and scaling model size or training data alone does not rectify their visual deficiencies.

Abstract

from arXiv · show

Is vision good enough for language? Recent advancements in multimodal models primarily stem from the powerful reasoning abilities of large language models (LLMs). However, the visual component typically depends only on the instance-level contrastive language-image pre-training (CLIP). Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning. We identify ''CLIP-blind pairs'' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns, often providing incorrect answers and hallucinated explanations. We further evaluate various CLIP-based vision-and-language models and found a notable correlation between visual patterns that challenge CLIP models and those problematic for multimodal LLMs. As an initial effort to address these issues, we propose a Mixture of Features (MoF) approach, demonstrating that integrating vision self-supervised learning features with MLLMs can significantly enhance their visual grounding capabilities. Together, our research suggests visual representation learning remains an open challenge, and accurate visual grounding is crucial for future successful multimodal systems.

1. Introduction

The paper argues that elementary visual shortcomings in MLLMs may originate in their visual representations, particularly CLIP-based encoders. It introduces CLIP-blind pairs and MMVP to expose systematic failures, then explores combining CLIP with vision-only features.

  • Motivation: MLLMs still exhibit surprisingly elementary visual shortcomings despite strong language and multimodal-task capabilities.The paper asks whether these errors arise from vision, language understanding, or modality alignment.
  • Motivation: Because most open-source MLLMs use CLIP as their visual encoder, limitations in pretrained vision models may cascade into downstream failures.The paper frames this as a natural hypothesis based on analogous findings for pretrained text encoders.
  • Approach: CLIP-blind pairs are visually different images with similar CLIP embeddings but different DINOv2 embeddings.The paper uses vision-only self-supervised representations to identify likely ambiguities in CLIP’s visual embedding space.
  • Benchmark: MMVP uses CLIP-blind pairs to ask straightforward visual questions and evaluates MLLMs only when both questions for a pair are answered correctly.The benchmark covers differences identified through manual inspection of paired images.
  • Findings: Seven of nine identified visual patterns remain unresolved by large-scale CLIP-based models, and challenging CLIP patterns strongly correlate with MLLM failures.The findings indicate that scaling model size or training data alone is insufficient and that CLIP encoders can bottleneck these systems.
  • Mitigation: Mixture-of-Features methods integrate CLIP and DINOv2 representations, with Interleaved-MoF improving visual grounding while maintaining instruction-following capabilities.Additive-MoF indicates stronger grounding from DINOv2 features but diminished instruction following; Interleaved-MoF addresses that trade-off.

Benchmark

The MMVP benchmark is constructed from CLIP-blind image pairs and straightforward questions targeting overlooked visual details. Human accuracy is high, while MLLMs frequently struggle, including on basic visual-grounding questions.

  • Construction: The benchmark construction identifies CLIP-blind pairs, designs questions about their visual differences, and evaluates MLLMs on paired questions.A pair receives credit only when both associated questions are answered correctly.
  • Construction: The pairs are collected from ImageNet and LAION-Aesthetics using CLIP-ViT-L-14 and DINOv2-ViT-L-14 embeddings.Selected pairs have CLIP cosine similarity above 0.95 and DINOv2 cosine similarity below 0.6.
  • Benchmark Design: MMVP contains 150 image pairs and 300 intentionally straightforward, unambiguous questions targeting details overlooked by CLIP.Examples probe distinctions such as whether a dog faces left or right.
  • Evaluation: The evaluation covers open-source and closed-source MLLMs, queries each question independently, and compares model performance with a randomized human user study.The paired-scoring rule requires both questions associated with an image pair to be answered correctly.
  • Results: 95.7%: human participants’ average question accuracy confirms that the benchmark questions are straightforward.The result is reported from the user study shown in Figure 4.
  • Results: Models except GPT-4V and Gemini score below the 25% random-guess level, while GPT-4V and Gemini still struggle with basic visual grounding.The reported outcomes indicate difficulty with visual details across model sizes and training-data scales.
  • Analysis: An ablation study supports visual incapability, rather than language-model hallucination, as the source of the poor performance.The study swaps answer options and changes question notations.

3. Systematic Failures in CLIP

The paper identifies nine systematic visual patterns that challenge CLIP models, using MMVP-VLM to test whether scaling resolves these failures and Figure 6 to relate them to MLLM performance.

  • Pattern identification: The authors transform MMVP questions and options into language-based descriptors to categorize otherwise abstract CLIP-blind visual patterns.GPT-4 is prompted to generalize visual features that embedding models struggle with.
  • Pattern identification: The nine patterns include orientation and direction, presence of specific features, state and condition, quantity and count, positional and relational context, color and appearance, structural and physical characteristics, text, and viewpoint and perspective.These categories are represented by 15 text-image pairs per pattern in MMVP-VLM.
  • Benchmark: MMVP-VLM evaluates CLIP models by testing whether they correctly match both images with their corresponding text descriptions across balanced visual-pattern subsets.The benchmark distills MMVP questions into simpler descriptions and adds questions when needed to maintain 15 pairs per pattern.
  • Scaling limits: Scaling training data and network size helps mainly with color and appearance and state and condition, while the remaining visual patterns challenge all tested CLIP-based models.Increasing resolution yields minimal improvement, and ImageNet-1k zero-shot accuracy does not definitively predict visual-pattern performance.
  • Impact on MLLMs: When CLIP underperforms on a visual pattern, MLLMs tend to show similar shortcomings, with LLaVA 1.5 and InstructBLIP each exceeding 0.7 Pearson correlation.The reported correlation supports transfer of visual-pattern weaknesses from CLIP encoders to downstream MLLMs.

4. Mixture-of-Features (MoF) for MLLM

The paper studies whether combining CLIP and vision-only self-supervised features can improve MLLM visual grounding. Additive mixing reveals a grounding–instruction-following trade-off, while interleaving improves grounding without compromising instruction following.

  • Setup: The study uses LLaVA with CLIP-ViT-L-14 and DINOv2-ViT-L-14 to compare visual encoders under consistent training and fine-tuning settings.LLaVA connects the pretrained vision encoder to the language model through an adapter.
  • Additive MoF: Additive MoF linearly mixes CLIP and DINOv2 features before the adapter, with α controlling CLIP’s share and 1 −α controlling DINOv2’s share.The approach transitions from 100% CLIP to 100% DINOv2 across several feature proportions.
  • Additive MoF: Increasing DINOv2 features improves visual grounding but reduces instruction-following ability, with a sharp decline when the DINOv2 proportion reaches 87.5%.Grounding gains diminish beyond a DINOv2 proportion of 0.75, creating a trade-off between the two capabilities.
  • Interleaved MoF: Interleaved MoF separately processes CLIP and DINOv2 embeddings through adapters, then spatially interleaves their visual tokens while preserving spatial order.The interleaved features are subsequently passed to the language model.
  • Interleaved MoF: Interleaved MoF increases MMVP visual grounding by 10.7% under the LLaVA setting without compromising instruction following.The improvement is replicated across LLaVA-1.5 and image-resolution settings, with consistent gains also reported on POPE.
  • Interleaved MoF: Interleaving, rather than simply increasing image resolution and token count, is reported to improve visual grounding by combining vision-only SSL and vision-language features.Similar improvements are reported with MAE and MoCoV3, as well as on MMBench and GQA.

5. Related Works

Prior MLLM work emphasizes pretrained language and CLIP vision components, adapters, and training data, while existing benchmarks target broader VQA, hallucination, reasoning, or robustness properties. The paper positions systematic study of visual encoders as an underexplored gap.

  • Multimodal LLMs: MLLMs commonly connect pretrained CLIP vision encoders and large language models through adapters such as MLPs, Q-Former, or gated attention.Recent work also highlights the importance of high-quality training data.
  • Evaluating MLLMs: MMVP evaluates simple visual questions constructed from CLIP-blind pairs, complementing benchmarks centered on traditional VQA, hallucination, reasoning, and robustness.The paper identifies limited research on how visual encoders affect MLLM performance.
  • Representation evaluation: Linear probing, fine-tuning, and ImageNet zero-shot accuracy provide basic representation assessments, but the paper reports a growing detachment from recent multimodal use cases.The MoF experiments are presented as evidence that vision-and-language and vision-only SSL models learn complementary capabilities.

6. Discussion

The discussion argues that CLIP vision encoders can bottleneck MLLMs because they overlook crucial visual details, and that scaling alone does not resolve these deficiencies. MoF can alleviate limitations, but broader evaluation and representation learning remain necessary.

  • Discussion: MLLMs can fail simple questions because pretrained CLIP encoders overlook crucial visual details and systematically mishandle important visual patterns.The discussion frames vision representation as a bottleneck in multimodal systems.
  • Discussion: Scaling data and model size alone does not rectify the inherent deficiencies identified in CLIP models.This conclusion contrasts the persistence of visual-pattern failures with the continued scalability and widespread use of CLIP-type models.
  • Discussion: Vision-and-language and vision-only self-supervised models excel in different aspects beyond conventional linear-probing and ImageNet zero-shot evaluations.A carefully designed MoF approach can alleviate visual limitations by using strengths from both learning paradigms.
  • Discussion: The paper concludes that new evaluation methods and continued development of visual representation learning are necessary for accurate visual grounding in multimodal systems.The conclusion presents accurate visual grounding as important for future multimodal systems.

A. Experiment Details

The experiments use established LLaVA training settings, datasets, and model-access procedures, while supplementing the benchmark with additional examples and evaluations.

  • The experiments use the same hyperparameter sets as LLaVA and LLaVA-1.5, with Vicuna-13B variants and up to 8 Nvidia A100 GPUs.
  • LLaVA experiments use CC595k for stage 1 and LLaVA 158k instruction data for stage 2.
  • LLaVA-1.5 experiments use CC595k for stage 1 and DataMix 665k for stage 2.
  • Additional MMVP examples are provided in Figures 10, 11, and 12.
  • The study evaluates models accessed through official checkpoints, APIs, websites, playgrounds, and chat interfaces across dates from September to December 2023.

B.3. Ablation Studies

Ablation studies test whether MMVP errors arise from visual grounding rather than response formatting or language-model hallucination, finding largely unchanged GPT-4V performance under altered options.

  • Option swapping: The option-swapping ablation changes answer order while preserving the underlying VQA question and alternatives.For example, the alternatives “Open” and “Closed” are reversed.
  • Option swapping: GPT-4V achieves 40.3% accuracy after swapping answer options, compared with 38.7% on the original benchmark.Most questions receive the same answer after swapping, suggesting the visual encoder is implicated rather than alignment or the LLM.
  • Notation changes: Changing option labels from letters to numbers yields 37.3% performance, closely matching the original 38.7%.The result further suggests that the core challenge is visual incapability rather than language-model hallucination.
  • Human evaluation: Four participants provide responses whose average score is used as human-level performance.The study uses a labeling interface shown in Figure 8.

C. CLIP-MLLM Failure Correlation

The study compares CLIP and MLLM errors and relates MMVP performance to ImageNet-1k accuracy, showing that CLIP-linked failures and benchmark-specific visual patterns are not fully captured by standard classification accuracy.

  • CLIP–MLLM correlation: LLaVA and InstructBLIP each show Pearson correlations above 0.7 between CLIP-model and MLLM errors.Bard also displays a very high correlation, indicating that multiple MLLMs share CLIP-related failure patterns.
  • ImageNet-1k comparison: Below 80 ImageNet-1k zero-shot accuracy, higher classification accuracy tends to accompany better MMVP performance.This trend does not necessarily hold for models with higher ImageNet-1k performance.
  • User study: The user-study interface randomly shuffles questions and collects both VQA answers and potential concerns.
  • Benchmark scope: MMVP-VLM evaluates visual patterns such as orientation that ImageNet accuracy alone does not capture.The benchmark therefore probes visual details relevant to downstream tasks beyond standard ImageNet classification.
  • Visual patterns: The benchmark organizes challenging examples around patterns including orientation, feature presence, state, and viewpoint.

E. More Benchmark Results

Additional backbone and benchmark experiments show that MoF variants improve visual-grounding evaluations while retaining broadly comparable performance on other benchmarks.

  • Different vision-only backbones: MAE- and MoCoV3-based MoF consistently improve visual grounding on the MMVP and POPE benchmarks.
  • Different vision-only backbones: Table 6 reports Interleaved MoF results using different vision-only self-supervised learning models.
  • Expanded evaluation: Interleaved-MoF at 336 resolution achieves comparable performance on most benchmarks and improves benchmarks focused on visual grounding.
  • Expanded evaluation: Table 7 compares LLaVA-1.5 with Interleaved-MoF LLaVA-1.5 across six additional benchmarks.The listed evaluations include LLaVA Benchmark, LLaVA-in-the-Wild, MMBench, TextVQA, POPE, VQA-v2, and MM-Vet.
  • MMVP examples: Figures 10–12 provide further examples of MMVP benchmark questions.
Loading 2401.06209v2…