PaperlayerView exact revision on arXiv

Source-linked AI summary

BLINK: Multimodal Large Language Models Can See but Not Perceive

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, Ranjay Krishna

arXiv:2404.12390v4cs.CVcs.AIcs.CL

TL;DR

Multimodal-LLM evaluations often mix visual perception with language knowledge and reasoning, leaving core perception abilities overlooked. Blink recasts 14 classic computer-vision tasks as image-based questions and finds that current models struggle, with GPT-4V averaging 51.26% accuracy.

  • Problem

    Existing multimodal-LLM evaluations often conflate visual perception with language knowledge and reasoning, leaving core perception abilities under-evaluated.

    Source: §1 · p. 2

  • Method

    Blink reformats 14 classic computer-vision tasks into image-based questions, with answer choices that may be images or text.

    Source: §1 · p. 3

  • Results

    Current multimodal LLMs struggle across Blink; GPT-4V achieves 51.26% average accuracy, 13.17% above random guessing.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Blink provides a testbed for multimodal LLMs to catch up with human-level visual perception.

    Source: §5 · p. 15

    Blink uses existing image datasets and does not cover all visual perception abilities in the wild.

    Source: D Limitations · p. 26

Abstract

from arXiv · show

We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans "within a blink" (e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). However, we find these perception-demanding tasks cast significant challenges for current multimodal LLMs because they resist mediation through natural language. Blink reformats 14 classic computer vision tasks into 3,807 multiple-choice questions, paired with single or multiple images and visual prompting. While humans get 95.70% accuracy on average, Blink is surprisingly challenging for existing multimodal LLMs: even the best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only 13.17% and 7.63% higher than random guessing, indicating that such perception abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also highlights that specialist CV models could solve these problems much better, suggesting potential pathways for future improvements. We believe Blink will stimulate the community to help multimodal LLMs catch up with human-level visual perception.

1 Introduction

Blink reframes classic computer-vision tasks to test visual perception that language-heavy evaluations may miss. The benchmark reveals a large gap between human and multimodal-LLM performance, while specialist vision models perform better on several tasks.

1 Introduction

  • Traditional computer-vision tasks target perception, but language-heavy evaluations conflate perception with language knowledge and reasoning, allowing caption-only GPT-4 to perform well.The introduction notes that many traditional tasks resist precise expression through language, such as pinpointing a spatial keypoint.Source: §1 · p. 2
  • Blink uses visual prompts such as circles, boxes, and masks and covers broader perceptual abilities than the recognition-focused prior benchmarks described in Figure 2.The figure contrasts Blink’s tasks, including multi-view reasoning, depth, and reflectance estimation, with prior benchmarks’ focus on recognition-based VQA.Source: Figure 2 · p. 3
  • Blink reframes 14 classic computer-vision tasks as 3.8K questions across 7.3K images, with image or text answer choices and tasks selected to resist dense-caption solutions.Questions draw on diverse datasets and may include multiple images.Source: §1 · pp. 2–3
  • Humans average 95.70% accuracy, but GPT-4V reaches 51.26% (44.44% below humans and 13.17% above random guessing).The authors conclude that multimodal LLMs’ perceptual abilities have been overestimated.Source: §1 · p. 4
  • Specialist vision models outperform GPT-4V by 62.8% on visual correspondence, 38.7% on relative depth, and 34.6% on multi-view reasoning in absolute accuracy.The authors suggest that multimodal LLMs may benefit from insights from specialized models.Source: §1 · p. 4

2 Related Work

Prior multimodal LLM work improved vision-language performance through modality-adaptation architectures and instruction tuning, while emphasizing textual reasoning over visual perception. Existing benchmarks broadened evaluation beyond single-task datasets but often focused on recognition or specialized domains rather than nuanced, holistic visual perception.

2.1 Multimodal Models

  • Multimodal LLM research advanced through modality-adaptation architectures and instruction tuning, but its reasoning studies largely emphasized text rather than visual perception.Early systems such as Flamingo and BLIP-2 [45] connected frozen visual encoders and LLMs, pre-trained on image-text matching, then fine-tuned on tasks such as VQA; LLaVA popularized instruction tuning with LLM-synthesized chat data.Source: §2.1 · p. 4

2.2 Multimodal Benchmarks

  • Holistic multimodal benchmarks expanded on single-task datasets, but their visual-perception questions largely remained recognition-focused.MME [28], MM-Vet [86], MMBench, and Seed-Bench [43] [44] added broader tasks, subjects, evaluation settings, or input types; their perception questions were mainly drawn from existing VQA datasets or generated from image descriptions and covered object recognition and OCR.Source: §2.2 · pp. 4–5
  • Other benchmarks target domain knowledge, hallucinations, illusions, or mathematical visual questions, rather than holistic assessment of nuanced human-level perception.MMMU [87] collects domain-knowledge questions, HallusionBench tests language hallucination and visual illusion, and MathVista [57] focuses on mathematical questions involving charts, tables, and diagrams; Blink instead targets multiple distinct nuanced perception abilities.Source: §2.2 · p. 5

3 The Blink Benchmark

Blink benchmarks visual perception abilities that are difficult to reduce to text-only questions, with the aim of evaluating multimodal LLMs across varied tasks and scales.

3.1 Overview of Blink

  • Blink spans 14 tasks from low-level pattern matching through mid-level spatial reasoning to high-level visual understanding, at pixel, patch, and image granularity.Its tasks are intended to probe nuanced perception beyond visual recognition.Source: §3.1 · pp. 5–6, §3 · p. 5, Figure 3 · p. 6
  • The benchmark recasts tasks as multiple-choice questions with text or image options, single or multiple images, and textual or visual prompts; it contains 3.9K questions and 7.3K images.Validation and test sets are evenly distributed, and test samples across tasks use unique images.Source: §3.1 · p. 6, §3.2 · p. 7
  • Blink adds visual prompting, perception beyond recognition, straightforward questions answerable without domain expertise, interleaved image-text questions and choices, and images from varied sources and styles.Visual prompts can highlight image regions; the collection includes in-the-wild scenes, object views, landscapes, diagrams, synthesized images, and photographs.Source: §3.1 · pp. 6–7

3.2 Dataset Collection Process

  • Tasks cover correspondence across viewpoints, lighting, or time; pixel reflectance and relative depth; true-or-false spatial relations; and left-versus-right camera motion from object views.They draw on HPatches, IIW, Depth in the Wild, Visual Spatial Reasoning, and Wild6D, respectively.Source: §3.2 · pp. 7–9
  • Other tasks test central-patch jigsaw completion, art-style matching, fine-grained object-box selection, counting in complex scenes, and distinguishing real photographs from generated images.The tasks use TARA, WikiArt, LVIS, TallyQA, and Stable Diffusion XL-generated images paired with photographs, respectively.Source: §3.2 · pp. 9–10
  • The remaining tasks evaluate graphical IQ-style pattern reasoning, visual similarity, semantic correspondence between object parts, and functional correspondence between points with similar affordances.Blink manually filters collected data to remove ambiguous examples.Source: §3.2 · pp. 10–11

4 Experiments

The experiments evaluate 16 recent multimodal LLMs on Blink, finding low overall accuracy despite relative strengths on some task types, and examine image-caption reduction, visual-prompt sensitivity, and errors.

4.1 Experimental Setup

  • The evaluation covers 16 recent multimodal LLMs, uses VLMEvalKit settings (temperature 0; 10 retries), and leaves images unresized.For models without multiple-image input, images are concatenated; answer choices are extracted using predefined rules and GPT-3.5-turbo.Source: §4.1 · p. 11

4.2 Main Results

  • 51.26%, 45.72%, and 44.11% are GPT-4V, Gemini Pro, and Claude 3 OPUS accuracies, respectively, only 13.17%, 7.63%, and 6.02% above random guessing.Their performance trails human performance by 44.44%, 49.98%, and 51.59%, respectively.Source: §4.2 · p. 11
  • 35–42% is the mean accuracy range for 7B and 13B open-source models, versus 38.09% random guessing; LLaVA-v1.6-34B reaches 45.05%.Some multimodal LLMs also fall below random guessing on jigsaw, semantic correspondence, multi-view reasoning, object localization, and relative reflectance.Source: §4.2 · p. 11
  • LLaVA-v1.6-34B, Gemini Pro [71], and GPT-4V perform relatively better than random guessing on spatial reasoning, art style, and counting.The analysis also examines reducing images to captions, sensitivity to visual prompts, and errors.Source: §4.2 · p. 11, §4 · p. 11

2 More details are at the official website at https://www.01.ai/

Blink models show uneven visual-perception performance, and the analyses find that captions, visual prompts, and specialist models expose important strengths and limitations. Human accuracy remains much higher, while GPT-4V’s errors often involve fine details and visual grounding.

2 More details are at the official website at https://www.01.ai/

  • Models show some capability in relative depth and forensics detection, but are described overall as doing relatively well on mid-level perception tasks.Source: §2 · p. 12

4.3 Analysis

  • Red circles outperform gray circles across relative depth, relative reflectance, and visual correspondence, while the best circle size varies by task; 10px is best on average.The evaluation used 100 validation samples per task and circle radii of 5px, 10px, and 15px; 10px circles were used throughout the paper’s evaluations.Source: Figure 7 · p. 13, §4.3 · p. 14
  • Caption + LLM improves MMBench accuracy over GPT-4V by 5.7%, reaches 47.2% on MMMU—9.6% below GPT-4V—and performs at random-guessing level on Blink.Dense captions cover the visual information needed for MMBench and carry a large portion of the information needed for MMMU. Blink’s performance decrease suggests a need for advanced perceptual abilities beyond what general captions currently provide.Source: §4.3 · p. 14

2 More details are at the official website at https://www.01.ai/

  • GPT-4V performs relatively better than its counterparts on visual similarity, art style, jigsaw, and multi-view reasoning, including 29% better on visual similarity than Gemini Pro.Gemini Pro and LLaVA show similar performance patterns.Source: §2 · p. 13
  • Humans average 95.70% accuracy and exceed 95% on most tasks, while models generally do better on image-level than pixel-level and crop-level tasks.The reported human IQ-test score was annotated by the authors and may not reflect typical human performance, which can vary.Source: §2 · p. 13, §4.3 · p. 13

4.3 Analysis

  • Specialist models outperform the best multimodal LLM by 18% to 57%; DepthAnything and DIFT reach human-level performance on depth estimation and visual correspondence.The results suggest that appropriate data and training strategies, such as distilling specialist models into multimodal LLMs [40], could help improve performance.Source: Table 2 · p. 14, §4.3 · p. 14
  • In 140 sampled GPT-4V errors, hallucinated fine-grained patterns and attributes account for 24.2%, and incorrect visual-prompt locations account for 20.0%.Other reported errors are failures to capture overall setting or style (8.6%) and failures to ground an object (5.7%).Source: §4.3 · p. 15

5 Conclusion … A.3 Evaluation Prompts

Blink tests core visual perception abilities that existing evaluations miss, revealing a large gap between current multimodal LLMs and human performance. Its analyses and evaluation materials describe possible improvement directions and benchmark procedures.

5 Conclusion

  • Around 50% accuracy for GPT-4V and Gemini on Blink falls far below human performance of 95.7%.Source: §5 · p. 15
  • Specialist computer vision models perform much better than GPT-4V and Gemini, and Blink offers a testbed for progress toward human-level visual perception.Source: §5 · p. 15

A.1 Visual Prompts Details

  • Blink uses circles, boxes, and masks; correspondence circles have 10px radii on images resized to 1024px height, while reflectance circles are white and localization boxes red and green.Source: §A.1 · p. 21
  • Figure 8 shows the prompts used to extract option labels from model predictions.Source: Figure 8 · p. 21, §A.3 · p. 22

A.2 Spatial Relation Curation Process

  • Spatial-relation samples come from Visual Spatial Reasoning and are reformatted from true-or-false claims into binary questions using GPT-3.5.Source: §A.2 · p. 21

A.3 Evaluation Prompts

  • Following MMBench, evaluation first matches option labels exactly, then uses GPT-3.5 to align unmatched predictions to an option or label them “Z”.Source: §A.3 · pp. 21–22

A.4 Human Evaluation Protocol

Human performance is measured using two coauthors per Blink task, with their scores averaged, while IQ-test scores are based on a separate sample.

A.4 Human Evaluation Protocol

  • Two coauthors evaluate each Blink task, and their average scores are reported as human performance.Source: §A.4 · p. 22
  • Human agreement ranges from 80–99%, while the IQ-test score comes from two coauthors tested on 100 samples, 50 each, because average human performance is difficult to control or represent for that task.Agreement is lowest for art style and functional correspondence, and highest for relative depth, object localization, and forensics detection.Source: §A.4 · p. 22

A.5 Dataset Statistics

This section presents detailed statistics for the Blink benchmark.

B Baseline Models · C Analysis

The section defines a comparison set of 16 multimodal LLMs, generally using each family’s latest and best-performing available checkpoint, and describes their variants, architectures, scales, and test dates.

B Baseline Models

  • The evaluation covers 16 multimodal LLMs and generally uses each model family’s latest and best-performing available checkpoint.The section lists the baseline models and their configurations.Source: B Baseline Models · p. 22
  • MiniGPT-4-v2 [17] combines EVA, LLaMA2-chat (7B) [75], and a linear projection, while OpenFlamingo [5] uses a 9B CLIP [64]/MPT-7B configuration.OpenFlamingo is described as an open-source alternative to Flamingo.Source: B Baseline Models · p. 22
  • InstructBLIP is evaluated at 7B and 13B scales, using BLIP-2 [45], CLIP, visual instruction tuning, and Vicuna.Source: B Baseline Models · p. 23
  • The LLaVa comparison includes InternLM2-Chat-7B and xTuner variants alongside original-paper checkpoints, all using CLIP.The v1.6 checkpoint uses more reasoning, OCR, and knowledge-enhanced training data than the v1.5 checkpoints.Source: B Baseline Models · p. 23
  • Yi-VL-6B and Yi-VL-34B use a LLaVa-style CLIP encoder with Yi-6B-Chat or Yi-34B-Chat, while CogVLM adds a trainable visual expert module.CogVLM uses EVA-CLIP [69] and Vicuna; Yi-VL models are noted for strong performance on existing benchmarks.Source: B Baseline Models · p. 23
  • The evaluated multimodal baselines included Qwen-VL-MAX [7], Gemini 1.0 Pro Vision [71], and Claude 3 OPUS; Qwen-VL-MAX was the best checkpoint, and OPUS the most powerful Claude 3 version.Source: B Baseline Models · p. 23
  • GPT-4 is tested with GPT-4V(ision), GPT-4 Turbo, and GPT-4o checkpoints; results may change with updates, and tests occurred in March and May 2024.GPT-4V was tested in March 2024; GPT-4 Turbo and GPT-4o were tested in May 2024.Source: B Baseline Models · p. 23

C.1 Validation Set Results · C.2 How to deal with multiple-image inputs?

The appendix reports task-level validation results and tests concatenating multiple images as a single-image format. Concatenation generally reduces performance, but its effects vary by model and task.

C.1 Validation Set Results

  • Table 4 provides detailed scores for each task on the validation set.Its caption identifies the results as being for different models and notes that the first row lists task names and instance counts.Source: §C.1 · p. 23, Table 4 · p. 24

C.2 How to deal with multiple-image inputs?

  • The experiment concatenated multiple images horizontally with black margins between them, then evaluated GPT-4V and Gemini Pro using the resulting single image.Source: §C.2 · pp. 23–24
  • Only GPT-4V and Gemini Pro among the 16 baseline models accept multiple-image inputs; eight of Blink’s 14 tasks require them.Source: §C.2 · p. 23
  • GPT-4V’s performance declined on every task with concatenated images, with the largest decrease on jigsaw and the smallest on multi-view reasoning.The comparison is reported for the Blink validation set.Source: §C.2 · pp. 24–25, Table 5 · p. 25
  • Gemini Pro’s performance decreased on most tasks with concatenated images, increased on semantic correspondence, and remained unchanged on multi-view reasoning.The comparison is reported for the Blink validation set.Source: §C.2 · p. 25, Table 5 · p. 25

C.3 Error analysis · C.4 Does self-consistency help?

The analysis finds that multimodal LLMs share vision-related errors, while GPT-4V’s self-consistency score is higher than its average single-run score under temperature 1.0. The error analysis identifies several recurring failure types, and notes that temperature-related refusals affect the comparison.

C.3 Error analysis

  • LLaVA-v1.5-7B shared 899 mistakes with 13B and 959 with InternLM2-7B, versus 782 with QwenVLMax and 655 with GPT4V, suggesting errors track vision encoders more than size or language components.Source: §C.3 · p. 25
  • Among 140 sampled GPT-4V errors, 28.5% involved detailed small regions or edges, 20% missed circled points, and 14.3% misread spatial relations.Source: §C.3 · p. 25
  • Reasoning errors accounted for 12.9% of the 140 sampled GPT-4V errors.Source: §C.3 · p. 25
  • Other sampled GPT-4V errors included missing the overall scene impression (8.6%), rejecting to answer (6.4%), and failing to ground or infer question-mentioned items (5.7%).Source: §C.3 · p. 25

C.4 Does self-consistency help?

  • 48.38% self-consistency [78] score averaged across tasks exceeded GPT-4V’s 38.15% average single-run score at temperature 1.0.Source: §C.4 · p. 26
  • At temperature 1.0, GPT-4V’s performance fell from its default temperature=0 level, which the authors attribute to firewall-triggered refusal replies.Source: §C.4 · p. 26

D Limitations

Blink uses existing image datasets and does not cover every visual perception ability encountered in the wild; its forensics images were collected from public online sources, with copyright concerns acknowledged.

  • Blink draws on existing image datasets and does not cover all visual perception abilities in the wild.Source: D Limitations · p. 26
  • For the forensics detection task, the authors manually collected publicly available images through online search.Source: D Limitations · p. 26
  • The authors report efforts to comply with copyright laws and provide proper credit, and invite rights holders to raise licensing concerns.They commit to addressing legitimate concerns promptly.Source: D Limitations · p. 26

Related papers

Loading 2404.12390v4…