Source-linked AI summary
BLINK: Multimodal Large Language Models Can See but Not Perceive
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, Ranjay Krishna
TL;DR
Existing multimodal evaluations often emphasize recognition and language reasoning rather than core visual perception. Blink reformats 14 classic computer vision tasks into a multimodal benchmark, and shows that current models remain far below humans while specialist vision models perform much better. The benchmark therefore provides a testbed for studying and improving multimodal visual perception.
Problem
Existing multimodal benchmarks provide limited evaluation of core visual perception because they often conflate perception with language knowledge and reasoning.
Method
Blink reformats 14 classic computer vision tasks into 3.8K multiple-choice questions paired with 7.3K images, including visual prompting and single- or multiple-image inputs.
Results
51.26% GPT-4V accuracy remains far below 95.70% human accuracy, while specialist vision models perform much better on Blink.
Takeaways & Limitations
Blink indicates that multimodal LLM perceptual abilities have been overestimated and offers a testbed for integrating insights from specialist vision models.
Takeaways & Limitations
Human IQ-test scores are annotated by the authors and may not reflect typical human performance, which is expected to vary.
Abstract
from arXiv · showhide
We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans "within a blink" (e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). However, we find these perception-demanding tasks cast significant challenges for current multimodal LLMs because they resist mediation through natural language. Blink reformats 14 classic computer vision tasks into 3,807 multiple-choice questions, paired with single or multiple images and visual prompting. While humans get 95.70% accuracy on average, Blink is surprisingly challenging for existing multimodal LLMs: even the best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only 13.17% and 7.63% higher than random guessing, indicating that such perception abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also highlights that specialist CV models could solve these problems much better, suggesting potential pathways for future improvements. We believe Blink will stimulate the community to help multimodal LLMs catch up with human-level visual perception.
1 Introduction
Blink targets visual perception abilities that existing multimodal benchmarks often conflate with language knowledge and reasoning. It reformats classic computer vision tasks for multimodal evaluation and finds a large human–model performance gap.
- Motivation: Existing multimodal benchmarks often emphasize natural-language recognition and reasoning, conflating visual perception with language knowledge.The paper contrasts this focus with earlier computer-vision tasks involving reflectance, 3D reasoning, depth, correspondence, affordance, and forensics.
- Benchmark: Blink reimagines 14 classic computer vision tasks as multimodal question-answering problems requiring image perception rather than dense-caption reduction.Questions may use image or text answer choices and include single or multiple images.
- Benchmark: 3.8K questions across 7.3K images cover low-level pattern matching, mid-level reasoning, and high-level visual understanding.Examples include visual correspondence, relative depth estimation, and visual similarity.
- Results: Specialist vision models substantially outperform multimodal LLMs on tasks including visual correspondence, relative depth estimation, and multi-view reasoning.The paper presents specialist models as a potential direction for improving multimodal perception.
2 Related Work
Prior vision-language datasets assess individual capabilities, while newer holistic benchmarks broaden task coverage but still largely derive perception questions from recognition-focused VQA or image descriptions.
- Existing datasets: Traditional vision-language datasets target single capabilities such as OCR, image captioning, or visual question answering.These datasets are described as insufficient for holistic assessment of multimodal perception and reasoning.
- Holistic benchmarks: Holistic benchmarks including MME, MM-Vet, MMBench, and Seed-Bench expand subjects, inputs, or task diversity.Their formats include yes/no questions, discrete evaluation sets, circular evaluation, multiple images, and video.
- Remaining gap: Perception questions in these holistic benchmarks are mainly extracted from existing VQA datasets or generated from image descriptions.This limits their distinction from recognition-centered evaluation.
3 The Blink Benchmark
Blink is designed as a broad perception benchmark spanning visual scales, perceptual complexity levels, visual prompts, and image sources. It converts 14 tasks into multiple-choice questions intended to resist text-only captioning solutions.
- Benchmark scope: Blink evaluates visual perception across 14 tasks, from pixel-level to image-level processing and from low-level matching to high-level understanding.The benchmark includes examples such as relative reflectance, jigsaw, visual correspondence, relative depth, and visual similarity.
- Benchmark scope: The tasks are selected to make dense captioning insufficient, requiring models to perceive image contents directly.The design systematically spans low-, mid-, and high-level perceptual abilities.
- Format: Blink uses multiple-choice questions with image or text answers, single or multiple images, and textual or visual prompts.The benchmark contributes 3.9K questions and 7.3K images with balanced validation and test sets.
- Design features: Visual prompts such as circles, boxes, and masks highlight image regions and support evaluation of detailed regional understanding.This format also provides an interface for studying visual-prompting effects.
- Design features: The benchmark covers perception beyond recognition, including 3D reasoning, geometric understanding, and affordance reasoning.Its images include in-the-wild scenes, abstract diagrams, synthesized images, and authentic photographs.
- Human-oriented design: Blink questions are intended to require neither domain expertise nor specialized knowledge and to be solvable by humans almost instantaneously, except for the IQ test.This supports measuring a perception gap on tasks that are straightforward for people.
4 Experiments
The experiments evaluate recent multimodal LLMs under standardized conditions and find that Blink remains difficult, with performance varying across perceptual task types.
- Experimental setup: The study evaluates 16 recent multimodal LLMs, including open-source models and GPT-4V, Gemini Pro, Claude 3 Opus, and Qwen-VL-MAX.Models include 7B, 13B, and 34B variants.
- Experimental setup: The evaluation follows VLMEvalKit settings with temperature 0, retry 10, no image resizing, and concatenation for models lacking multi-image input.Answer choices are extracted using predefined rules and GPT-3.5-turbo.
- Main results: 51.26% GPT-4V accuracy leads the reported models, while Gemini Pro and Claude 3 Opus reach 45.72% and 44.11%, respectively.These figures remain close to random guessing at 38.09%.
- Task patterns: Multimodal LLMs perform relatively better on spatial reasoning, art style, and counting than on several other Blink tasks.Some models underperform random guessing on jigsaw, semantic correspondence, multi-view reasoning, object localization, and relative reflectance.
2 More details are at the official website at https://www.01.ai/
Blink exposes a large perception gap between humans and multimodal LLMs, while analyses show that visual prompting and specialist vision models materially affect performance. Caption-based evaluation succeeds on language-heavy benchmarks but fails on Blink, underscoring its distinct perceptual demands.
- GPT-4V performs especially strongly on visual similarity, art style, jigsaw, and multi-view reasoning, exceeding Gemini Pro by 29% on visual similarity.
- Caption + LLM beats GPT-4V by 5.7% on MMBench, reaches 47.2% on MMMU, but falls to random-guessing performance on Blink.
- Red visual prompts outperform gray across tested tasks, while the optimal circle size depends on the task; 10px circles perform best on average.
- Specialist models outperform the best multimodal LLM by 18% to 57%, with DepthAnything and DIFT reaching human-level performance on depth estimation and visual correspondence.
- GPT-4V errors commonly involve hallucinated fine-grained attributes and incorrect visual-prompt locations.
5 Conclusion
Blink is introduced as a benchmark for core visual perception abilities that existing multimodal evaluations overlook. Its task suite and analyses show that current multimodal LLMs remain far below human perception, while specialist models suggest a path toward improvement.
- Blink evaluates core visual perception abilities that are not found in existing multimodal LLM evaluations.
- The benchmark uses circles, boxes, and masks as visual prompts, with task-specific prompt configurations.
A.4 Human Evaluation Protocol
Human performance on Blink is estimated from two human evaluators assigned to each task, with average scores used as the human-performance measure.
- Two humans, identified as coauthors, are assigned to each Blink task for human evaluation.
- Average scores from the assigned evaluators serve as the reported human-performance measure.
- 80-99% human agreement is observed across tasks, with the lowest agreement on art style and functional correspondence.
- Agreement is highest for relative depth, object localization, and forensics detection.
- The IQ-test human score is based on two coauthors evaluating 100 sampled items, 50 each, rather than average human performance.
A.5 Dataset Statistics
Blink's dataset statistics are presented in Table 3, which provides the benchmark's detailed statistical breakdown.
- Detailed statistics for the Blink benchmark are reported in Table 3.
- The supplied passage identifies the table but does not state its individual dataset statistics.
- Table 3 is identified as the detailed-statistics table for the Blink benchmark.
B Baseline Models
The baseline suite covers multimodal models spanning open-source and proprietary systems, with varied visual encoders, language backbones, scales, and checkpoints.
- MiniGPT-4-v2 combines EVA, LLaMA2-chat 7B, and a linear projection layer for visual understanding.
- OpenFlamingo uses a CLIP vision encoder, an MPT-7B language model, and a 9B checkpoint.
- InstructBLIP is evaluated at 7B and 13B scales and uses CLIP with BLIP-2 and Vicuna-based language models.
- The LLaVA baselines vary by source, language model, and fine-tuning setup, including InternLM2-Chat-7B, Vicuna, and xTuner variants.
- The suite includes Gemini Pro Vision, Claude 3 OPUS, and three GPT-4 checkpoints: GPT-4V, GPT-4 Turbo, and GPT-4o.
C.1 Validation Set Results
Blink validation evaluates multimodal models on task-specific results and examines how models handle multiple-image inputs, including concatenation as a workaround for single-image systems.
- Task-level Blink validation scores are reported in Table 4 for different models on the validation set.
- Only GPT-4V and Gemini Pro accept multi-image inputs among the 16 baseline models; most others accept only single-image inputs.
- Eight of Blink's 14 tasks require multiple-image input, motivating evaluation of concatenated images for models limited to single-image inputs.
- For concatenation, images are placed horizontally with a black margin between them, and GPT-4V and Gemini Pro are evaluated using this format.
- GPT-4V shows a performance decline across all tasks with concatenated images, largest on jigsaw and smallest on multi-view reasoning.
- Gemini Pro's concatenation effect is task-dependent: performance decreases on most tasks, increases on semantic correspondence, and is unchanged on multi-view reasoning.
C.3 Error analysis
The error analysis finds that multimodal LLMs often share mistakes across model sizes and language-model components, while GPT-4V errors span recognition, spatial, reasoning, scene, response, and grounding failures.
- Shared errors: 899 mistakes are shared by LLaVA-v1.5-7B and LLaVA-v1.5-13B despite their different sizes.The models make 1187 and 1147 mistakes in total, respectively.
- Shared errors: 959 mistakes are shared between LLaVA-v1.5-7B and LLaVA-internLM2-7B despite their different language-model components.By comparison, LLaVA-v1.5-7B shares 782 and 655 mistakes with QwenVLMax and GPT4V, respectively.
- GPT-4V errors: 28.5% of sampled GPT-4V errors involve failure to recognize detailed small regions or edges.Other sampled errors include spatial-relation errors, reasoning errors, scene-impression failures, answer rejection, and grounding failures.
- Self-consistency: 48.38% is GPT-4V's self-consistency score across five temperature-1.0 runs, versus 38.15% for an average single run.The paper notes that temperature 1.0 performs worse than default temperature 0 because GPT-4V often encounters firewalls and returns refusal messages.
D Limitations
Blink uses existing image datasets and therefore does not cover all visual perception abilities in the wild; the supplied material also includes illustrative and actual-sized benchmark examples.
- Scope: Blink does not cover all visual perception abilities in the wild because it uses data from existing image datasets.Forensics-detection images were manually collected from publicly available online-search results.
- Illustrative examples: Figure 9 presents the first of 11 sets of actual-sized Blink data with GPT-4V predictions.The supplied figure label identifies this as example set 1/11.
- Benchmark task example: The IQ-test prompt asks the model to select one of four pictures that follows the pattern established by preceding pictures.Suggested strategies include tracking quantity, rotations or flips, and recurring features or changes.
- Benchmark task example: The supplied IQ-test explanation identifies a repeated 90-degree clockwise rotation and selects answer A.It describes the same rotation pattern across the pictured sequences.
- Illustrative examples: Figures 16–19 present actual-sized Blink data with GPT-4V predictions, covering example sets 8/11 through 11/11.These figures complete the supplied sequence of actual-sized examples.