Source-linked AI summary
VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
Paul Gavrikov, Wei Lin, M. Jehanzeb Mirza, Soumya Jahagirdar, Muhammad Huzaifa, Sivan Doveh, Serena Yeung-Levy, James Glass, Hilde Kuehne
TL;DR
Current benchmarks often emphasize low-resolution foreground reasoning or high-resolution retrieval, offering limited evidence about fine-grained perception in dense, high-resolution scenes. VisualOverload uses 150 high-resolution scans of public-domain artworks with 2,720 manually curated question–answer pairs across six visual tasks. The strongest tested model, o3, achieves only 19.6% accuracy on the hardest split and 69.5% overall; models also fail in counting, precise OCR, and logical consistency.
Problem
Current benchmarks often emphasize low-resolution foreground reasoning or high-resolution retrieval, offering limited evidence about fine-grained perception in dense, high-resolution scenes.
Source: §1 · pp. 1–2Method
VisualOverload uses 150 high-resolution scans of public-domain artworks with 2,720 manually curated question–answer pairs across six visual tasks.
Source: §1 · p. 2Results
The strongest tested model, o3, achieves only 19.6% accuracy on the hardest split and 69.5% overall; models also fail in counting, precise OCR, and logical consistency.
Source: §1 · p. 2Takeaways & Limitations
VisualOverload provides a benchmark for probing basic visual understanding in detail-rich scenes, where current VLMs still struggle with fine-grained questions.
Source: §6 · p. 8The findings are based exclusively on art and may not directly generalize to benchmarks of natural images.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Is basic visual understanding really solved in state-of-the-art VLMs? We present VisualOverload, a slightly different visual question answering (VQA) benchmark comprising 2,720 question-answer pairs, with privately held ground-truth responses. Unlike prior VQA datasets that typically focus on near global image understanding, VisualOverload challenges models to perform simple, knowledge-free vision tasks in densely populated (or, overloaded) scenes. Our dataset consists of high-resolution scans of public-domain paintings that are populated with multiple figures, actions, and unfolding subplots set against elaborately detailed backdrops. We manually annotated these images with questions across six task categories to probe for a thorough understanding of the scene. We hypothesize that current benchmarks overestimate the performance of VLMs, and encoding and reasoning over details is still a challenging task for them, especially if they are confronted with densely populated scenes. Indeed, we observe that even the best model (o3) out of 37 tested models only achieves 19.6% accuracy on our hardest test split and overall 69.5% accuracy on all questions. Beyond a thorough evaluation, we complement our benchmark with an error analysis that reveals multiple failure modes, including a lack of counting skills, failure in OCR, and striking logical inconsistencies under complex tasks. Altogether, VisualOverload exposes a critical gap in current vision models and offers a crucial resource for the community to develop better models. Benchmark: http://paulgavrikov.github.io/visualoverload
1. Introduction
VisualOverload asks whether VLMs handle basic visual understanding when scenes are dense and high-resolution, a setting current benchmarks often miss. It introduces a benchmark for this gap and finds persistent fine-grained failures, including counting, OCR, and logical consistency.
1. Introduction
- Current benchmarks often emphasize low-resolution foreground reasoning or high-resolution retrieval, leaving broader fine-grained perception in dense scenes under-tested.The authors argue that this makes such benchmarks poor indicators of performance in settings where safety and reliability depend on fine-grained perception.Source: §1 · pp. 1–2
- Among 37 evaluated VLMs, the strongest tested model, o3, reaches 19.6% accuracy on the hardest split and 69.5% overall.The benchmark is divided into easy, medium, and hard splits calibrated by average model performance.Source: §1 · p. 2
- Models often answer logically opposite paired questions inconsistently, with instability increasing as query complexity rises.These inconsistencies can reduce performance to random or subrandom baselines, suggesting reliance on shortcuts rather than robust reasoning.Source: §1 · p. 2
- VisualOverload contains 2,720 manually curated question–answer pairs across six vision tasks, using high-resolution scans of public-domain artworks with dense scenes.Ground truths are held private to avoid target leakage.Source: §1 · p. 2
- Models can perform well on global scene classification yet struggle with fine-grained recognition in dense scenes, particularly counting and OCR.Error analysis finds failures on high ground-truth counts and precise text recognition, including typos.Source: §1 · pp. 2–3
2. The VisualOverload Benchmark
VisualOverload is designed to test basic image recognition in dense, high-resolution scenes, with manually curated questions grounded in visible content. It combines controlled evaluation procedures with new public-domain images and human-authored questions.
2. The VisualOverload Benchmark
- The benchmark targets basic image recognition in dense, high-resolution scenes to stress-test visual representations and multimodal alignment.Its sections cover dataset curation, evaluation, and comparison with existing benchmarks.Source: §2 · p. 3
2.1. Dataset Curation
- Six annotators wrote image-grounded questions that require observable or reasonably inferable content and no specialized knowledge.The six categories include activity and attribute recognition, counting, OCR, reasoning, and scene classification; questions use multiple-choice or freeform formats, with yes/no items paired with logical opposites.Source: §2.1 · pp. 3–4
2.2. Evaluation Process
- Evaluation uses average accuracy across questions, difficulty splits, and task categories, with paired accuracy for binary opposite questions.Heuristics normalize extracted responses, and the ground truth is withheld behind an evaluation server to prevent test leakage.Source: §2.2 · p. 4
2.3. Comparison With Existing Benchmarks
- VisualOverload combines dense, high-resolution scenes, human-authored questions, fresh images, and public-domain licensing as distinguishing features.The authors contrast this design with benchmarks they characterize as relying on shallow questions, recycled data, or automated question generation.Source: §2.3 · pp. 4–5
3. Experiments
The experiments evaluate 37 recent VLMs, comparing model performance across tasks and against chance baselines. Even the strongest model performs modestly overall, with counting, OCR, and visual reasoning remaining difficult.
3.1. Baselines
- The evaluation covers 37 recent VLMs, including open-weight models from 450M to 109B parameters, specialized high-resolution models, and four proprietary frontier models.Results are also compared with random chance and consistent chance for logically opposite questions.Source: §3.1 · p. 5
3.2. Main Results
- 41.7% is the best counting accuracy, achieved by Gemini 2.0 Flash; 62.7% is the best OCR accuracy, achieved by o4-mini.The authors report that all models struggle with these freeform tasks.Source: §3.2 · p. 5
- Models improve on activity and attribute recognition over chance but remain far from satisfactory, while most struggle with reasoning and some smaller models fall below consistent chance.Frontier models perform well on scene understanding, and 8B parameters suffice for 93.4% accuracy on scene classification; the authors describe this task as relying primarily on superficial scene understanding.Source: §3.2 · p. 5
- 19.6% accuracy on the hardest split and 69.5% overall are the results for the best model, o3, averaged over all tasks.The strongest open-weight model, InternVL3 38B, achieves 7.2% and 67.6%, respectively; specialized HD models perform worse than equally sized regular models, and some families show declining performance at larger sizes.Source: §3.2 · p. 5
4. Error Analysis
The error analysis finds counting errors increase with larger ground truths, OCR mistakes often require substantial correction, and consistency weakens on harder reasoning questions.
Counting
- Counting predictions are generally accurate at low ground-truth values, but errors rise substantially as counts increase and usually underestimate the true count.Some errors also stem from refusals or blank responses, which are treated as zero; severe overestimation appears in outliers.Source: §4 · pp. 5, 7, Figure 3 · p. 7
Counting and OCR
- Even with 10% tolerance, average counting accuracy improves by only 1.6%; larger tolerances help more but are impractical for real-world applications.Incorrect OCR predictions have normalized edit distances centered around 0.7, indicating substantial edits are needed for correction.Source: §4 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7
OCR
- OCR errors include hallucinations, irrelevant text extraction, and failures to follow text-only instructions; layout misreading and spelling autocorrection also contribute.Models may substitute more probable spellings for the actual text, including in non-English or non-Latin scripts.Source: §4 · p. 7
Logical consistency
- Frontier-model logical consistency drops from 83.3% on easier scene questions to 60.6% on harder reasoning questions, sometimes reaching near-random levels.The analysis measures consistency across paired logically opposite questions; some models show below-chance consistency, suggesting reliance on shortcuts rather than visual inputs.Source: §4 · p. 7
5. Related Work
Prior VLMs and benchmarks have broadened multimodal capabilities and evaluation, yet basic perception in visually overloaded scenes remains comparatively underexamined. VisualOverload complements existing benchmarks by testing six basic tasks across dense, high-resolution artworks.
Multi-modal vision benchmarks
- Multimodal benchmarks evaluate vision-language integration across VQA, captioning, reasoning, and instruction following, with scopes spanning real-world and conversational tasks, holistic evaluation, and expert-level multimodal reasoning.Prior work includes VisitBench [7], LLaVA-Bench [8], MMVet [68] (16 capabilities), SeedBench [31] (12 dimensions), MM-Bench, TouchStone [4], OmniBench [35], MMStar, and MMMU [69].Source: §5 · p. 8
Large multi-modal models
- Many VLMs still struggle with counting, spatial reasoning, concept binding, dense-scene understanding, and even image classification [16] [61] [10] [13] [14] [28] [52] [67] [48] [49] [73].VisualOverload builds on these findings with densely populated public-domain paintings and basic visual tasks.Source: §5 · p. 8
Multi-modal vision benchmarks
- On most benchmarks, results seem to saturate, but more carefully designed benchmarks [27] [28] [63] reveal persistent weaknesses in multiple dimensions, highlighting a discrepancy between many seemingly positive benchmark results and actual visual capabilities.Source: §5 · p. 8
- Unlike benchmarks focused on global understanding, broad task coverage, domain expertise, or needle-in-a-haystack retrieval, VisualOverload probes six basic tasks across full, dense scene complexity.Its human-annotated VQA pairs use high-resolution artworks and expose systematic error modes in open and proprietary VLMs.Source: §5 · p. 8
6. Conclusion
The paper concludes that VLMs handle global tasks better than fine-grained questions in visually overloaded scenes, and that fundamental visual understanding remains unresolved. Its evidence comes exclusively from art, which may not generalize directly to natural-image benchmarks.
Conclusion
- VLMs perform well on global tasks but consistently struggle with simple fine-grained questions in visually overloaded environments, indicating that fundamental visual understanding remains unsolved.The authors present VisualOverload as a resource for developing more robust and perceptive VLMs.Source: §6 · p. 8
Limitations
- The findings rely exclusively on art and may not directly generalize to natural-image benchmarks.The authors note that artistic representations differ from photorealism, with high stylistic variance and reliance on global shape [60].Source: §6 · p. 8
A. Quantifying Image Complexity
The authors estimate image complexity through pairwise comparisons by an LLM, defining each image’s score as its empirical win-rate. Figures show images at the distribution’s extremes and the win-rate across the dataset.
A. Quantifying Image Complexity
- Because standard metrics make visual complexity difficult to measure, the authors use automated pairwise comparisons across all 150 images, with Qwen3.5-27B judging which image is more visually complex.The prompt directs the model to ignore artwork identity and focus on visual composition; its final selections remained focused on density despite occasional identity attempts in reasoning traces.Source: A. Quantifying Image Complexity · p. 13
- An image’s complexity score is its empirical win-rate: the proportion of pairings in which the model selected it as more complex.Figure 7 shows the least- and most-complex images according to these measurements; Figure 8 shows win-rate over all 150 images.Source: A. Quantifying Image Complexity · p. 13, Figure 7 · p. 13, Figure 8 · p. 13
B. Discussion of Domain Shift
The authors treat historical paintings’ stylistic departure from photos as a robustness test and argue that performance tracks image and question details rather than domain shift. They motivate the domain choice by its naturalistic scenes, finite supply, high resolution, and clean copyright status.
B. Discussion of Domain Shift
- Historical paintings depict naturalistic scenes while differing from digital photos in style and color, making them a testbed for robustness to stylistic abstraction.The authors also note that paintings are finite and not easily scalable, offer high-resolution images without depth-of-field blur, and have clean copyright status.Source: B. Discussion of Domain Shift · pp. 13–14
- Per-image results for three models show above-random performance on most images, indicating that difficulty tracks questions and visual details rather than domain shift.The discussion points to differing performance across tasks with different image-understanding demands as further support.Source: B. Discussion of Domain Shift · p. 14, Figure 9 · p. 14
C. Benchmark Prompts
- Multiple-choice prompts supplied labeled options and requested a letter, while OCR prompts requested a direct answer.The main evaluation used prompts according to question type.Source: C. Benchmark Prompts · p. 14
D. Ablation of Resolution
The resolution ablation finds modest overall gains, with stronger resolution sensitivity for OCR than scene recognition. The section also describes a language-bias detection prompt and reports mixed effects from zero-shot chain-of-thought prompting.
D. Ablation of Resolution
- InternVL3’s patch limit constrained the resolution test: processing VisualOverload without downsampling would require 40 patches and eight 40-GB GPUs for the 8B model.The authors attribute this boundary to model architecture, not the benchmark, and expect performance to scale with resolution absent downsampling, given sufficient context and proper training.Source: D. Ablation of Resolution · p. 14
E. Language Bias Detection
- The language-bias check used Gemini 2.5 Pro to judge whether models could guess question answers through shortcuts rather than image evidence.The prompt asked for an answer and rationale for each question ID.Source: E. Language Bias Detection · pp. 14–15
D. Ablation of Resolution
- Resolution generally improves performance only modestly; OCR correlates strongly with resolution, scene recognition varies little, and other tasks tend to plateau after Full HD.The other-task trend is reported as typically increasing with resolution before the Full HD plateau.Source: D. Ablation of Resolution · p. 14, Figure 10 · p. 15
F. Performance with Advanced Prompting
- Zero-shot CoT lowered InternVL3 8B’s average performance but improved results on the hardest split and OCR; the authors hypothesize the tested LLM was too small to benefit overall.The authors note that CoT prompting is primarily effective in large-scale LLMs.Source: F. Performance with Advanced Prompting · p. 15