Source-linked AI summary
Vision language models are blind: Failing to translate detailed visual features into words
Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, Anh Totti Nguyen
TL;DR
Existing VLM benchmarks often mix visual and non-visual abilities, leaving precise low-level visual perception insufficiently measured. This paper introduces BlindTest, a seven-task benchmark of simple geometric-primitive questions, finding 58.07% average accuracy across four VLMs, while performance reaches at least 99.47% when shapes and letters are more widely separated.
Problem
Existing VLM benchmarks often conflate visual and non-visual abilities, limiting exclusive measurement of VLM visual capabilities.
Source: §1 · p. 2Method
The paper introduces BlindTest, seven novel low-level visual tasks using simple geometric primitives and minimal world knowledge, and evaluates state-of-the-art VLMs.
Source: §1 · p. 2, §8 · p. 23Results
58.07% average accuracy across four VLMs, with Claude 3.5 Sonnet best at 77.84%, while added spacing raises task accuracy to at least 99.47%.
Source: §1 · pp. 2–3Takeaways & Limitations
The findings indicate that VLMs struggle to decode precise visual information when geometric primitives or letters are close together.
Source: §1 · pp. 2–3The findings may generalize primarily to VLMs using CLIP-based vision encoders.
Source: §6 · p. 20
Abstract
from arXiv · showhide
While large language models with vision capabilities (VLMs), e.g., GPT-4o and Gemini 1.5 Pro, score high on many vision-understanding benchmarks, they are still struggling with low-level vision tasks that are easy to humans. Specifically, on BlindTest, our suite of 7 very simple tasks, including identifying (a) whether two circles overlap; (b) how many times two lines intersect; (c) which letter is being circled in a word; and (d) the number of circles in an Olympic-like logo, four state-of-the-art VLMs are only 58.07% accurate on average. Claude 3.5 Sonnet performs the best at 77.84% accuracy, far from the human expected accuracy of 100%. Across different image resolutions and line widths, VLMs including slow-thinking models consistently struggle with those tasks that require precise spatial information when geometric primitives overlap or are close. Yet, VLMs perform at near-100% accuracy when much more space is added to separate shapes and letters. Linear probing experiments show that vision encoders contain sufficient visual information to solve BlindTest and that language models fail to decode this information into correct answers. Code and data are at: https://vlmsareblind.github.io
1 Introduction
BlindTest isolates low-level visual perception from language and reasoning, revealing that leading VLMs struggle with simple overlapping, nested, and closely spaced geometric tasks despite strong performance on conventional benchmarks. Across seven tasks, models average 58.07% accuracy, with Claude-3.5 Sonnet best at 77.84%, while probing shows vision encoders contain the needed information.
Motivation and benchmark design
- BlindTest addresses a benchmark gap because many existing VLM questions can be answered from text, choices, or memorized knowledge, conflating visual and non-visual abilities.The paper therefore targets seeing rather than reasoning using seven minimal-world-knowledge tasks based on 2D geometric primitives.Source: §1 · p. 2
Task-level failures
- VLMs struggle with intersections, circled letters, overlapping or nested shapes, grid dimensions, and subway-map path counts, despite recognizing isolated shapes and words.Intersection accuracy ranges from ∼41% to 76%, two-circle accuracy from ∼75–93%, and subway-path accuracy from ∼31% to 58%.Source: §1 · p. 2
Main findings
- 58.07% average accuracy across seven BlindTest tasks, with Claude-3.5 Sonnet best at 77.84% versus humans’ expected 100%; slow-thinking models reach only 71.59% versus 72.75%.GPT-4o is better than Gemini-1.5 Pro on seven existing complex benchmarks but worse on BlindTest.Source: §1 · p. 2
Simplified inputs
- Simplifying images by adding space between letters or shapes, or reducing subway-map turns, substantially improves performance, with gains of +20 and +63 points on two tasks.Some models reach ∼100% accuracy on simplified BlindTest versions.Source: §1 · pp. 2–3
Representation analysis
- Linear probes achieve ≥99.47% accuracy on two-circle and line-intersection tasks from LLaVA-One-S and Phi-3.5 vision features, implicating language decoding rather than missing visual information.The probes use vision-encoder features before and after the projection layer.Source: §1 · p. 3
2 Vision language models
The section evaluates leading commercial and open-source vision-language models on simple images containing interacting geometric primitives. It also tests slow-thinking models to assess whether iterative reasoning improves performance on BlindTest.
Commercial VLMs
- The study tests four leading commercial VLMs selected from models ranking highly across seven recent vision-language benchmarks, including MMMU, AI2D, MathVista, ChartQA, DocVQA, ActivityNet-QA, and EgoSchema.The models are GPT-4o, Gemini-1.5 Pro, Claude 3 Sonnet, and Claude 3.5 Sonnet; these benchmarks cover college-level subjects, diagrams, mathematics, charts, documents, and videos.Source: §2 · p. 4
- Claude 3 Opus was replaced by Claude 3.5 Sonnet because Sonnet-3.5 performed more accurately on BlindTest and cost 5× less.All tested models are described in Appendix A.Source: §2 · p. 4
Open-source VLMs
Slow-thinking VLMs
- Two slow-thinking models generally do not outperform standard VLMs on BlindTest, supporting its characterization as a low-level visual task that does not require high-level reasoning.The evaluated models are closed-source Gemini 2.0 Flash-Thinking and open-source QVQ-Preview (QVQ) [51].Source: §2 · p. 4
3 BlindTest benchmark of 7 tasks
BlindTest evaluates VLMs on seven simple geometric and spatial tasks involving close, overlapping, intersecting, nested, or adjacent primitives. Across the benchmark, all four VLMs average 58.07% accuracy, with Sonnet-3.5 highest at 77.84%, still below the expected 100%.
3 BlindTest benchmark of 7 tasks
- BlindTest comprises seven simple tasks designed to test recognition of known geometric primitives when they are close together, overlapping, or intersecting.The benchmark avoids existing Internet eye-exam questions and targets spatial information that may not be captured or decoded reliably.Source: §3 · p. 4
3.1 Task 1: Counting line intersections
- Task 1 asks VLMs to count whether two two-segment piece-wise linear functions intersect 0, 1, or 2 times across 1,800 generated images.Images use canvas sizes C ∈ {384, 768, 1152}, with line coordinates sampled from an invisible 12×12 grid.Source: §3.1 · p. 5
3.2 Task 2: Two circles
- Task 2 tests whether VLMs distinguish overlapping, tangent, and disjoint pairs of same-sized circles across varying gaps, diameters, orientations, and resolutions.The full grid produces 768 images over three image sizes and four orientations.Source: §3.2 · p. 5
3.3 Task 3: The circled letter
- Task 3 asks which letter is circled when a red oval is superimposed on each character in three strings with variable character widths and heights.The strings are Acknowledgement, Subdermatoglyphic, and tHyUiKaRbNqWeOpXcZvM, generating 360, 408, and 480 images respectively.Source: §3.3 · pp. 5–6
3.4 Task 4: Counting overlapping shapes ###
- Task 4 counts overlapping circles arranged like an Olympic logo and repeats the experiment with overlapping pentagons for generalization.The benchmark varies shape count, size, resolution, line width, color, and vertical orientation, producing 480 images across both shapes.Source: §3.4 · p. 6
3.5 Task 5: Counting the nested squares
- Task 5 counts nested squares whose edges do not intersect, using 2 to 5 randomly positioned squares rendered at three line widths.The procedure generates 120 images while ensuring that nested squares do not touch by their edges.Source: §3.5 · pp. 6–7
3.6 Task 6: Counting the rows and columns of a grid
- Task 6 asks models to count rows and columns in empty and text-containing grids with varying dimensions, line widths, and canvas sizes.The combined grid variants contain 264 images.Source: §3.6 · p. 7
3 BlindTest benchmark of 7 tasks
- 58.07% mean accuracy across all four VLMs is substantially above 24% random chance, while Sonnet-3.5 leads at 77.84% but remains below 100%.The benchmark reports results over seven BlindTest tasks.Source: Table 1 · p. 8
4 Results
Across BlindTest’s low-level visual tasks, VLMs frequently fail to resolve precise spatial relationships, even when shapes, letters, grids, and paths are visually simple. Longer inference does not reliably address this limitation.
4.1 VLMs cannot reliably count line intersections
- 56.84% mean accuracy across VLMs on counting line intersections, with performance worsening as plots move closer together and remaining similar across image sizes.Sonnet-3.5 achieves only 75.36%, far below the expected 100%; the results contrast with high ChartQA accuracy, suggesting weak detail extraction despite recognizing overall trends.Source: §4.1 · pp. 8–9, Figure 2 · p. 8
4.2 VLMs cannot clearly see if two circles overlap or not
- 86.70% mean accuracy on deciding whether two large circles touch or overlap, with closer circles causing more errors and GPT-4o failing even when separated by one radius.Gemini-1.5 performs best at 93.62%, while performance varies little across resolutions and colors but depends on circle orientation.Source: §4.2 · pp. 9–10, Figure 3 · p. 9
4.3 VLMs do not always see the letter inside the red circle
- 79.7% mean accuracy for identifying a circled letter, despite VLMs accurately spelling the full string; errors often select adjacent or hallucinated characters.Accuracy improves by 0.46–13 points on familiar English words versus random strings, while Sonnet-3.5 and Gemini-1.5 lead at 87.88% and 83.29%.Source: §4.3 · pp. 10–11, Figure 4 · p. 10
4.4 VLMs struggle to count overlapped and nested shapes
- Mean accuracy is 39.44% for overlapping circles, 30.99% for pentagons, and 74.99% for nested squares, with counting difficulty increasing beyond five shapes.Models show a strong bias toward the five-circle Olympic logo, while nested-square accuracy remains imperfect even without intersecting edges and changes little with colors or resolution for most models.Source: §4.4 · pp. 11–12, Figure 5 · p. 11
4.5 VLMs cannot reliably count rows and columns in a grid
- 34.37% accuracy for counting rows and columns in empty grids, nearly doubling after adding one word per cell but still reaching only 88.68% for the best model.Text-containing grids are easier than empty grids, and columns are easier than rows: 70.53% versus 60.83% accuracy.Source: §4.5 · pp. 12–13, Figure 6 · p. 13
4.6 VLMs struggle to count single-colored paths
- 48.90% mean accuracy for counting single-colored paths, with accuracy falling as map complexity increases from one to three paths.Even with one path, the best model reaches only 93.33%, and predictions are often off by one to three paths.Source: §4.6 · p. 13
4.7 Long-inference, slow-thinking VLMs perform similarly to regular VLMs on BlindTest
- 71.59% versus 72.75% average accuracy for Gemini 2.0 Flash-Thinking and regular Gemini 2.0 Flash, while QVQ trails Qwen2-VL by 11.65 points.Thinking tokens remain in text space and provide no benefit on BlindTest, so longer inference does not solve the visual-resolution challenge.Source: §4.7 · p. 14, Table 4 · p. 14, Figure 7 · p. 14
5 VLMs can solve most tasks when there is more space between shapes and letters
Increasing the spacing between letters and shapes, or simplifying path geometry, substantially improves VLM accuracy on tasks that require precise visual separation and counting. These results indicate that overlap, adjacency, and zigzag complexity—not counting alone—are major sources of failure.
5.1 VLMs perform better as the spacing between letters in the circled letter task increases
- Sonnet-3.5 reaches 95% accuracy with 3 extra spaces between letters, while VLMs generally improve whenever at least 1 space separates adjacent characters.The figure reports accuracy gains of +21 for Sonnet-3 and +13 for GPT-4o from 0 to 3 spaces; remaining Sonnet-3.5 errors include adjacent-letter confusions and mistaking the circle for part of a letter.Source: §5.1 · p. 16, Figure 9 · p. 16
5.2 VLMs can more accurately count disjoint shapes
- VLMs count shapes more accurately when overlap is removed, with Sonnet-3.5 rising from 0.66 to 1.0 on ### and from 0.78 to 1.0 on DDD.At dx = 0.75, both Sonnet-3 and Sonnet-3.5 reach ≥96%, while the largest and smallest closed-source gains are △=91% for Sonnet-3 and △=22% for GPT-4o.Source: §5.2 · pp. 17–18, Figure 12 · p. 18
5.3 VLMs can count simplified, more straight paths
- When subway-like paths contain fewer turns, all VLMs count them more accurately, with gains ranging from +2 points for GPT-4o to +30 for Sonnet-3.5.Increasing the straight-direction probability from P=0.33 to 0.6 and 0.9 produces near-100 accuracy for some models, including 0.99 for Gemini-1.5 on 1 path, confirming that zigzag intersections create the difficulty.Source: §5.3 · p. 19, Figure 13 · p. 19
6 The vision encoder in open-source VLMs can extract sufficient information to solve BlindTest
Linear probing shows that open-source VLM vision encoders preserve sufficient low-level information to solve BlindTest, both before and after projection. The evidence instead implicates the language models’ failure to decode that information into correct outputs, while classifiers trained on harder cases generalize more robustly.
Method
- The study probes frozen features with logistic-regression classifiers on 11,100 two-circle images and 6,300 line-intersection images.Features are average-pooled from the layer before projection, with an additional experiment using post-projection features.Source: §6 · p. 20
Results
- ≥99.47% linear-probing accuracy on both tasks for CLIP features and 100% for a SigLIP variant show that vision encoders preserve the information needed for BlindTest.The evaluated tasks are two-circle overlap and line-intersection counting, using frozen vision-encoder features.Source: §6 · p. 20
- ≥99.58% accuracy after projection shows that most visual information is preserved through the projection layer, unlike the lower end-to-end VLM accuracy.The contrast indicates that the language models have access to the necessary visual information but fail to decode it into correct language outputs.Source: §6 · pp. 20–21
- Training on close-circle cases at distances −0.05 and 0.05 produces classifiers that achieve 100% accuracy on both close and farther-apart circles.This robustness holds regardless of the vision encoder and indicates that training on harder cases avoids overfitting to the training distribution.Source: §6 · p. 21
- Training on farther-apart circles performs poorly on closer circles, showing that the harder close-circle cases are necessary for robust generalization.The experiment compares training at distances −0.25 and 0.25 with evaluation on closer cases.Source: §6 · p. 21
7 Related Work
BlindTest differs from conventional VLM benchmarks by isolating low-level visual recognition with minimal prior knowledge or reasoning. Related work also highlights a gap in current improvement strategies and identifies low-level limitations that prior studies have not directly addressed.
Benchmarking VLM vision understanding
- BlindTest is a human-easy, low-level visual sanity check requiring minimal prior knowledge, commonsense, or complex reasoning, unlike conventional topic-specific benchmarks [6] [21].Those benchmarks can suffer data leakage because models may answer accurately without the image [6].Source: §7 · pp. 21–22
- Unlike ARC [8], which requires reasoning over abstract patterns after identifying shapes, BlindTest directly tests recognition of primitive shapes.Source: §7 · p. 22
Improving VLM vision capabilities
- BlindTest extends prior circled-object recognition [53] by exposing failures in low-level circled-letter OCR and in recognizing interacting primitives or following colored paths.The authors identify these as previously unaddressed limitations, relevant to graphs, visual math, and object-direction understanding.Source: §7 · pp. 22–23
8 Discussion and Conclusion … F.6 Finding: Various coloring of the two circles has minimal effects on the overall accuracy
BlindTest evaluates whether VLMs can perceive simple geometric primitives through seven low-level visual tasks designed to require minimal knowledge and resist memorization [6] [13]. The findings show persistent failures in precise visual reasoning that prompting, coding, resolution, rotation, and color changes do not consistently resolve.
8 Discussion and Conclusion
- BlindTest comprises seven novel low-level visual tasks targeting simple geometric primitives while minimizing world knowledge and memorization opportunities [6] [13].The benchmark is designed from scratch and requires minimal to zero knowledge.Source: §8 · p. 23
D.1 Finding: Meta-prompting and 2-shot examples do not improve the VLMs’ performance on the two circles task (
- Two-shot prompting, chain-of-thought, and meta-prompting [34] do not improve BlindTest accuracy, suggesting that the questions are understood but precise visual access remains limiting.On the two-circle task, GPT-4o and Sonnet-3.5 perform worse with two-shot examples and meta-prompting than with baseline prompts.Source: §8 · p. 23, §D.1 · p. 32, Table T2 · p. 32
B Hallucinations and educated guesses are among VLMs’ common failures
- VLMs commonly hallucinate or make educated guesses on overlapping-circle questions, including incorrectly describing colors or claiming that overlapping circles are separate.These failures indicate that models can produce plausible answers without accurately reading the image.Source: B Hallucinations and educated guesses… · p. 30
C Evaluating open-source models on BlindTest
- Open-source VLMs generally underperform commercial models, although LLaVA-OneV-72B-ov matches Sonnet-3 at 45.92 versus 45.73 accuracy and improves 27.1 points over its 0.5B version.The results indicate that increasing model size helps, but open-source models retain limitations similar to closed-source systems.Source: C Evaluating open-source models on… · p. 31, Table T1 · p. 31
E.1 Sonnet-3.5 cannot use its coding ability to regenerate the images from BlindTest
- Sonnet-3.5’s coding ability does not enable faithful reconstruction of BlindTest images, including preserving intersections, shape counts, or touching versus non-touching circles.The model can represent overall shapes but fails to reproduce important detailed features across SVG, HTML/CSS, JavaScript, and Python PIL formats.Source: E Image reconstruction using Sonnet-3.5 · p. 33, §E.1 · p. 33, Figure F2 · p. 34
F.2–F.4 Findings
- VLM performance is largely insensitive to image resolution and rotation, while increasing circle distance improves accuracy and boundary distances produce many failures.Sonnet-3.5 is especially conservative at negative distances, whereas other models often answer “Yes.”Source: §F.2 · p. 38, §F.4 · p. 38, Figure F5 · p. 38
F.5 Finding: VLMs prefer a specific rotation
- Across three resolutions, GPT-4o and Sonnet-3 prefer vertical arrangements, Gemini-1.5 prefers diagonal arrangements, and Sonnet-3.5 prefers horizontal arrangements.Source: §F.5 · p. 39, Table T4 · p. 39
F.1 Benchmark Construction Details; F.6 Finding: Various coloring of the two circles has minimal effects on the overall accuracy
- Changing circle colors has minimal effect on performance: the task does not change substantially, by ±5.79% for Sonnet-3.5 and ±10.81% for GPT-4o.Source: §F.6 · p. 39, Table T5 · p. 39
F.7 Results for fine-tuning Bunny on the two touching circle … I Counting the number of nested squares
Across touching-circle, circled-letter, and line-intersection tasks, VLMs and fine-tuned Bunny remain unreliable at precise spatial judgments, with errors persisting across prompts, fonts, resolutions, and some separations. Performance improves when visual ambiguity is reduced, such as by adding whitespace between letters, but fine-tuning can overfit without generalizing.
F.7 Results for fine-tuning Bunny on the two touching circle
- 17.1% accuracy for overlap and 11.7% for touching circles rose to up to 36.8% with 10K or 20K fine-tuning cases, while larger datasets did not improve performance.With 50K samples, Bunny sometimes generated only the end-of-text token; Fig. F6 compares accuracy by circle distance with and without fine-tuning.Source: §F.7 · p. 40, Figure F6 · p. 41
- Near-zero losses indicate that fine-tuned Bunny overfits the training set and fails to generalize, suggesting that this task may require multiple tasks or lack a simple solution.Performance improves for overlapping circles but generalizes poorly when circles are far apart.Source: §F.7 · p. 40
G.1 Task construction
- The circled-letter benchmark varies strings, letters, fonts, oval thickness, and scaling, renders 90–120 images per string, and evaluates exact case-insensitive character matches.The construction and evaluation use Acknowledgement, Subdermatoglyphic, and a random-character string.Source: §G.1 · p. 43, Evaluation · p. 43
G.2 Finding: VLMs mostly confuse the adjacent character for the circled letter
- Circled-letter errors mainly select neighboring characters, while Gemini-1.5 may treat the red oval as “@” and GPT-4o often predicts “o” regardless of the circled letter.These failures include 100% of Gemini-1.5’s mispredictions for “K” as adjacent “a”, and GPT-4o predicting “o” instead of several letters.Source: §G.2 · p. 46, Figure F11 · p. 46, Figure F12 · p. 47
G.4 Finding: Models perform similarly across two common font families
- Models show little performance variation across the two fonts and the two prompts, indicating that font choice and prompt wording do not explain their unreliable circled-letter recognition.The prompts ask either which letter is circled or which character is highlighted with a red oval.Source: §G.4 · p. 51, Figure F16 · p. 51, §G.5 · p. 52
G.6 Finding: VLMs fail to see the circled letter on various English words
- 86.43% mean accuracy across four SOTA VLMs remains below 100%, with accuracy higher for “question” than “civilization” at 90.36% versus 79.86%.Models perform better on shorter words without repetitive letters, and adding whitespace improves GPT-4o accuracy on Acknowledgement by 11% and Subdermatoglyphic by 21% from 0 to 3 spaces.Source: Table T7 · p. 53, §G.6 · p. 53, §G.7 · p. 54
H.1 Benchmark Construction Details
- The line-intersection benchmark balances 0, 1, and 2 intersections across two line widths and three resolutions, yet examples show that most VLMs still fail when no intersection exists, even with a large gap.The benchmark uses two prompt wordings and 1,800 generated images; Sonnet-3.5 is the exception in the no-intersection examples.Source: §H.1 · p. 56, Code · p. 56, Figure F23 · p. 60
H.2 Finding: image resolution does not influence VLM’s ability to count the line intersections
- Increasing plot resolution does not help VLMs count line intersections, and models perform relatively better on two intersections but worse on zero intersections.Sonnet-3.5 performs best on one intersection and has the least variance across intersection counts; the benchmark contains balanced 0, 1, and 2-intersection cases.Source: §H.2 · p. 59, §H.3 · p. 59, Figure F22 · p. 59
I.1 Benchmark Construction Details … J Counting the shapes in an Olympic-like logo
The nested-square benchmark evaluates VLMs on counting 2–5 nested squares, with 120 balanced images, and shows that even the best model remains below human-level reliability. Performance varies with line thickness and worsens as the number of squares increases.
I.1 Benchmark Construction Details
- Images are generated by varying five parameters: depth, initial size, reduction factor, center placement, and line thickness.Source: §I.1 · p. 61
- The benchmark contains 120 nested-square images, with 30 samples for each depth from 2 through 5 squares.Source: §I.1 · p. 61, Table T10 · p. 63
- The task asks models to count all squares and return the answer in curly brackets, with ground-truth counts of 2, 3, 4, or 5.Source: §I.1 · p. 61
I.2 Finding: the best-performing model is affected by line width
- Sonnet-3.5 becomes more accurate as line width increases, whereas GPT-4o shows the opposite trend and may confuse squares with thicker borders.Source: §I.2 · p. 64, Table T11 · p. 64
I.3 Additional Examples
- Model responses illustrate that VLMs fail to reliably count nested squares, even though the task uses only simple geometric shapes.Source: §I.3 · p. 65, Figure F27 · p. 65
- 92.08%: Sonnet-3.5 performs best on nested-square counting, but remains below the 100% human level, with difficulty increasing from 2 to 5 squares.Source: Figure F28 · p. 65
J.1 Benchmark Construction Details … K Counting the rows and columns of a grid task
The benchmark varies shape count, resolution, color, size, and line width across 480 overlapping-circle and pentagon images. Results show that resolution and color generally have little or negative impact, while Gemini-1.5 is strongly biased toward the five-circle Olympic-logo pattern.
K Counting the rows and columns of a grid task
- The supplied passages do not provide details for the grid rows-and-columns task.No evidence passage for section K was included in the input.Source: §J.1 · p. 66
J.1 Benchmark Construction Details
- The benchmark contains 480 centered images, evenly divided between circles and pentagons, with shape counts sampled from 5, 6, 7, 8, and 9.Images vary across three resolutions, two color options, two shape-size settings, and two line widths; the random-baseline accuracy is 20%.Source: §J.1 · p. 66, Figure F30 · p. 67, Table T12 · p. 67
J.2 Finding: different resolutions have no impact on most VLMs’ performance
- Resolution and color generally have minimal impact on VLM performance when counting overlapping shapes, although Sonnet-3.5 improves with larger images and GPT-4o improves on colored shapes.VLMs perform better on overlapping circles than pentagons.Source: Figure F31 · p. 68, §J.2 · p. 68
J.3 Finding: color-coding does not generally help the VLMs
- Except for GPT-4o, coloring the shapes generally reduces performance rather than making overlapping shapes easier to distinguish.The expected benefit of color-coding is therefore not observed for most models.Source: §J.3 · p. 68
J.4 Finding: Gemini-1.5 has the most biased predictions to the Olympic logo
- Prediction patterns differ across models: GPT-4o has lower variance for circles than pentagons, Sonnet-3 undercounts, and Sonnet-3.5 has the least relative variance across both shapes.Gemini-1.5 is biased toward five circles and behaves more randomly for pentagons.Source: Figure F32 · p. 69, §J.4 · p. 69
- Gemini-1.5 predicts five circles 99.74% of the time when images contain 6–9 circles, compared with 10.94% for pentagons.This indicates a strong bias toward the five-circle Olympic logo.Source: Table T13 · p. 69, §J.4 · p. 69
J.5 Additional Examples
- Examples show that VLMs tend to count correctly only when the image contains five circles, while overlapping-circle counting remains difficult across colors, line widths, and resolutions.Gemini-1.5 often predicts five regardless of the actual circle count.Source: Figure F33 · p. 70, Figure F34 · p. 70
K.1 Benchmark Construction Details … L Following single-colored paths task
The benchmark uses varied grid sizes, shapes, line thicknesses, contents, and image resolutions to test row and column counting. VLMs cannot reliably count either dimension, although text-containing grids improve performance and Sonnet-3.5 performs best.
K.1 Benchmark Construction Details
- The benchmark draws grid borderlines from row and column coordinates and includes blank and text-containing versions.It generates 264 images using varied grid configurations.Source: §K.1 · p. 71, Figure F35 · p. 73, Figure F36 · p. 74
- Ground-truth correctness requires both row and column counts to be correct, with random-chance accuracy of 1/22, i.e., 4.55%.The benchmark prompts models to provide row and column counts in specified formats.Source: §K.1 · p. 71
- 264 empty and text-containing grids are constructed with equal numbers of images in each subcategory.The benchmark varies image sizes, grid dimensions, line thicknesses, and whether cells are blank or contain random text.Source: §K.1 · p. 71, Table T14 · p. 72
K.2 Finding: VLMs cannot reliably count either rows or columns
- Including text inside grids improves all model accuracies, while Sonnet-3.5 outperforms other models on empty and text-containing grids.This comparison is reported for both grid-content conditions.Source: Table T15 · p. 75
- 70.53% column-counting accuracy and 60.83% row-counting accuracy remain far below the expected 100%.VLMs perform better at counting columns than rows, but neither dimension is counted reliably.Source: Table T16 · p. 75, §K.2 · p. 75
K.3 Additional Examples
- Models are often off by one or two when counting rows and columns in empty grids or small grids containing words.Additional examples show consistent failures on empty grids, while Sonnet-3.5 is more consistent for lower row and column counts.Source: Figure F37 · p. 76, Figure F38 · p. 76
- Examples show that including text in grid cells improves all VLMs’ performance, especially Sonnet-3.5.The examples complement the benchmark’s aggregate finding about text-containing grids.Source: §K.3 · p. 76, Figure F39 · p. 77
L.1 Benchmark Construction Details … N.1 Implementation details
The benchmark varies subway-map resolution, grid structure, path thickness, and path complexity to test colored-path counting, while supplementary results show that VLM accuracy declines with visual complexity but improves for Sonnet-3.5 on simplified images. Additional examples, real-world failures, and probing implementation details document the scope and evaluation setup.
L.1 Benchmark Construction Details
- The benchmark contains 180 images with exactly 1, 2, or 3 paths exiting each station, generated using an 18×18 grid, two image resolutions, and three line widths.Source: §L.1 · p. 78, Table T17 · p. 79, Figure F40 · p. 79
- Ground-truth path counts are restricted to {1, 2, 3}, giving a 33.33% random baseline and excluding cases with no connecting path.The two queried stations are distinct and sampled from {A, B, C, D}.Source: §L.1 · p. 78
L.2 Finding: VLMs fail to see the colored paths across various map complexities
- VLM accuracy decreases as paths per station increase from 1 to 3, and counting colored paths remains challenging overall.This trend is reported for subway-like maps across varying complexities.Source: §L.2 · p. 81, Figure F42 · p. 81
L.3 Sonnet-3.5 can more accurately count subway paths on simplified images compared to other VLMs
- Sonnet-3.5 gains +27 mean performance as the straight-path probability increases from P = 0.33 to 0.9.The model consistently benefits across image settings, indicating better counting on simplified images than other VLMs.Source: Figure F43 · p. 82, §L.3 · p. 82
L.4 Additional Examples
- VLMs generally fail on images with more intersecting paths, increasingly making educated guesses as path complexity rises.The examples specifically highlight this behavior for Gemini-1.5 and Sonnet-3.5, while failures can occur even in extremely easy cases across both line widths.Source: Figure F44 · p. 83, §L.4 · p. 83
M VLM failures on real-world data
- Figures provide examples of VLM failures on real-world data.These examples are presented as supplementary evidence of failures beyond the synthetic subway-like maps.Source: Figure F46 · p. 84, Figure F47 · p. 85
tasks
- The paper includes a linear probing experiment for the benchmark tasks.The supplied material identifies this experiment section but does not provide its results here.Source: tasks · p. 86
N.1 Implementation details
- The probing experiments use identical hyperparameters across models, with feature dimensions reported separately for each model and input size.After average pooling, LLaVA-OneV-S frozen features have identical dimensions across different image sizes.Source: §N.1 · p. 86, Table T18 · p. 86, Table T19 · p. 86
N.2 Additional results
Linear classifiers trained on frozen vision features solve selected BlindTest tasks with near-perfect accuracy, while transfer performance depends on the visual complexity gap. These results indicate that frozen encoder features contain sufficient task-relevant information.
- 99% accuracy is achieved on the two-circles and line-charts tasks when logistic regression is trained and tested at matched visual complexity.The diagonal axis is highlighted with a black outline in these evaluations.Source: §N.2 · p. 87
- Transfer from small to large gaps reaches accuracy comparable to the VLM baseline, supporting the sufficiency of frozen features for selected BlindTest tasks.Source: §N.2 · p. 87
- 100% accuracy is obtained when classifiers trained on close objects transfer to images with far-apart objects, across both models and stages.The reverse transfer from large to small distances performs at least 18% worse.Source: Figure F48 · p. 87
- Frozen projection-layer and vision-encoder features contain enough information to count line-chart intersections, with transfer from small to large distances harming performance less than the reverse direction.The reported impacts are 24% versus 39% for transfers between [1, 3) and [9, 11) y-coordinate distances.Source: Figure F49 · p. 88
Related papers
This paper cites
- Learning Transferable Visual Models From Natural Language Supervision2021
- Training language models to follow instructions with human feedback2022
- Sigmoid Loss for Language Image Pre-Training2023
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context2024
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks2023
- DocVQA: A Dataset for VQA on Document Images2020
Cited by
- Where Do Vision Language Models Fail at 3D Reasoning?2026
- Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning2026
- HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents2026
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm2026
- The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era2026
- Evidence-RL: Towards Evidence-intensive Visual Reasoning2026