PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision language models are blind: Failing to translate detailed visual features into words

Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, Anh Totti Nguyen

arXiv:2407.06581v6cs.AIcs.CV

TL;DR

VLMs can perform well on high-level vision benchmarks yet struggle with simple low-level visual tasks requiring precise spatial perception. BlindTest evaluates seven such tasks and finds substantial errors, while probing indicates that vision encoders contain sufficient information but language models fail to decode it reliably.

  • Problem

    Existing VLM benchmarks can confound visual ability with language, memorization, or question-only inference, leaving low-level visual capability insufficiently isolated.

  • Method

    BlindTest evaluates four SOTA VLMs on seven novel, minimal-knowledge tasks involving geometric primitives and precise spatial relationships.

  • Results

    58.07% average accuracy across seven tasks, with Sonnet-3.5 highest at 77.84%, remains below the expected 100% human accuracy; probing finds encoder features sufficient for ≥99.47% accuracy.

  • Takeaways & Limitations

    BlindTest exposes visual limitations not measured by prior benchmarks, especially when primitives overlap or are close together.

  • Takeaways & Limitations

    The encoder-decoding conclusion is based on linear probing of the vision encoders in the smallest open-source VLMs, LLaVA-OneV-S and Phi-3.5.

Abstract

from arXiv · show

While large language models with vision capabilities (VLMs), e.g., GPT-4o and Gemini 1.5 Pro, score high on many vision-understanding benchmarks, they are still struggling with low-level vision tasks that are easy to humans. Specifically, on BlindTest, our suite of 7 very simple tasks, including identifying (a) whether two circles overlap; (b) how many times two lines intersect; (c) which letter is being circled in a word; and (d) the number of circles in an Olympic-like logo, four state-of-the-art VLMs are only 58.07% accurate on average. Claude 3.5 Sonnet performs the best at 77.84% accuracy, far from the human expected accuracy of 100%. Across different image resolutions and line widths, VLMs including slow-thinking models consistently struggle with those tasks that require precise spatial information when geometric primitives overlap or are close. Yet, VLMs perform at near-100% accuracy when much more space is added to separate shapes and letters. Linear probing experiments show that vision encoders contain sufficient visual information to solve BlindTest and that language models fail to decode this information into correct answers. Code and data are at: https://vlmsareblind.github.io

1 Introduction

BlindTest isolates low-level visual perception from language and reasoning, revealing substantial failures on simple geometric tasks despite strong performance on broader VLM benchmarks.

  • Existing benchmarks can conflate visual ability with language, memorization, or question-only inference, motivating tests that exclusively measure visual capabilities.
  • BlindTest targets seven minimal-knowledge tasks using geometric primitives to test whether VLMs can see rather than reason.
  • VLMs struggle to detect intersections, overlap, circled letters, and counts when geometric primitives are close, overlapping, nested, or adjacent.
  • Line-intersection accuracy ranges from ∼41% to 76%, while two-circle accuracy reaches ∼75–93% but remains below the expected 100%.
  • 58.07% average accuracy across seven BlindTest tasks leaves all four VLMs well below the expected 100% human accuracy, with Sonnet-3.5 best at 77.84%.
  • Simplifying spatial layouts substantially improves performance, including gains of +20 and +63 for circled-letter and overlapping-circle tasks.

2 Vision language models

The study evaluates leading commercial and open-source VLMs, including standard and slow-thinking systems, on BlindTest.

  • Four commercial SOTA models—GPT-4o, Gemini-1.5 Pro, Claude-3 Sonnet, and Claude-3.5 Sonnet—form the primary evaluation set.
  • The selected commercial models rank highly across seven recent vision-language benchmarks spanning subjects, diagrams, mathematics, charts, documents, and videos.
  • Eight open-source models from the LLaVA-OneV, Phi-3.5, and InternVL-2 families are also tested, but underperform the four closed-source models.
  • Two slow-thinking models are evaluated for completeness, but generally do not outperform standard VLMs on BlindTest.

3 BlindTest benchmark of 7 tasks

BlindTest comprises seven controlled tasks built from simple geometric primitives, varying spatial configurations, image sizes, and rendering conditions to probe precise visual perception.

  • The benchmark tests line-intersection counting with two piecewise-linear plots whose geometry is controlled through image size and grid-based point sampling.
  • The two-circle task varies circle size, boundary distance, orientation, resolution, and overlap status to test detection of gaps, tangency, and overlap.
  • The circled-letter task superimposes a red oval on letters from three strings, including a familiar word, a nonrepetitive long word, and a random string.
  • Additional tasks test counting intersecting or nested shapes, counting rows and columns in grids, and following single-colored paths between subway-map stations.
  • The benchmark reports 58.07% mean accuracy across the seven tasks, with Sonnet-3.5 highest at 77.84%, still below the expected 100%.

4 Results

Across BlindTest tasks, VLMs struggle with precise spatial judgments when lines, circles, letters, or shapes overlap or are close together. Their errors persist across prompts and image settings, despite strong performance on simpler isolated content.

  • 4.1 VLMs cannot reliably count line intersections: 56.84% mean accuracy across prompts and line widths shows that VLMs cannot reliably count intersections between simple 2D line plots.The best accuracy is 75.36% for Sonnet-3.5, while performance worsens as plots move closer together.
  • 4.2 VLMs cannot clearly see if two circles overlap or not: 86.70% mean accuracy on two-circle overlap judgments remains below perfect performance, with models struggling to detect small gaps or intersections.Performance is especially poor when circles are tangent or close together.
  • 4.3 VLMs do not always see the letter inside the red circle: 79.7% mean accuracy on circled-letter identification contrasts with accurate reading of the underlying word and frequent predictions of adjacent letters.Models perform better on familiar English words than on a random character string.
  • 4.4 VLMs struggle to count overlapped and nested shapes: Counting becomes difficult beyond five overlapping or nested shapes, even though models count five circles with at least 83% accuracy.Gemini-1.5 often defaults to predicting five circles, consistent with a bias toward the Olympic logo.
  • 4 Results: Slow-thinking does not solve these low-level tasks: Gemini 2.0 Flash-Thinking averages 71.59% versus 72.75% for Gemini 2.0 Flash.The authors report that the thinking tokens remain in text space and provide no benefit on BlindTest.

5 VLMs can solve most tasks when there is more space between shapes and letters

Adding space between letters and shapes, or simplifying path geometry, substantially improves VLM accuracy. These results indicate that visual proximity and overlap are central difficulty factors in the tested tasks.

  • 5.1 VLMs can more accurately identify circled letters: Adding at least one space between characters improves every model, with reported gains exceeding 20 points for two models.The cited increases reach 92% from 72% and 72% from 46%.
  • 5.1 VLMs can more accurately identify circled letters: 95% accuracy is reached by Sonnet-3.5 with three extra spaces between letters, compared with substantially lower performance without added spacing.The remaining errors include adjacent-letter confusions, treating the circle as part of a letter, and confusing g with q.
  • 5.2 VLMs can more accurately count disjoint shapes: 96% or higher accuracy is achieved by Sonnet-3 and Sonnet-3.5 when circles are pushed apart to eliminate overlap.72B-LLaVA-OneV reaches 72% on disjoint circles, and the largest reported improvement is 91% for Sonnet-3.
  • 5.3 VLMs can more accurately count paths in simplified subway maps: Simplifying subway maps by increasing straight-path probability produces near-perfect path-counting accuracy, including 0.99 at P = 0.9.Fewer turns reduce intersections and substantially improve performance over the original maps.

6 The vision encoder in open-source VLMs can extract sufficient information to solve BlindTest

Linear probes recover BlindTest answers from frozen vision-encoder and post-projection features with near-perfect accuracy. The evidence therefore places the main decoding failure in the language-model portion of the VLM pipeline.

  • 6 The vision encoder in open-source VLMs can extract sufficient information to solve BlindTest: At least 99.47% linear-probing accuracy on two-circle and line-intersection tasks shows that frozen vision features retain sufficient low-level information.A SigLIP-based vision encoder reaches 100% on both tasks.
  • 6 The vision encoder in open-source VLMs can extract sufficient information to solve BlindTest: At least 99.58% accuracy after the projection layer shows that most visual information is preserved through projection before language decoding.The VLMs themselves perform substantially worse on the same BlindTest tasks.
  • 6 The vision encoder in open-source VLMs can extract sufficient information to solve BlindTest: The authors conclude that language models access the necessary visual information but fail to decode it into correct language outputs.This conclusion contrasts near-perfect probe accuracy with lower end-to-end VLM accuracy.
  • 6 The vision encoder in open-source VLMs can extract sufficient information to solve BlindTest: Classifiers trained on harder close-circle examples generalize to farther-apart circles, whereas classifiers trained only on easy examples perform poorly on close cases.This pattern supports greater robustness from training on difficult spatial configurations.

7 Related Work

BlindTest complements high-level chart, document, and diagram benchmarks by directly testing primitive visual recognition with minimal prior knowledge. The paper frames these low-level failures as distinct from the reasoning demands of benchmarks such as ARC.

  • Related work: VLMs can score highly on established benchmarks, including 95.2% on DocVQA, 90.8% on ChartQA, and 94.7% on AI2D for Sonnet-3.5.The paper contrasts these results with failures on primitive spatial tasks.
  • Related work: BlindTest is designed as a low-level visual sanity check that humans can solve easily with minimal prior knowledge or complex reasoning.This distinguishes it from real-world, topic-specific benchmarks that may require extensive prior knowledge.
  • Related work: Unlike ARC, which requires reasoning over abstract patterns after identifying shapes, BlindTest directly evaluates recognition of primitive shapes and spatial relations.The distinction isolates visual recognition from higher-level pattern reasoning.
  • Related work: The paper reports that its low-level limitations differ from prior work on recognizing content inside red circles over natural objects.Its circled-letter task targets optical character recognition rather than object recognition in natural images.

8 Discussion and Conclusion

BlindTest is designed to isolate whether VLMs can visually perceive simple geometric primitives without relying on memorization or high-level reasoning. The results show persistent failures on precise spatial judgments, while simplification and linear probing indicate that visual encoders contain usable information but language models often fail to decode it.

  • 8 Discussion and Conclusion: BlindTest tests VLMs on seven novel low-level visual tasks involving simple geometric primitives and minimal world knowledge.The benchmark is intended to reduce the influence of memorization and answering without using the image.
  • 8 Discussion and Conclusion: Prompting with in-context examples, chain-of-thought, or meta-prompting does not improve BlindTest accuracy.The authors interpret this as consistent with BlindTest being primarily a low-level visual task rather than one requiring extended reasoning.
  • 8 Discussion and Conclusion: The reported BlindTest failures suggest limitations for real-world visual tasks involving paths, intersections, and counting objects in crowded scenes.The paper gives subway maps, street maps, directed graphs, music sheets, and crowded scenes as examples.

F.7 Results for fine-tuning Bunny on the two touching circle

Fine-tuning Bunny on the two-circle task yields limited and unstable gains that do not generalize reliably. Performance varies with distance, and the experiments indicate that fitting the training data does not solve the underlying visual-discrimination problem.

  • F.7 Results for fine-tuning Bunny on the two touching circle: Fine-tuning improves Bunny from 17.1% overlap and 11.7% touching accuracy to at most 36.8% on smaller datasets, while larger datasets do not consistently help.With 50K samples, the model sometimes generated only the end-of-text token.
  • F.7 Results for fine-tuning Bunny on the two touching circle: Near-zero training losses coexist with poor generalization, indicating overfitting rather than robust learning of the task.The authors suggest that training may require multiple tasks or that the problem may not have a simple solution.
  • F.7 Results for fine-tuning Bunny on the two touching circle: Bunny’s performance improves for overlapping circles but fails to generalize accurately when the circles are far apart.The distance breakdown is shown in Fig. F6.
  • F.7 Results for fine-tuning Bunny on the two touching circle: VLMs often confuse adjacent letters or interpret the red oval as part of the circled character, with extra spacing improving performance.Sonnet-3.5 reaches 95% accuracy with three extra spaces between letters, while residual errors include adjacent-letter and oval-related confusions.
  • F.7 Results for fine-tuning Bunny on the two touching circle: Prompt wording and font choice have little effect on the circled-letter task, whereas word structure and spacing affect accuracy.Mean accuracy is 90.36% on question and 79.86% on civilization, and prompt variants produce similar results.

H.1 Benchmark Construction Details

The line-intersection benchmark systematically varies image resolution, line geometry, thickness, and intersection count in a balanced dataset. Results show that increasing resolution does not resolve the models’ difficulty, with zero-intersection cases especially challenging.

  • H.1 Benchmark Construction Details: The benchmark varies line color, image size, grid-based point placement, x- and y-coordinates, line thickness, and the number of intersections.The construction uses a 12×12 grid and fixes x-coordinates while sampling y-coordinates.
  • H.1 Benchmark Construction Details: Increasing image resolution does not help VLMs detect or count intersections in two-dimensional lines.The result is shown in Fig. F22.
  • H.1 Benchmark Construction Details: Models generally perform better on examples with two intersections and worse on examples with zero intersections.Sonnet-3.5 is the exception in the breakdown, performing best on one-intersection examples.

I.1 Benchmark Construction Details

The nested-square benchmark varies square count and rendering parameters to test precise counting. VLMs remain unreliable, with performance affected by model and line thickness.

  • Benchmark construction: Each square is entirely contained within the previous square, using a reduction factor and centers chosen to keep all squares visible.The benchmark varies initial size, center placement, and line thickness.
  • Benchmark construction: 120 images contain 2, 3, 4, or 5 nested squares, with 30 samples for each depth.Squares are generated with varied random locations and line widths.
  • Evaluation: The benchmark uses two equivalent counting prompts and extracts numerical answers from curly-braced responses.Ground-truth counts are 2–5, giving a 25% random-baseline accuracy.
  • Results: Sonnet-3.5 becomes more accurate as line width increases, whereas GPT-4o shows the opposite trend.The results suggest GPT-4o confuses squares when borders are thick.
  • Results: 92.08% is Sonnet-3.5’s best nested-square accuracy, still below the 100% human expectation as counts rise from 2 to 5.The task includes 120 images with 30 samples per depth.

J.1 Benchmark Construction Details

The overlapping-shape benchmark varies shape type, count, color, size, resolution, distance, and line width. VLMs generally struggle to count overlapping shapes, with biases and model-specific sensitivity to rendering changes.

  • Benchmark construction: Images use three resolutions, two color schemes, multiple shape sizes, and two line widths while keeping interlaced shapes close together.The setup tests whether these visual attributes affect counting overlapping shapes.
  • Benchmark construction: The benchmark contains 480 images: 240 circles and 240 pentagons, distributed across shape counts, colors, resolutions, and line widths.Ground-truth counts range from 5 to 9, with a 20% random-baseline accuracy.
  • Results: VLMs perform better on overlapping circles than pentagons, while resolution and color usually have minimal impact.Sonnet-3.5 improves with larger images, whereas GPT-4o benefits from colored shapes.
  • Results: Coloring usually does not help VLMs and can reduce performance except for GPT-4o.The authors expected color coding to make shapes more distinct.
  • Results: Gemini-1.5 often predicts five circles regardless of the actual count, indicating a bias toward the Olympic logo.This five-circle tendency largely disappears for pentagons.

K.1 Benchmark Construction Details

The benchmark evaluates counting in grids and subway-like path diagrams under varied sizes, layouts, resolutions, and line thicknesses. VLM accuracy declines as spatial complexity increases, although text-containing grids and simplified paths can improve some results.

  • Grid construction: The grid benchmark contains 264 blank and text-containing images across varied sizes, row and column counts, and line thicknesses.Both square and rectangular grids are included.
  • Grid results: Including text in grid cells improves all model accuracies, with Sonnet-3.5 outperforming the other models on both grid types.The benchmark separately evaluates empty and text-containing grids.
  • Grid counting: 70.53% column-counting accuracy and 60.83% row-counting accuracy remain far below the expected 100%.Models cannot reliably count either dimension alone.
  • Subway construction: The subway benchmark uses four stations, an 18×18 hypothetical grid, two resolutions, two line thicknesses, and 1–3 outgoing paths per station.It contains 180 generated images and excludes cases with zero paths between queried stations.
  • Subway paths: All VLMs except Sonnet-3 perform worse as the number of paths per station increases in subway-like maps.Sonnet-3.5 benefits consistently when the probability of straight paths increases.

N.2 Additional results

Linear probes show that frozen vision features contain enough information for selected BlindTest tasks. The difficulty therefore lies in converting visual information into correct language outputs, especially across different spatial complexities.

  • Linear probing: 99% accuracy is achieved by logistic-regression probes on two-circle and line-chart tasks when training and testing use the same visual complexity.The probes use frozen features from the vision encoder and projection layer.
  • Transfer results: Transferring probes from small to large gaps retains accuracy comparable to VLM baselines, supporting the sufficiency of frozen features for selected tasks.The reverse transfer, from large to small gaps, causes larger performance losses.
Loading 2407.06581v6…