Source-linked AI summary
TopoPerception: A Shortcut-Free Evaluation of Global Visual Perception in Large Vision-Language Models
Wenhao Zhou, Hao Zheng, Rong Zhao
TL;DR
LVLM evaluation on semantically rich tasks does not isolate global visual perception, and local shortcuts can overstate perceptual ability. TopoPerception tests that capability using topological image properties across granularities, finding performance near random chance even at the easiest levels and lower accuracy among stronger-reasoning variants within model families.
Problem
Semantically rich LVLM benchmarks combine perception, reasoning, and language generation, while local shortcuts can let models answer without processing global image structure.
Source: §1 · p. 2Method
TopoPerception uses image topology to assess global visual perception without shortcuts, with fixed questions and options and synthetic images.
Source: §1 · p. 2, §4.1 · p. 5Results
At the simplest levels, LVLM performance is close to random chance, and stronger-reasoning variants tend to have lower accuracy within model families.
Source: §5.2 · p. 6, §7 · p. 8Takeaways & Limitations
The findings expose a fundamental limitation in global visual perception that fluent linguistic output on other problems can mask.
Source: §7 · p. 8The synthetic data differs significantly from the distribution of natural images.
Source: §6.2 · p. 7
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) typically align visual features from an encoder with a pre-trained Large Language Model (LLM). However, this makes the visual perception module a bottleneck, which constrains the overall capabilities of LVLMs. Conventional evaluation benchmarks, while rich in visual semantics, often contain unavoidable local shortcuts that can lead to an overestimation of models' perceptual abilities. Here, we introduce TopoPerception, a benchmark that leverages topological properties to rigorously evaluate the global visual perception capabilities of LVLMs across various granularities. Since topology depends on the global structure of an image and is invariant to local features, TopoPerception enables a shortcut-free assessment of global perception, fundamentally distinguishing it from semantically rich tasks. We evaluate state-of-the-art models on TopoPerception and find that even at the coarsest perceptual granularity, all models perform no better than random chance, indicating a profound inability to perceive global visual features. Notably, a consistent trend emerge within model families: more powerful models with stronger reasoning capabilities exhibit lower accuracy. This suggests that merely scaling up models is insufficient to address this deficit and may even exacerbate it. Progress may require new training paradigms or architectures. TopoPerception not only exposes a critical bottleneck in current LVLMs but also offers a lens and direction for improving their global visual perception. The data and code are publicly available at: https://github.com/Wenhao-Zhou/TopoPerception.
1. Introduction
Current LVLM pipelines may lose global visual structure during visual encoding and alignment, while semantically rich benchmarks can confound perception with reasoning and local shortcuts. TopoPerception uses topology to test global perception across controlled granularities without those shortcuts.
1. Introduction
- Converting continuous visual data into discrete tokens is lossy, and semantic alignment can discard global information without a straightforward linguistic correlate.Resizing, patching, and token reduction may also distort global structures or discard visual details.Source: §1 · pp. 1–2
- Semantically rich tasks combine perception, reasoning, and language generation, so success does not isolate visual-perception fidelity.Models may answer from stored language-model knowledge or question wording rather than the image.Source: §1 · p. 2
- Local shortcuts let models answer through specific objects or scene elements without understanding global structure, potentially overestimating visual perception.Topology is independent of local features and captures global image structure, making it a proxy for perception without those shortcuts.Source: §1 · p. 2
- TopoPerception uses topological image properties to evaluate global visual perception across granularities without local shortcuts.Its fixed question and options keep the text constant, while the benchmark tests whether visual encoding and alignment preserve global image features.Source: §1 · pp. 2–3
2. Related work
Related work identifies a gap in evaluating global visual perception independently of semantic content. TopoPerception addresses this with topological properties, while the LVLM visual pipeline presents several potential sources of information loss.
2.1. Evaluation benchmarks for LVLMs
- Semantic benchmarks leave global visual-perception fidelity unclear because semantic content introduces local shortcuts.TopoPerception aims to measure whether global visual features are preserved independently of semantic content.Source: §2.1 · p. 3
2.2. Benchmarking global perception in LVLMs
- Shape-based benchmarks can permit local curvature shortcuts, while maze tasks can be easier when one valid path suffices and offer limited difficulty control.TopoPerception instead uses topological properties to assess global structure without local cues.Source: §2.2 · p. 3
2.3. The visual pipeline in LVLMs: sources of information loss
- Visual encoders may distort global layout through resizing, padding, or patching, and token reduction may discard visual details.Cross-modal alignment can also complicate information fusion between visual representations and textual concepts.Source: §2.3 · p. 4
3. Sources of shortcuts in benchmarks
The section distinguishes statistical shortcuts based on unintended regularities from semantic shortcuts inferable from input meaning. In LVLM benchmarks, semantic shortcuts can occur even without statistical co-occurrence.
3. Sources of shortcuts in benchmarks
- Statistical shortcuts arise from unintended data regularities, such as text cues accompanying images or image elements co-occurring.In these cases, language cues or one visual element may let a model infer an answer without understanding the full image.Source: §3 · p. 4
- Semantic shortcuts can be inferred from input meaning alone, such as question wording that makes an answer possible without the image.The section states that semantic shortcuts are contained within statistical shortcuts and occur in both training-dependent and training-free settings.Source: §3 · p. 4
- Zero-shot LVLM benchmarks are primarily susceptible to semantic-level shortcuts.For example, asking for an image category can let a model use shape instead of the explicitly requested texture.Source: §3 · p. 4
4. TopoPerception benchmark
TopoPerception uses synthetic topological images and a fixed multiple-choice question to make the image the only variable. It varies image partitioning granularity to control task difficulty and assess retention of global information.
4.1. Benchmark design and task format
- TopoPerception uses synthetic topological images and a fixed question with fixed options to prevent reliance on memorized natural-image features or text alone.The five-option VQA presents one image; options B, C, and D correspond to the three topological categories, while A and E are distractors.Source: §4.1 · p. 5
4.2. Topological properties and granularity levels
- Images are generated from uniform spanning trees on connected graphs, with each difficulty level corresponding to a different spatial resolution and partitioning granularity.Coarser partitioning places weaker demands on information retention, while finer partitioning places stronger demands.Source: §4.2 · p. 5
- A model preserves image topology only when its perceptual granularity is at least as fine as the image’s partitioning granularity.With nearest-neighbor downsampling, topology may be distorted or lost below that threshold and remains stable at or above it.Source: §4.2 · p. 5
5. Experiments
TopoPerception evaluates state-of-the-art LVLMs on global visual structure, with results near random guessing even at the easiest level. Error analyses show strong option biases and predictions that scarcely vary across image categories.
5.1. Experimental setup
- The evaluation sampled 100 images per difficulty level and dataset category, yielding 300 samples per granularity level.The study evaluated GPT-4o, o4-mini, o3, Claude-sonnet-4-0, Claude-opus-4-0, Gemini-2.5-flash, and Gemini-2.5-pro using standard API calls.Source: §5.1 · p. 5
5.2. Results on the TopoPerception
- At Level 0, most models hover around the 20% random baseline, and none exceeds 33.3%, indicating weak global-structure perception.Level 0 uses a 29 × 29 resolution.Source: §5.2 · p. 6, Table 1 · p. 6
- Within each evaluated model family, variants are close in accuracy, but the stronger-reasoning variants tend to score lower.The reported ordering is GPT-4o > o4-mini > o3, Claude-sonnet-4-0 > Claude-opus-4-0, and Gemini-2.5-flash > Gemini-2.5-pro.Source: §5.2 · p. 6
5.3. Error analysis
- Models show option biases in Level 0 confusion matrices, with Claude and Gemini models favoring C and o3 strongly favoring A.The OpenAI models show more varied preferences; o4-mini’s choices are more uniform, with 19.67% accuracy.Source: §5.3 · p. 6, Figure 6 · p. 7
- For each model, prediction distributions are nearly identical across image categories and across the overall test data.Models were allowed open-ended responses at default inference settings and temperature to expose preference patterns across the output distribution.Source: §5.1 · p. 6, §5.3 · p. 7
6. Discussion
The discussion distinguishes text-only from image-local shortcuts and explains how TopoPerception is designed to address both. It argues that synthetic data isolates controlled global properties, while LVLMs’ information bottleneck and training objectives may contribute to weak global perception.
6.1. Classification of shortcuts in benchmarks
- TopoPerception addresses Type 1 text-only shortcuts with fixed questions and options, then Type 2 image-local shortcuts through topological properties.The distinction follows “Morgan’s Canon of data”.Source: §6.1 · p. 7
6.2. Synthetic data
- Synthetic images control and decouple properties that natural images mix, enabling evaluation of attributes that may not be directly assessed in natural data.The authors frame synthetic and natural data as complementary and synthetic tasks as reducing interference from prior learning.Source: §6.2 · p. 7
6.3. Interpretation of experimental results
- The authors describe a systemic information bottleneck and training-objective mismatch, with language-generation training potentially filtering out global structure as unimportant detail.They characterize LVLMs as focusing on local or salient objects rather than understanding the image in its entirety.Source: §6.3 · pp. 7–8
7. Conclusion
TopoPerception evaluates LVLMs’ global visual perception across granularities using image topology. Even at the simplest levels, models perform close to random chance, prompting questions about model design and calibration.
7. Conclusion
- Even at the simplest evaluation levels, LVLMs perform close to random chance, with answers based almost entirely on intrinsic biases.The authors describe these results as revealing a fundamental limitation masked by fluent linguistic output on other problems.Source: §7 · p. 8
- Larger scale and stronger reasoning capabilities can degrade performance, suggesting careful calibration and possibly more expressive or iterative visual encoders.The authors also suggest mechanisms that let models re-examine the image during reasoning.Source: §7 · p. 8
- TopoPerception offers a diagnostic tool for analyzing what LVLMs perceive and guiding development of future vision-language models.The paper aims to guide systems toward grasping image structure as well as describing images fluently.Source: §7 · p. 8