PaperlayerView exact revision on arXiv

Source-linked AI summary

Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models

Yunkai Zhang, Linda Li, Yingxin Cui, Xiyuan Ruan, Zeyu Zheng, Kezhen Chen, Yi Zhang, Diji Yang

arXiv:2604.09687v2cs.CVcs.AI

TL;DR

Existing multimodal benchmarks may not test exhaustive visual readout, leaving dense spatial fidelity insufficiently examined. The paper introduces Grid2Matrix, a controlled color-grid transcription benchmark, and finds sharp early end-to-end failures despite substantially more recoverable structure in isolated vision encoders, a gap termed Digital Agnosia.

  • Problem

    Many VLM benchmarks permit success from sparse salient cues, semantic regularities, or linguistic priors rather than exhaustive image readout.

  • Method

    Grid2Matrix uses synthetic color grids, a color-to-number mapping, and exact matrix transcription while varying grid size and color count to reduce semantic confounds.

  • Results

    Across proprietary and open-weight models, end-to-end performance collapses sharply on dense grids, while isolated vision encoders retain substantially more recoverable structure than outputs express.

  • Takeaways & Limitations

    Digital Agnosia identifies a representation-to-expression gap, and G2M offers a controlled stress test for dense spatial perception in layout-sensitive tasks.

  • Takeaways & Limitations

    Common reasoning and entity-detection benchmarks allow sparse perception, potentially leaving global spatial structure and fine-grained details untested.

Abstract

from arXiv · show

Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap \textit{Digital Agnosia}. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.

1 Introduction

The paper introduces G2M to test exhaustive visual transcription and finds that VLMs fail abruptly on surprisingly small grids. Probing shows more grid structure remains in vision encoders than end-to-end outputs express, revealing Digital Agnosia and structured dependence on spatial factors.

  • Motivation and benchmark: G2M asks VLMs to convert a colored N × N grid and color-to-integer dictionary into the exact corresponding matrix.Its synthetic design minimizes semantic confounds while varying grid size and color count to measure when dense-transcription failure emerges.
  • Main findings: Frontier VLMs exhibit a sharp early collapse in zero-shot end-to-end performance, often failing on surprisingly small grids.Exact Match can reach zero before the visual signal should be physically unrecoverable, with abrupt rather than gradual degradation.
  • Main findings: Isolated vision encoders retain substantially more recoverable grid structure than corresponding end-to-end outputs reveal.This dissociation indicates that missing information in the visual encoder alone does not explain the observed collapse.
  • Main findings: Digital Agnosia denotes a representation-to-expression gap in which visual information is encoded internally more extensively than expressed through language.The term names the paper’s proposed failure mode.
  • Further analyses: Model errors are highly structured and depend on spatial morphology, patch-grid alignment, model scaling, and multimodal alignment.Neither larger models nor stronger aligned encoders straightforwardly eliminate the problem.
  • Implications: G2M is positioned as a controlled diagnostic benchmark for studying dense spatial fidelity and how failures emerge.The benchmark targets exhaustive readout rather than success based on sparse salient cues or semantic regularities.

2 The Grid2Matrix Benchmark

G2M is a procedurally generated benchmark for exact transcription of color grids into matrices, with difficulty controlled by grid size and color count. It provides whole-grid and cell-level metrics while stressing spatial resolution under fixed image resolution.

  • Task design: Each G2M example pairs an N × N solid-color grid with a color-to-integer dictionary and requires the exact corresponding integer matrix.The known construction enables unambiguous whole-grid and per-cell evaluation.
  • Parameterized difficulty: Difficulty varies through grid size from 3 × 3 to 64 × 64 and color count from 3 to 10 while image resolution remains fixed at 512 × 512 pixels.Increasing N stresses spatial resolution, while increasing C increases information density.
  • Parameterized difficulty: At high densities, grid cells can outnumber visual patches, forcing multiple cells’ locations and colors into a single patch embedding.This creates a direct stress test for perceptual resolution.
  • Parameterized difficulty: Figure 1 spans simple sanity checks to dense configurations that exceed standard patch resolution.The examples visualize the benchmark’s difficulty spectrum.
  • Evaluation: Exact Match requires the entire predicted matrix to match ground truth, whereas Cell Accuracy measures the fraction of correctly predicted cells.Together they capture full serialization and finer-grained failure patterns.
  • Model evaluation: Zero-shot experiments cover proprietary GPT-5 and Gemini-3 models plus open-weight InternVL3.5 and Qwen3-VL families.For open-weight models, isolated vision encoders are probed before multimodal projection using a shallow convolutional spatial probe.

3 Empirical Observations: Diagnosing Digital Agnosia

G2M exposes abrupt end-to-end failures on small color grids, while spatial probing shows that frozen vision-encoder features retain substantially more grid structure. This gap localizes Digital Agnosia beyond visual encoding alone.

  • Both proprietary models achieve perfect Exact Match on 3 × 3 grids, but performance drops sharply as grid size increases.
  • GPT-5-mini reaches 0.00% Exact Match at 9 × 9, whereas Gemini-3-Flash-Preview retains 97.33% at the same resolution.
  • Qwen3-VL-8B-Instruct falls to 71.19% Cell Accuracy at 4 × 4, and neither open-weight model achieves an exact match by 9 × 9.
  • Spatial probes recover 100.00% Cell Accuracy for both open-weight models at 32 × 32, despite rapid zero-shot degradation toward the 33% random baseline.
  • At 48 × 48, InternVL’s probe reaches 98.75% Cell Accuracy versus Qwen3-VL’s 72.31%, and both remain above random guessing at 64 × 64.
  • The probe–generation dissociation indicates a representation-to-expression bottleneck termed Digital Agnosia.

4 Deep Dive: Spatial Morphology of Digital Agnosia

Digital Agnosia produces structured spatial error patterns rather than random mistakes. Recoverability depends strongly on grid-cell interactions with visual patch boundaries, not grid density alone.

  • 4.1 Visualizing the Error Landscape: Diagnostic Heatmaps: Proprietary models exhibit distinct spatial biases, including GPT-5-mini’s top-left concentration and Gemini-3-Flash’s weakness on the top and bottom rows.
  • 4.1 Visualizing the Error Landscape: Diagnostic Heatmaps: Open-weight models generally lose accuracy toward the right and bottom of the grid, consistent with difficulty maintaining spatial state during autoregressive generation.
  • 4.1 Visualizing the Error Landscape: Diagnostic Heatmaps: Vision-encoder probes retain stable, architecture-specific blind spots at the same positions across grid sizes.
  • 4.2 Grid-Patch Alignment: Neighboring grid sizes can differ substantially because performance depends on how cells align with 16 × 16 patch boundaries.
  • 4.2 Grid-Patch Alignment: Cells fully contained within patches are generally more recoverable than cells touching or crossing patch boundaries.
  • 4.2 Grid-Patch Alignment: Accuracy peaks locally at 32 × 32 and 48 × 48, while 64 × 64 underperforms nearby misaligned grids because every cell is an edge interaction.

5 Scaling and Alignment Paradoxes

Scaling and multimodal alignment improve some visual-probe measures but do not straightforwardly close the representation-to-expression gap. Alignment can strengthen recoverability while inherited blind spots and downstream expression failures persist.

  • 5.1 The Inefficacy of Model Scaling: Scaling affects Qwen3-VL and InternVL3.5 differently, so model size alone does not uniformly improve dense spatial fidelity.
  • 5.1 The Inefficacy of Model Scaling: InternVL3.5 VE scaling raises probe accuracy from 98.75% to 100.00% at 48 × 48 and from 69.27% to 73.34% at 64 × 64.
  • 5.1 The Inefficacy of Model Scaling: Despite those probe gains, scaling the InternVL3.5 model from 14B to 38B lowers zero-shot Cell Accuracy from 46.15% to 39.56% at 12 × 12.
  • 5.1 The Inefficacy of Model Scaling: Scaling can strengthen visual representations without ensuring that the downstream language model extracts or expresses their spatial details.
  • 5.2 Inherited Blind Spots and the Alignment Paradox: Alignment improves extracted-encoder spatial probing across all three Table 3 settings, but many blind spots remain visible.
  • 5.2 Inherited Blind Spots and the Alignment Paradox: The paper terms this mismatch the Alignment Paradox: stronger probe recoverability does not guarantee full zero-shot language expression or removal of inherited blind spots.

6 Related Work

Existing VLM evaluations emphasize high-level reasoning and often permit sparse visual perception, leaving low-level spatial fidelity under-tested. This motivates distinguishing encoder information capacity from the spatial detail that reaches language output.

  • Many VLM benchmarks target semantic reasoning, chart understanding, or entity detection rather than exhaustive low-level spatial fidelity.
  • Sparse-perception tasks let models attend only to salient regions or visual tokens while ignoring global structure and fine-grained details.
  • This evaluation pattern implicitly relies on vision-encoder pretraining to provide holistic perceptual acuity.
  • Prior work shows that alignment can prevent VLMs from using visual representations already captured by isolated vision encoders.
  • Perceptual resolution denotes the finest visual detail a VLM preserves and maps to linguistic tokens, distinct from input resolution or patch size.
  • Digital Agnosia describes the dissociation in which vision encoders retain high-fidelity spatial details that downstream language output only partially preserves or expresses.

7 Conclusion

Grid2Matrix tests dense spatial transcription under reduced semantic confounds and reveals a sharp end-to-end failure despite recoverable structure in isolated vision encoders. The paper names this representation-to-expression gap Digital Agnosia and finds that its severity is structured and affected by alignment and scaling.

  • Grid2Matrix tests whether VLMs can faithfully transcribe dense spatial structure while minimizing semantic confounds.
  • Across proprietary and open-weight models, end-to-end performance sharply collapses on the benchmark’s simple visual transcription task.
  • Isolated vision encoders retain substantially more recoverable grid structure than corresponding end-to-end outputs reveal.
  • Digital Agnosia names the gap between internally encoded visual information and its eventual expression through language.
  • Failure severity depends on grid-patch alignment, model scaling, and multimodal alignment rather than appearing purely random.
  • Although synthetic, G2M is presented as a controlled stress test for dense spatial perception relevant to tables, charts, forms, and GUIs.

B Color Ablations: Semantic Interference and Sensitivity

Color ablations reveal a divergence between isolated visual probing and end-to-end generation: more colors degrade encoder probing but can improve zero-shot accuracy on moderate grids. The results implicate structured retrieval and localization effects beyond simple color discrimination.

  • Color sensitivity: Increasing the palette from 3 to 10 colors decreases isolated vision-encoder Cell Accuracy across all grid sizes.
  • Color sensitivity: Encoder blind spots remain at the same locations across color counts, indicating structural biases independent of color distribution.
  • End-to-end generation: Expanded palettes frequently improve zero-shot Cell Accuracy on moderately sized grids despite making random guessing harder.
  • End-to-end generation: On 12 × 12 grids, InternVL and Qwen score above the 10% random baseline with 10 colors, while their relative advantage is smaller with 3 colors.
  • Interpretation: The proposed explanation is that distinctive colors provide retrieval landmarks and reduce coordinate-tracking or autoregressive instability during sequential readout.
  • Color-specific behavior: Color-specific behavior differs between probing and generation: White is easiest for the encoder, but end-to-end models increasingly favor Green and Yellow at 9 × 9 and 12 × 12.

C Scaling Analysis of Proprietary Models

Proprietary-model scaling improves robustness in some regimes but does not eliminate extreme-density failures, which can be abrupt, spatially patterned, or inverted across model sizes.

  • The proprietary-model results show that scaling produces neither uniformly better performance nor immunity to ultra-dense-grid degradation.
  • GPT-5.2 maintains 99.95% Cell Accuracy through 12 × 12 grids but falls to 64.42% at 20 × 20.
  • At 20 × 20, GPT-5.2 exhibits horizontal failure strides that the authors hypothesize arise from visual tokenization and row-wise reading.
  • Gemini-3-Pro maintains 100% Cell Accuracy through 12 × 12 but reaches 84.28% at 20 × 20.
  • At 20 × 20, Gemini-3-Flash reaches 87.77% and outperforms Gemini-3-Pro despite greater spatial variance.

D Grid2Matrix Evaluation Details

The evaluation separates visual-encoder perception from full-model transcription to test whether Digital Agnosia reflects missing visual information or downstream access and expression.

  • The protocol uses a frozen vision encoder with a diagnostic spatial probe alongside zero-shot inference on the full VLM.This isolates perceptual capacity from end-to-end language-mediated output.

D.1 Data Generation and Evaluation Metrics

G2M evaluates dense grid transcription across controlled synthetic inputs using end-to-end generation and spatial probes, with exact and cell-level accuracy plus spatial heatmaps.

  • Data Generation: The dataset spans 20 × 20 to 64 × 64 grids with uniformly sampled dictionary colors and 512 × 512 input images.The evaluation uses 8,000 training, 2,000 validation, and 10,000 testing samples for probing, plus separate zero-shot test sets.
  • Evaluation Metrics: Exact Match requires zero matrix errors, whereas Cell Accuracy measures the percentage of correctly predicted individual cells.
  • Spatial Analysis: Spatial heatmaps quantify correctness by image location for both zero-shot generation and spatial probes, exposing patch-boundary, regional, and decay patterns.
  • Zero-Shot Evaluation: Zero-shot evaluation uses deterministic greedy decoding and a cascading parser to assess spatial accuracy despite formatting deviations.The parser tries strict Python-array parsing, row-wise regular expressions, and fallback integer extraction before recording a parse error.

E More Result Details

Additional analyses show that spatial errors are structured by patch interactions, color identity, grid composition, and model or encoder scaling rather than determined by density alone.

  • Grid-Patch Interactions: Patch-boundary misalignment creates structured interaction patterns in both InternVL3.5-8B and Qwen3-VL-8B-Instruct heatmaps.The dominant cell within a visual patch dictates the feature representation, while recoverability declines as patch-boundary conflict increases.
  • Color and Spatial Effects: Blind spots persist across color counts, while spatial performance fluctuates across grid sizes because cells are divided by patch boundaries and area dominance effects.
  • Color Effects: Both GPT-5 and Gemini model families overpredict blue, while white typically becomes the earliest substantially underpredicted color as grids grow.
  • Scaling: Scaling improves some spatial outcomes but is not uniformly monotonic across families, model sizes, or vision encoders.InternVL degrades from 14B to 38B on the 12 × 12 grid, Qwen improves monotonically end-to-end, and larger Qwen vision encoding slightly degrades spatial fidelity.
  • Alignment: Multimodal alignment leaves Qwen’s structural blind spots largely consistent but shifts InternVL’s spatial error landscape.
Loading 2604.09687v2…