PaperlayerView exact revision on arXiv

Source-linked AI summary

Why MLLMs Struggle to Count: Overcoming Individuation and Aggregation Bottlenecks with ConvStack

Liwei Che, Yihao Quan, Sen Fang, Hongyi Wang, Ranjay Krishna, Ruixiang Tang, Vladimir Pavlovic

arXiv:2609.37096v2cs.CV

TL;DR

MLLMs can describe image content while struggling to count objects, and the causes of this visual-grounding gap are not well understood. The paper analyzes individuation and aggregation bottlenecks and introduces ConvStack, which adds local visual structure through residual fusion. Counting-only fine-tuning improves dense counting and broader spatial benchmarks while general visual performance remains broadly comparable.

  • Problem

    MLLMs can recognize image semantics yet fail precise visual grounding, and the causes of their visual-counting difficulty are poorly understood.

    Source: §1 · p. 2

  • Method

    The paper analyzes individuation and aggregation bottlenecks and proposes ConvStack, which combines localized convolutional features with residual fusion in visual token space.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Counting-only fine-tuning of ConvStack improves dense object counting and broader spatial benchmarks, while general visual performance remains broadly comparable.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis identifies object individuation and count aggregation as distinct bottlenecks in MLLM visual counting.

    Source: §1 · p. 2

    The theorem gives a sufficient condition at the injection point; real images may not meet its evidence assumptions, and training or later language layers are not guaranteed to preserve the features.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) consistently struggle with fine-grained visual counting, yet the underlying causes remain poorly understood. In this work, we present a mechanistic analysis of this failure mode, identifying two critical bottlenecks inherent to the global attention pipeline of MLLMs. First, we reveal an individuation bottleneck stemming from image patchification: because Vision Transformers process patches independently, they struggle to group fragmented geometric features across boundaries into distinct object representations. Second, we identify a collapse in the subsequent counting aggregation process, where representation separation rapidly diminishes as numerosity increases due to attention compression. Identifying and formalizing these twin bottlenecks constitutes our first major contribution. To overcome them, we propose ConvStack, a lightweight architecture that operates directly in the visual token space to explicitly aggregate and inject local spatial structures via zero-initialized residual connections. By explicitly addressing the individuation bottleneck, ConvStack provides unambiguous geometric evidence for downstream aggregation. Remarkably, by fine-tuning exclusively on counting tasks, the model achieves substantial improvements in dense object counting and broader spatial understanding benchmarks, without compromising on general visual capabilities.

1 Introduction

The introduction uses visual counting to expose a gap between semantic scene understanding and precise visual grounding. It diagnoses individuation and aggregation bottlenecks and presents ConvStack as a targeted architectural response with improvements on counting and spatial benchmarks.

1 Introduction

  • An MLLM can describe an aerial scene correctly yet miscount its airplanes, illustrating the gap between semantic understanding and precise visual grounding.In Figure 1, Qwen3-VL-8B counts seven airplanes instead of eight, while ConvStack recovers the correct count.Source: Figure 1 · p. 1, §1 · p. 2
  • The analysis identifies patchification as an individuation bottleneck and attention normalization as an aggregation bottleneck that compresses neighboring-count representations.Independent, nonoverlapping patches impede grouping geometric features across patch boundaries; attention compression reduces hidden-state distances between adjacent counts.Source: §1 · p. 2
  • ConvStack uses localized convolutional features and deep residual fusion in visual token space to bridge patch boundaries and reinforce visual evidence downstream.Its adapter output projections are zero-initialized, preserving input features at initialization.Source: §1 · p. 2
  • Fine-tuning ConvStack exclusively on counting tasks improves dense object counting and broader spatial benchmarks, while general visual performance remains broadly comparable.The reported general-visual results include gains on some benchmarks and decreases on others.Source: §1 · p. 2

2 Related Work

Related work situates ConvStack among approaches that restore local convolutional structure to ViTs, and among counting and grounding research that emphasizes localized evidence and shortcut-resistant evaluation.

Vision Transformers and convolutional bias

  • Hybrid ViT approaches reintroduce convolutional structure, and ConvStack combines local visual features with residual routes and multilayer visual injection.Prior approaches include convolutional attention projections, convolution-attention combinations, and soft convolutional priors.Source: §2 · p. 2

Visual counting

  • Counting research treats the task as localizing countable evidence and aggregating it, while controlled experiments probe individuation and count readout across numerosity.The experiments vary patch alignment, shape, color, and size, without assuming human cognition or a universal subitizing boundary.Source: §2 · pp. 2–3

Visual grounding in language models and shortcut diagnostics

  • High VQA accuracy can coexist with shortcut behavior, so counting accuracy is meaningful only when data rules out grid or object-area cues and requires local individuation.Source: §2 · p. 3

3 Method

The method treats visual counting as object individuation followed by evidence aggregation, and probes how local spatial features and attention affect these computations. It uses controlled synthetic counting tasks to analyze patch geometry, count representations, and residual visual evidence before introducing ConvStack.

3.1 Patchification Makes Object Individuation Harder

  • On SynPoly, the CNN reaches 99% probe accuracy at epoch 16, while the ViT reaches 75.2% by epoch 30; ViT accuracy is also lower for larger counts.In a separate three-seed comparison, ViT mean accuracy is 97.7% for counts 1–3 and 80.6% for counts 4–10.Source: §3.1 · p. 3
  • 100% probe accuracy first occurs at epochs 1, 9, and 14 for center, random, and edge placements, respectively; splitting disks across patches distributes object evidence across tokens.When each disk lies in a distinct patch, object count equals occupied-patch count; all placement runs finish at 100%.Source: §3.1 · pp. 3–4
  • 99.9% mean accuracy across three seeds leaves a 0.1-point small-to-large-count gap for ConvStack, versus 12.9 points for the 1 × 1 control.The original ViT gap was 17.0 percentage points; the control retains the same depth, channel widths, pooling operations, and injection points.Source: Figure 2 · p. 3, §3.1 · p. 4
  • 77.9% to 99.8% counting accuracy and 1.570 to 5.835 class separation accompany ConvStack, with the latter exceeding the CNN’s 2.995 class-separation score.The accuracy gain is 21.9 percentage points; ConvStack also produces more compact same-count clusters and clearer class separation, particularly at larger counts.Source: Figure 4 · p. 4, §3.1 · p. 5

3.2 Count Separation Through Visual Residuals

  • With perfectly individuated objects, attention increments shrink as object count grows, reducing adjacent-count hidden-state distances and making exact readout more sensitive to perturbations.Under the stated model, larger-count clusters increasingly overlap; recovering exact counts requires a scalar decoder with sensitivity at least 1/D_n.Source: §3.2 · p. 5
  • Residual visual evidence guarantees count-independent separation only when κ > 0 and the stated evidence and interference bounds hold uniformly through the maximum count.The condition is sufficient at the injection point; it does not guarantee that training learns the features or that later language layers preserve them.Source: §3.2 · p. 6

3.3 The ConvStack Architecture

  • ConvStack pairs localized convolutional feature extraction with deep residual fusion and operates directly in pretrained MLLMs’ visual token space.Source: §3.3 · p. 6

X Wdown Kdw Wup

ConvStack reshapes visual tokens into a 2D grid, applies a convolutional bottleneck, and adds the scaled correction back to the grid. The resulting features are flattened for fusion with selected LLM layers.

X Wdown Kdw Wup

  • At each adapter location, ConvStack applies a bottleneck to a 2D visual-token grid and adds its scaled output to the input features.The bottleneck uses channel projections around a 3 × 3 depthwise convolution that mixes neighboring tokens; zero-initializing the up-projection makes the adapter an identity mapping at initialization.Source: X Wdown Kdw Wup · p. 6

4 Experiments

The experiments evaluate ConvStack on counting, spatial understanding, and general visual benchmarks, using counting-focused training with the pretrained backbone frozen. They report gains in dense counting and spatial reasoning, broadly comparable general visual performance, and clearer count representations.

4.1 Experimental Setup

  • The standard mixture combines PixMo-Count training data with 2,000 examples each from SynPoly and SynDot, while the remaining pretrained parameters are frozen.Adapters are trained at vision layer 18 and native DeepStack levels, together with the visual merger, for one epoch.Source: §4.1 · p. 7

4.2 Evaluation

  • 75.19% on PixMo-Count is 9.73 percentage points above the base model’s 65.46%; ConvStack-8B also leads on CV-Bench Spatial and SAT-Spatial.The evaluation compares ConvStack-8B with the base model and alternative spatial adapters; Table 2 covers counting and spatial benchmarks.Source: Table 2 · p. 7, §4.2 · p. 7
  • POPE F1 rises from 87.83% to 89.52%, MME from 2219.6 to 2251.0, and RealWorldQA from 69.93% to 70.33%, while MMBench and MathVista decrease slightly.ConvStack-8B improves on four of six general visual benchmarks and remains broadly comparable to Qwen3-VL-8B.Source: Table 3 · p. 7, §4.2 · p. 7

4.3 Ablation Study

  • Table 4 compares ConvStack with alternative fine-tuning strategies using the same data and hyperparameters; the results identify explicit 3 × 3 local token mixing as critical.The reported method trains 42.8M parameters in its residual adapters and visual merger.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Mechanistic Analysis

  • On 500 SynPoly images, ConvStack raises accuracy for counts 6–10 from 56.8% to 98.8%, while mean cosine silhouette increases from 0.479 to 0.707.Figure 7 shows accuracy by count class and t-SNE representations; separation improves for every adjacent count pair.Source: Figure 7 · p. 8, §4.4 · p. 8
  • Across 500 MSCOCO images, ConvStack improves S-Space readout accuracy by 3.5% horizontally, 2.3% vertically, and 2.1% in depth.Figure 8 includes a depth-readout example shifting from −0.28 to +0.08 and aggregate gains across all three dimensions.Source: §4.4 · p. 8, Figure 8 · p. 9

5 Conclusion

The conclusion attributes MLLMs’ visual-counting difficulty, especially at large numerosities, to bottlenecks in individuation and aggregation. It presents ConvStack as improving counting and broader spatial understanding while maintaining broadly comparable general visual performance.

5 Conclusion

  • The conclusion identifies individuation and counting aggregation bottlenecks as causes of MLLMs’ visual-counting difficulty, especially for large numbers.ConvStack improves internal counting-representation separation; counting-only training yields gains in visual counting and broader spatial understanding with broadly comparable general visual performance.Source: §5 · p. 9

A Datasets

The appendix describes the training, diagnostic, and evaluation datasets used in the study. It covers counting data, spatial-understanding benchmarks, and general visual benchmarks.

A Datasets

  • The standard adaptation mixture uses the PixMo-Count training split and 2,000 examples each from SynPoly and SynDot.The appendix also describes datasets used for counting diagnostics and evaluation.Source: A Datasets · p. 12

A.1 Counting Data and Controlled Diagnostics

  • PixMo-Count provides natural images with target-object counts and point annotations, while SynPoly and SynDot provide controlled synthetic counting settings.SynPoly varies polygon appearance; SynDot varies disk placement relative to patch centers and edges.Source: §A.1 · p. 12

A.2 Spatial Understanding Benchmarks

  • Spatial evaluation uses CV-Bench, SAT-real, SpatialEval, and SAT-Spatial, covering spatial relations, positions, depth, viewpoints, and navigation.The appendix specifies the evaluation subsets and scoring procedures for these benchmarks.Source: §A.2 · p. 12

A.3 General Visual Benchmarks

  • General visual evaluation uses POPE, MME, MMBench-EN, MMMU, RealWorldQA, and MathVista across object hallucination, perception, reasoning, and everyday-scene understanding.The appendix describes the evaluation splits and scoring procedures for these benchmarks.Source: §A.3 · pp. 12–13

A.4 Images for Spatial Representation Analysis

The spatial representation analysis uses 500 MSCOCO images, while the formal results model how attention compresses count differences and how that compression affects decoding. The decoder guarantees are conditional on the stated homogeneous-token assumption.

A.4 Images for Spatial Representation Analysis

  • The S-Space analysis compares horizontal, vertical, and depth information in 500 MSCOCO images using the base model and its ConvStack adaptation.The images form a representation-analysis set.Source: §A.4 · p. 13

B.1 Attention Model and Compression of Count Margins

  • For object-favoring attention (r > 1), total attention mass on object tokens increases with count but is discretely strictly concave; uniform attention gives increments of 1/M.The proposition assumes perfectly individuated objects and homogeneous object and background tokens.Source: §B.1 · p. 13

B.2 Decoder Sensitivity Under Compressed Count Intervals

  • Adjacent-count hidden-state distance is D_n = KΔA(n); exact decoding requires sensitivity of at least 1/D_n when D_n > 0.Counts are distinguishable under every additive perturbation of norm at most ε if and only if D_n > 2ε; if D_n ≤ 2ε, some admissible perturbed state has absolute error at least 1/2 for one count.Source: §B.2 · p. 14
  • The decoder result is pairwise and conditional on Assumption 1; actual image perturbations need not fill a norm ball.Heterogeneous values, multiple heads, or count-dependent residuals can change the geometry and preserve other count cues.Source: §B.2 · p. 14

B.3 Proof of Theorem 1

The proof analyzes when residual visual evidence can support a stable count readout, while the object-formation analysis separates possible sources of counting error. Its guarantees rely on shared assumptions and do not establish that convolution alone supplies them.

B.3 Proof of Theorem 1

  • When the stated assumptions hold and κ > 0, Theorem 1 yields a decoder whose error remains below 1/2 for perturbations with ε < κ/2.Rounding the decoder output to the nearest integer then returns the count.Source: §B.3 · pp. 14–15
  • The theorem requires the same evidence direction and constants across the count family; convolution alone guarantees neither the required additivity nor image coverage.It establishes a stable readout at the injection representation, not that optimization finds it or later transformations preserve it.Source: §B.3 · p. 15, §B.4 · p. 16

B.4 Object Formation and Aggregation Error

  • The object-formation and aggregation error decomposition separates two possible failure sources, but their errors can cancel and neither source is asserted to dominate universally.The unit-formation term measures cardinality error only, not incorrect identities or compensating splits and merges.Source: §B.4 · p. 15
  • Patch occupancy alone supports counting only when each object occupies one patch and each occupied patch contains one object; linear decodability additionally requires a shared feature margin.The fragmentation index describes how an object's support is distributed across tokens, without specifying a convergence rate or arbitrary multi-patch ordering.Source: §B.4 · pp. 15–16

B.5 Metrics for Controlled Counting Experiments

The controlled counting metrics distinguish exact count classification from expected-count error and quantify adjacent-class separation. The parameter-budget subsection reports ConvStack's added and trained parameter counts.

B.5 Metrics for Controlled Counting Experiments

  • Accuracy uses the most probable count class as the discrete prediction for 1,000 SynPoly images, with 100 images at each count from 1 to 10.The controlled models derive probabilities from similarities to ten count-text prototypes, each averaged from 20 caption embeddings.Source: §B.5 · p. 16
  • MAE and RMSE use the continuous expected count from the same predictive distribution, measuring deviation rather than exact classification.Accuracy measures exact classification, while MAE and RMSE measure expected-count deviation.Source: §B.5 · p. 16
  • Class separation averages normalized centroid distances over nine adjacent count pairs; larger scores indicate greater separation relative to within-class spread.Distances are computed in the original embedding space before t-SNE projection.Source: §B.5 · p. 16

B.6 Parameter Count

  • ConvStack adds 2.708M parameters and trains 42.827M parameters, including the existing visual merger.The table caption states that percentages are relative to Qwen3-VL-8B's 8.767B parameters and counts the shared DeepStack adapter once.Source: §B.6 · p. 16, Table 5 · p. 17

C Additional Ablation Study

The ablation varies ConvStack's ViT injection layer and selects layer 18 using PixMo-Count validation accuracy. Layer 18 achieves the highest reported accuracy on both PixMo-Count test and SpatialEval.

C.1 ViT Injection Layer

  • Layer 18 is selected using PixMo-Count validation accuracy after varying the injection layer from 0 to 26.Adapters are retained at all native DeepStack levels.Source: §C.1 · p. 17
  • Layer 18 exceeds runner-up layer 17 by 1.71 percentage points on PixMo-Count test.Source: §C.1 · p. 17
  • 75.14% on PixMo-Count test and 66.67% on SpatialEval: layer 18 achieves the highest reported accuracy on both benchmarks.Figure 9 reports accuracy for Qwen3-VL-8B with ConvStack across vision layers.Source: Figure 9 · p. 17, §C.1 · p. 17

D Quantitative Analysis of Count Representations

In the same hidden representation space, ConvStack improves both within-class count clustering and separation between adjacent counts. Figure 10 reports these gains across all count classes and all nine adjacent pairs.

D Quantitative Analysis of Count Representations

  • Silhouette measures image-level separation using within-count distances and the nearest other-count class; Γn compares adjacent class centroids relative to their spread.Both analyses use the final decoder state of the last prompt token before final RMSNorm, from the image and counting prompt, and compare models on the same 500 SynPoly images.Source: §D.1 · p. 18, §D.2 · p. 18, D Quantitative Analysis of Count… · p. 18

D.1 Silhouette Across Count Classes

  • Overall mean silhouette increases from 0.479 to 0.707, and the mean for counts 6–10 rises from 0.319 to 0.627.ConvStack improves the score in every count class.Source: §D.1 · p. 18, Figure 10 · p. 18

D.2 Separation Between Adjacent Counts

  • Mean Γn rises from 0.996 to 1.449 across adjacent count pairs, and from 0.558 to 1.058 for counts 9 and 10.ConvStack improves separation for all nine adjacent pairs.Source: §D.2 · p. 18, Figure 10 · p. 18

Related papers

Loading 2609.37096v2…