PaperlayerView exact revision on arXiv

Source-linked AI summary

Counting to Four is still a Chore for VLMs

Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo

arXiv:2604.10039v1cs.CV

TL;DR

VLM counting evaluations often focus on final answers, leaving limited insight into where counting-relevant evidence is retained. This paper introduces a controlled counting benchmark, probes model components, and tests a visual-attention intervention; it finds that spatial evidence weakens at the LLM stage, where attention often shifts toward textual priors.

  • Problem

    Prior evaluations largely assess final answers, leaving limited evidence about where counting-relevant information is retained within VLMs.

    Source: §6 · p. 8

  • Method

    The paper introduces COUNTINGTRICKS, probes visual evidence across model components, and tests MAS, which enforces a minimum visual-attention share during generation.

    Source: §1 · p. 2, §5.4 · p. 8

  • Results

    Across open-weight models, spatial evidence is stronger in early visual representations and projected tokens but weakens at the LLM stage, where attention often drifts toward textual priors.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Robust counting may require preserving spatial structure deeper into reasoning and promoting where-to-look grounding, beyond simply increasing visual attention.

    Source: §6 · p. 8

    MAS yields mixed effects across held-out splits and architectures, and attention regularization alone is not sufficient.

    Source: §6 · p. 8, §5.4 · p. 8

Abstract

from arXiv · show

Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

1. Introduction

VLMs can handle complex multimodal tasks yet often miscount simple objects. This study probes where counting evidence is lost and tests whether visual-attention regularization can improve counting.

1. Introduction

  • Though VLMs excel at captioning, visual question answering, and instruction following, they often miscount simple objects, with errors worsening under clutter or mild adversarial text.Source: §1 · p. 1
  • Layer-wise probes locate counting errors in failure to retain visual evidence against linguistic dominance, rather than failure of initial visual perception.The study probes the vision encoder, modality projector, and backbone LLM, and examines inaccuracies at the token level.Source: §1 · p. 2
  • MAS redistributes attention between modalities through regularization, encouraging visual re-attention and yielding up to a 1% consistent accuracy gain on object counting.The proposed intervention targets visual-information imbalance during complex counting tasks.Source: §1 · p. 2
  • COUNTINGTRICKS contains 18k cases across 32 patchification settings, covering object size, shape, adjacency, layout, and shape to assess counting in open-weight and commercial VLMs.Source: §1 · pp. 1–2

2. Related Work

Related work documents VLM counting weaknesses, develops diagnostics for visual-versus-language failures, and identifies modality imbalance; this paper positions COUNTINGTRICKS as a probe and MAS as a corrective approach.

Vision-Language Models

  • General-purpose VLMs underperform specialized counting systems despite tokenization advances, while compression findings motivate testing their reliance on text priors over dense spatial evidence [12] [16] [21].The cited tokenization approaches include Qwen2.5-VL’s dynamic resolution and LVLM-COUNT’s sub-image partitioning.Source: §2 · p. 2

Diagnostic Benchmarks for Vision-Language Models

  • Diagnostic suites expose errors on basic attributes and fine-grained counting [13] [8] [4], while perturbations or conflicting instructions can suppress visual evidence [6]; COUNTINGTRICKS probes patch layouts and object adjacency.The suite extends beyond surface accuracy to test internal failure modes for component-level analysis.Source: §2 · p. 2

Interpretability in Vision-Language Models

  • Prior work links modality imbalance to textual-prior dominance [15] [19], diluted spatial details [18], and attention redistribution [17]; this work operationalizes MAS as a differentiable fine-tuning objective.The stated aim is to correct imbalance and address visual information loss identified by probing.Source: §2 · p. 2

3. CountingTricks Evaluation Suite

COUNTINGTRICKS is a programmatically generated object-counting suite that varies visual patchification and prompt conditions, with evaluation spanning final-count accuracy and internal grounding diagnostics.

3.1. Image and Prompt Construction

  • COUNTINGTRICKS balances object count, shape, color, and placement across 32 cases, covering squares, triangles, and circles with 3–12 objects.The cases vary objects’ relationships to the vision-encoder patch grid and add size and positional complexity.Source: §3.1 · p. 2
  • The suite tests cell-centered, grid-line, and grid-intersection alignments, alongside size variation, positional jitter, enlarged circles, and densely adjacent objects.Grid-intersection placement is described as the worst-case quantization condition; enlarged circles span 2.5–4 patch sizes.Source: §3.1 · p. 3, Figure 2 · p. 3
  • Adversarial prompts pair each image with an object, color, and distractor count set to N ± 1 or N ± 2 to probe text-prior dominance.The benchmark frames rejecting the false-count premise as visually grounded behavior and shifting toward it as language bias.Source: §3.1 · p. 3

3.2. Evaluation Models & Metrics

  • The evaluation covers 10 open-weight VLMs in the 3B–11B parameter range, targeting models commonly deployed for edge and local inference.The model set includes both legacy and newer baselines.Source: §3.2 · p. 4, Table 1 · p. 4
  • The diagnostics measure final-count accuracy, attention grounding with Attn-IoU, and intermediate-feature localization quality using AP@50.Attn-IoU compares back-projected attention with the object mask; AP@50 evaluates intermediate representations at an IoU threshold of 0.5.Source: §3.2 · p. 4

4. Evaluation Results

COUNTINGTRICKS exposes substantial counting failures that vary with patchification and count complexity, while attention and probing analyses show that visual evidence is underused or degraded during language-stage reasoning.

4.1. Overall Results

  • Accuracy rose from 39.08% avg in small-object Case 1A to 52.98% avg in dilated Case 5A, but collapsed in dense adjacent cases, with InternVL3-8B at 8.9% in Case 9B.The results link small-object failures to patchification noise and dense-case failures to difficulty resolving instances whose gaps approach the patch grid’s Nyquist limit.Source: §4.1 · pp. 4–5
  • Qwen2.5-VL-7B led the 10-model benchmark at 50.52% average accuracy, while LLaVA-1.5-7B scored 11.82%.The 3B Qwen2.5-VL variant scored 36.01%, exceeding Llama-3.2-11B at 24.00%.Source: §4.1 · p. 4, Table 2 · p. 5
  • Accuracy correlated negatively with ground-truth count (average r ≈−0.78), and numeral-specific failures indicate number avoidance rather than errors caused solely by visual density.Qwen2.5-VL-7B scored 0.0% at count 11 but 20.1% at count 12; LLaVA-1.5-7B scored 0.0% at counts 7, 8, 9, and 11. The findings suggest that ambiguous visual evidence can give way to linguistic priors.Source: §4.1 · p. 5

4.2. Attention Interpretability and Spatial Signal Analysis: Evidence of Text-Prior Dominance

  • LLM layers allocated approximately 89.3% of attention to text and only ∼10.7% to visual tokens, while prompts to inspect the image did not reliably shift attention.This visual-attention imbalance is presented as evidence that generation is driven largely by linguistic priors rather than pixel-level evidence.Source: §4.2 · p. 5, Figure 3 · p. 6
  • Correct answers were often ungrounded: the mean Attention Reward Score was −0.15, and median visual-region attention was ∼42.5%.Attention maps often failed to align with ground-truth objects, suggesting that correct counts can arise from probabilistic guessing rather than spatial accounting.Source: §4.2 · p. 5, Figure 3 · p. 6

4.3. Where does visual evidence fade? Early vs Fused

  • With an identical lightweight YOLO probe containing ∼1.7M to 2.2M trainable parameters, projector features supported faster convergence and a higher AP asymptote.The small probe-capacity differences were negligible relative to the frozen VLM backbone, so performance differences were attributed to the retained representations.Source: §4.3 · pp. 5–6, Table 3 · p. 6, Figure 5 · p. 7
  • Average Precision (AP) fell from 0.554 at the projector tap to 0.282 at the LLM tap for Qwen2.5-VL, and from 0.705 to 0.372 for Qwen3-VL.AP peaked at the projector and declined at the LLM, indicating that spatial evidence encoded by the visual backbone is diluted during fusion and language reasoning.Source: Figure 4 · p. 6, §4.3 · p. 6
  • Encoder attention maps sharply localized objects, but fused-token attention became diffuse and spatially unstructured across background regions and uninformative tokens.The transition supports the finding that spatial detail exists upstream but washes out after fusion.Source: §4.3 · pp. 6–7, Figure 6 · p. 7

5. Methodology: Enforcing Visual Grounding via Modality Attention Share

The method addresses the visual attention sink—models’ tendency to favor textual tokens over visual evidence—by regularizing visual attention during generation. Its evaluation finds backbone- and dataset-dependent gains rather than a universal improvement.

5.1. Quantifying Visual Reliance (MAS)

  • MAS measures the ratio of attention mass allocated strictly to visual tokens and preserves the computational graph to query/key projections, enabling it as a training signal..Source: §5.1 · p. 7

5.2. The Visual Constraint Loss

  • A hinge penalty activates only below threshold τ, pushes attention toward the image when MAS < τ, and vanishes at MAS ≥ τ; the objective is Ltotal = LCE + λLmas.This guardrail preserves standard cross-entropy’s dominance when visual attention is sufficient.Source: §5.2 · pp. 7–8

5.3. Training Strategy and Data

  • For controlled instruction tuning, the study converts FSC-147 density maps into natural-language counting conversations and applies the MAS constraint only to assistant-response tokens.This selectively targets blind guessing during answer generation without disrupting encoding of the user’s prompt.Source: §5.3 · p. 8

5.4. Empirical Validation: MAS as an Intervention

  • The intervention uses a hinge constraint during answer generation (τ=0.4, λmas=0.1) and fine-tunes for 10 epochs under the same instruction template, without architectural changes or additional supervision.It directly targets the attention sink, where attention drifts toward textual context even when the answer is visually determined.Source: §5.4 · p. 8
  • For Ovis-2.5, MAS raises Circles 84.9%→85.2% and FSC-Val 17.5%→17.7%, but lowers FSC-Test 16.6%→16.1%.Table 4 summarizes exact-match accuracy across three backbones; its caption notes that effects depend on backbone and dataset.Source: §5.4 · p. 8, Table 4 · p. 8
  • On Qwen3-VL, Circles rises 18.2%→30.4% while FSC-Val/Test fall; on Intern3.5-VL, FSC-Val rises 16.9%→17.7%, with Circles and FSC-Test slightly lower.These outcomes show that a single fixed threshold is not universally beneficial and can trade off against output formatting or reliance on linguistic priors.Source: §5.4 · p. 8, Table 4 · p. 8
  • MAS is a useful but non-universal control knob: it can discourage blind generation in some regimes, but attention regularization alone is insufficient.The paper suggests targeting numeral tokens, normalizing for token-length effects, and encouraging where-to-look alignment as stronger interventions.Source: §5.4 · p. 8

6. Conclusion

The conclusion finds that counting-relevant spatial signal weakens from early visual representations and projected tokens to LLM layers, where attention often shifts toward textual priors. MAS brings modest in-distribution validation gains for some backbones but mixed held-out results, suggesting that increasing visual attention alone is insufficient.

Conclusion

  • COUNTINGTRICKS reveals that plausible counts can mask failures under simple visual stressors and text-in-image conflicts.Across open-weight models, spatial signal is comparatively strong in early visual representations and projected tokens but substantially weaker at the LLM stage, where attention often drifts toward textual priors.Source: §6 · p. 8
  • MAS yields modest in-distribution validation improvements for some backbones, alongside mixed effects across held-out test splits and architectures.MAS is a hinge regularizer that enforces a minimum visual-attention budget during generation.Source: §6 · p. 8
  • The conclusion argues that counting blindness can arise from LLM computation allocation, while robust counting needs deeper spatial-structure preservation and grounding constraints on where to look.Increasing attention to vision alone is not a complete solution.Source: §6 · p. 8

A. Appendix · A.1. Supplementary Experimental

The supplementary experiments report accuracies across 32 test cases and show accuracy declining as counts grow, alongside diffuse, misaligned attention in deep language-model layers.

A.1. Supplementary Experimental

  • Table 5 reports complete accuracies across all 32 test cases, with each code’s evaluation represented by 1,000 samples evenly distributed across its count range.The caption refers readers to Section 3 for the case-coding rule.Source: Table 5 · p. 10
  • Accuracy consistently declines as count magnitude increases (r ≈−0.78), and LLaVA-1.5 and Qwen2.5-VL have 0% accuracy for specific counts such as 7 and 11.The figure describes these count-specific failures as blind spots evidencing strong linguistic priors.Source: Figure 7 · p. 11
  • Across deep LLM layers 15–25, fused-token attention becomes diffuse and misaligned, losing the instance-level separation needed for accurate counting.The heatmaps characterize this as the attention signal washing out.Source: Figure 8 · p. 12

Related papers

Loading 2604.10039v1…