PaperlayerView exact revision on arXiv

Source-linked AI summary

Counting to Four is still a Chore for VLMs

Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo

arXiv:2604.10039v1cs.CV

TL;DR

VLMs can reason across modalities yet still fail at simple object counting, and answer-only evaluations provide limited insight into the source of these failures. This paper introduces controlled counting diagnostics and component-wise analysis, finding that visual evidence weakens at the LLM stage as attention shifts toward textual priors. MAS provides modest, mixed intervention effects, indicating that attention allocation matters but attention regularization alone is insufficient.

  • Problem

    VLMs often miscount simple visual scenes, while existing evaluations provide limited evidence about where counting failures arise inside the model.

  • Method

    The paper evaluates COUNTINGTRICKS cases across patchification and adversarial-prompt conditions, probes model components, and tests MAS, a minimum visual-attention-share regularizer.

  • Results

    Visual evidence is relatively strong in early representations and projected tokens but substantially weaker at the LLM stage, while MAS yields modest in-distribution gains with mixed held-out and cross-architecture effects.

  • Takeaways & Limitations

    Counting failures reflect both visual limitations and underuse of visual evidence during language-stage generation, so robust counting requires preserving spatial structure and improving where-to-look grounding.

  • Takeaways & Limitations

    MAS is helpful only in some regimes, and attention regularization alone is not sufficient for robust counting.

Abstract

from arXiv · show

Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

MBZUAI

The passage provides MBZUAI contact information.

  • The listed contact addresses use the mbzuai.ac.ae domain.

1. Introduction

Although VLMs perform strongly on complex multimodal tasks, they remain unreliable at basic counting and spatial grounding. The paper studies whether these errors arise from visual perception or later reliance on linguistic priors, then proposes diagnostics and an attention-based intervention.

  • VLMs achieve strong general multimodal reasoning but often fail on elementary spatial and object-awareness tasks such as counting.
  • Counting errors worsen under mild clutter and adversarial textual cues, motivating analysis of whether models use visual evidence or learned priors.
  • COUNTINGTRICKS evaluates object counting across varied patchification settings, object layouts, and shapes.
  • Layer-wise probes attach lightweight detection heads to the vision encoder, modality projector, and LLM to localize the counting bottleneck.
  • The study identifies degraded retention of visual evidence under linguistic dominance and introduces MAS to redistribute attention across modalities.
  • COUNTINGTRICKS contains 18k cases across 32 patchification settings, while MAS is reported to provide up to a 1% consistent accuracy gain.

2. Related Work

Related work establishes that general-purpose VLMs struggle with fine-grained counting and that diagnostic studies increasingly examine conflicts between visual evidence and linguistic priors. Interpretability work further motivates probing modality imbalance and visual attention sinks.

  • General-purpose VLMs remain weaker at precise object counting than specialized counting architectures despite broad reasoning capabilities.
  • Diagnostic benchmarks expose failures on visually trivial skills, including counting and fine-grained attribute recognition.
  • Adversarial perturbations and conflicting instructions can cause VLMs to ignore visual data in favor of probabilistic language generation.
  • Interpretability studies describe modality imbalance in which textual priors dominate visual evidence and spatial details dilute across fusion layers.

3. CountingTricks Evaluation Suite

COUNTINGTRICKS is a programmatically generated counting suite that varies patch-grid alignment, object size, shape, placement, density, and adversarial prompts. The evaluation combines final accuracy with attention grounding and intermediate-feature diagnostics across efficient VLMs.

  • Benchmark construction: COUNTINGTRICKS programmatically balances object count, shape, color, and placement across 32 patchification cases, with counts N ∈ [3, 12].
  • Patchification settings: Numerical prefixes encode object alignment with the patch grid, from cell-centered placement to intersection alignment that maximizes feature fragmentation.
  • Object variation: Alphabetical suffixes vary object size and positional translation, while cases 5–15 test enlarged circles, density, and adjacency.
  • Prompt construction: Adversarial prompts pair an object with a distractor count and color to probe text-prior dominance; grounded models should reject false premises.
  • Models: The study evaluates 10 state-of-the-art open-weight VLMs, focusing on models with 3B–11B parameters.
  • Evaluation metrics: The suite uses accuracy for final count matching and Attn-IoU to measure overlap between back-projected attention and object masks.
  • Evaluation metrics: AP@50 measures whether intermediate features retain spatial geometry for object localization, independently of final textual output.

4. Evaluation Results

COUNTINGTRICKS exposes substantial counting failures linked to object scale, adjacency, count complexity, and linguistic priors. Probing indicates that spatial evidence is strongest at the projector but degrades in later language layers, where attention shifts toward text.

  • Baseline Performance: Qwen2.5-VL-7B achieves the highest average accuracy at 50.52%, while LLaVA-1.5-7B reaches 11.82% across the 32 controlled regimes.
  • Layout Sensitivity and Patchification: Small objects improve from 39.08% average accuracy in Case 1A to 52.98% in Case 5A, while adjacency causes severe accuracy collapse in Cases 9–15.Qwen2.5-VL-7B rises from 56.3% to 73.3% between the cited cases.
  • Count Complexity and Number Avoidance: Average accuracy correlates negatively with ground-truth count at r ≈−0.78, with models increasingly favoring some numbers over others.Qwen2.5-VL-7B reaches 0.0% for count 11 despite 99.3% for count 2 and 20.1% for count 12.
  • Modality Imbalance: Models allocate approximately 89.3% of attention to system prompts and instructions, leaving approximately 10.7% for visual tokens.The analysis characterizes this imbalance as a visual attention sink associated with linguistic priors.
  • Grounding Analysis: Correct answers are often ungrounded, with a negative mean Attention Reward Score of −0.15 and median visual-region attention of approximately 42.5%.Attention maps frequently fail to align with ground-truth objects even when the predicted count is correct.
  • Where Visual Evidence Fades: Projector probes converge faster and reach higher AP than other taps, while AP falls from 0.554 to 0.282 in Qwen2.5-VL and from 0.705 to 0.372 in Qwen3-VL.Identical lightweight probes and negligible capacity differences support interpreting these gaps as differences in retained representation quality.

5. Methodology: Enforcing Visual Grounding via Modality Attention Share

The method diagnoses a visual attention sink and introduces MAS, a differentiable hinge-based constraint that preserves a minimum visual-attention budget during answer generation. Results across backbones show modest in-distribution gains but mixed held-out and cross-architecture effects.

  • Motivation: Probing identifies a visual attention sink in which models disproportionately attend to textual tokens and ignore relevant visual evidence during reasoning.MAS directly targets this failure mode by encouraging visual consultation during generation.
  • Quantifying Visual Reliance: MAS measures the ratio of attention mass allocated to visual tokens relative to visual and textual tokens at each layer, head, and decoding step.The formulation remains differentiable through the query/key projections, enabling use as a training signal.
  • Visual Constraint Loss: The MAS hinge loss activates only when visual attention falls below threshold τ, preserving standard cross-entropy when the model is sufficiently grounded.The total objective combines instruction-tuning loss with the grounding penalty as Ltotal = LCE + λLmas.
  • Training Strategy and Data: MAS is applied selectively to assistant-response tokens so it penalizes blind answer generation without disrupting encoding of the user prompt.The reported setup uses τ=0.4, λmas=0.1, and 10 fine-tuning epochs under the same instruction template.
  • Empirical Validation: On Ovis-2.5, MAS raises Circles from 84.9% to 85.2% and FSC-Val from 17.5% to 17.7%, but lowers FSC-Test from 16.6% to 16.1%.These results indicate that validation improvements do not necessarily transfer to held-out generalization.
  • Empirical Validation: MAS effects are backbone-dependent: it improves Qwen3-VL Circles from 18.2% to 30.4% and Intern3.5-VL FSC-Val from 16.9% to 17.7%, while reducing other scores.The mixed outcomes show attention share is a useful but non-universal control knob that can trade off against other behaviors.
  • Discussion: The ablation supports a conservative conclusion that MAS can discourage blind generation in some regimes, but attention regularization alone is insufficient.The authors point toward grounding-aware token selection, token-length normalization, and where-to-look alignment as stronger intervention directions.

6. Conclusion

The paper finds that VLM counting failures arise partly from visual evidence being weakened during language-stage reasoning, not only from initial visual perception. MAS provides modest, mixed intervention results, indicating that stronger mechanisms must preserve spatial grounding deeper in the reasoning process.

  • Conclusion: COUNTINGTRICKS exposes plausible but unreliable counting under simple visual stressors and conflicts between images and text.The diagnostic evaluation moves beyond answer-only assessment of counting competence.
  • Conclusion: Spatial evidence is relatively strong in early visual representations and projected tokens but weakens substantially in LLM stages as attention drifts toward textual priors.This pattern is consistent with a visual attention sink.
  • Conclusion: MAS enforces a minimum visual-attention budget during generation and yields modest in-distribution validation improvements for some backbones, with mixed held-out and cross-architecture effects.The intervention tests whether the identified bottleneck is amenable to direct control.
  • Conclusion: The findings suggest counting blindness is shaped by how the LLM allocates computation during generation, not purely by the vision backbone.Simply increasing visual attention is not sufficient; robust counting also requires preserving spatial structure and promoting where-to-look alignment.
  • Conclusion: The benchmarks and analyses are intended to support future architectures and training objectives that count from visual information rather than linguistic shortcuts.This is presented as a practical foundation for studying grounding failures in VLMs.

A.1. Supplementary Experimental

The supplementary experiments examine counting accuracy across 32 coded test cases and show that performance declines as counts increase, with diffuse deep-layer attention failing to preserve instance separation.

  • Supplementary Experimental: Table 5 reports complete accuracies over all 32 test cases, with each code evaluated using 1000 samples evenly distributed across its count range.The table uses the case-coding rule described in Section 3.
  • Supplementary Experimental: Accuracy has a consistent negative correlation with ground-truth count magnitude, with r ≈−0.78 across diverse geometric cases.LLaVA-1.5 and Qwen2.5-VL exhibit 0% accuracy for specific numbers such as 7 and 11.
  • Supplementary Experimental: Averaged attention in deep LLM layers 15–25 becomes diffuse and misaligned, washing out instance-level separation needed for accurate counting.The heatmaps indicate that count-relevant visual structure is not retained clearly in these later layers.
Loading 2604.10039v1…