Source-linked AI summary
NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models
Yiming Fu, Fangjun Li, Xiujin Liu, Ruidong Ma, Hang Yu, Zhichen Lu, Kanwei He, Alessandro Di Nuovo, Angelo Cangelosi, Zhegong Shangguan
TL;DR
Numerosity perception in vision-language models is difficult to assess because existing benchmarks confound number with visual factors. NumerosityVLM controls these factors across synthetic conditions and finds that architecture explains performance differences, while numerosity signals emerge early in vision encoders and model gaps primarily align with language decoding.
Problem
Existing numerosity benchmarks confound number with object size, density, appearance, and other visual variables, limiting reliable assessment of VLM perceptual abilities.
Method
NumerosityVLM evaluates seven VLMs using a controlled benchmark that orthogonally manipulates size, spatial arrangement, and numerosity with progressive cue ablation.
Results
Model architecture explains the largest share of counting-performance variance, while numerosity signals emerge early in vision encoders and cross-model differences align with language-model output mapping.
Takeaways & Limitations
The findings associate VLM numerosity differences more strongly with later representation-to-output processing than with the initial availability of visual numerosity information.
Takeaways & Limitations
Open-ended evaluation metrics were omitted because context-length limits caused some responses to be truncated before valid numerical answers could be extracted.
Abstract
from arXiv · showhide
Vision-language models (VLMs) achieve strong performance on high-level multimodal tasks, yet numerosity perception, a cognitive ability that emerges in human infants before language acquisition, remains poorly understood in current models, as existing counting benchmarks entangle numerosity with correlated visual factors. We introduce a cognitively inspired diagnostic benchmark, NumerosityVLM, comprising 10,800 synthetic images across six controlled conditions. The benchmark orthogonally manipulates object size, spatial arrangement, and numerosity, while progressively ablating texture, shape, and color. Evaluating seven VLMs in a zero-shot setting, multi-factor analysis reveals that model architecture explains the largest proportion of performance variance (partial $ω^{2}=0.325$), far exceeding visual conditions. Layer-wise probing further shows that linearly separable numerosity signals consistently emerge at early stages of the vision encoder, while performance differences across evaluated models are primarily associated with the language model component. Code and data are publicly available at https://github.com/fuy3/NumerosityVLM-Benchmark, and https://huggingface.co/datasets/fuy3/NumerosityVLM.
1 Introduction
NumerosityVLM addresses open questions about VLMs’ fundamental perceptual abilities by providing a controlled diagnostic benchmark for numerosity perception. It evaluates representative models behaviorally and mechanistically, showing that architecture dominates performance differences while numerosity signals emerge early in vision encoders.
- Motivation: Numerosity perception offers a testbed for assessing whether VLMs possess fundamental visual abilities beyond high-level benchmark performance.The ability emerges in human infants before language acquisition, while prior work suggests a gap between reasoning and low-level perception.
- Benchmark and evaluation: NumerosityVLM introduces a controlled diagnostic benchmark with six experimental conditions and evaluates seven open-source VLMs spanning four architectural families.The evaluation combines controlled behavioral testing with mechanistic analysis.
- Benchmark design: The benchmark orthogonally controls object size, spatial arrangement, and numerosity while progressively ablating visual cues.These controls enable systematic diagnosis of numerosity perception in VLMs.
- Main findings: Model architecture explains more counting-performance variance than visual conditions and object categories.This finding comes from multi-factor attribution analysis.
- Main findings: Numerosity signals consistently emerge in early vision layers, while performance differences align strongly with the language model’s mapping from latent representations to textual outputs.Layer-wise probing separates early numerosity encoding from later language-model-dependent output mapping.
2 Related Work
Prior work characterizes human numerosity perception through rapid subitizing and approximate estimation, while VLM counting benchmarks increasingly seek diagnostic explanations. However, existing studies lack systematic factor-isolated evaluation and mechanistic analysis of numerical representations.
- Human numerosity perception: For N ≤4, human enumeration is rapid and nearly error-free, whereas larger numerosities rely on the Approximate Number System, whose variability increases proportionally with magnitude.Numerosity judgments are also influenced by non-numerical visual cues, including object size, density, and convex hull extent.
- Counting evaluation in vision–language models: VLM counting benchmarks include general-purpose datasets focused mainly on small numerosities and counting-specific datasets covering larger, denser scenes.Counting-specific datasets rely on complex real-world images, making systematic error attribution difficult.
- Diagnostic and mechanistic analysis: Recent studies attribute counting failures to dataset bias, visual clutter, and attention limitations, while other work examines cognitively inspired evaluation and numerosity representations in neural networks.Existing studies still lack systematic factor-isolated evaluation of visual determinants and mechanistic analysis of numerical representations.
3 The Diagnostic Counting Benchmark
NumerosityVLM is a controlled synthetic benchmark for mechanistically evaluating VLM numerosity perception, separating numerosity from correlated visual factors through orthogonal controls and progressive cue ablations. It spans subitizing and approximate-number regimes across six conditions, three object categories, and diverse randomized instances.
- Benchmark Overview: NumerosityVLM provides a controlled synthetic benchmark designed for mechanistic evaluation of vision–language model numerosity perception.The benchmark is inspired by studies of human numerosity perception.
- Controlled Conditions: Six conditions combine three orthogonal controls with three progressive ablations of texture, shape, and color.The control conditions manipulate object size, spatial arrangement, and numerosity independently; ablations progressively remove visual cues.
- Orthogonal Controls: Baseline preserves numerosity–total-area covariance, whereas Size-Incongruent fixes total surface area and Space-Incongruent dissociates numerosity from convex-hull area while introducing controlled crowding.Baseline uses ISA = 50 × 50 pixels^2; Size-Incongruent fixes TSA at 2.5 × 10^5 pixels^2; Space-Incongruent fixes ISA at 50 × 50 pixels^2.
- Visual-Cue Ablations: The ablation chain reuses Baseline coordinates and ISA, then removes texture, semantic shape, and color while preserving specified spatial, size, color, or luminance properties.Texture-Ablated uses simplified templates; Shape-Ablated uses solid-colored dots; Color-Ablated converts dots to grayscale.
- Numerosity Range: Numerosity levels include integers 1–4 and logarithmically spaced values from 5 to 100 with an approximate ratio of 1.45.This design covers both subitizing and approximate number system regimes.
- Stimulus Diversity: Across six conditions, three object categories each contain five subtype variations, with 1–5 subtypes randomly sampled and 50 instances generated per condition, category, and numerosity level.The categories are apples, butterflies, and human figures.
4 Experimental Setup … Prompting Strategy
The benchmark evaluates seven open-source VLMs spanning four architectures under controlled open-ended and closed-ended prompting. Evaluation uses consistent templates where compatible, with CLIP restricted to closed-ended prompts and numeric outputs extracted heuristically.
- 4 Experimental Setup: Experiments use official pre-trained weights in BF16 precision on a single NVIDIA L4 GPU with 24 GB memory.The same hardware and precision context is reported for the benchmark experiments.
- 4.1 Model Selection: Seven open-source VLMs spanning four architectures are evaluated to identify factors affecting numerosity perception and potential biases or shortcuts.The models use varied vision-language designs, including contrastive alignment, MLP projection, multimodal joint pretraining, and multi-scale feature injection.
- 4.1 Model Selection: The evaluated systems cover CLIP, LLaVA, InternVL3, and Qwen3VL architecture families with distinct vision-language integration mechanisms.CLIP aligns visual and text embeddings; LLaVA projects visual features into LLM token space; InternVL3 uses tighter fusion; Qwen3VL uses DeepStack.
- Prompting Strategy: Both open-ended and closed-ended prompting strategies are used during benchmark evaluation.The open-ended prompt asks for the object count, while the closed-ended prompt asks for the count with a selection instruction.
- Prompting Strategy: Identical prompt templates are applied across all compatible models to ensure consistent evaluation.CLIP is evaluated only with closed-ended prompts because it does not support text generation.
- Prompting Strategy: Numeric responses are batch-extracted from model outputs using heuristic rules and regular expressions.This extraction procedure is part of the benchmark’s evaluation implementation.
Evaluation Matrix
The evaluation matrix uses five complementary metrics to measure exact-count performance and error magnitude, including scale-normalized and misprediction-specific errors.
- Metric Definitions: Five metrics quantify overall performance and error magnitude: Accuracy, MAE, NAE, MAEerror, and NAEerror.Accuracy measures exact-match correctness, while MAE measures average absolute deviation.
- Metric Definitions: NAE normalizes absolute deviation by the ground-truth count to account for scale effects across numerosities.The normalization reflects the ratio-dependent nature of human numerosity perception and allows larger deviations for higher counts without disproportionate performance effects.
- Metric Definitions: MAEerror and NAEerror average absolute and normalized errors only over mispredicted samples, respectively.Their averages use m, the number of mispredicted samples.
Multi-factor Contribution Analysis
The analysis uses multi-factor ANOVA to attribute counting-performance differences across models, image conditions, and subsets, including their pairwise interactions. Effect sizes are reported with partial ω2.
- Analysis design: Multi-factor ANOVA quantifies contributions to image-level Accuracy.The analysis includes three categorical factors: Model, Condition, and Subset.
- Analysis design: The model includes pairwise interactions among Model, Condition, and Subset.These interactions are analyzed alongside the three main categorical factors.
- Effect-size measure: Partial ω2 quantifies the effect sizes of performance differences.This provides a statistical attribution of counting-performance variation to the analyzed factors.
4.3 Systematic Bias Analysis
The analysis tests whether controlled visual factors induce systematic performance biases and examines magnitude-dependent counting behavior through residual distributions across numerosity levels.
- Visual Bias Analysis: Hierarchical controlled comparisons assess robustness to size, spatial, and texture manipulations within two model groups stratified by overall performance.Comparisons include Baseline versus Size-Incongruent, Space-Incongruent, and Texture-Ablated conditions.
- Numerosity Bias Analysis: Grouped mean residuals, defined as residual = ŷ − y, reveal systematic over- or under-counting and potential scale-dependent bias across numerosity levels.Residuals were analyzed to examine magnitude-dependent counting behavior.
Layer-wise Numerosity Probing
The study probes numerosity representations across the full VLM pipeline using aggregated hidden features and frozen-backbone linear SVMs, then examines language-backbone number geometry with controlled numerical prompts and PCA.
- Pipeline Coverage: Representations are extracted from the visual embedding layer through Vision Encoder blocks, the alignment module, and language-model decoder layers.The study includes Attention and MLP outputs within Vision Encoder blocks and each layer of the vision-language alignment module.
- Representation Aggregation: Two aggregation strategies compare a dedicated global summarization token with the arithmetic mean of patch representations.The strategies reduce the dimensionality of the token set while contrasting global and spatially aggregated representations.
- Probe Design: Linear SVM probes use Shape-Ablated data with C = 1 while keeping the backbone frozen, and probing accuracy is compared with final textual counting performance.A representative model is additionally evaluated over C ∈ {0.01, 0.1, 1, 10, 100}.
- Language-Backbone Analysis: Language-backbone number geometry is tested by prompting each backbone with “The number is n.” for n ∈ {1, …, 100} and extracting representations at transformer layer L = 12.Principal Component Analysis is applied to the resulting 100 representations.
Language-backbone Numerical Geometry · 5 Results
Across controlled numerosity conditions, VLM performance varies mainly by model architecture, while visual biases and language-backbone geometry distinguish the evaluated performance tiers. Layer-wise analyses indicate that numerosity signals emerge early visually, with tier differences arising primarily during language-level transformation and decoding.
- Overall Performance: 77.55% ± 5.19 mean accuracy and 0.34 ± 0.02 mean NAEerror characterize Tier-1 models, versus 21.06% ± 5.87 and 2.39 ± 0.53 for Tier-2 models.Tier-2 performance is slightly above the 8.34% random baseline in accuracy.
- Overall Performance: Open-ended metrics were omitted because context-length limitations truncated some responses before valid numerical answers could be extracted.Representative outputs and error distributions are provided in the appendix.
- Visual Bias Analysis: 12.01% accuracy improvement occurred for Tier-2 models under Size-Incongruent conditions, while Space-Incongruent performance remained comparable to Baseline.Size-Incongruent improvements were significant for Tier 1 (p < 0.01) and Tier 2 (p < 0.001).
- Visual Bias Analysis: 80.92% ± 4.08% accuracy under color ablation exceeded 77.71% ± 8.53% under shape ablation and 72.05% ± 3.44% under Texture-Ablated conditions.Texture ablation significantly reduced performance relative to Baseline across both tiers (p < 0.001).
- Contribution Attribution: partial ω2 = 0.325 for Model explained substantial performance variance, compared with partial ω2 = 0.005 for Condition and partial ω2 = 0.001 for Subset.Interaction effects were statistically significant in some cases but had small effect sizes.
- 5.3 Numerosity Bias Analysis: 94.94% average accuracy in the 1–4 subitizing range fell to 52.15% as numerosity increased from 20 to 100, while mean NAE rose to around 0.17.Tier-1 models achieved 85.56% average accuracy with mean NAE 0.04 in the 5–20 lower ANS range.
- 5.4 Layer-wise Analysis of Numerosity Representations: Linearly separable numerosity signals emerged in early vision-encoder layers, while Tier-1 models retained more structured information than Tier-2 models during later language-model decoding.Early patch-level representations detected numerosity, and both tiers encoded robust visual representations by the encoder’s end.
- 5.5 Language-backbone Number Geometry: ρ = 0.92 ± 0.02 and logarithmic-fit R2 = 0.95 ± 0.01 characterized Tier-1 numerical geometry, versus ρ = 0.64±0.16 and logarithmic R2 = 0.43 ± 0.36 for Tier-2.Tier-1 geometry was more ordered and compressed, whereas Tier-2 scaling was weaker and more variable.
6 Conclusion and Limitations
NumerosityVLM is a cognitively inspired diagnostic benchmark for evaluating numerosity perception in VLMs using 10,800 strictly controlled synthetic images. The findings indicate that model architecture accounts for the largest share of counting-performance variance, while numerosity representations emerge early in vision encoders and cross-model differences primarily arise during language decoding.
- Conclusion and Limitations: NumerosityVLM evaluates VLM numerosity perception with 10,800 strictly controlled synthetic images.The benchmark is described as cognitively inspired and diagnostic.
- Conclusion and Limitations: Model architecture explains the largest proportion of counting-performance variance.This result comes from multi-factor analysis.
- Conclusion and Limitations: Layer-wise probing shows that numerosity representations emerge early in vision encoders.The finding concerns the location of numerosity representations across encoder layers.
- Conclusion and Limitations: Cross-model performance differences primarily arise during language decoding.This conclusion follows from the layer-wise probing analysis.