Source-linked AI summary
DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting
Qiuyi, Zhang, Zhen Xu, Jing Gu, Seojin Bang, Arnav Mohanty Das, Yaning Hu, Gautam Kumar, Jared Lichtarge, Le Xue, Bettina Catoto, Akul Datta, Daniel Luo, Farha Khan, Marcella Valentine, Yinfei Yang, Andrew M. Dai
TL;DR
Visual counting benchmarks report correctness without distinguishing perceptual failures from reasoning or interpretation errors, and image-grounded reasoning remains difficult to assess by facet. DecoyCount pairs real-world images with registered decoys, matched cues, and scenario-based multi-hop questions; its results show sizable cue-related accuracy gaps that added context does not close. The findings indicate that stronger general reasoning does not by itself ensure aligned visual reasoning.
Problem
A final counting score cannot distinguish perception failures from unintended interpretations or reasoning shortcuts, and existing benchmarks rarely separate facets of image-grounded reasoning.
Source: §1 · p. 1Method
Results
Matched cues add 11.8–22.1 percentage points across frontier models, while scenario grounding does not improve aggregate performance and greater scale, thinking, and agentic processes often widen gaps.
Source: §1 · pp. 2–4Takeaways & Limitations
The findings identify visual-reasoning misalignment as a substantial but incomplete source of error, and stronger general reasoning does not ensure selection of the intended visual process.
Source: §1 · p. 2Because items were retained partly for frontier-model failures, DecoyCount accuracy is not a population estimate across a regular sample of images.
Source: §4.3 · p. 9
Abstract
from arXiv · showhide
Visual counting benchmarks report whether vision-language models produce the correct number, but not whether errors arise from perception or reasoning. We introduce DecoyCount, a benchmark built from 1,660 real-world images with designed decoys across seven mechanisms, along with paired prompts: a neutral question, a matched alignment cue, and a novel multi-hop, multi-image question that situates the same counting problem in a realistic scenario. Using this design, we find that counting failures are induced partly by visual misalignment: models can execute the intended counting process when cue-guided but fail to select it unaided, and we term this difference in accuracy the reasoning-alignment gap. Despite being highly capable on general reasoning, frontier models remain below 50\% accuracy on our benchmark, while alignment gaps range from 11.8-22.1 percentage points; strikingly, we observe that more capable models, greater thinking budgets, and agentic processes often induce larger gaps. Because humans may bring context that models lack, we test whether textual underspecification contributes to misalignment using a scenario-based multi-hop condition without explicit hinting, but this added context does not improve aggregate performance. Our analysis concludes that stronger general reasoning does not by itself generalize to aligned visual reasoning, leaving misalignment with human visual expectations as a substantial component of the remaining deficit.
1 Introduction
Visual counting scores collapse perception and reasoning failures, while visual-reasoning benchmarks rarely distinguish reasoning facets. DecoyCount pairs real-world counts with mechanism-specific decoys and matched cues, revealing substantial cue-related gains that additional scenario context does not eliminate.
1 Introduction
- A single final count cannot distinguish failure to perceive an object from selecting an unintended interpretation or reasoning shortcut, and image-grounded reasoning remains difficult to assess separately.Existing reasoning benchmarks emphasize explicit logical or mathematical chains and rarely separate broader reasoning facets grounded in images.Source: §1 · p. 1
- DecoyCount pairs real-world items with pre-registered decoys from seven mechanisms and matched, answer-free cues that specify the intended counting process.Its contributions describe 1,660 items and 842 multi-image questions.Source: §1 · pp. 2–3
- Scenario-based multi-hop questions add realistic context without naming the counting rule, but do not improve aggregate performance and rescue only 11.6% to 32.6% of neutral failures per family.The benchmark uses paired questions around the same underlying count, with a second image for many contextual items.Source: §1 · p. 2, Figure 1 · p. 3
- Across frontier models, neutral accuracy is 17.2% to 43.2%, five of six return decoys more often than truth, and matched cues add 11.8–22.1 percentage points.Cues also repair 2.6–6.7× more answers than they break and remove 43.3% to 66.7% of decoy hits.Source: §1 · p. 3
- Accuracy improves while reasoning-alignment gaps widen by 4.52 points with scale and 11.84 points with thinking; general CI harnesses also widen gaps by 2.48–7.17 points.The introduction concludes that stronger general reasoning does not ensure selection of the intended visual reasoning process.Source: §1 · p. 4
2 Background and Related Work
Prior work addresses counting, adversarial evaluation, and multimodal multi-hop reasoning, but often specializes in only some diagnostic dimensions. The supplied comparison describes DecoyCount as combining adversarial decoys, alignment cues, and realistic multi-hop, multi-image counting.
Counting benchmarks
- Counting benchmarks largely report aggregate or subset accuracy rather than item-specific failure mechanisms, while HoloCount adds diagnostic subsets and robustness tests.The passage also surveys crowd, synthetic-geometry, language-posed, and real-photograph counting benchmarks.Source: §2 · p. 4
Adversarial and hard benchmarks
- Visual adversarial counting benchmarks isolate canonical-count priors: VLMBias creates conflicts with familiar counts, while CounterCount finds models may attend to relevant evidence yet underuse it.The comparison passage characterizes prior benchmarks as covering only subsets of diagnostic design axes.Source: §2 · p. 4
Multi-hop reasoning
- Multimodal multi-hop benchmarks cover retrieval, grounding, spatial, and multi-image tasks, but the cited passage says none targets counting; HoloCount composition chiefly compares independent counts.The supplied table caption characterizes DecoyCount as combining decoys, alignment cues, bridge questions, and multi-image counting.Source: §2 · p. 4, Table 1 · p. 5
3 The DecoyCount Benchmark
DecoyCount uses paired counting prompts and registered decoys to compare neutral answering, targeted guidance, and scenario grounding around the same underlying count. Its collection process gates for difficulty and decoy quality, then reviews retained items for countability and ambiguity.
3 The DecoyCount Benchmark
- DecoyCount pairs neutral questions, mechanism-targeted alignment cues, and scenario-grounded multi-hop questions around the same count to separate guidance from contextual grounding.Many contextual items include a second image; the contextual condition is a grounding control, not simply a harder reasoning tier.Source: §3 · p. 5, §3.1 · p. 5
3.1 Tasks and Decoys
- Seven pre-fixed decoy families capture different counting failures, while a separate seven-domain scene taxonomy distributes each family across settings and keeps each domain below 21% of the release.Families include Lookalike, Estimate, Prior, Structure, Duplicate, Text, and Wrong Unit.Source: §3.1 · p. 5
- Decoys are pre-registered wrong counts supported by the image, and retained items must elicit at least one exact decoy hit from a gating model.Matched cues state the relevant counting rule without changing the image, target, answer space, or revealing the count, linking cue gains to process selection.Source: §3.1 · p. 6
3.2 Collection and Validation
- Three-model gates require all models to miss the neutral answer and at least one to return a registered decoy exactly; independent review retains 1,660 of 2,100 candidates after removing ambiguous or unsuitable items.The review process corrected 121 counts and replaced 79 images.Source: §3.2 · p. 6
4 Cross-Model Results
The section compares frontier models on identical counting items, then uses controlled Qwen sweeps and validity checks to examine whether scale, reasoning effort, or benchmark construction closes the reasoning-alignment gap. Across these tests, higher accuracy does not reliably mean smaller gaps, and several checks show that the effect persists under altered settings.
4.1 Frontier Model Comparison
- GPT-6-astra leads every family but retains large gaps in five of seven; Structure and Estimate have smaller gaps that guidance does not readily repair.Accuracy, decoy-hit rate, and ∆R produce different model rankings.Source: Figure 2 · p. 7, §4.1 · p. 7
- 43.17% neutral accuracy makes GPT-6-astra the leader, but cueing adds 22.09 percentage points; all six models remain below 50% neutral accuracy and exceed the noise floor in ∆R.The multi-hop condition uses a valid paired subset and is not directly comparable with neutral or cued accuracy.Source: §4.1 · p. 7, Table 2 · p. 8
4.2 Scale and Reasoning Effort
- With Qwen3.8 FlashNext, neutral accuracy more than doubles from 12.65% to 27.17% off-to-native, while ∆R grows from 8.96 to 20.80 points and peaks at 21.71 under the 30k-token cap.This is a Qwen thinking-budget sweep, not a hosted-API effort parameter.Source: §4.2 · p. 8, Figure 3 · p. 9
- From temperature 0.3 to 1.0, neutral accuracy moves by at most 0.88 points; at T = 1.5, accuracy collapses mainly because completions stop terminating.The T = 1.5 result is reported in Table 9.Source: §4.2 · p. 8
- From 4B to 27B, Qwen3.5 neutral accuracy rises 14.08%→19.38% and cued accuracy 22.91%→32.73%, widening ∆R from 8.84 to 13.35 points.The 4.52-point increase has a paired 95% CI of [2.77, 6.29]; the 9B-to-27B change is unresolved, so the authors make no monotonic claim beyond the endpoints. Neutral decoy-hit share among errors also rises from 50.92% to 62.94% (+12.02 points; paired 95% CI [10.30, 13.74]).Source: §4.2 · p. 8, Figure 3 · p. 9
4.3 Validity Checks
- After matching ground-truth counts, edited-image accuracy is 13.4–25.4 points lower than on commissioned originals, but edited versus unedited ∆R is similar in the reported model comparisons.The image groups sample the taxonomy unevenly and are not interchangeable populations; found-image accuracy remains close to originals.Source: §4.3 · p. 9
- Despite partly adversarial, out-of-distribution item selection, DecoyCount model rankings correlate with CountQA at ρ = 0.98 neutral and 0.97 cued, and with FSC147 at 0.83 and 0.94.The authors caution that absolute accuracy is not a population estimate for a regular sample of counting images.Source: §4.3 · p. 9
- Changing “are there” to “are visible” raises one Gemini-3.8-flash rollout from 32.89% to 39.58%, but can change which instances count as members of the set.The wording change repairs 246 answers and breaks 135; the effect varies by decoy family, with Estimate degrading.Source: §4.3 · p. 10
5 Understanding Visual-Reasoning Misalignment
The paired design uses counting cues to test whether failures reflect misalignment in selecting a visual reasoning process. The section finds that cues repair many failures, while tools and added context do not reliably replace targeted guidance.
5.1 Reasoning Alignment Gap
- Matched cues repair 305–437 items while breaking 63–134, and remove 43.3%–66.7% of exact decoy hits.The paired directional effect is significant for every model (exact McNemar, p < 10^-20).Source: §5.1 · p. 10
- Five families gain 11.42–30.93 percentage points from cues, while Estimate gains 1.70 points and Structure gains 5.92.Estimate’s low decoy share is consistent with a perceptual or enumeration bottleneck; Structure mixes shortcuts and miscounting.Source: §5.1 · p. 10
- At least 82% of the gain depends on cueing the actual trap; generic and mismatched controls do not differ, and prompt length cannot explain the result.Text is an exception in degree: generic or mismatched counting instructions also help, while Estimate barely responds to any cue.Source: §5.1 · pp. 10–11, Table 3 · p. 11
5.2 Agentic Harnesses and Tools
- General agentic computation can raise accuracy while widening the gap: GPT-6 with CI gains 8.41 neutral-accuracy points but widens the gap by 2.59 points.Provider-native CI also changes reasoning length and structure, so a positive aggregate ∆2R does not establish that tools harm alignment.Source: §5.2 · p. 11, Table 4 · p. 11
5.3 Contextual Grounding
- Scenario context rescues only 11.6%–32.6% of neutral failures, while losses among previously correct items leave aggregate performance unimproved.Context rescues an average 24.0% in the three unit-settling families versus 13.7% in the other four; it helps Duplicate most, while Text, Structure, and Estimate decline.Source: Figure 4 · p. 12, §5.3 · pp. 12–13
5.2 Agentic Harnesses and Tools
- Adding SAM3 to the fixed-agent CI setup raises neutral accuracy by 3.61 points and narrows the gap by 4.52 points.SAM3 narrows all seven families, most for Estimate (−10.8 points), Structure (−8.6), and Text (−5.4).Source: §5.2 · p. 12
5.3 Contextual Grounding
- Multi-hop accuracy is 24.4–48.1 percentage points higher when the paired neutral count is correct, indicating that a wrong initial visual count remains a bottleneck.Context can sometimes settle the unit, but downstream reasoning rarely compensates for a count already extracted incorrectly.Source: §5.3 · p. 13
A Implementation Details
The appendix records implementation choices and how the benchmark was assembled, screened, and scored. Its release composition uses overlapping mechanism labels across distinct scene domains.
A Implementation Details
- Images were converted to JPEG at quality 92, SVGs rasterized, and the longest side capped at 2048 pixels; no image failed.Responses use a position-first integer parser because models may label intermediate quantities as “total” or “answer”; the evaluator also records a regression test.Source: A Implementation Details · p. 15
- The multi-hop condition is scored on 1,539 items; 121 quality-control corrections were excluded because their derived answers used the old counts.Items without a secondary image remain in the evaluation with self-contained, single-image multi-hop questions.Source: A Implementation Details · p. 15
A.1 Release Composition
- The 1,660-item release has 2,203 decoy assignments, and items may represent multiple failure mechanisms rather than disjoint visual categories.Mechanism labels are tracked separately from seven scene domains, reducing the risk that a mechanism score merely proxies for one recurring setting.Source: Table 6 · p. 16, §A.1 · p. 16
A.2 Collection and Annotation Details
- Items record counts, decoys, prompts, optional second images, and source metadata; counts and decoys are set before model exposure.Multi-hop items are retained only when the true and decoy counts produce different final answers.Source: §A.2 · p. 17
- Candidates must fool all three gate models on the neutral answer, with at least one returning a registered decoy, before reviewers verify countability and visual grounding.Blind counters assess neutral and contextual tasks separately, with disagreements adjudicated; quality control also corrected counts, replaced images, and removed ambiguous or specialist items.Source: §A.2 · p. 17
A.3 Paired Repair Results
These appendices report paired repair, scale and thinking-budget controls, and a provenance comparison. Together, they show that accuracy gains can coexist with residual decoy-aligned errors and that the gap is not confined to edited images.
A.3 Paired Repair Results
- Matched cues repair 2.6–6.7× more items than they break and remove 43%–67% of neutral decoy hits.The paired direction is consistent across models: 305–437 answers are repaired, compared with 63–134 broken.Source: Table 7 · p. 17, §A.3 · p. 17
A.4 Qwen3.5 Scale-Study Configuration
- The Qwen3.5 scale study used self-hosted inference with fixed image and decoding settings, paired bootstrap intervals, and a release alignment check.The release alignment removed 29 items and corrected 11 decoy annotations; intervals use 10,000 paired primary-image-cluster bootstrap draws.Source: §A.4 · p. 17
A.5 Qwen3.8 FlashNext Thinking-Budget Sweep
- From 4B to 27B, accuracy rises by 5.30 points and other errors fall by 12.29, yet exact-decoy answers rise by 6.98 points.The decoy share among remaining errors increases by 12.02 points (paired 95% CI [10.30, 13.74]); scale removes miscellaneous errors faster than registered shortcuts.Source: Figure 5 · p. 18, §A.5 · p. 18
- Neutral accuracy is nearly unchanged from temperature 0.3 through 1.0, while temperature 1.5 produces missing integer answers in 12.73% of rollouts.The high-temperature collapse is attributed to generation instability rather than a change in counting alignment.Source: Table 9 · p. 18, §A.5 · p. 18
A.6 Provenance Control
- Edited images are harder in absolute terms, but their alignment gaps are comparable to those for unmodified public images.The provenance comparison supports robustness, not a causal claim that editing has no effect, because provenance groups also differ in taxonomy and image content.Source: §A.6 · p. 19, Table 10 · p. 19
A.7 Family-Level Alignment Results
The appendix separates family-level error patterns from wording sensitivity and cue controls. Across these views, mechanisms differ in decoy capture and cue response, so a single difficulty ordering is not universal.
A.7 Family-Level Alignment Results
- Text, Duplicate, Prior, and Wrong Unit combine frequent decoy hits with large gaps, while Estimate has the lowest decoy concentration and smallest gap.Structure and Lookalike lie between these regimes, and model-level variation cautions against treating the ordering as a universal difficulty scale.Source: §A.7 · p. 20, Table 12 · p. 20, Table 13 · p. 20
A.8 Wording-Sensitivity Breakdown
- Changing “are there” to “are visible” helps Text and Prior but harms Estimate, so wording acts as a selective rather than uniform alignment intervention.This sensitivity motivates testing whether richer contextual questions repair failures more broadly.Source: §A.8 · p. 21, Table 14 · p. 21
A.9 Cue Controls by Decoy Family
- Matched cues outperform generic and mismatched instructions in every family, while Estimate barely responds to any cue.Text is a partial exception in degree: generic or mismatched counting guidance also suppresses some transcription shortcuts.Source: §A.9 · p. 21, Table 15 · p. 21
A.10 Tool Results by Decoy Family
Tool effects on the reasoning-alignment gap depend on the agentic surface: a fixed-agent visual-measurement tool narrows it, while other contrasts vary by setup.
A.10 Tool Results by Decoy Family
- With the Qwen agent held fixed, adding SAM3 narrows the reasoning-alignment gap in every family, most strongly for Estimate and Structure.The tool arms have one rollout, so family effects are mechanism-level evidence rather than precise rankings.Source: §A.10 · p. 21, Table 17 · p. 22
- On 301 paired GPT-5.6-sol coding-agent items, tool use gains 15.61 points neutral and 12.62 points cued, narrowing the gap by 2.99 points.This paired contrast agrees with CI→CI+SAM3, although selection affects the performance level.Source: §A.10 · p. 22, Figure 6 · p. 23
- The interpreter contrast widens the gap by 7.17 points, but also changes the agent, preventing a tool-only interpretation.Figure 6 distinguishes this confounded comparison from the fixed-agent SAM3 and coding-agent contrasts.Source: Figure 6 · p. 23
B Run-to-run Variance
The study uses a pooled run-to-run noise floor of 1.98 percentage points, while emphasizing that variability differs by model and condition and the estimate covers sampling variation only.
B Run-to-run Variance
- Pooling 18 model-condition cells gives σ = 0.71% and a minimum resolvable gap of 1.98%, the threshold used throughout.The pooled estimate is preferred to any single cell because three repeats in one configuration provide only two degrees of freedom.Source: B Run-to-run Variance · p. 23
- The floor varies by condition: 1.57% cued, 2.22% neutral, and 2.10% multi-hop, so neutral comparisons need larger gaps to clear the same bar.Per-cell spread ranges from 0.19% for GPT-6-astra multi-hop to 2.47% for Kimi-k3 neutral; Kimi-k3 is noisiest in every condition.Source: B Run-to-run Variance · p. 23
- The repeats bound sampling variation, not implementation variation, because they use each model’s reference decoding rather than a seed sweep.A seed-controlled arm to measure implementation variation has not been run.Source: B Run-to-run Variance · p. 23
C Comparison to Other Benchmarks
DecoyCount accuracy tracks standard counting ability but remains substantially lower on neutral prompts, while separating models more broadly than standard benchmarks.
C Comparison to Other Benchmarks
- The best model scores 84.0% on CountQA and 81.1% on FSC147, versus 43.2% neutral and 65.3% cued on DecoyCount.The figure plots DecoyCount exact match against CountQA exact match or FSC147 accuracy within 10% of the true count.Source: C Comparison to Other Benchmarks · p. 23
- Every model’s CountQA score exceeds its neutral DecoyCount score by 41–54 points, and DecoyCount expands the model-score range from 24 points on CountQA to 29 neutral and 42 cued.None recovers its counting number even when the cue names the trap.Source: C Comparison to Other Benchmarks · p. 24
- Across nine models, CountQA rank correlations with DecoyCount are ρ = 0.98 neutral and 0.97 cued; FSC147 correlations are 0.83 and 0.94.The relationship is monotone rather than flat or inverted; DecoyCount adds a second axis to counting skill.Source: C Comparison to Other Benchmarks · p. 24
D Additional multi-hop results
Multi-hop context yields family-dependent rescues and losses, leaving aggregate improvement small; downstream accuracy remains substantially higher when the neutral count is correct.
D Additional multi-hop results
- Downstream accuracy is 24.43–48.10 points higher when the neutral count is correct, and multi-hop context rescues 24.0% of miscounts in unit-settling families versus 13.7% elsewhere.The tables report downstream accuracy conditioned on the paired neutral count and family-level multi-hop changes.Source: D Additional multi-hop results · p. 24, Table 19 · p. 25, Table 20 · p. 25
- Context rescues Duplicate cases but loses many Text cases and offers little net help for Structure or Estimate, explaining small aggregate change despite item-level movement.Table 20 reports family-level differences; Table 21 summarizes multi-hop transitions by family.Source: D Additional multi-hop results · p. 24, Table 20 · p. 25, Table 21 · p. 26
- The multi-hop condition’s downstream accuracy remains count-bottlenecked, so richer context cannot generally compensate for an incorrect visual count.The paper concludes that mechanism-matched cues directly target process selection, whereas contextual gains and losses mostly cancel.Source: D Additional multi-hop results · p. 24