PaperlayerView exact revision on arXiv

Source-linked AI summary

CompareBench: A Benchmark for Visual Comparison Reasoning in Vision-Language Models

Jie Cai

arXiv:2509.22737v3cs.CVcs.AI

TL;DR

Existing VLM evaluations rarely isolate visual comparison, leaving comparative judgments without systematic assessment. The paper introduces a suite of counting, caption-and-tag, and comparison resources; evaluations show persistent fine-grained errors, while temporal comparisons can favor models drawing on background knowledge.

  • Problem

    Existing VLM evaluations give limited attention to visual comparison, leaving comparative reasoning without systematic assessment despite its role in everyday perception.

    Source: §1 · p. 1

  • Method

    The authors build TallyBench for object counting, OmniCaps for date-keyed captions and tags, and CompareBench, a 1,200-pair QA benchmark spanning four comparison tasks.

    Source: §1 · pp. 2–3, §3.1 · p. 4, §5 · p. 10

  • Results

    Strong VLMs still make systematic errors in counting, geometric, and spatial comparison, while VLMs can surpass human accuracy on temporal comparison by drawing on encyclopedic pretraining knowledge.

    Source: §4.1 · p. 8, §5 · p. 11

  • Takeaways & Limitations

    CompareBench provides a testbed for diagnosing perception-driven limitations, including confusing object length with height, missing small counting differences, and misclassifying foreground and background.

    Source: §5 · p. 11

    CompareTemporalBench often requires historical, landmark, or public-figure knowledge beyond direct visual evidence, so it is not a pure test of visual temporal perception.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Visual comparison reasoning is a fundamental capability of vision-language models (VLMs), covering judgments of object quantity, geometric dimensions, spatial relations, and temporal order. Yet existing benchmarks rarely isolate comparison as a reasoning axis, leaving it unclear whether models can reliably perform comparative visual judgments. We introduce a benchmark suite organized around three top-level resources: TallyBench, a 2,000-image object counting benchmark; OmniCaps, a 716-image caption and tag resource; and CompareBench, a 1,200-QA visual comparison benchmark. CompareBench contains four sub-benchmarks spanning quantity, geometric, spatial, and temporal comparison, with the temporal component unifying historical scenes, landmarks, and public figures. Evaluating nine closed-source model routes from Anthropic, Google, and OpenAI on TallyBench and CompareBench reveals strong overall performance but persistent failures in counting, spatial reasoning, geometric comparison, and temporal ordering. These results show that visual comparison remains a systematic weakness of current VLMs and establish CompareBench as a focused benchmark for multimodal reasoning evaluation. All data, code, and prompts will be released at https://github.com/caijie0620/CompareBench.

1 Introduction

Existing VLM benchmarks give limited attention to visual comparison, despite its role in everyday perception. CompareBench targets that gap with four comparison tasks, supported by standalone counting and caption-and-tag resources.

1 Introduction

  • Existing evaluations emphasize recognition, description, or commonsense reasoning and rarely isolate comparative ability, despite visual comparison's role in everyday perception.The comparison dimensions include quantity, geometric properties, temporal order, and spatial relations.Source: §1 · pp. 1–2
  • TallyBench provides object-counting data, while OmniCaps supports temporal comparison with HistoryCaps, LandmarkCaps, and CelebrityCaps examples.CompareBench covers quantity, geometric, spatial, and temporal comparison.Source: §1 · p. 2, Figure 1 · p. 2
  • CompareBench contains 1,200 QA pairs across quantity, temporal, geometric, and spatial comparison, alongside TallyBench and OmniCaps as supporting resources.TallyBench is a standalone object-counting benchmark; OmniCaps contains three caption subsets.Source: §1 · pp. 2–3
  • The authors evaluate closed-source VLMs from OpenAI, Google, and Anthropic on TallyBench and CompareBench, reporting counting and comparison accuracy to reveal persistent failures.The evaluation includes the four comparison sub-benchmarks.Source: §1 · p. 3

2 Related Work

Prior multimodal benchmarks cover broad skills or narrower tasks such as description and counting, but comparative reasoning across visual entities remains underrepresented.

2.1 Multimodal Benchmarks

  • Multimodal benchmarks cover captioning, VQA, and reasoning, but rarely test direct comparison across multiple visual entities.Captioning focuses on visual-language description, while VQA tests factual and compositional questions about image content.Source: §2.1 · p. 3

2.2 Counting and Comparative Reasoning Benchmarks

  • Counting benchmarks address numerosity, but do not extend to broader comparisons such as size, distance, or temporal order.Prior work includes natural-scene and domain-specific counting, open-world and dense counting, and efforts to teach CLIP to count.Source: §2.2 · pp. 3–4
  • Even holistic benchmarks often omit questions about which object is larger, closer, or earlier, leaving comparison reasoning without systematic representation.CompareBench is presented as a complementary diagnostic suite spanning quantity, temporal, geometric, and spatial reasoning.Source: §2.2 · p. 4

3 Methodology

The methodology builds date-keyed image resources and counting data into a 1,200-sample benchmark for quantity, geometric, spatial, and temporal comparison. Its tasks draw on TallyBench and OmniCaps or use additional annotated images, with standardized answer instructions.

3.1 OmniCaps (716 Images)

  • OmniCaps provides 716 date-keyed caption-and-tag images across historical events, landmarks, and public figures, with dates supplying temporal order and captions adding context.Its three subsets are HistoryCaps, LandmarkCaps, and CelebrityCaps.Source: §3.1 · p. 4

3.2 TallyBench (2,000 Samples)

  • TallyBench pairs 2,000 images with counting questions and integer answers across 49 fine-grained categories of biological and artificial objects.The unified instruction counts recognizable partially occluded or cropped objects, excludes reflections, and requires a single integer response.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 CompareBench (1,200 Samples)

  • CompareBench contains 1,200 samples across quantity, geometric, spatial, and temporal comparison, drawing quantity and temporal tasks from TallyBench and OmniCaps and using additional annotated images for geometry and space.Tasks use standardized instructions and require a single choice from A–D.Source: §3.3 · pp. 5–6
  • CompareGeometryBench (200) compares labeled candidates on length, width, height, thickness, or diameter, while CompareSpatialBench (100) compares labeled locations by relative depth or vertical height.Spatial instructions direct models to ignore unrelated image context.Source: §3.3 · p. 6
  • CompareTallyBench (600) uses four-image grids to identify the requested maximum or minimum count, while CompareTemporalBench (300) combines historical scenes, landmarks, and public figures ordered by dates.The temporal scenarios use event dates, landmark construction or opening dates, and public figures’ birth dates.Source: §3.3 · pp. 6–7
  • Together, the four sub-benchmarks provide a systematic framework for evaluating visual comparison, complementing recognition- and description-oriented benchmarks.Source: §3.3 · p. 7

4 Experiments

Closed-source VLMs from OpenAI, Gemini, and Claude were evaluated on TallyBench and CompareBench. Results show strong overall performance alongside persistent comparison failures and a cost-accuracy tradeoff across model routes.

4.1 Results and Analysis

  • Accuracy is exact integer match on TallyBench and the proportion of correct single-choice answers on CompareBench.CompareBench questions have one correct answer among A, B, C, or D.Source: §4.1 · p. 7
  • Gemini 3.5 Flash leads average CompareBench accuracy, followed by Gemini 3.1 Pro and GPT-5.4; humans remain near ceiling on counting, geometry, and spatial tasks but drop sharply on temporal comparison.On TallyBench, human annotators achieve the highest accuracy, while Gemini 3.5 Flash and Gemini 3.1 Pro are the strongest model baselines.Source: §4.1 · p. 8, Table 3 · p. 8
  • Counting-related tasks are relatively easier, while temporal and spatial reasoning remain the most challenging; models also confuse object length with height and foreground with background.The reported analysis describes persistent errors despite strong overall performance.Source: §4.1 · p. 8
  • All three models fail on six illustrated TallyBench cases involving spoons, Labubus, sheep or goats, books, birds, and chickens.The cited cases include look-alike confusion, missed partially occluded objects, and fine-scale duplicate errors.Source: Figure 4 · p. 9
  • Temporal comparison often requires historical knowledge beyond images, and Gemini 3.5 Flash and Gemini 3.1 Pro score substantially higher than humans on that task.The paper attributes this likely to background knowledge learned from large pretraining corpora.Source: §4.1 · p. 9

4.2 Model Performance and Cost

  • Gemini 3.5 Flash achieves 93.17% average accuracy at approximately $0.00315 per query, the strongest cost-performance point among nine evaluated routes.The Pareto Frontier comprises Gemini 3.1 Flash-Lite, GPT-5.4 mini, GPT-5.4, and Gemini 3.5 Flash; the cost estimate excludes retries, caching, and provider-specific overhead.Source: §4.2 · p. 10

5 Conclusion

The conclusion presents CompareBench as a four-task visual comparison benchmark, built alongside TallyBench and OmniCaps. Experiments reveal persistent fine-grained comparison errors and a temporal case where VLMs can surpass human annotators using pretraining knowledge.

5 Conclusion

  • CompareBench contains 1,200 QA pairs across quantity, temporal, geometric, and spatial reasoning, built from TallyBench and OmniCaps.TallyBench has 2,000 counting images; OmniCaps has 716 date-keyed images covering historical events, landmarks, and celebrities.Source: §5 · pp. 10–11
  • Strong API-based VLMs still make systematic counting, geometric, and spatial errors, while pretraining knowledge can let them surpass human accuracy on CompareTemporalBench.The conclusion frames the benchmark as a testbed for both perception-driven limitations and knowledge-driven challenges.Source: §5 · p. 11
  • The authors identify fine-grained difficulty and error-type annotations, plus comparison dimensions such as color, texture, and material, as future extensions.Source: §5 · p. 11

Limitations

The benchmark is a focused diagnostic rather than a comprehensive assessment: its coverage is constrained by sample construction and temporal tasks that depend on external knowledge, and its findings characterize a limited set of model routes and settings.

Limitations

  • The 1,200-QA benchmark uses manually constructed samples with verifiable answers and does not cover all comparison dimensions, visual domains, linguistic formulations, or real-world application distributions.Its results are intended as a focused diagnostic evaluation, not a complete assessment of visual comparison abilities.Source: Limitations · pp. 11–12
  • Temporal comparison often requires historical dates, landmark years, or public figures’ birth years, so this split is not a pure test of visual temporal perception.This knowledge dependence also explains why VLMs can outperform a human baseline relying mainly on visual reasoning.Source: Limitations · p. 12
  • The evaluation covers nine closed-source API routes under a fixed prompt protocol, not open-source or fine-tuned models, multi-turn settings, visible reasoning traces, or prompt sensitivity.The findings therefore mainly characterize representative closed-source VLMs under this protocol.Source: Limitations · p. 12
Loading 2509.22737v3…