PaperlayerView exact revision on arXiv

Source-linked AI summary

CaptionQA: Is Your Caption as Useful as the Image Itself?

Shijia Yang, Yunong Liu, Bohan Zhai, Ximeng Sun, Zicheng Liu, Emad Barsoum, Manling Li, Chenfeng Xu

arXiv:2511.21025v2cs.CV

TL;DR

Caption evaluation often does not test whether captions can stand in for images in downstream tasks. CaptionQA uses domain-specific questions answered from captions alone to assess that utility, and reports substantial caption-versus-image utility gaps across models. Its results make caption utility directly measurable within the benchmark’s downstream-LLM setting.

  • Problem

    Existing evaluation practices do not directly establish whether captions preserve the image information needed for real downstream tasks.

    Source: §1 · p. 1

  • Method

    CaptionQA uses domain-specific taxonomies and dense multiple-choice questions, answered by a text-only LLM using captions alone, to evaluate caption utility.

    Source: §1 · pp. 1–2

  • Results

    Strong proprietary models lose 9.2–16.4% when moving from QA-on-image to QA-on-caption, and caption utility can diverge by up to 32% for models close on standard image-QA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The open-source construction pipeline lets researchers extend utility evaluations by specifying a domain-specific taxonomy and running question generation.

    Source: §6 · p. 8, §12 · p. 15

    CaptionQA assumes that utility is what a downstream LLM can recover, so correct guesses from world knowledge or priors may reflect underspecified captions or questions solvable without the image.

    Source: §13 · p. 20

Abstract

from arXiv · show

Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, and multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains--Natural, Document, E-commerce, and Embodied AI--each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains. The code is available at https://github.com/bronyayang/CaptionQA.

1 ADVANCED MICRO DEVICES, INC. 2STANFORD UNIVERSITY 3INDEPENDENT RESEARCHER 4NORTHWESTERN UNIVERSITY 5UT AUSTIN Project page: https://captionqa.github.io/website/

CaptionQA addresses the gap between conventional caption evaluation and captions’ use as stand-ins for images in downstream systems. It evaluates captions by whether they preserve information needed for domain-specific tasks, while prior approaches often miss that utility.

1. Introduction

  • Existing evaluation relies on image-QA, outdated caption benchmarks, or complex and narrow evaluations that do not reveal how captions serve real-world use.Image-QA mainly measures passive visual question answering, while newer evaluations tend to be non-deterministic, natural-image-focused, or limited to simple facts.Source: §1 · p. 1
  • CaptionQA measures whether captions preserve details needed for downstream tasks, rather than how much detail they contain.Its questions are grounded in domain-specific taxonomies and require image-level utility to answer.Source: §1 · pp. 1–2
  • The benchmark covers Natural, Document, E-commerce, and Embodied AI, with taxonomies tailored to each domain’s useful information.The domains emphasize, respectively, object properties and spatial relations; layout and document structure; product attributes; and perception and task-relevant robotics cues.Source: §1 · p. 1, Figure 1 · p. 2
  • Caption utility falls 9.2–16.4% for strong proprietary models, and open-source models show gaps of 11–32.4%, with Embodied AI exceeding 40%.Models close on standard image-QA can diverge by up to 32% on CaptionQA.Source: §1 · p. 2
  • CaptionQA pairs generated captions with dense, taxonomy-grounded questions answered by a separate text-only LLM using only the caption.The benchmark contains 33,027 questions, averaging 50.3 per image, and the construction and quality-check pipeline is open-source and extendable.Source: §1 · pp. 2–3

2. Related Work

  • Prior caption metrics measure text overlap or parsed facts, while VLM-as-a-Judge methods can be non-deterministic and conflate utility with verbosity.Existing QA-based evaluation also tests a different capability or lacks deep taxonomy and cross-domain coverage.Source: §2 · p. 3
  • CaptionQA’s distinction is a downstream-driven definition of a good caption, with deterministic evaluation and taxonomy-driven diagnostics across specialized domains.The paper frames this as evaluating the active captioning task rather than passive fact retrieval or statement judging.Source: §2 · p. 3

3. Task and Evaluation Setup

CaptionQA evaluates captions by testing whether a text-only QA model can use them in place of images. Its setup combines domain-specific questions, multiple caption prompts, explicit scoring for correctness and missing information, and a QA reader selected for speed, faithfulness, performance, and stability.

3.1. Task Formulation

  • A target MLLM generates each caption, then a text-only LLM answers the corresponding benchmark questions using only that caption.Answers are scored with a multiple-choice scheme to measure how much task-relevant information the caption preserves.Source: §3.1 · p. 3, §3 · p. 3

3.2. Benchmark Scope & Statistics

  • The benchmark spans natural images, documents, e-commerce, and embodied AI, with 25 top-level categories, 69 subcategories, and 33,027 questions over 657 images.Questions average 50.3 per image and are organized around each domain’s information needs.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3. Caption Prompts

  • Four shared caption prompts—Long, Short, Simple, and Taxonomy-Hinted—test prompt sensitivity, and Simple is recommended as the default for balancing performance and prompt length.Taxonomy-Hinted explicitly lists domain taxonomy perspectives for the captioning model.Source: §3.3 · p. 4

3.4. Benchmark Scoring

  • Non-yes/no questions include a “Cannot answer from the caption” option, and scoring rewards correct answers, penalizes incorrect ones, and gives partial credit for cannot-answer responses.The per-question scores are averaged, and answer options are shuffled to reduce order bias.Source: §3.4 · p. 4

3.5. LLM QA Model Selection

  • Candidate QA readers are compared on faithfulness, efficiency, performance, and stability before selection for the main experiments.The evaluation checks empty-caption behavior, questions per second, accuracy on shared caption-question pairs, and run-to-run accuracy variation.Source: §3.5 · p. 4
  • Qwen2.5 72B was selected as the QA reader, delivering 21.14 QPS and ±0.02% accuracy variation, despite slightly lower faithfulness and accuracy than GPT-5.It is roughly two orders of magnitude faster and evaluates the full benchmark in 25 minutes on one AMD MI325 GPU.Source: §3.5 · p. 4, Figure 2 · p. 5

4. Benchmark Construction

CaptionQA uses a domain-agnostic construction pipeline in which a domain-specific taxonomy guides image selection, question generation, filtering, deduplication, and quality control.

4.1. Taxonomy Design

  • Human experts define two-level, domain-specific taxonomies that organize information downstream tasks may require.GPT-5 drafts candidate categories; experts review, merge, and prune them for coverage and clarity.Source: §4.1 · p. 5

4.2. Image Collection

  • Human experts curate diverse, challenging images, including photographs, document screenshots, product pages, and embodied-AI frames.The embodied-AI images include single- and multi-view frames from existing datasets.Source: §4.2 · p. 5

4.3.1. Question generation.

  • Three generators produce taxonomy-targeted multiple-choice questions for each image, retaining links from questions to their taxonomy nodes.GPT-5, 4o, and o4-mini generate candidate questions, choices, and answers for downstream organization by domain and category.Source: §4.3.1 · p. 5

4.3.2. Filtering text-answerable questions.

  • A text-only blind test removes questions that a model answers above an option-dependent threshold without seeing the image.Qwen2.5-72B attempts each question ten times with shuffled options, filtering for questions that genuinely require visual input.Source: §4.3.2 · p. 5

4.3.3. Question deduplication.

  • Embedding-based, per-image deduplication groups semantically similar questions and selects a central representative while preserving source and taxonomy links.The pipeline forms similarity groups and selects each group's medoid question.Source: §4.3.3 · pp. 5–6, Figure 3 · p. 6

4.4. Quality Control & Human Verification

  • Dual-VLM agreement retains validated questions directly, while disagreements and flagged items go to human refinement; post-quality question accuracy increased significantly.The workflow reduced the human question-review workload by more than 90%; Table 2 reports accuracy before and after quality control across four domains.Source: §4.4 · p. 6, Table 2 · p. 6

5. A Gap between Caption and Image Utility

CaptionQA reveals that captions retain less task-relevant information than their source images, and that longer or more elaborate prompts do not reliably close this gap. The size of the gap varies across models and domains, while simple prompts capture most gains associated with longer captions.

5. A Gap between Caption and Image Utility

  • Strong proprietary models show 9.2–16.4% image-to-caption utility gaps, while open-source gaps range from about 11% to 32.4%.Figure 4 plots the QA-on-image versus QA-on-caption CaptionQA Acc. difference averaged across four domains.Source: Figure 4 · p. 6, §5 · p. 7
  • GPT-5 scores 90.3% with the Simple prompt and Gemini 2.5 Pro around 90.1% with Long; Qwen3-VL-30B-A3B and GLM-4.1V-9B lead open models at 87.1% and 85.8%.These are overall CaptionQA scores; E-commerce is typically easiest (77–96%), while Embodied AI is most challenging (around 66–87%).Source: §5 · p. 6, Table 3 · p. 7
  • Models close on QA-on-image can differ sharply on QA-on-caption: GPT-5 and LLaVA-OneVision-7B differ by about 9% and 32.3%, respectively.Claude Sonnet 4.5 and LLaVA-OneVision-7B differ by only 1.1% on QA-on-image but by 17.2% on caption-based scores.Source: §5 · pp. 7–8
  • Embodied AI has the largest domain gaps, including over 40% for LLaVA-OneVision-7B, while E-commerce has the smallest gaps across models.The gap is 13.81% for GPT-5 and 29.06% for Claude Sonnet 4.5 in Embodied AI.Source: Table 4 · p. 7, §5 · p. 8

5.1. Can complex prompt mitigate the gap?

  • Taxonomy-Hinted prompts reduce performance in 23 of 25 categories, with a mean decline of 10.8%.The largest reported drops are 33.1% in Document Domain-Specific Evaluation and 7.8% in Embodied AI Perception.Source: §5.1 · p. 8, Figure 5 · p. 8
  • Complex instructions can lead models to imitate the requested format rather than describe image content adequately.The authors characterize this as a shift from content grounding toward format imitation and a tension between instruction-following and visual understanding.Source: §5.1 · p. 8

5.2. Can longer caption mitigate the gap?

  • Moving from Short to Simple captions (21 to 317 words) yields +33.8% mean gains, capturing 99% of Long's +34.2% benefit.Increasing length from Simple to Long (317 to 471 words) adds only +0.35% mean improvement, with every category changing by less than 2%.Source: §5.2 · p. 8
  • Document and E-commerce categories gain 41–56% from Short to Simple, whereas Embodied AI categories gain only 6–34%.The authors associate the former pattern with under-description and the latter with missing activity and task-context information; they recommend Simple as the default prompt.Source: §5.2 · p. 8

6. Conclusion

CaptionQA evaluates whether captions preserve task-relevant information by testing them against dense, taxonomy-driven questions. Its construction and prompt analyses describe question coverage across domains and recommend Simple prompts for standard evaluation.

6. Conclusion

  • CaptionQA measures caption usefulness for downstream tasks using a deterministic QA evaluator and a benchmark designed to avoid shortcuts and enforce quality checks.The open-source construction pipeline supports extension by specifying a domain-specific taxonomy.Source: §6 · p. 8

7. Motivation and Overview

  • CaptionQA tests whether a caption can stand in for its image through dense, taxonomy-driven question answering, yielding fine-grained diagnostics across domains and aspects.Source: §7 · p. 11

8. Question Characteristics

  • Across domains, 87–92% of questions have four choices; Natural has a higher binary-question share of 30.4% because many attribute checks are yes/no.Other taxonomy categories often require distinguishing among multiple plausible options.Source: §8 · p. 11

8.1. Question Density Across Domains

  • Question density averages 66.1 per image in Natural, 41.7 in Document, 48.6 in E-commerce, and 46.4 in Embodied AI.Category distributions differ by domain, including 30.5% content-level evaluation in Document and 45.0% perception in Embodied AI.Source: §8.1 · p. 11

9. Caption Prompts

  • Simple prompts average 356 words and are recommended for standard CaptionQA evaluation because they balance performance and reasonable length.Short, Long, and Taxonomy-Hinted captions average 22, 510, and 650 words, respectively.Source: §9 · p. 11

10. Taxonomy Structure

CaptionQA organizes each domain’s useful visual information into a hierarchical taxonomy that guides question generation. Its dense questions support stable model rankings with substantially fewer images than the full dataset.

10. Taxonomy Structure

  • Each domain has 6–7 top-level categories and 15–22 subcategories, totaling 69 subcategories across four taxonomies.Top-level categories capture broad semantic areas, while subcategories provide specific evaluation dimensions.Source: §10 · p. 11

10.1. Design Rationale

  • Human-in-the-loop refinement makes the taxonomies domain-specific, comprehensive, balanced, and practical for question generation and annotation.The process refines categories against downstream task requirements while avoiding redundancy.Source: §10.1 · p. 12

11. Image Amount Justification

  • Questions are distributed densely across images, with Natural images supporting the highest question density and Document images the lowest.Within-domain question-density distributions have low variance, which the authors describe as evidence of systematic annotation quality.Source: Figure 8 · p. 12
  • Using only 10% of images yields model rankings with Spearman correlation ρ > 0.95 against full rankings.Rankings stabilize within 20–40 images across domains; 30–50 images per domain support preliminary evaluation.Source: §11 · p. 14

12. Cost of Extending CaptionQA to New Domains

Extending CaptionQA to a new domain requires defining its information needs and preparing representative images; the released code handles the remaining construction steps.

12. Cost of Extending CaptionQA to New Domains

  • Adding a domain is a mechanical procedure driven by CaptionQA’s released code.The section frames extension as defining a taxonomy and preparing an image set before running the existing pipeline.Source: §12 · pp. 14–15

1. Write a taxonomy.

  • The new domain’s taxonomy specifies which information matters for its downstream applications.The taxonomy is a one-time configuration file and can cover aspects such as objects, layout, OCR, and affordances.Source: §1 · p. 15

2. Collect images.

  • The extension process uses a curated image set and code to generate and filter questions, verify them with two VLMs, and optionally spot-check them by humans.The text says no manual question authoring is required once the taxonomy and image list are prepared.Source: §2 · p. 15

3. Run the pipeline.

CaptionQA’s extension pipeline uses generated questions, filtering, verification, and local inference, while its QA reader models a downstream consumer using captions as its only evidence. The authors report low construction and evaluation costs and say the framework can use other sufficiently strong LLM readers.

3. Run the pipeline.

  • Extending a domain with about 10,000 finalized questions costs roughly $40–$60 in API usage and 3–4 MI325 GPU-hours for construction.The API estimate corresponds to roughly 17.7 million total tokens; the GPU estimate assumes about three candidates per retained question and ten blind QA passes per candidate.Source: §3 · pp. 15–16
  • Evaluating one captioning model on a domain with about 10,000 questions takes less than 8 minutes of MI325 time, with no additional API cost.The Qwen models run locally, and both construction and evaluation GPU costs scale linearly with the number of questions.Source: §3 · p. 16
  • The authors conclude that domain extension is feasible once a taxonomy and image set are specified, with the remaining process run through public code on a single GPU.They say costs scale linearly with image and question counts, making extensions with a few hundred images per domain within reach for typical academic and industrial users.Source: §3 · pp. 16–17

13. Rational and Reliability of LLM as QA Reader

  • CaptionQA’s QA reader answers task questions from captions alone, measuring whether a caption supports task completion rather than how it appears to a judge.The framework is reader-agnostic: users can substitute another sufficiently strong LLM to represent their downstream consumer.Source: §13 · pp. 18, 20

14. Prompt Transition Analysis: Where Does Length Help?

Prompt transitions show that moving from Short to Simple captures most gains, while lengthening Simple captions adds little and taxonomy hints usually reduce performance. Coverage and accuracy shifts vary across categories, including severe information omissions in Short captions.

14.1. Short to Simple: Identifying High-ROI vs. Low-ROI Categories

  • Document domain-specific evaluation and E-commerce textual elements gain the most from Short to Simple, while Embodied AI categories and Natural hallucination gain the least.The top categories improve by up to nine times more than the bottom categories; the Short-to-Simple transition has a mean gain of +33.8%.Source: §14.1 · p. 20, Figure 17 · p. 21

14.2. Simple to Long: Diminishing Marginal Returns

  • Simple prompts achieve 99% of Long prompts’ performance at 70% of their length, with all categories changing by less than 2% from Simple to Long.The mean Simple-to-Long gain is +0.35%.Source: §14.2 · p. 20, Figure 18 · p. 21

14.3. Long to Taxonomy-Hinted: When Structure Backfires

  • 23 of 25 categories lose performance with Taxonomy-Hinted prompts versus Long, with a mean loss of −10.8%.The largest cited decline is −33.1% for Document Domain-Specific Evaluation; only two categories gain, and these are high-level judgments.Source: §14.3 · pp. 20–21, Figure 19 · p. 21

14.4.1. Short to Long: Strong Positive Correlation

  • Short-to-Long coverage and accuracy changes correlate strongly (r=0.905), but some categories gain more coverage than accuracy while others show the reverse.Newly answerable questions in the coverage-ahead categories have lower accuracy, reported as 79–84%.Source: §14.4.1 · p. 21, Figure 20 · p. 22

14.4.2. Short to Simple: Captures Most of Long’s Benefits

  • Short to Simple produces mean gains of 32.8% in coverage and 33.8% in accuracy, achieving 99% of Short to Long’s gains at 70% of its length.The cited Short-to-Long mean gains are 33.1% coverage and 34.2% accuracy.Source: §14.4.2 · p. 22, Figure 21 · p. 22

14.4.3. Simple to Long: Minimal Changes

  • Simple-to-Long changes are under 2% for both coverage and accuracy in every category, with E-commerce Visual Appearance the only notable outlier.The mean changes are +0.3% coverage and +0.4% accuracy.Source: §14.4.3 · p. 22, Figure 22 · p. 22

14.4.4. Long to Taxonomy-Hinted: Negative Correlation

  • Long-to-Taxonomy-Hinted prompts reduce both coverage and accuracy across 23 of 25 categories, with the two measures declining together (r=0.966).The passage attributes this pattern to prompts inducing ungroundable content that captioning models fabricate, which QA models then detect as unreliable.Source: §14.4.4 · p. 22, Figure 23 · p. 23

14.5. Extreme Cases: Information Omission in Short Captions

  • Short captions leave key information unanswered, including Embodied AI Perception, Natural Spatial relations, and E-commerce textual elements.Cannot-Answer rates exceed 70% in each of these categories; the authors characterize these cases as omitted caption information, not absent image information.Source: §14.5 · pp. 22–23

15. Category-Level Statistical Summary

Category-level results show uneven caption utility: scene and content-level tasks are comparatively strong, while spatial, structural, and action-relevant details remain difficult. Performance also varies substantially across models and categories.

15. Category-Level Statistical Summary

  • Table 9 summarizes 25 top-level categories with mean score, inter-model standard deviation and range, Cannot-Answer rate, and question count.Figures 24–27 further break performance down across 69 subcategories and 24 models.Source: §15 · p. 23, §15.1 · p. 23
  • 85.3% mean score and 10.0% Cannot rate mark robust hallucination detection; Embodied Perception combines 65.3% score with 36.7% Cannot, indicating widespread information absence.Cannot-Answer rates correlate with difficulty (r=0.68), while spatial reasoning has high inter-model variance.Source: §15 · p. 23

15.1.1. Natural Domain

  • 88% scene-type classification and 85% environment recognition contrast with 45% distance estimation, 52% orientation, and 58% relative positioning.Coarse color and size attributes score 75%, compared with 55% for fine-grained texture and material.Source: §15.1.1 · p. 23

15.1.2. Document Domain

  • 82% content-level evaluation and 80% domain-specific reasoning exceed structural parsing at 60–65%; chart axes, legends, and data points score 62%, 58%, and 55%.Gemini 2.5 Pro leads table-specific subcategories by 5–8 points.Source: §15.1.2 · p. 23

15.1.3. E-commerce Domain

  • 81% mean performance across E-commerce subcategories is 8–12 points higher than other domains, but fine print, specifications, style, and material appearance are harder.Brand names score 85% and titles 82%, versus 68% for fine print, 65% for specifications, 62% for style matching, and 58% for material appearance.Source: §15.1.3 · p. 23

15.1.4. Embodied AI Domain

  • 83% activity recognition and 78% task-level reasoning exceed scores for object properties (60%), affordances (55%), and manipulation planning (52%).Depth cues (48%), camera viewpoint (52%), and embodiment context (55%) have over 30% Cannot-Answer rates; temporal-dynamics tasks score 50–58%.Source: §15.1.4 · pp. 23–24

16. Detailed Model Performance Analysis

Across 69 subcategories, model strengths depend on domain and task type, and category-level variation exceeds differences between models on the same category.

16. Detailed Model Performance Analysis

  • Figure 28 plots 24 models across 69 subcategories, grouping the axes into four domain-colored sections separated by radial lines.The Document-domain breakdown identifies Gemini models as relatively strong on table and chart parsing, while chart-specific elements remain challenging.Source: §16 · p. 24, Figure 25 · p. 25
  • 35–95% within-model variation across categories exceeds the 5–15-point between-model variation on the same category; Embodied AI spatial scores fall to 35–60% versus 75–90% on perceptual categories.GPT-5 leads Natural and E-commerce categories, while Gemini models perform better on Document structural elements.Source: §16 · p. 24

17. Question Difficulty Distribution

The section assesses whether CaptionQA differentiates model capability levels by grouping questions according to how many of 24 models answer correctly, then showing domain distributions and examples.

17. Question Difficulty Distribution

  • Question difficulty is assessed by the percentage of 24 models answering each question correctly.The analysis is intended to assess discrimination across capability levels.Source: §17 · p. 25

17.1. Difficulty Categorization

  • Questions are labeled easy when at least 80% of models answer correctly, medium at 50–80%, and hard below 50%.The percentages refer to the proportion of the 24 evaluated models answering correctly.Source: §17.1 · p. 25

17.2. Distribution Across Domains

  • Figure 29 presents the question-difficulty distribution separately for each domain.The supplied passage does not state the domain-specific proportions.Source: §17.2 · p. 25

17.3. Examples of Hardest and Easiest Questions

  • Figure 30 presents concrete examples of the hardest and easiest questions across all four domains, alongside the corresponding images, answer choices, and correct answers.Source: §17.3 · p. 25

18. Full Results

The appendix provides full model, prompt, and domain result breakdowns, alongside category-level summaries and visualizations. It evaluates 24 models under four caption prompts across four domains using three utility-related metrics.

Difficulty and full results tables

  • Table 10 summarizes question difficulty, and Figures 29–30 show domain-level difficulty distributions and hardest-versus-easiest examples.The full results tables cover all evaluated model, prompt, and domain combinations.Source: §18 · p. 25, Table 10 · p. 31, Figure 29 · p. 31

Models covered

  • The evaluation covers 24 multimodal LLMs, including 18 open-source models and six proprietary models, with open-source model sizes ranging from 1B to 78B parameters.The models span multiple model families and include API-only systems.Source: §18 · pp. 25–26

Domain results

  • E-commerce models achieve the highest overall scores across domains (70–96%), while Embodied AI is the most challenging domain overall, with scores ranging from 50–85%.E-commerce strengths include contextual understanding (85–96%) and product-level information (82–94%). In Embodied AI, activity/task understanding (80–93%) is relatively strong, while perception subcategories score 40–70%.Source: Figure 26 · p. 26, Figure 27 · p. 27

Prompts

  • Each model is evaluated with Long, Short, Simple, and Taxonomy-Hinted caption prompts in all four domains, yielding 16 model-prompt-domain settings.Simple is described as the recommended default; the Taxonomy-Hinted prompt supplies domain taxonomy aspects.Source: §18 · pp. 26–27

Domains and metrics

  • The tables report Score, Acc, and Cannot for each model: average utility-oriented multiple-choice score, correct-answer fraction, and fraction judged unanswerable from the caption.Tables are organized by domain and prompt, with rows grouped by open-source and proprietary models and sorted by Score.Source: §18 · p. 27

Full results overview

  • Across 69 subcategories, model performance varies dramatically, with no single model excelling uniformly; GPT-5 and Gemini 2.5 Flash are top proprietary models, while Qwen3-VL 30B-A3B and GLM-4.1V 9B lead among open-source models.Figure 28 shows a 30–95% range across model-category results and a systematic spatial-reasoning gap.Source: Figure 28 · p. 28

Category and taxonomy tables

  • Table 9 summarizes category performance under the Simple prompt, while Table 8 presents the Embodied-AI caption-utility taxonomy.Table 9 includes category means, inter-model standard deviations and ranges, Cannot rates, and question counts.Source: Table 8 · p. 29, Table 9 · p. 30
Loading 2511.21025v2…