Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

1 Introduction

• Reference-based text metrics can penalize valid lexical variation, while question-based benchmarks VDC (Chai et al., 2024) and VCapsBench (Zhang et al., 2025) lack fine-grained diagnosis separating factuality from coverage.The introduction also describes semantic and cross-modal evaluation approaches but identifies the shared gap as insufficient diagnostic separation.Source: §1 · p. 1
• CapQuiz tests whether readers can answer fine-grained video questions from captions, rewarding salient-information coverage while requiring factual accuracy.Its human-verified multiple-choice questions include plausible distractors, requiring discrimination between correct details and alternatives.Source: §1 · pp. 1–2
• CapQuiz organizes questions into 10 Descriptive and Inferential types across 24 video domains, and derives CapP for factuality, CapR for coverage, and CapF1 as a unified metric.QA outcomes are categorized as true positives, false positives, and false negatives.Source: §1 · p. 2
• The benchmark comprises 1,204 videos across 24 domains and 23,632 human-verified multiple-choice question-answer pairs organized into 10 types.The types fall under broader Descriptive and Inferential categories.Source: §1 · p. 2
• CapQuiz is reported to align better with human preferences than existing metrics, while revealing VLLM disparities and diagnostic insights missed by traditional one-dimensional measures.The supplied introduction reports these findings without giving quantitative comparisons.Source: §1 · p. 2

Text-based Evaluation

• Text-based metrics from lexical-overlap measures to semantic judges remain reference-dependent, penalizing valid captions that differ from references or mention uncovered visual details.The passage describes n-gram metrics, METEOR, CIDEr, SPICE, BERTScore, BERTHA, and LLM-based judges.Source: §2 · pp. 2–3

Cross-modal Grounded Evaluation

• Cross-modal metrics such as CLIPScore (Hessel et al., 2021) offer visual grounding but often miss fine-grained relations, action directionality, temporal dynamics, and logical reasoning.The passage also describes earlier image-text and scene-graph approaches, and InfoMetIC (Hu et al., 2023), which provides coarse-grained and token-level scores.Source: §2 · p. 3

Question-based Evaluation

• Question-based methods such as QACE (Lee et al., 2021), VQAScore (Lin et al., 2024), and CaptionQA (Yang et al., 2025) primarily target static images.Source: §2 · p. 3
• Video QA metrics include reference-based Dream1K (Wang et al., 2024) and reference-free VDC (Chai et al., 2024), QEVA (Jung and Kim, 2025), and VCapsBench (Zhang et al., 2025).Dream1K reintroduces reference bias; the other formats can invite guessing or instability, and existing QA metrics largely emphasize factual correctness, whereas CapQuiz treats factual trustworthiness and salient-information coverage as primary evaluation dimensions.Source: §2 · p. 3

3.1 Hierarchical Taxonomy Design

• The taxonomy spans 5 video super-categories, 24 sub-categories, and 10 question types across Descriptive and Inferential dimensions.Descriptive types probe visual grounding, while Inferential types target higher-order reasoning, enabling fine-grained diagnosis.Source: §3.1 · pp. 3–4

3.2.1 Video Curation

• Videos come from test or heldout validation splits of 10 public benchmarks, with an approximately 6:3:1 ratio of short, medium, and long clips.The duration bands are 0–30s, 30–60s, and 60–120s, respectively.Source: §3.2.1 · p. 4

3.2.2 Multiple-Choice QA Generation

• Candidate question-answer pairs are generated from videos using Gemini-2.5-Pro and GPT-4o, then filtered for blind solvability and redundancy.For each semantic cluster, the pipeline retains the question with the lowest accuracy in the blind pass to maximize discriminative power.Source: §3.2.2 · pp. 4–5
• Human reviewers verify that questions are video-grounded, have exactly one visually verifiable correct answer, and include a plausible hard negative.Each pair is reviewed by at least three annotators; inter-annotator agreement reaches Gwet’s AC1 of 0.92.Source: §3.2.2 · p. 5

3.3 Dataset Statistics

• CapQuiz contains 1,204 videos averaging 19.63 QA pairs each, and its reference-free, human-verified format supports information-fidelity evaluation across 24 domains and 10 question types.Its heterogeneous sources, short-, medium-, and long-form clips, and uniformly distributed answer options support visual diversity, temporal reasoning, and reduced position bias.Source: §3.3 · pp. 5–6

3.4 Evaluation Methodology

• CapQuiz achieves the highest correlation with human judgments in all six reported settings, with all correlations significant at p < 0.01.The settings cover Spearman and Kendall correlations for Factuality, Coverage, and Overall Quality.Source: Table 2 · p. 6
• CapQuiz reports Factuality (CapP), Coverage (CapR), and Overall (CapF1), which is the harmonic mean of factuality and coverage.CapP penalizes hallucinations while disregarding uncertainty; CapR measures correctly retrieved details among all questions. GPT-4.1 judges answers using only the caption as evidence.Source: §3.4 · p. 6

4.2 Evaluation on SOTA Models

• GPT-5.2 achieved the highest Overall CapF1 (83.08), while Qwen3-VL scaling from 2B to 32B raised Overall CapF1 from 61.53 to 77.95.These results indicate a proprietary-model advantage and a scaling trend within the Qwen3-VL series.Source: Table 3 · p. 7, §4.2 · p. 7

4.1 Alignment with Human Judgments

• CapQuiz correlations with human judgments exceeded the VLLM-as-a-Judge baseline in all six settings, with every reported correlation statistically significant (p < 0.01).The study sampled 200 videos and 600 captions from GPT-4o, InternVL3.5-8B, and QWen3-VL-8B; CapP, CapR, and CapF1 map to Factuality, Coverage, and Overall Quality, respectively.Source: §4.1 · pp. 6–7

4.2 Evaluation on SOTA Models

• Across models, CapP (Factuality) exceeded CapR (Coverage); GPT-4o scored 78.13 on CapP and 66.71 on CapR.The authors interpret this pattern as trustworthy but insufficiently exhaustive descriptions, leaving room to increase caption density without hallucinations.Source: §4.2 · p. 7
• Inferential performance lagged Descriptive performance, with an approximately 10% drop for GPT-5.2 and an approximately 30% drop for AuroraCap-7B.The gap widened for smaller models, suggesting complex visual reasoning remained a stronger differentiator than visual recognition.Source: §4.2 · p. 7

4.3 Prompt Sensitivity Analysis

• Detailed-prompt effects varied by model, but the benchmark’s model-scaling, factuality-coverage, and reasoning trends remained robust across prompt complexity.Overall CapF1 decreased 3.07% for Gemini-2.5-flash and increased 4.81% for GPT-4.1-2025-04-14; Qwen3-VL models gained consistently, while LLaVA-Video-7B decreased 4.01% and Tarsier2-7b declined 76.52%.Source: §4.3 · p. 8

• CapQuiz aligns robustly with human judgments and reveals that current SOTA VLLMs, despite high factual precision, struggle with comprehensive coverage and inferential tasks compared with descriptive ones.Source: §5 · p. 8
• CapQuiz is a reference-free benchmark that uses human-verified, fine-grained multiple-choice questions to decouple and quantify caption factuality and coverage.Source: §5 · p. 8
• The authors envision CapQuiz as a testbed to guide research toward more robust and grounded video-understanding models.Source: §5 · p. 8

Limitations

• CapQuiz’s question set may miss visual details, making CapR a proxy that can overestimate coverage in videos with dense or subtle backgrounds; QA pairs are English-only.The pipeline generates an average of 19.63 QA pairs per video, but cannot guarantee complete coverage of every detail in complex videos.Source: Limitations · p. 9
• GPT-4.1 judging aligns highly with human experts but is limited by its upper bound, closed-source updates that affect reproducibility, and potential bias on ambiguous descriptions.Source: Limitations · p. 9

Ethical Considerations

• Videos came exclusively from established open-source datasets with Creative Commons or compatible licenses, and annotators were paid above local minimum wage.The authors say dataset use followed the datasets’ usage policies and compensation adhered to fair labor practices.Source: Ethical Considerations · p. 9
• Despite commercial APIs’ safety guardrails, annotations may retain model-inherent bias or toxicity; manual inspection filtered potential NSFW content and sensitive PII.Source: Ethical Considerations · p. 9

A Robustness to Judge Choice

• With the same prompt, answer space, and parsing rules, Qwen3-30B-A3B-Instruct-2507 preserves overall VLLM ranking trends and Section 4.2 conclusions, showing comparative findings are not tied to one proprietary judge.The Qwen3-based evaluation pipeline reevaluates the VLLMs from Section 4.2.Source: A Robustness to Judge Choice · pp. 11–12, Table 5 · p. 12
• Qwen3-30B-A3B-Instruct-2507 generally lowers absolute CapP, CapR, and CapF1 scores, likely due to judge differences in instruction-following and zero-shot reasoning; GPT-4.1 improves calibration.Despite the absolute-score shift, the main comparative conclusions remain robust to judge substitution.Source: A Robustness to Judge Choice · p. 12

B Robustness to QA Set Size

• Because CapR proxies salient-information coverage rather than exhaustive recall, the ablation tests whether evaluation remains stable with a finite curated QA set.The study varies the retained proportion of QA pairs per video.Source: B Robustness to QA Set Size · p. 12
• The study uniformly subsampled each video’s QA set at 25%, 50%, 75%, and 100%, repeating partial-retention settings with 10 seeds across four representative VLLMs.The evaluated models were Gemini-2.5-pro, GPT-5.2, Qwen3-VL-30B-A3B, and Qwen3-VL-32B.Source: B Robustness to QA Set Size · p. 12
• Model rankings remain unchanged even at 25% QA retention, while CapF1 variance is very small at 50% and 75% but noticeably larger at 25%.The authors conclude that the default density of 19+ QA pairs per video is above the stability threshold needed for reliable evaluation.Source: B Robustness to QA Set Size · pp. 12–13, Table 6 · p. 13

C.1 Video Taxonomy

• The video taxonomy groups content into Knowledge, Everyday, Creativity, Civics, and Function, spanning explanation, ordinary life, artistic expression, public affairs, and practical utility.Knowledge includes society; Everyday centers on authentic daily life; Creativity is expressive; Civics covers public events and issues; Function serves practical goals.Source: §C.1 · pp. 13–14
• Civics includes news, social issues, and civic action, while Function includes how-to guidance and practical decision support.The taxonomy also names cooking, buyer’s guides, travel planning, life guides, and functional recordings as Function subtypes.Source: §C.1 · p. 14

C.2 Question Taxonomy

• Descriptive questions test observable entities, attributes, actions, events, and settings.These cover identification, visual properties or states, movements and interactions, composite happenings, and environmental context.Source: §C.2 · p. 15
• Inferential questions test relations, causality, intent and emotion, thematic or symbolic meaning, and spatiotemporal references.They assess reasoning about relationships, event logic, agents’ motivations or states, abstract meaning, and precise spatial or temporal cues.Source: §C.2 · p. 15

D Prompts

• The prompts cover video taxonomy assignment, QA generation, blind solvability checking, and question deduplication.The appendix points to Figures 5–7 for these prompt stages.Source: D Prompts · p. 15
• Additional prompts elicit multiple-choice QA, VLLM-as-a-judge evaluation, and detailed captions.The appendix points to Figures 8–9 for these prompts.Source: D Prompts · p. 15

E Annotation Details

• Annotators were roughly evenly split among India, Singapore, China, and a mixed European and North American cohort, with written-English proficiency required.The annotation interface is shown in Figure 10.Source: E Annotation Details · p. 15
• The blind solvability check prompt required answering a multiple-choice question with only the correct capital letter and no explanation.The prompt provides placeholders for the question and options.Source: E Annotation Details · p. 18
• The question-answering prompt directs a video-understanding assistant to choose the best answer based on the video caption and respond only with the option letter.It supplies placeholders for the caption, question, options, and option identifier.Source: E Annotation Details · p. 19