Source-linked AI summary
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Zizhen Wang, Bo Feng, Zhengfeng Lai, Shiyu Li, Yang Lu, Meng Cao, Ping Huang, Xiaoming Simon Wang
TL;DR
Video-caption evaluation is hindered by reference matching, which can penalize valid descriptions and does not finely separate factuality from coverage. CapQuiz addresses this with a reference-free benchmark using human-verified multiple-choice questions organized across video domains and question types. It aligns more robustly with human judgments and exposes model differences, while its coverage measure and judge-based evaluation retain explicit scope limitations.
Problem
Existing caption metrics can penalize valid lexical variations and fail to provide fine-grained separation of factuality from information coverage.
Method
CapQuiz evaluates captions through human-verified, fine-grained multiple-choice questions across 24 video domains and 10 Descriptive and Inferential question types.
Results
CapQuiz achieves robustly better alignment with human judgments than existing evaluators and provides diagnostic insights into model performance.
Takeaways & Limitations
CapQuiz supports separate analysis of caption factuality and coverage while revealing degradation on inferential tasks and limited coverage in current VLLMs.
Takeaways & Limitations
CapR is a proxy based on identified salient information, and English-only questions limit evaluation in multilingual contexts.
Abstract
from arXiv · showhide
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many'' nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.
1 Introduction
Video captioning needs evaluation that handles valid descriptive variation while separately measuring factuality and information coverage. CapQuiz addresses this with a reference-free, question-based benchmark and reports stronger alignment with human preferences than existing evaluators.
- 1 Introduction: Existing reference-based and semantic evaluators can penalize valid captions that differ lexically from ground-truth descriptions and do not finely separate factuality from coverage.The one-to-many nature of video description motivates a reference-free alternative.
- 1 Introduction: CapQuiz evaluates caption quality through human-verified, fine-grained multiple-choice questions answered using only the generated caption.The benchmark uses plausible distractors to test whether captions preserve salient visual information accurately.
- 1 Introduction: CapQuiz quantifies factuality, coverage, and their unified quality using CapP, CapR, and CapF1.Question-answer outcomes are categorized as true positives, false positives, and false negatives before computing the metrics.
- 1 Introduction: The benchmark comprises 1,204 videos across 24 domains and 23,632 human-verified question-answer pairs organized into 10 question types.The taxonomy spans broader Descriptive and Inferential categories.
- 1 Introduction: CapQuiz achieves superior alignment with human preferences and reveals diagnostic differences among VLLMs that traditional one-dimensional metrics miss.Its fine-grained analysis is intended to expose capability disparities rather than provide only a single overall score.
2 Related Works
Prior video-caption evaluation includes reference-based, cross-modal, and question-based approaches, but each leaves important gaps. Reference dependence preserves one-to-many bias, while existing question formats lack the diagnostic rigor and stability needed for fine-grained analysis.
- 2 Related Works: Reference-dependent metrics compare captions with human-authored references, limiting coverage and undervaluing valid details absent from specific reference texts.Surface-overlap metrics and semantic methods differ in implementation but share dependence on reference captions.
- 2 Related Works: Cross-modal methods reduce reference reliance by incorporating visual information through image-text matching, scene graphs, or shared semantic spaces.Examples include CLIPScore and InfoMetIC.
- 2 Related Works: Question-based evaluation probes information fidelity, but many methods target static images or compare candidate answers with references.These choices can reintroduce reference dependence in video caption assessment.
- 2 Related Works: Reference-free video QA methods avoid reference matching, yet binary or open-ended formats remain susceptible to random guessing or instability and lack probing question types.This motivates a more structured multiple-choice evaluation design.
3 The CapQuiz Benchmark
CapQuiz is a reference-free benchmark that uses human-verified multiple-choice questions to assess video-caption information fidelity across diverse domains and question types. Its pipeline combines taxonomy-guided QA generation, automated filtering, human verification, and metrics separating factuality, coverage, and overall quality.
- 3.2.2 Multiple-Choice QA Generation: The benchmark construction sources held-out videos, balances short, medium, and long durations, and uses an over-generate-then-filter pipeline for QA creation.Candidate questions are generated from raw videos with taxonomy definitions and few-shot examples before filtering.
- 3.2.2 Multiple-Choice QA Generation: Automated filtering removes blindly solvable or redundant questions, retaining the most challenging instance within each semantic cluster.The retained question is the one with the lowest blind-pass accuracy, increasing discriminative power.
- 3.2.2 Multiple-Choice QA Generation: Human annotators verify visual relevance, unique factual correctness, and question difficulty, including at least one plausible but incorrect hard negative.At least three annotators review each surviving QA pair, and a subset achieves Gwet’s AC1 of 0.92.
- 3.1 Hierarchical Taxonomy Design: CapQuiz contains 1,204 videos averaging 19.63 human-verified QA pairs per video, spanning 24 video domains and 10 question types.The benchmark uses a hierarchical taxonomy covering descriptive and inferential capabilities.
- 3.4 Evaluation Methodology: CapQuiz evaluates captions through judge-selected answers categorized as true positives, false negatives, or false positives, then derives factuality, coverage, and overall-quality metrics.CapP penalizes hallucinations, CapR measures correctly retrieved details across questions, and CapF1 combines the two dimensions.
- 3.4 Evaluation Methodology: CapQuiz achieves the highest correlation with human judgments in all six reported settings across factuality, coverage, and overall quality.Table 2 reports Spearman and Kendall correlations with 95% bootstrap confidence intervals, all significant at p < 0.01.
4 Experiments
Experiments show that CapQuiz aligns more reliably with human judgments than direct VLLM judging and reveals model scaling, factuality–coverage trade-offs, and reasoning gaps. These patterns remain robust under varied caption prompts.
- 4.1 Alignment with Human Judgments: CapQuiz consistently achieves higher correlations with human judgments than the VLLM-as-a-Judge baseline across all six settings, with statistical significance.The evaluation uses Spearman and Kendall rank correlations over 600 video-caption pairs.
- 4.2 Evaluation on SOTA Models: Overall CapF1 rises from 61.53 to 77.95 as activated Qwen3-VL model size increases from 2B to 32B.This scaling trend links larger activated model size with improved video caption quality within the Qwen3-VL series.
- 4.2 Evaluation on SOTA Models: CapP consistently exceeds CapR; for GPT-4o, factuality is 78.13 versus coverage at 66.71.The results indicate reliable but non-exhaustive descriptions, leaving room to increase caption density without introducing hallucinations.
- 4.2 Evaluation on SOTA Models: Inferential performance trails Descriptive performance, with the gap reaching about 30% for AuroraCap-7B versus about 10% for GPT-5.2.The disparity indicates that complex visual reasoning remains a stronger differentiator than basic visual recognition.
- 4.3 Prompt Sensitivity Analysis: The benchmark’s main trends persist under detailed prompting despite model-specific changes, including a 3.07% decrease for Gemini-2.5-flash and a 4.81% increase for GPT-4.1-2025-04-14.Prompt sensitivity varies across models, but the benchmark conclusions remain robust.
5 Conclusion
The paper introduces CapQuiz as a reference-free benchmark that evaluates video captions through information fidelity. It separates factual precision from coverage and exposes weaknesses in comprehensive and inferential caption understanding.
- 5 Conclusion: CapQuiz assesses video caption quality with human-verified fine-grained multiple-choice questions and quantifies factuality and coverage separately.The benchmark is designed around information fidelity rather than reference-caption matching.
- 5 Conclusion: Experiments show that current VLLMs often maintain high factual precision while struggling with comprehensive coverage and inferential tasks.CapQuiz is intended to guide research toward more robust and grounded video understanding models.
Limitations
CapQuiz’s question set may miss visual details and is currently English-only, while GPT-4.1-based judging introduces reproducibility and bias concerns.
- Limitations: CapR is a proxy for salient information because the QA set may not capture every visual detail, potentially overestimating coverage in dense videos.The benchmark averages 19.63 QA pairs per video, but subtle or dense background content may remain unrepresented.
- Limitations: English-only QA pairs limit CapQuiz’s evaluation of VLLMs in multilingual contexts.This is a scope boundary of the current benchmark construction.
- Limitations: GPT-4.1 judging is bounded by the model’s capabilities, closed-source updates, and possible bias on ambiguous visual descriptions.These factors can affect reproducibility and introduce evaluation noise.
Ethical Considerations
The benchmark prioritizes privacy and safety by sourcing videos from established open-source datasets with compatible licenses and relying on commercial APIs with safety guardrails.
- Ethical Considerations: Video samples come exclusively from established open-source datasets distributed under Creative Commons or compatible licenses.This curation strategy addresses privacy, safety, and usage-policy compliance.
A Robustness to Judge Choice
CapQuiz remains robust when GPT-4.1 is replaced by Qwen3-30B-A3B-Instruct-2507 as the automatic judge, preserving VLLM rankings and the study’s main conclusions. Absolute CapP, CapR, and CapF1 scores generally decrease under the alternative judge.
- GPT-4.1 is retained as the default judge because it provides strong instruction-following, stable answer formatting, and improved score calibration.CapQuiz is not tied to a specific judge model.
- Absolute CapP, CapR, and CapF1 scores generally decrease under the Qwen3-30B-A3B-Instruct-2507 judge.The shift is attributed to differences in judges’ instruction-following and zero-shot reasoning capabilities.
B Robustness to QA Set Size
CapQuiz produces stable model comparisons across QA retention ratios, including an aggressive 25% setting. Its QA density is therefore sufficient for robust CapF1 estimates, although variance depends on retention level.
- CapR functions as a salience-oriented coverage proxy rather than an exhaustive recall measure over all possible video details.The benchmark uses a finite set of curated QA pairs per video.
- The retention-ratio study reports mean ± standard deviation for CapF1 across 10 random seeds at 25%, 50%, and 75% retention.The 100% condition uses the full QA set.
- VLLM rankings remain unchanged across all tested QA retention ratios, including the aggressive 25% setting.The analysis evaluates retention ratios of 25%, 50%, 75%, and 100% with repeated subsampling for partial settings.
- CapQuiz is not overly sensitive to the exact number of QA pairs when comparing systems.This conclusion follows from the stable relative rankings across retention conditions.
C Taxonomy
The taxonomy organizes benchmark videos into broad content domains and question types, separating directly observable Descriptive questions from more interpretive categories. Its domains span creative, civic, functional, knowledge-oriented, and everyday content.
- Everyday and Knowledge: Everyday content focuses on authentic ordinary life, including human bonds, animal companions, and solitary personal experiences.The taxonomy also includes knowledge and personal-life categories covering subjects such as health, history, society, and individual routines.
- The taxonomy covers creative, civic, functional, knowledge-oriented, and everyday video content through diverse domains such as sports, news, cooking, science, and human relationships.The listed domains include both broad categories and specialized subcategories.
- Creativity: Creative content includes movies and shows, dance and performance, music and singing, animation, comedy sketches, games, sports, and nature.These categories emphasize artistic expression, entertainment, physical performance, or the natural world.
- Civics and Function: Civic and functional content distinguish public events and societal issues from practical guidance, purchasing, travel, life systems, and functional recordings.Examples include news, social issues, civic action, how-to videos, cooking, buyer’s guides, travel planning, and life guides.
- Descriptive questions target factual information directly observable in the video.The supplied taxonomy includes entity identification, attributes such as color or quantity, and actions or direct interactions.
E Annotation Details
CapQuiz’s annotation pipeline uses visually grounded, independently answerable QA generation followed by solvability checks, deduplication, multiple-choice answering, and grading. Annotators and prompts are structured to support detailed, reproducible visual analysis.
- The annotation workforce was internationally distributed and required written-English proficiency for text generation and comprehension.India, Singapore, and China each contributed approximately 25% of personnel, with the remaining cohort drawn from European and North American countries.
- QA generation requires questions to be independently answerable from the video without seeing the answer options first.Allowed questions include perceptual-factual items and interpretive-inferential questions supported by visual evidence.
- The pipeline applies blind solvability checking and semantic deduplication before multiple-choice question answering and grading.The supplied prompts separately specify single-letter answer selection and two-level clustering by question and answer semantics.
- The multiple-choice answering stage asks an evaluator to select the best option using only the video caption, question, and options.The response format is restricted to the option letter.
- The detailed captioning prompt instructs an analyst to convert chronologically ordered video frames into a structured, purely visual report without audio information.The report is intended to serve as a detailed knowledge base for subsequent questions.