CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models

• Physically grounded models could help users anticipate outcomes and plan around real-world objects without costly trial and error.Source: §1 · p. 1
• Real-video benchmarks largely emphasize perception and description, while controlled synthetic causal benchmarks isolate reasoning but lack real-world diversity and complexity.Source: §1 · p. 1
• CausalVQA comprises 1,786 EgoExo4D egocentric-video multiple-choice items across five categories, curated with human-and-model checks against superficial visual and linguistic cues.All categories except descriptive involve causal reasoning.Source: §1 · p. 2, Table 1 · p. 2
• Humans achieved 84.78% accuracy, while Gemini 2.5 Flash scored 61.66% overall, leaving a meaningful gap in real-world video causal reasoning.Source: §1 · p. 2

2.2.1 Video Selection

• 298 EgoExo4D egocentric videos (about 12 hours) were selected for familiar, goal-directed physical activities with rich object interactions.The selection avoided activities with few interactions between the camera-wearer and other objects.Source: §2.2.1 · p. 3

2.1 Design Objectives

• CausalVQA targets counterfactual, hypothetical, anticipation, and planning questions, with descriptive items enabling comparison to more surface-level visual perception.The design prioritizes real-world relevance, physical phenomena, visual comprehension, and accessible evaluation.Source: §2.1 · pp. 2–3, §2.2.2 · p. 3

2.2.3 Distractor Generation

• A vision-language model generated seed distractors from clips and descriptions, and Llama refinement broadened alternatives and replaced a potentially correct option.Annotators supplied the initial questions and answers, while a VLM also described activity after each clip to refine anticipation questions.Source: Table 2 · p. 4, §2.2.4 · p. 4, §2.2.3 · p. 4

2.2.5 Language Perturbation and Reordering

• Paraphrased distractors and reordered options form two question versions; paired accuracy requires correct answers on both, making performance more robust to option wording and order.Under stated assumptions, paired scoring reduces theoretical response variance to less than one quarter of the unpaired variance.Source: §2.2.5 · pp. 4–5

2.2.6 Ensuring Visual Groundedness

• Text-only answerability screening removed approximately 39% of questions; two-reviewer quality assurance removed 2,306 items (53%) and repaired less fundamental errors.Reviewers checked timestamps, video dependence, answer correctness, question type, coherence, and whether multiple answers were correct.Source: §2.2.6 · p. 5, §2.2.7 · p. 5

2.2.8 Human-Derived Empirical Difficulty Levels

• Difficulty labels use agreement among 15 naive annotators per question version: easy is above 86.67%, medium is 70–86.67%, and hard is 36.67–70%.Items whose modal human response did not match the ground-truth answer were removed.Source: §2.2.8 · pp. 5–6

2.3.1 Benchmark Composition

• The benchmark contains 1,586 items (793 paired questions) across 779 video segments; 63.7% are reasoning questions, and 57.2% are considered Easy for humans.Sports-related scenarios account for 60.4% of the benchmark.Source: §2.3.1 · p. 6, Table 5 · p. 7

2.3.2 Debug Dataset

• A separate 100-question, easy-only debug set is excluded from reported benchmark statistics and evaluation; 12 experienced AI users judged items to capture cause-and-effect and spatial-temporal interactions.The debug set contains 20 questions from each question category.Source: §2.3.2 · p. 7, §2.3.3 · p. 7

3 Baseline Evaluations

• Scores use paired accuracy, crediting a question only when both versions are answered correctly; paired scores were 10–12% lower than unpaired scores.The evaluation reports zero-shot results across the benchmark and its question subsets.Source: §3 · p. 7

3.1 Human Baseline

• Humans scored 84.78% overall and 81.43% on Reasoning questions, with descriptive questions easiest and counterfactual and hypothetical questions more difficult.Human accuracy was 90.63% on descriptive, 76.57% on counterfactual, and 75.92% on hypothetical questions.Source: §3.1 · p. 9

3.2 Models

• The evaluation included locally run open-weight models and commercial closed models, using uniformly sampled video frames.Open-weight models used 16 frames; GPT-4o and Gemini 2.5 Flash used 10 frames.Source: §3.2 · p. 9

Overall Results

• Models generally performed substantially worse on hard than easy items, aligning with the benchmark’s human-derived difficulty levels.Performance also varied by question type: Gemini scored 53.46% on Reasoning versus 76.04% on descriptive questions, while humans scored 81.43% versus 90.63%.Source: Overall Results · p. 9
• Average model accuracy was 39.34% on anticipation and 39.55% on hypothetical questions, compared with human scores of 83.54% and 75.92%, respectively.The authors relate this gap to combining visual input with physical-world knowledge to reason about what could or will happen.Source: Overall Results · p. 9

Model-Specific Discussions

• Gemini 2.5 Flash performed best overall, while PerceptionLM showed strength across reasoning categories and sometimes outperformed the larger GPT-4o on the reasoning aggregate.GPT-4o’s performance benefited from relative strength on easier descriptive questions.Source: Model-Specific Discussions · p. 9

Overall Results

• Gemini 2.5 Flash scored 61.66% overall, more than 22 percentage points below humans, and 53.46% on Reasoning questions, about 27 points below humans.Reasoning questions exclude descriptive questions.Source: Overall Results · p. 9, Table 6 · p. 10

4.1 Video Question Answering Benchmarks

• Many VQA benchmarks emphasize descriptive questions, while others probe goals or mental states, include few counterfactuals, or focus on longform video rather than short-horizon causal reasoning.Examples include TGIF-QA [13], ActivityNet-QA [43], TVQA [16], NExT-QA [41], PerceptionTest [31], and HourVideo [6].Source: §4.1 · p. 10
• ACQUIRED [40] and Causal-VidQA [20] target real-world causal and counterfactual reasoning, but the authors say neither did much to ensure visual groundedness.CausalVQA instead uses multiple safeguards intended to require deeper causal understanding.Source: §4.1 · p. 10

4.2 Physical Reasoning Benchmarks

• Controlled simulation benchmarks test physical intuition and causal reasoning while limiting the realism and complexity of real-world scenes.CausalVQA builds on their systematic control while using actual video; examples include PHYRE [4], IntPhys [33], CLEVRER [42], ComPhy [7], CRIPPVQA [28], and ContPhy [45].Source: §4.2 · p. 11

4.3 Video Generation Benchmarks

• VideoPhy [5] and PhyGenBench [24] assess physics in generated videos, while Physics-IQ [25] prompts physically plausible continuations; these are not designed for non-generative physical and causal understanding.Source: §4.3 · p. 11

Shortcuts

• Some questions can be answered from a single frame, reflecting real-world correlations such as an ending frame predicting subsequent actions regardless of earlier events.The authors suggest further safeguards and filtering to reduce shortcut exploitability.Source: §5 · p. 11

Data scope

• CausalVQA covers only egocentric, goal-driven activities from EgoExo4D; future work could broaden video sources and include modalities such as audio.Source: §5 · p. 11

Reasoning traces

• The benchmark does not include explanations or thought traces, so it does not assess the process by which models arrive at answers.The authors note that user research identified explaining an answer as an expectation for “true” reasoning.Source: §5 · p. 11

Human–model comparison

• Future work will examine whether models and humans make similar mistakes, using detailed correlations between their performance on individual items.Source: §5 · pp. 11–12

6 Conclusion

• CausalVQA uses a hybrid human-and-model quality-control loop to mitigate shortcut reliance and human evaluations to assign item difficulty levels.Source: §6 · p. 12
• The resulting dataset contains 1,586 items spanning counterfactual, hypothetical, anticipation, planning, and descriptive questions, with easy, medium, and hard levels.Source: §6 · p. 12
• The best-performing model, Gemini 2.5 Flash, performs meaningfully worse than humans across all question types and difficulty levels, especially on counterfactual and next-event prediction items.Source: §6 · p. 12
• The benchmark is intended to highlight the model–human performance gap and help measure and stimulate progress on causal structure and physical principles in real-world activities.Source: §6 · p. 12

Funding

• Meta funded creation of the dataset.Source: DataSheet · p. 16

Instance information

• Correct answers are omitted only for instances in the held-out set.Source: DataSheet · p. 16

Instance relationships

• Related MCQ and MCQ′ questions share the same question ID (qid).Source: DataSheet · p. 16

Data splits

• The datasheet describes separate debug and test splits; test targets are held out, and item-level responses can be submitted to a leaderboard.Source: DataSheet · p. 17

A Human Tasks

• The appendix covers human tasks in benchmark construction and baselining or difficulty assignment.Source: A Human Tasks · p. 20

A.1 Question Generation Guidelines

• Sixteen annotators spent over 300 hours generating questions, with each clip costing $52 to review and question.Source: §A.1 · p. 20

Overview

• Annotators were instructed to watch clips and create questions obvious to humans with ground-truth answers; each clip was to receive three reviews and about five questions when suitable.Source: Overview · p. 20

Question Types

• Question types distinguish observed facts, likely next events, goal-directed sequences, departures from observed facts, and imagined alternatives.Source: Question Types · p. 20

A.2 Human Quality Assurance Guidelines

• Quality assurance reviews questions, correct and alternative answers, timestamps, and type labels to correct errors and improve clarity.Source: §A.2 · p. 21

Workflow

• Reviewers check whether video is necessary, the question is clear, the video provides enough information, and only one answer is most correct.Source: Workflow · p. 22

A.3 Human-Derived Empirical Difficulty Levels and Human Baselining Guidelines

• The human baseline used 273 annotators and filtered questions; evaluators received approximately equal video duration, did not see both versions of a question, and were given task descriptions to clarify the goal.Source: §A.3 · p. 23

A.4 Qualitative Assessment Guidelines

• Twelve 90-minute remote interviews with AI-experienced participants in the US, Canada, and UK explored how they define reasoning and whether benchmark items match those definitions.Participants were balanced by age and gender and reviewed work-in-progress benchmark items.Source: §A.4 · p. 23

Goals of User Research

• The user research asked how people conceptualize reasoning across modalities, which AI reasoning capabilities matter, whether CausalVQA tasks test reasoning, and whether usefulness matters more than defining reasoning.It also asked whether passing the benchmark would satisfy people that an AI could effectively reason.Source: Goals of User Research · p. 23

Interview Script Summary

• The interview script elicited participants’ definitions of reasoning, learning, and logic, then had them perform CausalVQA tasks aloud and judge whether the tasks test physical-world reasoning.Participants were also asked how they might improve the test.Source: Interview Script Summary · pp. 23–24

B Prompt Details for Question Creation Pipeline

• The appendix presents automated question-creation and quality-assurance prompts in execution order, with headings tied to the paper sections and occasional output examples.Source: B Prompt Details for Question Creation… · p. 24

B.1 Distractor Generation

• Distractor generation prompts a vision-language model for a visually grounded answer and rationale, then requests four plausible, distinct, clip-based distractors matched to the answer’s length and complexity.The prompts emphasize avoiding duplicates and answer equivalents, and require a rationale.Source: §B.1 · pp. 24–25

B.2 Language Refinement

• Language refinement makes answers concise and idiomatic, preserves question and answer meaning, and replaces answer-hinting words in questions with neutral terms.The meaning check directs the model to restore essential physical or mechanical details if they are missing.Source: §B.2 · pp. 26–28

B.3 Language Purtubation and Reordering

• Language perturbation removes distractors equivalent to the answer, polishes and diversifies the remaining choices, creates synonym variants, and reorders choices.The examples show an alternative correct answer being corrected and demonstrate perturbed, reordered answer sets.Source: §B.3 · pp. 29–30, Table 10 · p. 29, Table 11 · p. 30

B.4 Ensuring Visual Groundedness

• The LLM is prompted to remove questions that can be answered without visual information.This instruction targets visual groundedness in the question set.Source: §B.4 · p. 30
• The answering prompt requires choosing the most correct option and returning the answer, its number, and a rationale in a fixed format.It also instructs the model to answer even when uncertain, use END terminators, and avoid extraneous text.Source: §B.4 · p. 30

C.1 Unpaired Benchmark Results

• Unpaired results best paired results by 10–12% in general, with moderate sensitivity to language perturbation and answer ordering.The table reports unpaired performance by model across question categories and difficulty levels.Source: §C.1 · p. 31, Table 12 · p. 31
• Gemini 2.5 Flash dominates the unpaired results, while PerceptionLM and InternVL2.5 rank ahead of GPT-4o on the reasoning subsection.Reasoning questions exclude descriptive questions.Source: §C.1 · p. 31
• Table 12 covers unpaired performance by model across benchmark question categories and difficulty levels.Source: Table 12 · p. 31

C.2 Blind and Single Frame Results

• The blind robustness check used Qwen2.5-72B-Instruct, selected for a size similar to the construction model but different enough to test shortcut removal.Source: §C.2 · p. 31
• Residual blind-test information may reflect differing Llama and Qwen sensitivities, while the post-filtering human quality-assurance round may reintroduce linguistic hints.Source: §C.2 · pp. 31–32
• Blind LLM performance averages approximately 25%, significantly below every tested video model; blind and single-frame ablations perform similarly poorly at 21–25%.The paired-results table includes blind and single-frame conditions.Source: §C.2 · pp. 32–33, Table 13 · p. 32
• Human performance remains high with only a single frame, a shortcut the authors note may be difficult to avoid for real-world physical-activity questions.Source: §C.2 · p. 33

D Question Pairing

• Pairing questions is an inexpensive strategy for reducing variance and improving detection of small performance improvements when item generation limits benchmark length.Source: D Question Pairing · p. 34
• Reordering answer sets and perturbing distractors lets paired questions test sensitivity to minor language differences and option order.Source: D Question Pairing · p. 34
• The variance model divides items into a known fraction k and an unknown fraction u = 1 − k, with both quantities unobserved.Source: D Question Pairing · p. 34
• The analysis treats known answers as contributing no variance and assumes zero covariance, leaving variance dependent on chance-guess probability for unknown items.Source: D Question Pairing · p. 34
• Pairing reduces detectable effect size according to the ratio of unpaired to paired variances.Source: D Question Pairing · p. 34
• For five-option questions, the reported variance benefit is described as equivalent to a test approximately four times larger.Source: D Question Pairing · pp. 34–35