Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluation

1 Introduction

• Existing multimodal benchmarks often overlook counterfactual reasoning and lack systematic coverage from abstract to concrete and low-level perception to high-level cognition.The paper also identifies limited investigation of how robust video understanding interacts with stepwise reasoning in dynamic environments.Source: §1 · p. 1
• COVER classifies counterfactual video tasks across four quadrants and introduces sub-questions derived from necessary conditions for answering complex queries.The quadrants span abstract-concrete and perception-cognition dimensions.Source: §1 · pp. 1–2
• COVER’s sub-questions break a counterfactual query into necessary steps, allowing evaluation of intermediate accuracy alongside overall reasoning performance.The example asks about actions in a reversed video, including which action occurs first and which occurs last.Source: §1 · p. 2, Figure 1 · p. 2
• Experiments across open-source and commercial models find a strong positive correlation between sub-question accuracy and counterfactual reasoning and robust video-understanding performance.The experiments also examine automatically generated versus human-guided sub-question decomposition.Source: §1 · p. 2

2 Related Work

• Image benchmarks assess visual recognition and spatial reasoning, while video benchmarks address temporal dynamics and contextual reasoning.Examples include MM-Vet, MME, MMBench, and GQA for images, and Video-MME, MvBench, and SEED-Bench for video.Source: §2 · pp. 2–3
• Related work spans chain-of-thought methods and counterfactual reasoning across text, visual QA, and hybrid settings, including approaches to taxonomy and sub-task decomposition.The cited examples include ACQUIRED, AuroraCap, and CoFCA.Source: §2 · p. 3
• Existing video counterfactual benchmarks target narrower areas such as physical properties, captioning, or scenario taxonomy and miss the breadth of real-world reasoning demands.The comparison framework distinguishes video, Q&A format, question source, counterfactual questions, perceptual-cognitive demands, and abstract-concrete demands.Source: §2 · p. 3, Table 1 · p. 3
• COVER uses sub-question decomposition and separates abstract-concrete attributes from perceptual-cognitive demands to broaden counterfactual reasoning evaluation.The paper describes this spectrum as enabling a more nuanced and comprehensive assessment than prior efforts.Source: §2 · p. 3

3 The COVER Benchmark

• The benchmark overview reports approximately 2,800 videos and around 12,000 to 13,000 individual QA instances, with an enhanced version containing about 2.9k question pairs.Each enhanced pair includes at least one original question, one counterfactual question, and at least one sub-question.Source: §3 · pp. 3–4

3.1 Benchmark Definition

• COVER divides 13 tasks into four quadrants based on abstract-concrete and perception-cognition dimensions.The tasks range from emotion and perceptual properties to action recognition, prediction, procedure understanding, and social relation.Source: Figure 2 · p. 3, §3.1 · p. 4

3.2 Data Construction

• Data construction began with 146 videos and 150 reviewed aspect-specific QA pairs, then expanded seed data with 720–760 GPT-generated instances per quadrant.The final collection comprises 2,923 counterfactual question-answer pairs, each with an original question, a counterfactual question, and sub-questions.Source: §3.2 · pp. 4–5
• Eight annotators checked logical consistency and video-grounded reasoning, while three experts cross-validated structural balance.Video lengths ranged from 16 to 1,739 frames, averaging 294.34 frames.Source: §3.2 · pp. 4–5

4.1 Settings

• The evaluation tests varied-scale commercial and open-source models on COVER using oriacc, cfacc, and subacc, with at least three runs and a consistent 16-frame strategy.The model set spans commercial and open-source systems and multiple parameter scales; alternative frame sampling is examined separately.Source: §4 · p. 5, §4.1 · p. 5, Table 5 · p. 7

4.2 Main Results

• Counterfactual accuracy falls sharply below original-question accuracy for many models, while most models score higher on sub-questions than counterfactual questions.Gemini-1.5-Pro and GPT-4o show drops of 25.18% and 24.33%, respectively; examples include Claude-3.5-Sonnet at 49.40% subacc versus 38.04% cfacc.Source: §4.2 · p. 5, Table 3 · p. 6
• The quadrant results show differing strengths: Gemini models excel in several perception and cognition tasks, while InternVL2.5-78B leads in abstract cognition and concrete-perception counterfactual accuracy.Gemini 1.5 Pro reaches 82.14% oriacc in concrete cognition; Gemini 2.0 Flash reaches 75.90% in abstract perception and 74.22% in concrete perception. InternVL2.5-78B reaches 72.88% in abstract cognition and 58.25% cfacc in concrete perception.Source: §4.2 · pp. 5–6, Table 3 · p. 6

4.3 Sub-question Guideline

• CoT raises cfacc over the no-CoT baseline for most models, but manually designed Guide-CoT sub-questions do not consistently outperform automatically generated CoT.Examples include gains of 3.46% for Qwen2-VL-7B and 7.15% for GPT-4o-mini under CoT; GPT-4o-mini scores 57.93% cfacc with Guide-CoT versus 58.62% with CoT.Source: §4.3 · p. 6, Table 4 · p. 6
• For InternVL2.5-78B, providing standard answers with Guide-CoT yields 68.29% cfwithans, an 8.63% gain over no-CoT, compared with CoT's 0.96% gain.The authors suggest standard answers help integrate information and verify intermediate reasoning steps, particularly in complex or multi-step tasks.Source: §4.3 · pp. 6–7

5.1 Ablation Study of Video Frames

• As model size increases, accuracy generally rises, but more video frames can raise oriacc while lowering cfacc.InternVL2.5-8B oriacc rises from 71.26% at 2 frames to 74.20% at 64 frames, while InternVL2.5-2B cfacc falls from 44.20% to 40.06%.Source: §5.1 · p. 7

5.2 Robustness and Logical Reasoning in MLLMs

• 0.608 is the reported correlation between subacc and cfacc, while oriacc and subacc correlate at 0.836; correct sub-questions are also more predictive of correct counterfactual answers than incorrect sub-questions.The analysis characterizes the first relationship as moderate and notes that counterfactual reasoning involves additional complexity beyond single-step reasoning.Source: §5.2 · pp. 7–8, Table 6 · p. 8, Figure 5 · p. 9
• Perception-heavy tasks show sharper declines from subacc to oriacc to cfacc than abstract reasoning tasks, where the transition to counterfactual accuracy is more stable.The authors identify improving counterfactual reasoning in perception-heavy scenarios as a challenge, likely requiring better causal inference and reasoning mechanisms.Source: §5.2 · pp. 8–9, Figure 4 · p. 8

5.3 The Effects of Model Scale

• For the 8B model, the oriacc–cfacc gap is 16.56% and the cfacc–subacc gap is 3.90%, compared with 29.39% and 14.81% for the 2B model.The oriacc–cfacc gap first increases to 30.38% for the 4B model before narrowing at 8B.Source: §5.3 · p. 9, Table 7 · p. 9

• COVER decomposes complex queries into structured sub-questions, enabling fine-grained analysis of multimodal logical reasoning.Source: §6 · p. 9
• The benchmark evaluates MLLMs across abstract-concrete and perception-cognition dimensions for counterfactual video reasoning.Source: §6 · p. 9
• COVER reveals a strong correlation between sub-question accuracy and overall reasoning performance.Source: §6 · p. 9
• The authors highlight the need for improved reasoning abilities in dynamic video tasks and position COVER as a standard for evaluating multimodal logical reasoning.Source: §6 · p. 9

• COVER’s applicability to image or text reasoning remains unexplored.Source: Limitations · p. 10
• Automated sub-question decomposition may not match human-designed questions, especially in complex scenarios.Source: Limitations · p. 10
• Further validation across model architectures and real-world tasks is needed to assess generalizability.Source: Limitations · p. 10

• COVER is intended to enhance counterfactual video reasoning while supporting fairness, transparency, and responsible AI development.The authors acknowledge ongoing challenges in bias mitigation, fairness, and environmental sustainability and encourage broader collaboration.Source: Ethical Considerations · p. 10
• Human annotators received compensation above local minimum-wage standards, and the process was designed to protect their physical and mental well-being.Source: Ethical Considerations · p. 10
• AI was used only for data augmentation and grammar or typo correction, not for generative or creative tasks.The authors also considered risks to research originality and transparency.Source: Ethical Considerations · p. 10

• The appendix covers task-splitting scores, annotation agreements, and data-augmentation prompts and process flow.Source: §A.1 · p. 13
• Three expert annotators independently scored each task using the abstract-concrete and perception-cognition framework.The authors say these scores demonstrate task categorization’s consistency and logical coherence, helping prevent overlap and ambiguity.Source: §A.1 · p. 13
• COVER assessments across multiple dimensions are compiled in Table 9, and Figure 6 outlines the data-augmentation workflow.Source: §A.1 · p. 13
• Figure 7 outlines contextual data augmentation using GPT-4o to construct domain-specific instructional prompts.Source: §A.1 · p. 13

A.2 Additional Results

• GPT-4o’s Procedure Understanding accuracy fell from 78.17% raw to 28.97% counterfactual, a 49.2% decline, while accuracy typically peaked at 8–32 frames and declined at 64.The frame-count decline was particularly pronounced for original and sub-questions, across short and long videos.Source: §A.2 · p. 13, Table 10 · p. 15, Table 11 · p. 15
• Figure 5 reports statistically significant covariation (r = 0.836) between semantic comprehension and multi-step reasoning capabilities across MLLMs.The figure describes associations between oriacc and subacc, and between cfacc and subacc.Source: §A.2 · p. 13
• On manually curated seed data, InternVL2.5-78B-CoT significantly improved bridging the cf–sub–ori accuracy gap; larger, better-aligned models with explicit reasoning were more capable of maintaining coherence across tasks.The evaluation of test-time reasoning strategies is summarized in Table 12.Source: §A.2 · p. 14, Table 12 · p. 16

A.3 Sample Reaults on Test Time Long Reasoning Models

• QVQ-72B-Preview produced detailed, human-aligned reasoning, yet its cfacc was 59.33% versus InternVL2.5-78B’s 71.33%.The authors suggest QVQ’s redundancies or logical inconsistencies may reduce precision, while InternVL2.5-78B may benefit from rapid pattern recognition and information retrieval.Source: §A.3 · pp. 15–16, Table 13 · p. 16
• Table 13 reports InternVL2.5-78B at 70.80% subacc versus QVQ-72B-Preview at 58.76% and Claude-3.7-sonnect at 46.72%; its oriacc is 70.00%, versus 69.33% and 46.00%, respectively.The passage also reports a statistically significant positive correlation between subacc and oriacc.Source: §A.3 · p. 16, Table 13 · p. 16
• QVQ’s reasoning content about video opening and closing scenes aligned with manually curated sub-problems, which the authors describe as cognitive congruence.Figure 11 highlights the aligned content in blue.Source: §A.3 · p. 16, Figure 11 · p. 19

A.4 Examples of Sub-question Guidelines

• The examples show sub-question errors propagating to counterfactual failures and subtle reasoning errors leading to failures, underscoring sensitivity to reasoning-step integrity.Figure 8 marks single-step prediction errors and sub-questions that do not support counterfactual reasoning; Figure 9 compares CoT, Guide-CoT, and Standard reasoning processes.Source: §A.4 · p. 16, Figure 8 · p. 18, Figure 9 · p. 18