Source-linked AI summary
A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
Benno Krojer, Mojtaba Komeili, Candace Ross, Quentin Garrido, Koustuv Sinha, Nicolas Ballas, Mahmoud Assran
TL;DR
Existing video-QA benchmarks can be solved using superficial visual or textual cues, complicating assessment of video-language models’ spatiotemporal understanding. MVP introduces paired videos with identical questions and opposing answers to test physical-world understanding. The best model reaches 40.2% average accuracy, compared with 92.9% for humans.
Problem
Existing video-language benchmarks can yield strong performance through superficial visual or textual cues, limiting what they establish about models’ spatiotemporal understanding.
Source: §1 · p. 1Method
Results
The best model achieves 40.2% average accuracy on MVP, while human performance is 92.9%.
Source: §6 · p. 9Takeaways & Limitations
Video-language models remain far from matching human performance across the tested tasks, although they exceed chance on some categories.
Source: §6 · p. 9Automated curation leaves a small subset of overly simple or ambiguous examples, and single-letter answers may limit performance without free-form reasoning.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Existing benchmarks for assessing the spatio-temporal understanding and reasoning abilities of video language models are susceptible to score inflation due to the presence of shortcut solutions based on superficial visual or textual cues. This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical understanding of video language models. The benchmark is comprised of 55K high-quality multiple-choice video QA examples focusing on physical world understanding. Examples are curated from nine video data sources, spanning first-person egocentric and exocentric videos, robotic interaction data, and cognitive science intuitive physics benchmarks. To mitigate shortcut solutions that rely on superficial visual or textual cues and biases, each sample in MVP has a minimal-change pair -- a visually similar video accompanied by an identical question but an opposing answer. To answer a question correctly, a model must provide correct answers for both examples in the minimal-change pair; as such, models that solely rely on visual or textual biases would achieve below random performance. Human performance on MVP is 92.9\%, while the best open-source state-of-the-art video-language model achieves 40.2\% compared to random performance at 25\%.
1 Introduction
Existing video-QA benchmarks can reward superficial visual or textual cues, leaving video-language models’ physical-world understanding uncertain. MVP tests this ability with visually similar video pairs sharing a question but requiring opposing answers.
1 Introduction
- MVP pairs visually similar videos with identical questions and opposing answers, requiring correct answers for both examples.This design penalizes models that rely on superficial visual or textual cues and produce the same answer for both samples.Source: §1 · pp. 1–2
- The benchmark was built from nine video sources and contains 54,828 multiple-choice video-QA examples, organized into 27,414 minimal-change pairs.The sources span intuitive physics, spatiotemporal reasoning, action anticipation, and robotic manipulation.Source: §1 · p. 2
- MVP extends minimal-change video pairs beyond small prior sets for coarse-grained temporal reasoning by mining pairs from existing sources.The authors describe the process as automated with limited human intervention.Source: §1 · p. 2
- 40.2% is the best evaluated model accuracy on MVP, compared with 92.9% human accuracy.The findings suggest video-language models may struggle with physical reasoning despite strong performance on standard spatio-temporal benchmarks.Source: §1 · p. 2
- The paper analyzes shortcut solutions across all 11 datasets in MVBench using language-only, single-frame/image, and Socratic LLM baselines.Source: §1 · p. 2
3. Benchmark closed-source and open-source state-
The section compares evaluated video-language models’ MVP performance with human performance. The reported model results remain well below the human baseline.
3. Benchmark closed-source and open-source state-
- Figure 2 shows performance of the strongest evaluated VideoLLMs on the MVP mini-version alongside human performance.Source: Figure 2 · p. 3
- Human accuracy is 92.9%, while the best evaluated model reaches 40.2% on MVP.Source: §3 · p. 3
- The reported comparison includes GPT-4o and Gemini, which achieve around 30% accuracy against 25% random performance.Source: §3 · p. 3
2 Robustness Analysis of MVBench
The shortcut analysis tests whether MVBench performance depends on video, language, temporal information, or fine-grained visual features. Its baselines achieve non-trivial results without full video input.
2 Robustness Analysis of MVBench
- The analysis evaluates 20 tasks across 11 datasets, comparing unperturbed video-language models with language-only, video-only, single-frame, and simple Socratic baselines.Source: §2 · p. 3
- A video-only VideoChat2-Mistral reaches 50% overall accuracy, compared with 61% for the same model given the question and 30% for random guessing.This indicates that answer candidates can sometimes be filtered without knowing the question.Source: §2 · p. 3
- On counterfactual inference, the correct answer often holds regardless of the question’s intervention, such as when two objects collide either way.Source: §2 · p. 3
- Single-frame Idefics3-8B and Qwen2-VL-7B achieve 47% and 51% overall accuracy, respectively, comparable to full-fledged VideoLLMs.They are on par with or exceed VideoLLMs on eight listed tasks.Source: §2 · p. 3
- The table’s stated pattern is that optimal performance on MVBench tasks is frequently achieved by models relying on visual or textual biases.Source: Table 1 · p. 4
- Shortcut baselines can achieve strong performance on spatiotemporal benchmarks using language cues, visual cues, single frames, or simplified captions.The summary concludes that these tasks may not require temporal reasoning or fine-grained visual features.Source: §2 · p. 4
3 TestingPhysicalWorldUnderstanding via Minimal Change Pairs
MVP tests physical-world understanding with minimal-change video pairs and a scoring rule that requires both members to be answered correctly. Its curation mines similar videos with contradictory answers and filters examples solvable from single frames.
3 TestingPhysicalWorldUnderstanding via Minimal Change Pairs
- A minimal-change pair uses identical questions, visually similar videos, and mutually exclusive answer candidates.Each answer option is correct for only one video in the pair and acts as a hard negative for the other.Source: §3 · p. 4, Table 3 · p. 5
- The scoring rule awards a positive score only when a model answers both samples in a pair correctly; random performance is 25%.The model receives the paired examples independently and outputs one answer letter for each.Source: §3 · p. 4
- The question taxonomy covers human-object and robot-object interactions, intuitive physics, spatial and temporal understanding, memory, counterfactuals, and anticipation.Source: §3 · p. 4
- The curation starts with manually filtered videos and metadata-based questions, yielding 548K video-QA examples before pair mining.The sources include captions, class labels, QA annotations, and concept labels.Source: §3 · p. 5
- Pair mining searches for visually similar videos with semantically matched questions and contradictory answers, using symbolic or neural similarity rules and answer-entailment checks.The pipeline ranks candidate pairs by ViCLIP video-embedding cosine similarity and selects pairs to represent question or skill types.Source: §3 · pp. 5–6
- A five-model ensemble flags single frames as solvable when at least four models answer correctly; pairs are discarded when 30% of frames in both videos are solvable.This heuristic removes around 20% of examples from the previous stage.Source: §3 · p. 6
- MVP contains 54,828 video-QA examples grouped into 27,414 minimal-change pairs, with a reasonably balanced mix of natural, synthetic, robotics, and intuitive-physics videos.Videos average 8.8 seconds, and answer candidates average 8.1 words.Source: §3 · p. 6
- MVP emphasizes physical and perceptual difficulty over linguistic diversity, with 2,355 unique question-and-answer words compared with MVBench’s 4,338.The benchmark’s short videos and basic perceptual skills are intended to reduce reliance on cultural or domain knowledge.Source: §3 · pp. 5–6
4 Empirical Results on MVP
On MVP, most evaluated video-language models score around random chance, far below human performance. Results also vary by task, and ablations show lower average performance after minimal-pair pairing and single-frame filtering.
Overall performance of VideoLLMs
- 40.2% average accuracy for InternVL2.5 and 38.1% for Tarsier-34B contrast with 92.9% human accuracy on MVP.Most other models perform around random chance (25%); human results use a representative subset.Source: §4 · p. 6, Table 4 · p. 7
- VideoLLMs perform better than random on coarse-grained temporal reasoning, indicating some ability to distinguish event order.All models fall short on fine-grained robot-object interactions, while intuitive physics is the hardest category, with sub-random scores.Source: §4 · p. 6
Dataset sub-tasks
- Intuitive-physics performance is close to 0% overall, though LongVU (Qwen2) reaches 39.1% on Gravity-Continuity and 42.2% on Unchangeableness.Tarsier-34B reaches 35.2% on Gravity-Support; some human-object interaction subsets also show above-random performance, including Counterfactual and Memory.Source: §4 · p. 7
Importance of Data Curation
- 45.4% average accuracy with random pairing falls to 27.3% with minimal-change pairs, then drops another 2.2% to 25.1% after single-frame-solvable examples are removed.The additional filtering step lowers performance by 3.5% for fine-grained human-object interactions and 3.3% for coarse-grained temporal reasoning, while robot-object and intuitive-physics categories have almost no single-frame bias.Source: §4 · p. 7
- The curation ablation confirms that minimal-change pairing and single-frame filtering lower average model performance on MVP-mini.The reported average is across five VideoLLMs.Source: §4 · p. 7
5 Related Work
Prior vision-language and video-QA benchmarks have been studied for language and visual shortcuts, while minimal-change pairs offer an approach to testing whether models rely on such cues. MVP applies that approach to video pairs procedurally mined at scale.
Benchmarks addressing vision-and-language biases
- Blind language-only models can solve many vision-language tasks, and video-QA studies likewise report strong language-only baselines.Video models also show visual grounding, spatial, and single-frame biases; some video tasks may not require frame ordering.Source: §5 · pp. 7–8
- Minimal-change input pairs restrict shortcut use because both examples must be answered correctly; textual pairs can remain vulnerable to language bias.Prior video approaches include small manipulated or scraped sets, such as TempCompass and Vinoground (Zhang et al., 2024a).Source: §5 · p. 8
- MVP procedurally extracts minimal video pairs from existing sources at scale and avoids Winoground's metric to remain agnostic to multi-video processing.Its Minimal Pair Score is inspired by Winoground (Thrush et al., 2022).Source: §5 · p. 8
- MVP focuses on short videos testing basic spatial, temporal, or intuitive-physics skills to reduce reliance on cultural knowledge.These skills concern physical-world properties (Yi* et al., 2020; Chen et al., 2022; Jassim et al., 2024; Riochet et al., 2022; Bear et al., 2021; Margoni et al., 2024; Baillargeon et al., 1985).Source: §5 · p. 8
6 Discussion and Limitations
The evaluation finds that current VideoLLMs remain far from human performance across tested tasks, despite above-random results on some subcategories. The authors identify possible performance constraints from the answer format and residual noisy examples from automated curation.
Discussion
- The best model averages 40.2% accuracy on MVP, compared with 92.9% for humans; some subcategories nevertheless exceed random chance.The authors conclude current VideoLLMs remain far from human performance across the tested tasks.Source: §6 · p. 9
Outline
The outline assigns appendices to benchmark variants, curation, shortcut analysis, human baselines, prompting, and project reflections. The curation appendix details a three-step process and dataset-specific procedures.
Outline
- Appendices C–F cover shortcut-analysis explorations, the human baseline, model prompting and answer extraction, and project reflections.Appendix C also gives implementation details for the shortcut analysis motivating MVP.Source: Outline · p. 14
A MVP-mini
- MVP-mini contains 18,290 video-QA examples (9,145 pairs) and is designed for faster evaluation, while MVP-full supports large-scale filtering and curation.Its subset-aware downsampling avoids underrepresenting any dataset or subset.Source: A MVP-mini · p. 14, Outline · p. 14
B Details: The Curation of MVP
- MVP curation uses manual categorical filtering, automatic minimal-difference pairing, and automatic filtering of single-frame-solvable examples; five of nine datasets also require QA conversion.The first two steps vary across datasets, while the third is the same across them.Source: B Details: The Curation of MVP · p. 14, Outline · p. 14
B.1 Manual categorical filtering
Manual categorical filtering selects benchmark subsets and removes question types or examples judged unsuitable for assessing temporal or physical understanding.
B.1 Manual categorical filtering
- MVP selects suitable subsets and question categories from six of its nine datasets.Source: §B.1 · p. 14
- Perception Test questions are manually annotated, with non-temporal or ill-defined questions filtered out.Source: §B.1 · p. 14
- Language Table uses its human-captioned, human-controlled split and excludes single-object captions to retain more complex interactions.Source: §B.1 · p. 14
- CLEVRER counterfactual examples are filtered when the target object is uninvolved in collisions or the answer objects already collide in the video.Source: §B.1 · p. 14
- InfLevel is limited to real-life videos in which people conduct experiments in front of a camera.Source: §B.1 · p. 14
- STAR’s Feasibility and Interaction splits are excluded because they are often ill-defined, language-biased, or too easy.Source: §B.1 · p. 14
B.2 Automatic Pairing of Minimally Different Examples
The pairing pipeline finds visually similar examples with identical questions and mutually exclusive answers, using dataset-specific rules to construct candidate pairs.
B.2 Automatic Pairing of Minimally Different Examples
- Pairing is applied to five of nine datasets; Vinoground and the three intuitive-physics datasets already contain minimal-change pairs.For the other datasets, pairing seeks similar videos and answers that cannot both be true.Source: §B.2 · p. 14
- For other grouped QA examples, the procedure groups identical questions and opposing answers, ranks candidate pairs by visual similarity, and applies dataset-specific rules.The typical selection is 50 pairs per question; one Perception Test rule rejects “Both the other options” answers.Source: §B.2 · p. 15
- Language Table answers use the same question, which is fixed across the dataset; hand-crafted rules also reject entailments involving general and specific directions.Source: §B.2 · p. 15
- Language Table candidates share objects and enough answer-token overlap, then pass a ViCLIP cosine-similarity threshold above 0.9 and hand-crafted entailment checks.The token difference must be greater than zero and less than four; rules discard cases where one answer entails the other.Source: §B.2 · p. 15
- Something Something v2 examples become action questions, and pairing uses well-defined antonyms; 82 action types, or 47% of all actions, have an identified opposite.Caption answers use something-placeholders, and candidate pairs are also matched by objects.Source: §B.2 · p. 15
- For Something Something v2, object matching can be relaxed to one shared object, after which the top 4,000 pairs are selected by ViCLIP similarity.Source: §B.2 · p. 15
- CLEVRER pairs also require near-matching object sets and attributes, allowing one mismatch under the fuzzy-subset criterion.The dataset provides object and attribute metadata for this comparison.Source: §B.2 · pp. 15–16
- CLEVRER candidates share a question and mutually exclusive answers; numerical answers must differ by one.Source: §B.2 · p. 16
C Details: Shortcut Analysis
The appendix reports additional shortcut checks, human-scoring and prompting procedures, and the project’s development rationale. It also records unresolved plans for crowd-sourcing human shortcut performance.
C Details: Shortcut Analysis
- VideoChat2 video-blackout results were slightly worse across both LLM versions; single-frame selection varied 1–2%, except Idefics3’s first frame was more than 5% worse.Source: C Details: Shortcut Analysis · p. 16
- Long generic captions performed best among caption variants for LLaMA3 8B and 70B, but the paper reports medium captions as its generic baseline.Action captions ranked next in performance.Source: C Details: Shortcut Analysis · p. 16
- Perplexity is around random overall, while answer-frequency statistics provide little evidence of skew across MVBench subtasks.The perplexity baseline scores answer candidates using LLaMA3-7B.Source: C Details: Shortcut Analysis · p. 16
D Human Annotation
- Six lab researchers answered one video from each pair at a time, preventing them from using knowledge that paired answers must be AB or BA.Source: D Human Annotation · p. 16
E Details on prompting multi-modal LLMs
- VideoLLMs receive a prompt requesting a single A/B answer, and regex extraction fails on fewer than 1% of examples.Source: E Details on prompting multi-modal LLMs · p. 17
F Behind the Scenes
- The behind-the-scenes section aims to share subjective or typically unreported process details that may help researchers, especially newcomers.Source: F Behind the Scenes · p. 17
F.1 Motivation and timeline
- The project began by diagnosing shortcut vulnerabilities in existing benchmarks, manually inspecting examples and testing frequency, text-only, single-frame, video-only, and caption-based shortcuts.Source: §F.1 · p. 17
- The authors debated what counts as temporal understanding and how to organize examples, while human shortcut crowd-sourcing remained a plan that could improve filtering beyond their five-model ensemble.Language Table’s 440K video-caption examples made finding candidate pairs especially difficult.Source: §F.1 · p. 17
F.2 Observations and lessons learned
The authors describe curation and evaluation challenges, including edge cases, frame sampling, and parsing model outputs. Their advice also stresses anticipating shortcuts and notes multiple directions for video-QA work.
F.2 Observations and lessons learned
- At this data-curation scale, there are too many edge cases to catch every one.Source: §F.2 · p. 17
F.3 Advice for others working on a similar direction
- The authors say a clear story and contribution from the project’s outset make paper writing smoother, and describe video-QA as crowded but open to varied directions.Source: §F.2 · p. 17, §F.3 · p. 18
F.2 Observations and lessons learned
- The first author describes intuitive-physics datasets as MVP’s cleanest and most fascinating sources, a subjective assessment.Source: §F.2 · p. 17
- The authors praise Something Something v2 for low noise at 200K examples and its coverage of interesting, simple phenomena.Source: §F.2 · p. 17
- Many models may lack access to sub-second cues because they represent videos as 16 uniformly sampled frames, while frame rate matters for many MVP examples.The authors say increasingly nuanced questions may hinge on a span shorter than a second.Source: §F.2 · p. 17
- Frame rate matters for many MVP examples, while some models may miss the relevant sub-second span when using 16 uniformly sampled frames.Source: §F.2 · p. 17
- Parsing LLM outputs into structured answer options raises open questions about whether better prompting or more flexible extraction could improve performance.Source: §F.2 · p. 18
F.3 Advice for others working on a similar direction
- Despite studying shortcuts, the authors probably missed some shortcuts or issues in MVP and urge anticipating those future, more capable models could exploit.They note older video benchmarks might have addressed single-frame biases during design.Source: §F.3 · p. 18