Source-linked AI summary
MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs
Dhairya Bhatia, Bishoy Galoaa, Oliver Fritsche, Shahid Kamal, Muhammad Obaidullah Abdul Salam, Umer Saleem, Om Rastogi, Frania Felix Chettiar, Nesli Erdogmus, Sarah Ostadabbas
TL;DR
Video-LLMs are assumed to provide motion-aware perception for world models, but existing evidence leaves open whether they can distinguish controlled physical differences such as speed, magnitude, and direction. MotionBlind tests this with contrastive self-recorded clips and strict four-answer instance scoring, finding that open models remain near chance and that neither more frames nor smarter selection closes the gap. The result limits current Video-LLMs as trustworthy sources of physically grounded supervision, reward, or evaluation.
Problem
Whether Video-LLMs can perceive physically grounded speed, magnitude, and direction, rather than relying on static appearance or language priors, remains unresolved.
Method
MotionBlind uses self-recorded contrastive clip pairs with controlled static content, complementary yes/no questions, and strict IAcc scoring across speed, magnitude, and direction.
Results
Open Video-LLMs remain near the 6.25% IAcc chance floor, while only Gemini 3.1 Pro clears MotionBlind overall and still fails on physical magnitude.
Takeaways & Limitations
A Video-LLM that cannot reliably distinguish different physical motions is not yet a trustworthy source of supervision, reward, or evaluation for a world model.
Takeaways & Limitations
The 60 self-recorded instances use a single indoor actor, binary answer parsing may understate unexpressed reasoning, and open models contain at most 12B parameters.
Abstract
from arXiv · showhide
Video large language models (Video-LLMs) are increasingly used as the perceptual front end of world models, a role that assumes they can read motion: how fast something moves, which way it travels, how hard it is pushed. We show they cannot. A Video-LLM can watch two clips of the same person in the same room, name every object in both, and still fail to say which clip moves faster. We introduce MotionBlind, a contrastive benchmark of self-recorded video for physically grounded motion(speed, magnitude, and direction), the variables a world model must predict. Each instance is a pair of near-identical clips that differ only in motion. Each clip carries two complementary yes/no questions, giving four items per instance, and a model earns credit only if all four are correct. We report Instance Accuracy(IAcc), which has a 6.25% chance floor. Single-frame, appearance, and language-only shortcuts all collapse to it. MotionBlind complements the recent TimeBlind benchmark. We run a controlled study of six open and two frontier Video-LLMs, varying whether the video is present, whether frames are shown in the correct temporal order, and how frames are sampled (1 to 24 frames, four selection strategies). Open models sit near the 6.25% floor, and scale does not help. Removing the video drops every model to zero IAcc, and shuffling frames collapses IAcc to chance, so the task genuinely needs video in order. Neither more frames nor smarter frame selection closes the gap, because these change which frames are seen, not whether motion is read. Only Gemini3.1 Pro clears the benchmark overall, and even it fails on speed. A frontend that cannot tell two speeds of the same action apart is not yet a trustworthy source of supervision, reward, or evaluation for a world model.
1 Introduction
Video-LLMs can describe static scene content while failing to distinguish physically different motion. MotionBlind isolates this gap with contrastive clips and controlled diagnostics, finding that open models remain near chance despite scale, frame changes, and selection strategies.
- 1 Introduction: Video-LLMs can identify people, clothing, and furniture while often misclassifying whether otherwise identical actions move in different directions.This exposes the mismatch between fluent scene description and motion perception.
- 1 Introduction: The benchmark is designed to defeat static-appearance and language-prior shortcuts by requiring all four answers in each contrastive instance.IAcc counts an instance only when all four items are correct, yielding a 6.25% chance floor.
- 1 Introduction: MotionBlind targets speed, magnitude, and direction using contrastive self-recorded clips that differ only in physically grounded motion.The benchmark complements TimeBlind by trading breadth for controlled depth on physical motion.
- 1 Introduction: Only Gemini 3.1 Pro clears MotionBlind overall, while even it fails on purely rate-defined categories.Open models remain near the 6.25% chance floor, and neither scale, additional frames, nor dynamic selection closes the gap.
- 1 Introduction: MotionBlind addresses a reliability assumption behind using Video-LLMs as supervision, reward, and evaluation components for physical world models.The benchmark asks whether models can read how environments change rather than merely recognize what scenes contain.
- 1 Introduction: The study varies video presence, temporal order, frame budgets, and frame-selection methods across six open and two frontier Video-LLMs.Text-only accuracy is zero, shuffled or reversed frames collapse performance toward chance, and learned or training-free selectors do not consistently beat uniform sampling.
2 Related Work
Prior video benchmarks can overestimate temporal competence because static frames, object co-occurrence, and language priors may suffice. MotionBlind therefore targets a complementary physical-motion gap and probes whether frame-ingestion choices can recover it.
- 2 Related Work: Single-clip video QA can overestimate temporal understanding when models answer from representative frames, object statistics, or language priors.Contrastive minimal-pair protocols instead hold static content constant and vary temporal structure.
- 2 Related Work: Existing frame selectors optimize informativeness or temporal coherence, but their gains do not transfer to pairwise, order-sensitive motion discrimination.The cited explanation is that their rewards and encoders lack temporal-position or paired-video information.
- 2 Related Work: MotionBlind and TimeBlind separate physically grounded motion from semantic temporal reasoning while both use contrastive evaluation.MotionBlind focuses on speed, magnitude, and direction; TimeBlind emphasizes semantic changes and dependencies.
3 Introducing MotionBlind
MotionBlind is a self-recorded contrastive benchmark that isolates physical motion from appearance and context. Its strict instance-level metric requires complete success on four complementary predictions, making partial shortcut-based performance insufficient.
- 3 Introducing MotionBlind: Each pair holds object, scene, background, viewpoint, and lighting constant, leaving the evaluated motion attribute as the only determining signal.The benchmark is self-recorded because these physical variables are difficult to source and label reliably from internet video.
- 3 Introducing MotionBlind: MotionBlind contains 60 instances, 82 clips, and 240 question-answer items covering speed, magnitude, and direction.The dataset uses near-identical minimal pairs with balanced yes/no answers and controlled visual conditions.
- 3 Introducing MotionBlind: IAcc counts an instance correct only when all four predictions in its 2 × 2 contrastive structure are correct, with a 6.25% chance level.This strict metric prevents partial credit from appearance or language biases.
- 3 Introducing MotionBlind: Human annotators achieved a mean IAcc of 91.3%, confirming that the instances are independently answerable from motion.Clips were shown one at a time, and per-item accuracy reached 97.8%.
4 Experimental Setup
The experiments compare six open and two frontier Video-LLMs while independently varying frame budget, selection strategy, and temporal-integrity probes. Outputs are constrained to parsed yes/no or multiple-choice answers, with unparseable responses scored incorrect.
- 4 Experimental Setup: The study evaluates six open Video-LLMs and two frontier proprietary models spanning different architectural and training emphases.The evaluated frontier models are Gemini 3.1 Pro and GPT-5.6 Luna.
- 4 Experimental Setup: Frame budgets span 1, 4, 8, 16, and 24 frames, with uniform, random, HORNet, and Frame2Clip selection.Matched 16-frame probes test text-only input, shuffled frames, and reversed frames.
- 4 Experimental Setup: The evaluation parses explicit binary or multiple-choice outputs and scores unparseable responses as incorrect.MotionBlind and TimeBlind use yes/no or A/B prompts, while Video-MME retains its standard four-way format.
5 Experimental Results
MotionBlind shows that open Video-LLMs remain near chance on physically grounded motion, even when given more frames or adaptive selection, while only Gemini 3.1 Pro breaks away and still fails on speed. Integrity probes confirm that success requires video in the correct temporal order rather than appearance or language shortcuts.
- Overall benchmark results: 60.0% IAcc makes Gemini 3.1 Pro the only tested model to break away overall, while open models range from 3.3% to 11.7% against the 6.25% chance floor.GPT-5.6 Luna reaches 15.0%, and human performance is 91.3% IAcc.
- Category breakdown: Open models score 0% IAcc on speed and magnitude, while direction reaches 22–33%; Gemini 3.1 Pro clears chance on three categories but falls to 14% on speed.The failures therefore concentrate on physical rate and magnitude rather than all temporal properties uniformly.
- Integrity probes: Removing video or disrupting temporal order reduces MotionBlind performance to at or near 0% IAcc, showing that the benchmark requires visual evidence and ordered frames.Video-MME remains substantially above chance without video and changes only slightly under frame shuffling or reversal.
- Frame budget: IAcc rises with frame budget but saturates by N=16–24, with open-model ceilings at or below 12%, far below solving the benchmark.The frame-budget sweep covers uniform, random, HORNet, and Frame2Clip selection strategies.
- Frame selection: Dynamic selection does not rescue performance: HORNet reaches 8.3% versus 10.0% for uniform sampling at 16 frames, and no strategy pushes Eagle above 11.7%.HORNet’s nearly flat keep-probabilities make its top-K selections close to arbitrary on most clips.
- Frame selection: The F2C comparison is a scope boundary because its 1 FPS candidate pool is exhausted, making its N=16 and N=24 anchor sets identical.This limits what the F2C column can establish about frame-selection effectiveness.
6 Discussion: Implications for World Models
MotionBlind exposes a motion-perception gap with direct implications for Video-LLMs used in world-model pipelines. The study is limited by controlled small-scale data, indoor single-actor videos, binary responses, and open models of at most 12B parameters.
- Implications for World Models: A Video-LLM that cannot distinguish motion variables cannot be trusted to label transitions, score success, or evaluate physically grounded world-model behavior.The paper specifically warns that misreading force or speed can produce noisy or inverted supervision in physical AI pipelines.
- Implications for World Models: More frames, smarter selection, and better prompting do not recover motion signals that the representation never encoded.The conclusion frames these interventions as post-hoc levers that alter temporal coverage or inputs without ensuring motion is represented.
- Limitations: MotionBlind has 60 self-recorded instances, single-actor indoor clips, constrained yes/no parsing, and open models with at most 12B parameters.These choices introduce small-sample variance, leave in-the-wild and egocentric generalization open, may understate unexpressed reasoning, and limit separation of architecture from scale.
- Limitations: The dataset excludes sensitive or personally identifiable information, with informed consent and automatic face blurring before release.These collection safeguards are stated alongside the benchmark’s safety-critical physical-AI motivation.
7 Conclusion
MotionBlind benchmarks whether Video-LLMs perceive physically grounded motion using controlled, self-recorded contrastive videos. Across six open and two frontier models, open-model motion understanding remains unsolved, and even the frontier model that clears the benchmark overall fails on physical magnitude.
- Conclusion: Open-model motion understanding remains unsolved, while the frontier model that clears MotionBlind overall still fails on physical magnitude.The conclusion also reports that neither more frames, larger models, nor dynamic selection closes the gap.
- Conclusion: MotionBlind is a self-recorded contrastive benchmark that isolates physically grounded motion perception and complements evaluation with TimeBlind.The benchmark is used to study six open Video-LLMs and two frontier models.
A Per-category breakdown
The per-category breakdown shows that open models retain signal only for translational direction, while speed and magnitude remain at chance. Gemini 3.1 Pro performs above chance in three categories but still fails on speed.
- Per-category breakdown: For every open model, speed and magnitude are at the chance floor, while only translational direction carries signal.The comparison uses a matched uniform-16 frame budget.
- Per-category breakdown: Gemini 3.1 Pro clears three of four categories but still fails on speed.
B Full frame-selection sweeps
Frame-budget sweeps show that open models plateau at low IAcc regardless of the frame selector. The Gemma-4-12B-it result uses a cache subset and is not directly comparable.
- Full frame-selection sweeps: The sweep covers per-selector budgets for every open model on MotionBlind, with the shared sampler key also used for TimeBlind.
- Full frame-selection sweeps: Every open model plateaus in the low teens or below, and no frame selector escapes the MotionBlind floor.Figure 5 plots IAcc against frame budget N with one line per selector.
- Full frame-selection sweeps: Gemma-4-12B-it is evaluated on the MotionBlind-82 cache subset, so its result is not 1:1 comparable.
C Video-MME sanity check
Video-MME provides a non-contrastive sanity check, while the supplied figures illustrate how frame budgets and selectors are examined in related temporal and motion evaluations.
- Video-MME remains substantially answerable without motion understanding: removing video leaves 39–44% accuracy, compared with 25% chance, while frame perturbations change accuracy only modestly.
- TimeBlind accuracy rises with frame budget but saturates well below the human ceiling, showing that additional frames do not fully resolve temporal reasoning.
- Dynamic selection is evaluated by marking which of 32 evenly spaced candidate frames are retained under an eight-frame budget for a motion question.
D Full sweep results (all numbers)
The full sweep reports model performance across frame budgets, selectors, and three benchmarks, with MotionBlind and TimeBlind contrasted against standard Video-MME accuracy.
- The reported sweep provides exact values for every model, selector, and frame budget on MotionBlind, TimeBlind, and Video-MME.
- Across Video-MME frame budgets, the same models reach approximately 50–65%, distinguishing MotionBlind’s failure from general video-question-answering incapacity.
- MotionBlind results are tabulated as Instance Accuracy IAcc (%) across models, selectors, and budgets, with 6.25% chance and unavailable runs marked explicitly.
- TimeBlind results use the same IAcc (%) metric and report a 6.25% chance level across the full sweep.
- Video-MME results report standard accuracy (%) against 25% chance and a 27.2% majority baseline, including frame-budget settings from 1 to 24.