PaperlayerView exact revision on arXiv

Source-linked AI summary

MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models

Xiyang Wu, Zongxia Li, Jihui Jin, Guangyao Shi, Gouthaman KV, Vishnu Raj, Nilotpal Sinha, Jingxi Chen, Fan Du, Dinesh Manocha

arXiv:2511.18373v2cs.CV

TL;DR

VLMs struggle to reason about physics from video when motion, spatial relationships, and physical dynamics must be integrated. MASS adds grounded spatial-temporal representations to VLMs and post-trains them to use those cues. The authors report consistent gains over strong baselines and recent state-of-the-art models, with performance comparable to Gemini-2.5-Flash.

  • Problem

    VLMs struggle to connect visual evidence with underlying physical dynamics, and often miss or hallucinate violations of physics in videos.

    Source: §1 · p. 1

  • Method

    MASS grounds entities, encodes their spatial and motion cues as structured language-space representations, and uses reinforcement fine-tuning to strengthen reasoning.

    Source: §1 · p. 2

  • Results

    MASS yields consistent gains over strong baselines and recent state-of-the-art VLMs on real-world and AIGC videos, with performance comparable to Gemini-2.5-Flash.

    Source: §6 · p. 8, §5.3 · p. 8

  • Takeaways & Limitations

    The authors conclude that structured spatial-temporal signals can support physics-aware reasoning across real-world and AIGC videos.

    Source: §6 · p. 8

    Physical abnormality detection remains more difficult than grounded physics-comprehension tasks, despite a reported 12% improvement on that task.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Vision Language Models (VLMs) perform well on standard video tasks but struggle with physics-related reasoning involving motion dynamics and spatial interactions. We present a novel approach to address this gap by translating physical-world context cues into interpretable representations aligned with VLM perception, comprehension, and reasoning. We introduce MASS, a model-agnostic approach that injects spatiotemporal signals into the VLM language space via depth-based 3D encoding and visual grounding, coupled with a motion tracker for object dynamics. We also contribute a comprehensive benchmark, MASS-Bench, consisting of 4,350 real-world and AIGC videos and 8,361 free-form video question-answering pairs focused on physics-related comprehension tasks, with detailed annotations including visual detections and grounding over sub-segments, as well as full-sequence 3D motion tracking of entities. To strengthen cross-modal alignment and reasoning, we apply reinforcement fine-tuning to MASS. Experiments and ablations show that our refined VLMs outperform comparable baselines, larger models, and prior state-of-the-art models, achieving performance comparable to closed-source state-of-the-art VLMs, with only a 2\% gap to Gemini-2.5-Flash on physics reasoning and comprehension.

1. Introduction

VLMs struggle to connect video evidence with physical dynamics, especially when spatial, motion, and temporal cues interact. MASS adds structured spatial–temporal grounding, and the authors report improved physics reasoning across diverse tasks.

1. Introduction

  • Physics reasoning remains difficult because models must connect visual evidence to physical dynamics, while dense motion-level annotations are scarce.Models can rely on familiar visual and language priors rather than evidence, particularly when AIGC videos contain implausible motion.Source: §1 · p. 1
  • The paper motivates adding structured spatial and temporal representations to existing VLMs rather than training models from scratch.Prior specialized modules provide spatial-temporal cues, but their systematic integration into VLMs for higher-level physics reasoning is rarely explored.Source: §1 · pp. 1–2
  • MASS combines entity grounding and 3D motion representations with post-training; its refined VLMs outperform comparable, larger, and prior state-of-the-art models.The approach uses spatial and motion representations to support reasoning about motion, interactions, and spatial constraints.Source: §1 · p. 2
  • 8.7% and 6.0% are the reported improvements for the refined VLMs over comparable and larger baselines and prior state-of-the-art models.The passage does not assign each percentage to a specific baseline category.Source: §1 · p. 2
  • MASS-Bench contains 4,350 real and AIGC videos and 8,361 free-form question-answer pairs on spatial-temporal comprehension, physics reasoning, and abnormality detection.Its annotations include entity-level visual grounding and dense spatial-motion representations across videos.Source: §1 · p. 2

2. Related Work

Prior work advances video understanding, spatial and temporal modeling, and physics evaluation, but physics-centric reasoning still requires their joint integration with physical commonsense.

Physical Reasoning and Abnormality Detection in Videos

  • Existing physics and commonsense evaluation benchmarks and mitigation methods address video failures, but reliably identifying and addressing physical failures remains open.The cited approaches include evaluation protocols and methods using physics cues, chain-of-thought, action-centric correction, and motion-aware judgments.Source: §2 · p. 3

Spatial, Temporal, and Motion Understanding in VLMs

  • Physics-centric reasoning remains underexplored because it requires jointly integrating spatial, temporal, and motion cues with physical commonsense.Related work has improved each of these dimensions in VLMs, but the passage identifies their joint use for physics reasoning as a gap.Source: §2 · p. 3

Video Understanding in VLMs

  • Recent VLM work targets long-video modeling, temporal reasoning, hallucination, and reinforcement-based training, alongside persistent multi-segment reasoning failures.The review also notes long-context retrieval and large-scale 3D spatial VQA data.Source: §2 · p. 3

3. MASS-Bench: A Motion-Grounded Physics Reasoning and Comprehension Benchmark

MASS-Bench is a free-form video QA benchmark designed for physics-related comprehension, with real and generated videos, two question styles, and detailed motion-grounding annotations.

3. MASS-Bench: A Motion-Grounded Physics Reasoning and Comprehension Benchmark

  • Existing resources often lack fine-grained, temporally consistent entity annotations and balanced examples that follow or violate physical laws.The section presents these gaps as limiting entity-level spatiotemporal reasoning and factual comprehension.Source: §3 · p. 3
  • MASS-Bench comprises 4,350 videos and 8,361 free-form QA pairs, with positive and negative examples and human-verified answers.Videos come from real-world and AIGC sources; questions address physics comprehension and abnormalities.Source: §3 · p. 3
  • MASS-Bench distinguishes factual questions, which assess grounding of real versus hypothetical scenarios and physical mechanisms, from critical-thinking questions requiring inference of causes or intentions and detection of physical or commonsense abnormalities beyond explicit annotations.Source: §3 · p. 3
  • Each pair includes temporal segmentation, entity-level visual grounding, full-video temporal profiles, and per-segment 3D positions and motion vectors.The structured annotations are intended to make physics-intensive perceptual cues more interpretable for reasoning.Source: §3 · p. 4, Figure 2 · p. 4
  • The benchmark spans five categories: spatial understanding (33.3%), temporal understanding (19.5%), motion and action recognition (14.4%), physics comprehension (15.6%), and physical abnormality detection (17.1%).The categories progress from basic spatial-temporal awareness toward higher-level physics comprehension and violation detection.Source: §3 · p. 4

4. MASS: Model-Agnostic Approach

MASS augments existing VLMs with entity-focused spatial and motion signals, converts those signals into language-aligned representations, and post-trains the model to reason over them.

4. MASS: Model-Agnostic Approach

  • MASS is a model-agnostic augmentation that adds dedicated spatial and motion encoders and targeted training to existing VLMs.The design aims to improve spatiotemporal perception, cross-modal alignment, and physics-aware reasoning without a heavy preprocessing pipeline.Source: §4 · p. 4, Figure 3 · p. 5

4.1. Entity-Centric Visual Grounding

  • Question terms guide entity detection and grounding; dynamic sub-sequences support tracking, and segmentation masks enable later spatial-temporal analysis.Grounding localizes queried entities, preserves their motion and spatial context, and filters irrelevant information.Source: §4.1 · pp. 4–5

4.2. Spatial Motion Feature Extraction

  • CoTracker3 tracks points within grounded regions, while depth estimates and aligned RGB frames yield 3D positions and entity motion trajectories.The extracted motion features include averaged motion vectors and first and last tracked positions.Source: §4.2 · p. 5

4.3. Visual Feature Representation

  • MASS serializes each entity’s spatial-motion signals as language templates containing 3D start and end positions, motion vectors, bounding boxes, and temporal indices.The representations are intended to give the VLM explicit cues in a modality it handles effectively.Source: §4.3 · pp. 5–6

4.4. Training Pipeline

  • Post-training addresses the mismatch between grounded motion features and pretrained VLM representations using chain-of-thought, SFT, and T-GRPO-based RFT.T-GRPO adds temporal-aware rewards to GRPO; the training also uses semantic-coherence, temporal, and format rewards.Source: §4.4 · p. 6

5. Empirical Results

MASS-augmented VLMs outperform the previous best model and reach performance comparable to Gemini-2.5-Flash. The results also highlight the value of combining spatial-temporal cues with reasoning-focused training.

5.1. Baseline Insights: Why Motion-Aware Spatial-temporal Grounding Matters

  • Qwen2.5-VL-7B+MASS and LLaVA-OneVision-7B+MASS exceed Qwen3-VL-8B by 8.7% and 6.0%, respectively, while Qwen2.5-VL-7B+MASS matches Gemini-2.5-Flash overall.It also outperforms Gemini-2.5-Flash on physical abnormality detection, the most reasoning-heavy category.Source: §5.1 · pp. 6–7, Table 1 · p. 6

5.2. Ablation Study: The Role of Reasoning

  • GRPO yields larger overall gains than SFT, reaching 61.93% on Qwen2.5-VL-7B and 59.24% on LLaVA-OneVision-7B, though SFT performs better on PA in reruns.GRPO gains are especially clear on SU, TU, MAR, and PC; the authors suggest task-level reward optimization suits physical reasoning better than SFT.Source: §5.2 · p. 7, Table 3 · p. 7
  • The balanced real-plus-synthetic training setting works best on both backbones, and replacing 3D motion with 2D motion lowers accuracy.Real videos add natural appearance and motion variation, while synthetic data exposes physical patterns more clearly; the 2D ablation removes depth-aware projection.Source: §5.2 · p. 8, Table 4 · p. 8

5.3. Discussion

  • MASS improves physical abnormality detection by 12%, but this remains harder than grounded physics-comprehension tasks and requires further advances.The task requires overcoming hallucinations from learned real-world physics priors and explaining why a visual process violates physical laws.Source: §5.3 · p. 8

6. Conclusion, Limitations and Future Work

MASS combines structured spatiotemporal cues with reasoning-oriented VLM post-training, alongside a benchmark for physics-focused video QA. The authors report gains over strong baselines and identify remaining limits in crowded scenes and under strong camera motion.

Conclusion

  • MASS yields consistent gains over strong baselines and recent state-of-the-art VLMs on real-world and AIGC videos.The refined models reach performance comparable to Gemini-2.5-Flash.Source: §6 · p. 8
  • MASS injects physical-world cues into VLM inference through depth-based 3D encoding, visual grounding, and motion-aware trajectory modeling.Source: §6 · p. 8
  • MASS-Bench pairs positive and negative video demonstrations with QA on spatial-temporal comprehension, physics reasoning, and abnormality detection.It includes fine-grained visual detections, sub-segment grounding, and 3D motion tracking.Source: §6 · p. 8

Limitations

  • Performance may decline in crowded scenes, and strong camera ego-motion remains an open challenge.MASS-Bench focuses on top-K entities and does not evaluate highly crowded scenes; noisy motion cues cause gradual rather than catastrophic degradation.Source: §6 · p. 8

Future Work

  • Future work targets longer-range motion grounding, scalable tracking in crowded scenes, denser multi-entity data, stronger camera motion, and more diverse training examples.Source: §6 · p. 8

Supplementary Material

This supplementary section expands on the MASS-Bench dataset description with additional definitions and statistics.

Supplementary Material

A.1. Dataset Composition and Metadata

The supplementary material describes MASS-Bench’s composition, question taxonomy, example and question types, and layered motion-grounding annotations. It also specifies split and inference protocols.

A.1. Dataset Composition and Metadata

  • MASS-Bench contains 6,093 training and 2,268 test examples from 4,350 unique videos, averaging 545.8 frames and about 19.6 seconds.Approximately 1.5% of unique videos overlap across splits, but no video-question pairs are shared.Source: §A.1 · p. 12
  • All methods use the same fixed test set and evaluation scripts, and MASS uses input-video depth and tracking signals without ground-truth test annotations at inference.Source: §A.1 · p. 12

A.2. Question Categories

  • The five question categories progress from spatial and temporal understanding through motion recognition to physics comprehension and physical-abnormality detection.They are ordered from easier foundational awareness toward more challenging reasoning and violation detection.Source: §A.2 · p. 12

A.3. Example and Question Types

  • The benchmark distinguishes positive examples that follow real-world physics from negative examples containing at least one physically implausible scene element.It also separates factual questions with explicit cues from critical-thinking questions requiring inference about events or dynamics.Source: §A.3 · p. 12

A.4. Motion Grounding Annotations

  • Each pair has five annotation layers: frame-range segmentation, entity bounding boxes, full-video entity profiles, per-segment 3D positions, and 3D motion vectors.Entity profiles persist after detection and remain blank in segments where the entity is absent.Source: §A.4 · p. 12

B. Implementation Details

Implementation details cover model training, motion-grounding prompts for video QA, and standardized answer evaluation. The prompts serialize entity-level motion descriptors and guide structured reasoning and answers.

B. Implementation Details

  • Qwen2.5-VL-7B and LLaVA-OneVision-7B use identical full-parameter fine-tuning settings on 8 NVIDIA H100 GPUs.GRPO takes approximately 9–12 hours per model, while SFT ablations take 2–3 hours.Source: B. Implementation Details · p. 12

C. Visual Feature Representation Template

D. Evaluation Template

E. Experiments on Real-world Video QA

MASS is evaluated on MMVU and MVBench, two real-world video QA benchmarks, with gains depending on the model and detection quality. Qwen2.5-VL-7B+MASS improves over its baseline on both benchmarks, while LLaVA-OneVision-7B+MASS drops on these out-of-distribution tasks.

E. Experiments on Real-world Video QA

  • MMVU tests expert-level temporal, procedural, and interaction-centric reasoning, while MVBench focuses on temporal video understanding and action-centric tasks.Both contain diverse video QA pairs captured from real-world scenarios.Source: E. Experiments on Real-world Video QA · p. 14
  • Qwen2.5-VL-7B+MASS improves over its baseline and narrows the gap to close-source models on MMVU and MVBench, while LLaVA-OneVision-7B+MASS drops on these out-of-distribution tasks.GPT-4o has the highest accuracy on both datasets; MMVU's expert-level interdisciplinary reasoning highlights limitations of smaller 7B models.Source: E. Experiments on Real-world Video QA · p. 14, Table 6 · p. 15
  • Missed salient entities or noisy motion cues, common in MMVU and MVBench, can degrade performance and motivate more selective motion-profile extraction.The authors identify detection and tracking quality as a factor in performance on these benchmarks.Source: E. Experiments on Real-world Video QA · p. 15

F. Noise-Induced Robustness

MASS-augmented models tolerate moderate motion-profile perturbations, but sign reversals can trigger errors. The section also reports inference overhead and bounds the current approach to entity-centric tracking with limited camera motion.

F. Noise-Induced Robustness

  • Under 10% Gaussian perturbations to boxes/positions, 3D trajectories, or both, MASS-augmented backbones show slight accuracy drops yet outperform their baselines across settings, including after RFT.Failures mainly occur when noise flips motion-component signs, reversing inferred directions and cascading into process misinterpretation.Source: F. Noise-Induced Robustness · p. 15

G. Inference Time and Memory Usage

  • MASS increases per-query latency through motion-profile extraction, while GPU memory rises modestly; profiles are cacheable per video, and faster modules or lower frame stride, resolution, or top-K could reduce overhead.Timing and memory are measured on 100 randomly sampled test cases on an NVIDIA L40S.Source: G. Inference Time and Memory Usage · p. 15, Table 8 · p. 15

H. Multi-object Tracking and Camera Motion

  • MASS-Bench tracks grounded target entities only (top-K, default 5) and largely limits camera motion; crowded multi-entity tracking and stronger camera-motion handling remain future work.Dense tracking in crowded scenes can increase compute and introduce failures; the authors note that MASS degrades gracefully under noisy motion cues.Source: H. Multi-object Tracking and Camera… · p. 15

I. Case Study

The case study presents qualitative video QA examples across MASS-Bench categories, pairing model responses with expert annotations. Red highlights mark hallucinated or incorrect predictions, providing a reference for assessing model behavior.

Case Study

  • Across the examples, expert annotations describe ground-truth physics-driven dynamics, giving a reference for evaluating model behavior and identifying failure modes.Figures 6–10 cover the categories defined in MASS-Bench.Source: §I · p. 15
  • Spatial-understanding and temporal-understanding examples compare model responses with expert annotations and mark hallucinations or critical contextual errors in red.The examples appear in Figures 6 and 7, respectively.Source: Figure 6 · p. 16, Figure 7 · p. 16
  • Motion-and-action recognition and physics-comprehension examples use the same annotated QA format, with hallucinated content and critical contextual errors highlighted in red.The examples appear in Figures 8 and 9, respectively.Source: Figure 8 · p. 17, Figure 9 · p. 17
  • Physics-abnormality detection examples likewise pair prompts and expert annotations with model responses, highlighting hallucinated content and critical contextual errors in red.Figure 10 shows responses from GPT-4o, Gemini-2.5-Flash, Qwen2.5-VL, and Qwen2.5-VL + MASS.Source: Figure 10 · p. 18
Loading 2511.18373v2…