PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World

Yuzhi Huang, Kairun Wen, Rongxin Gao, Dongxuan Liu, Yibin Lou, Jie Wu, Jing Xu, Jian Zhang, Zheng Yang, Yunlong Lin, Chenxin Li, Panwang Pan, Junbin Lu, Jingyan Jiang, Xinghao Ding, Yue Huang, Zhi Wang

arXiv:2603.12746v1cs.CV

TL;DR

MLLMs excel at static visual understanding but remain inconsistent when tracking and reasoning about dynamic objects and interactions in evolving 4D scenes. The paper introduces Dyn-Bench, a large-scale benchmark combining spatio-temporal VQA and dynamic object grounding across diverse filtered video data, and finds that structured visual and textual integration improves dynamic understanding.

  • Problem

    Current MLLMs lack systematic evaluation and reliable capability for object-centric spatio-temporal reasoning and dynamic object grounding in evolving scenes.

  • Method

    Dyn-Bench evaluates general, spatial, and region-level MLLMs using 1k videos, 7k VQA pairs, and 3k grounding annotations from diverse real and synthetic datasets.

  • Results

    Region-level models show the strongest object-centric reasoning and grounding, while structured methods such as Mask-Guided Fusion and ST-TCM improve motion perception and spatio-temporal reasoning.

  • Takeaways & Limitations

    Reliable dynamics understanding benefits from jointly modeling localized visual evidence with structured spatial, temporal, and motion information.

Abstract

from arXiv · show

Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at "thinking in dynamics", i.e., perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies (e.g., chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs' dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/.

1. Introduction

Existing MLLMs handle static spatial understanding better than evolving scenes, where they struggle to track objects and reason coherently about motion, interactions, and temporal change. Dyn-Bench addresses this gap with unified evaluation of spatio-temporal reasoning and dynamic object grounding across complementary levels.

  • Motivation and gap: Current MLLMs struggle to maintain consistent representations of moving objects across frames under occlusion, ego-motion, and semantic ambiguity.These failures produce fragmented interpretations of motion and interactions in dynamic environments.
  • Motivation and gap: Existing benchmarks emphasize scene-level temporal correlations but insufficiently evaluate object-centric tracking and fine-grained motion boundaries.This limits assessment of coherent reasoning about motion, causality, and scene evolution.
  • Benchmark contribution: Dyn-Bench evaluates three complementary levels of dynamic scene understanding, combining spatio-temporal reasoning with dynamic object grounding.The levels cover inter-object perception, object–scene tracking, and camera–object reasoning.
  • Benchmark contribution: Dyn-Bench contains 1k videos, 7k VQA pairs, and 3k grounding annotations constructed from eight real and synthetic datasets.Its design targets comprehensive evaluation in realistic 4D environments.
  • Benchmark contribution: The benchmark unifies evaluation of dynamic object grounding and reasoning across multiple levels and diverse data domains.This addresses the limited object-centric coverage of prior spatio-temporal benchmarks.
  • Structured reasoning: Structured approaches such as mask-guided methods and ST-TCM improve coherence in reasoning about dynamic objects and scenes.ST-TCM unifies spatial, temporal, and motion information into one representation.

2. Dyn-Bench

Dyn-Bench is built from diverse 2D and 4D dynamic-scene datasets, filtered for geometric and temporal reliability, and organized around three complementary levels of object-centered understanding. Its benchmark pairs spatio-temporal VQA with dynamic-object grounding and uses structured representations to encode motion and relations.

  • Dataset scope: Dyn-Bench contains 1k dynamic video scenes, 7k VQA pairs, and 3k grounding annotations from four 2D and four 4D datasets.The sources span diverse environments, motion patterns, and camera trajectories.
  • Benchmark organization: The benchmark covers Dynamic Inter-Object Perception, Dynamic Object–Scene Tracking, and Dynamic Camera–Object Reasoning.Each level integrates spatio-temporal reasoning and dynamic object grounding tasks.
  • Data construction: Data collection uses instance masks, depth maps, and camera poses to support cross-modal question–answer generation and object-category annotation.Missing annotations are completed with existing pipelines before filtering.
  • Data construction: A multi-criteria filter evaluates geometric stability, motion smoothness, image sharpness, and depth consistency, supported by VLM-based quality evaluation.Low-quality videos are discarded to improve data reliability.
  • Structured representation: ST-TCM reconstructs 3D object trajectories from per-frame RGB-D inputs and segmentation masks, producing position, size, and orientation attributes.These geometric cues support textual modeling of inter-object and camera–object relations.

3. Evaluation on Dyn-Bench

Dyn-Bench evaluates general, spatial, and region-level MLLMs on spatio-temporal reasoning, with region-level models additionally assessed on dynamic object grounding. Performance varies by capability: spatial priors help geometry-dependent reasoning, while region-level cues provide stronger object-centric dynamics understanding.

  • Evaluation setup: General, spatial, and region-level MLLMs are evaluated under distinct task coverage, with region-level models assessed on both reasoning and dynamic object grounding.General and spatial models lack explicit dynamic object grounding and are evaluated only on spatio-temporal reasoning.
  • Main results: General MLLMs perform strongly on relational and motion-oriented reasoning, while open-source models show more balanced object- and scene-level generalization.Qwen3-VL-235B is reported as comparable or superior overall, with smaller variants remaining competitive.
  • Main results: Spatial MLLMs perform better on geometry-dependent object–scene reasoning but remain weaker on camera–object interaction and motion-centric reasoning.The results indicate that spatial priors alone are insufficient for dynamic 4D scenes.
  • Main results: Region-level MLLMs achieve the strongest object-centric spatio-temporal reasoning and dynamic object grounding through fine-grained regional cues and localized feature alignment.UniPixel-7B leads reasoning within this group, while Sa2VA variants achieve the highest grounding accuracy.

4. How MLLMs Think in Dynamics Textually

The textual analysis examines MLLM self-explanations in dynamic settings and finds a gap between fluent language and physically grounded reasoning. Explicit spatio-temporal structure helps produce more coherent motion and relational predictions.

  • ST-TCM Analysis: ST-TCM is evaluated as an auxiliary input together with ablations of temporal semantics, spatial geometry, and motion dynamics.The study uses these components to identify factors that improve spatio-temporal reasoning across representative models.
  • Self-Explanations: GPT-4o’s self-explanations reveal fluent but physically inconsistent reasoning in some dynamic-scene failures.In a representative case, it estimates a car’s catching-up behavior using apparent size changes rather than metric reasoning.
  • Self-Explanations: Self-explanation analysis compares success and failure cases to expose strengths and limitations in dynamic reasoning.The analysis uses GPT-4o’s articulated intermediate reasoning as evidence of how models interpret dynamic events.
  • Self-Explanations: Explicit spatio-temporal cues support structured reasoning that integrates motion and relational information for temporally consistent predictions.The successful case estimates relative velocities while combining motion and relational cues.

5. How MLLMs Think in Dynamics Visually

The visual analysis tests whether explicit masks improve MLLMs’ perception of dynamic regions and their temporal and relational reasoning. Combining raw frames with masks is more effective than providing isolated localization cues.

  • Input Strategies: Mask-Guided Fusion combines raw frames with corresponding masks, whereas Masked Frames Only overlays masks while preserving temporal continuity.Both strategies direct attention toward moving entities, but they differ in whether appearance information from the raw frames is retained.
  • Results: Mask-guided visual grounding improves dynamic object interactions and temporal consistency, while Masked Frames Only provides only minor gains.The comparison indicates that isolated localization cues have limited value.
  • Results: Mask-Guided Fusion improves all reasoning categories, with the largest gains in Inter-Object and Camera-Object reasoning.These categories require fine-grained motion and relational understanding.
  • Interpretation: Integrating appearance and motion cues strengthens object grounding and produces more coherent spatio-temporal reasoning.The result connects localized visual guidance with improved continuity and interaction understanding.

6. Related Work

Related work has advanced MLLMs from static perception toward spatio-temporal reasoning, but stable object grounding and temporal coherence remain difficult. Existing benchmarks largely emphasize scene-level or observer-centric evaluation rather than fine-grained dynamic-object understanding.

  • Spatio-Temporal Understanding with MLLMs: Recent MLLMs improve temporal awareness beyond static visual perception through temporal encodings and adaptive attention.These advances support short-term scene understanding and basic motion interpretation.
  • Spatio-Temporal Understanding with MLLMs: Stable spatial grounding and temporal coherence remain challenging under rapid motion, occlusion, and complex interactions.The difficulty is especially pronounced for region-level MLLMs.
  • Benchmarks for Spatio-Temporal Intelligence: Most existing spatio-temporal benchmarks emphasize scene-level or observer-centric reasoning and provide coarse evaluations of dynamic environments.Examples include quantitative motion reasoning and agent-centric temporal reasoning.

7. Discussion and Future Work

Dyn-Bench evaluates dynamic understanding through object-level and scene-level reasoning and grounding across multiple MLLM categories. The results support coupling structured temporal semantics with localized grounding for more coherent dynamic interpretation.

  • Discussion: Dyn-Bench evaluates how general, spatial, and region-level MLLMs perceive, track, and interpret dynamic content.Its assessment covers motion patterns, temporal consistency, and multi-entity interactions.
  • Discussion: ST-TCM enhances temporal coherence and relational reasoning through structured linguistic abstraction of dynamic events.The framework provides a high-level representation of dynamic information.
  • Discussion: Mask-guided visual grounding strengthens motion perception, improves fine-grained object continuity, and mitigates temporal drift.This complements the linguistic abstraction supplied by ST-TCM.
  • Future Work: The findings suggest that reliable dynamic understanding emerges from coupling high-level temporal semantics with localized region-level grounding.Future architectures are motivated to jointly model motion dynamics, relational structure, and higher-level temporal information.

Supplementary Material

The supplementary material adds implementation details, visualizations, and prompt templates that complement the main paper. It covers filtering, ST-TCM construction, qualitative results, failure cases, and data-generation prompts.

  • The supplement details the dynamic video filtering strategy and construction of the Spatio-Temporal Textual Cognitive Map.
  • It presents qualitative Dyn-Bench visualizations, representative failure cases, and comparisons involving ST-TCM and Mask-Guided Input.
  • It describes six prompt templates for generating VQA and grounding pairs.

A.1. Data Filter Strategy

Dyn-Bench uses multi-stage filtering to select dynamic videos with reliable visual, geometric, motion, and semantic properties. The resulting dataset supports stable spatio-temporal reasoning and dynamic object grounding through reconstructed trajectories and structured representations.

  • The filtering process targets videos with object and camera motion, inter-object interactions, human activities, and sequential manipulations while excluding static or heavily edited content.
  • Videos must provide sufficient visual fidelity and stable correspondences for geometric and motion reasoning, excluding severe distortion, abrupt transitions, extreme zooming, and heavy occlusion.
  • The pipeline combines motion, geometry, dynamic coverage, and multimodal semantic analysis to progressively refine dataset quality.
  • A random forest predicts video dynamism from 31 extracted features using 3,000 manually annotated videos scored from 0 to 5.
  • Only videos passing both quantitative and semantic evaluations are retained, preserving visual fidelity, geometric stability, and motion diversity.
  • ST-TCM reconstructs object geometry and motion, models inter-object and camera–object relations, and converts these cues into temporally aligned textual representations.

B. More Visual Results

The supplementary visual results illustrate Dyn-Bench's hierarchical tasks, model failure cases, and structured prompting procedures. Comparisons focus on temporal consistency, spatial grounding, camera-relative reasoning, and dynamic-object interactions.

  • More Visual Results: Qualitative examples span Dynamic Inter-Object Perception, Dynamic Object-Scene Tracking, and Dynamic Camera-Object Reasoning across nine representative task types.
  • Failure Cases on Dyn-Bench: Failure cases across Gemini-2.5 Pro, UniPixel-7B, and VST-7B-RL show temporal, spatial-grounding, and cross-object motion errors.
  • Failure Cases on Dyn-Bench: ST-TCM and Mask-Guided Inputs are compared for their effects on temporal consistency, spatial focus, and reasoning robustness.
  • Prompt Templates for Data Generation: The prompt templates generate questions and grounding expressions about inter-object relations, scene-level tracking, and camera-relative object behavior.
  • Prompt Templates for Data Generation: Camera-object prompts address relative depth, movement toward the camera, apparent scale, frame position, and camera translation.
Loading 2603.12746v1…