VLM4D: Towards Spatiotemporal Awareness in Vision Language Models

1. Introduction

• Human spatiotemporal cognition supports interpreting motion, spatial and temporal change, physical phenomena, and object interactions.It includes tracking lateral shifts, rotation, and repeated actions along trajectories from egocentric and allocentric perspectives.Source: §1 · p. 2
• VLMs’ strong performance on static visual understanding and video tasks such as captioning does not naturally translate into robust spatiotemporal reasoning.The paper contrasts this with human spatiotemporal cognition, which develops naturally within the first few months of life.Source: §1 · p. 2
• Dynamic-scene reasoning requires reconstructing 3D space plus time and integrating spatial, temporal, and semantic cues across changing perspectives.A car’s forward motion and left turn can appear from the camera’s perspective as movement to the right and into the distance when the camera rotates.Source: §1 · p. 2
• VLM4D is introduced as a benchmark to probe current VLMs’ spatiotemporal grounding and identify deficiencies in their reasoning.The paper describes it as the first benchmark explicitly designed to evaluate VLMs’ spatiotemporal reasoning capabilities.Source: §1 · p. 2

2. Related Work

• Prior video methods and recent VLM work address motion, video, or fine-grained spatial-temporal understanding, while this paper evaluates 4D reasoning in state-of-the-art VLMs.Examples include VideoLLM-MoD for long-video efficiency, VideoRefer for object-level understanding, and Grounded-VideoLLM with an additional temporal stream.Source: §2 · p. 3
• VLM benchmarking has expanded from static-image visual QA to video-comprehension tasks that probe temporal coherence and event understanding.MMVU is cited as extending this work to knowledge-intensive expert-level reasoning in video-based models.Source: §2 · p. 3
• Existing video benchmarks assess perception and semantic understanding but largely overlook explicit spatiotemporal awareness, while 3D benchmarks focus on static scenes.The paper presents VLM4D as unifying spatial understanding, temporal continuity, and motion reasoning.Source: §2 · p. 3

3. The VLM4D Benchmark

• VLM4D contains 1,000 videos and over 1,800 QA pairs designed to assess spatial and temporal understanding jointly.Videos are drawn from datasets with spatiotemporal content and augmented with synthetic videos generated using Cosmos [4], modified using techniques from [34].Source: §3 · p. 3

3.1. Benchmark Construction

• Synthetic videos use bounding-box guidance to align generated motion with specified trajectories, followed by manual video and question review.GPT-4o generates directional and false-positive questions; false-positive questions query non-existent objects and constitute 10%, matching the real-data ratio.Source: §3.1 · pp. 4–5
• Human annotations supply the QA pairs, while a three-round, multi-person review refines vague, misleading, or incorrect pairs and filters ambiguous videos.GPT-4o supplements the questions with alternative answers for multiple-choice evaluation.Source: §3.1 · p. 5

3.2. Categorizing Spatiotemporal Performance

• The benchmark evaluates egocentric and exocentric videos across translational movement, rotational movement, spatiotemporal counting, and false positives.These dimensions test linear motion, orientation and perspective shifts, counting objects performing motion, and whether an object or event occurred.Source: §3.2 · p. 5

4.1. Evaluation Setup

• The evaluation covers 23 recently released VLMs, including proprietary and open-source models ranging from 2 to 72 billion parameters.Models include GPT-4o, Gemini 2.5 Pro, Claude Sonnet 4, Grok-2-Vision, Llama 4, and Qwen2.5-VL.Source: §4.1 · p. 5
• Models are evaluated zero-shot with direct-answer and chain-of-thought inference settings, using video or sampled frames as input.Chain-of-thought prompting requests intermediate reasoning steps before the final answer.Source: §4.1 · pp. 5–6
• Multiple-choice accuracy is the primary metric, and GPT-o3 and o4-mini judge full chain-of-thought reasoning with manual resolution of disagreements.This approach addresses outputs that include multiple answer choices or whose final answer contradicts the reasoning.Source: §4.1 · p. 6

4.2. Benchmark Results

• 62.0% overall accuracy for Gemini-2.5-Pro leads the evaluated models, compared with 98.8% human accuracy; performance varies across data and viewpoint categories.GPT-4o reaches 57.5%; Llama-4-Scout-17B scores 54.1% and Qwen2.5-VL-72B scores 53.0%.Source: Table 1 · p. 6, §4.2 · p. 7

5.1. Limited Spatiotemporal Cognition

• CoT shows no large accuracy advantage over Direct Output across evaluated VLMs, and some models produce irrelevant reasoning or conclusions inconsistent with their reasoning.The authors describe a disconnect between visual and linguistic knowledge despite larger models using human-like strategies.Source: §5.1 · p. 7, Figure 6 · p. 7

5.2. Deficiencies in Spatiotemporal Labeling

• The dataset-label analysis covers over 2 million samples and string-matches terms for directionality, translation, rotation, and perspective shifts.The authors manually evaluate ShareGPT4Video, which they identify as having the highest density of spatiotemporal data; Figure 7 presents term occurrences across popular video SFT datasets.Source: §5.2 · p. 7, Figure 7 · p. 7
• Less than 10% of 100 ShareGPT4Video labels detected as spatiotemporal were judged accurate by human evaluators.The authors say this underscores the inadequacy of dense captioning approaches that generate descriptors without capturing precise motion dynamics.Source: §5.2 · p. 7

4D Feature Fields Reconstruction

• The feature-field approach requires per-scene optimization as post-processing, making it computationally intensive.The paper also proposes targeted fine-tuning on data rich in spatiotemporal actions and interactions.Source: §6 · pp. 7–8

Spatial-Temporal SFT

• Targeted spatiotemporal SFT improves reasoning accuracy, while adding synthetic data does not necessarily outperform real-only training.The experiments use Qwen 2VL (7B) and Qwen 2.5VL (7B), with an 80%/20% real-world train/test split.Source: Table 2 · p. 8, §6 · p. 8

4D Feature Fields Reconstruction

• Reconstructed feature fields achieve the highest accuracy across both reasoning types among InternVideo2’s tested input modalities.The comparison uses all 50 DAVIS 2016 videos; the authors attribute the result to structured 4D representations and avoiding RGB reconstruction artifacts.Source: Table 3 · p. 8, §6 · p. 8

7. Conclusion

• Newer open-source and proprietary VLMs perform better than their counterparts but remain significantly behind human proficiency on VLM4D.The conclusion identifies robotics and interactive AI systems requiring deep understanding of dynamic visual environments as relevant application settings.Source: §7 · p. 8

Supplementary Material

• The supplementary material covers VLM4D statistics, evaluation setup, video instruction-tuning dataset analysis, CoT and Direct Output examples, and Feature4X reconstruction details.Source: Appendix Outline · p. 14

• Table A presents the VLM4D dataset breakdown, and Figure A visualizes model performance across question categories.Source: A. VLM4D Benchmark Statistics · p. 14
• Image-only models receive video as multi-frame image sequences using the maximum frames allowed by their context windows.Source: A. VLM4D Benchmark Statistics · p. 14
• Video-capable models use their default frame-rate settings, typically 1 fps.Source: A. VLM4D Benchmark Statistics · p. 14

• Table B organizes the evaluation types and their GPU requirements.Source: B. Evaluation setup · p. 14

C. Video Instruction Tuning Dataset Analysis

• Four video instruction datasets contain over 2 million samples in total.Source: C. Video Instruction Tuning Dataset… · p. 14, Table C · p. 14
• Over 40% of ShareGPT-4o captions include at least one target category, but directional terms such as “left” and “right” seldom co-occur with rotational or translational actions.The authors identify this as a gap in capturing complex spatiotemporal relationships.Source: C. Video Instruction Tuning Dataset… · p. 14, Table D · p. 14

D. Examples of VLMs’ responses

• The response appendix presents evaluated VLM outputs under both chain-of-thought and direct-output prompting.Source: D. Examples of VLMs’ responses · p. 14

E. Details of 4D Reconstruction

• Feature4X represents dynamic foreground motion with dynamic 3D Gaussians guided by a sparse 4D Motion Scaffold, while static 3D Gaussians represent the background.Source: E. Details of 4D Reconstruction · pp. 14–15
• Feature4X extracts video-segment features from InternVideo2-Chat and supplies its unified feature field directly to the InternVideo decoder, bypassing video encoding.The appendix says this improves inference efficiency and retains structural information from the 4D scene representation.Source: E. Details of 4D Reconstruction · p. 15
• A multi-category example illustrates incorrect direction descriptions and translational labels that indicate little beyond that objects or subjects are moving.Source: E. Details of 4D Reconstruction · p. 19
• For the toy-truck direction question, two example chain-of-thought responses give opposite answers: clockwise and counter-clockwise.Their associated direct-output responses also differ, answering C and B respectively.Source: E. Details of 4D Reconstruction · p. 23