PaperlayerView exact revision on arXiv

Source-linked AI summary

4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding

Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xiang An, Bo Li, Xin Xie, ZiDong Wang, Mingze Sun, Shuang Chen, Hongyu Li, Xiaobin Hu, Ruqi Huang

arXiv:2605.05997v2cs.CV

TL;DR

Dynamic spatial reasoning from monocular video remains difficult for VLMs because text-only reasoning is verbose and imprecise, while external geometric modules add complexity. 4DThinker generates annotation-free reasoning data and trains VLMs with joint text-imagery supervision followed by 4D reinforcement learning. Across multiple benchmarks, it consistently improves performance, including 62.0 versus 58.9 for DSR Suite-Model, while remaining limited by estimator noise and multiple-choice evaluation.

  • Problem

    VLMs struggle with dynamic spatial reasoning in monocular video, while text-only and module-augmented approaches have limitations in precision, complexity, or intrinsic capability.

  • Method

    4DThinker combines annotation-free raw-video data generation, Dynamic-Imagery Fine-Tuning with joint text and latent supervision, and 4D Reinforcement Learning with text-only policy gradients.

  • Results

    4DThinker consistently improves across multiple dynamic spatial reasoning benchmarks and reaches 62.0 versus 58.9 for DSR Suite-Model without external 3D priors.

  • Takeaways & Limitations

    Grounding chain-of-thought in continuous 4D imagery is more effective than purely textual or module-augmented approaches for the evaluated dynamic reasoning tasks.

  • Takeaways & Limitations

    The data pipeline relies on off-the-shelf geometric estimators whose errors may propagate into training data, and evaluation focuses on multiple-choice benchmarks.

Abstract

from arXiv · show

Dynamic spatial reasoning from monocular video is essential for bridging visual intelligence and the physical world, yet remains challenging for vision-language models (VLMs). Prior approaches either verbalize spatial-temporal reasoning entirely as text, which is inherently verbose and imprecise for complex dynamics, or rely on external geometric modules that increase inference complexity without fostering intrinsic model capability. In this paper, we present 4DThinker, the first framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, i.e., internally simulating how scenes evolve within the continuous hidden space. Specifically, we first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos. We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly supervises textual tokens and 4D latents to ground the model in dynamic visual semantics. Building on this, 4D Reinforcement Learning (4DRL) further tackles complex reasoning tasks via outcome-based rewards, restricting policy gradients to text tokens to ensure stable optimization. Extensive experiments across multiple dynamic spatial reasoning benchmarks demonstrate that 4DThinker consistently outperforms strong baselines and offers a new perspective toward 4D reasoning in VLMs. Our code is available at https://github.com/zhangquanchen/4DThinker.

1 Introduction

4DThinker targets dynamic spatial reasoning in monocular video, where VLMs struggle with evolving 3D scenes and existing approaches face precision, complexity, and scalability limitations. It combines annotation-free data generation with intrinsic 4D latent reasoning and two complementary training stages.

  • Motivation: Dynamic spatial reasoning requires separating camera ego-motion from object motion as scenes evolve in 3D over time.The capability has direct implications for autonomous driving and robotics.
  • Motivation: Existing approaches either verbalize spatial-temporal reasoning as verbose, imprecise text or depend on external geometric modules.These directions do not directly establish intrinsic dynamic reasoning within the VLM.
  • Design Goals: Prior methods are limited to static scenes and rely on annotated reference images or distilled foundation models, hindering scalability in annotation-scarce video.This motivates imagery-dynamic, model-intrinsic, and data-scalable design goals.
  • Framework: 4DThinker synthesizes annotation-free 4D reasoning data from raw videos by decomposing camera and object motion and interleaving textual analysis with dynamic mental imagery.The framework uses salient landmarks and highlighted frames as imagery learned for latent simulation.
  • Training: DIFT jointly supervises text tokens and latent positions, while 4DRL applies outcome-based rewards with policy gradients restricted to text tokens.This two-stage recipe grounds dynamic visual semantics and avoids optimizing continuous latent positions with discrete log-probabilities.

2 Related Work

Prior latent-reasoning and spatial-understanding methods address text, images, or static 3D scenes, while dynamic monocular-video reasoning requires handling camera and object motion together. 4DThinker extends latent visual reasoning to 4D dynamics and avoids the external geometric modules used by existing dynamic methods.

  • Latent Reasoning: Latent-reasoning research moves intermediate computation from explicit tokens into continuous hidden representations, including multimodal latent visual tokens.Examples progress from dedicated pause tokens and implicit trajectories to continuous visual embeddings with or without auxiliary supervision.
  • Latent Reasoning: Existing latent visual methods remain confined to pure text, 2D images, or static scenes before 4DThinker.4DThinker extends latent visual tokens to spatial-temporal dynamics in video.
  • Visual-Spatial Understanding: Dynamic spatial understanding from monocular video is harder because both camera and objects may move, and benchmarks show VLMs struggle to distinguish them.VLM4D exposes basic motion-direction failures, while DSI-Bench reveals systematic conflation of camera and object motion.
  • Visual-Spatial Understanding: Existing dynamic spatial-understanding methods rely on external geometric modules that increase inference complexity.4DThinker instead internally simulates object trajectories, camera motion, and their interplay through latent visual tokens.

3 Methodology

4DThinker builds dynamic spatial reasoning from raw monocular videos through scalable imagery-grounded data generation and a two-stage training framework. The model learns continuous 4D visual dynamics alongside text and recurrently feeds its latent imagery into subsequent reasoning.

  • Motion-centric QA construction: The annotation-free pipeline separates camera-motion and object-motion understanding before constructing motion-centric questions and corresponding visual imagery.Camera samples use labeled temporal segments and boundary overlays; object samples analyze trajectories and sample 2–5 frames including the active interval endpoints.
  • Video preprocessing: 4DThinker preprocesses raw videos to identify persistent static and dynamic landmarks, track them with temporally consistent masks, and create highlighted overlays.MegaSaM supplies camera trajectories, while Mhigh and SAM3 support landmark selection and mask tracking.
  • Imagery-based CoT synthesis: Imagery-based CoT synthesis interleaves textual analysis with dynamic mental imagery in structured reasoning traces, which are automatically validated for format, chronology, and answer isolation.The pipeline produces approximately 38K supervised training pairs.
  • Dynamic-Imagery Fine-Tuning: DIFT jointly trains text-token prediction and next-embedding prediction at latent positions, aligning hidden states with visual embeddings to teach internal simulation of 4D patterns.The latent objective uses cosine similarity between the preceding hidden state and the ground-truth visual embedding.
  • Latent inference: During inference, latent imagery is recurrently self-conditioned by feeding the preceding hidden state into latent positions, allowing the model to track object motion through 3D space over time.This recurrent mechanism is derived from the DIFT formulation.

4 Experiments

Experiments show that 4DThinker improves fine-grained geometric and holistic dynamic reasoning across base VLMs, benchmarks, and training configurations. Ablations identify the contributions of latent supervision, reinforcement learning, loss and reward components, and latent-token capacity.

  • DSR-Bench: 4DThinker consistently improves DSR-Bench performance across base VLMs, with gains of +6.5 pp for Qwen2.5-VL-3B and +34.0 pp for Qwen3-VL-32B.The full pipeline reaches 34.2 versus 24.6 for Qwen2.5-VL-3B and 62.0 versus 28.0 for Qwen3-VL-32B.
  • DSR-Bench: The best model reaches 62.0 on DSR-Bench, surpassing DSR Suite-Model at 58.9 without external 3D priors.The largest gains occur on absolute subtasks where base models perform near chance, around 20%.
  • Dyn-Bench: On Dyn-Bench, DIFT+4DRL reaches 65.9 versus 54.3 for Qwen2.5-VL-7B and 75.4 for Qwen3-VL-32B, establishing a new SOTA.The method produces the top three leaderboard performers and exceeds GPT-5 and Gemini-2.5-Pro with the Qwen2.5-VL-7B backbone.
  • Training strategy: DIFT reaches 31.1 versus 26.8 for CoT SFT, while DIFT+4DRL reaches 34.2 versus 29.7 for CoT SFT + GRPO.The ablation attributes these improvements to jointly supervised latent visual tokens and textual reasoning, followed by reinforcement learning over the 4D representation.
  • Loss and reward components: Removing Lce reduces performance from 34.2 to 19.3, while removing Lsim reduces it to 28.5; Racc contributes more than Rfmt.The reported drops associate Lce with coherent text generation, Lsim with visual alignment, and Racc with reasoning accuracy.
  • Latent token size: Accuracy rises from 29.6 at K=1 to 31.1 at K=4, then declines to 30.8 at K=8 and 29.3 at K=16.The paper attributes the decline beyond K=4 to excessive latent tokens diluting textual context and disrupting language coherence.

5 Conclusion and Limitation

4DThinker combines scalable 4D data generation, joint text-imagery supervision, and outcome-based reinforcement learning, with consistent benchmark improvements. The authors note estimator-noise and open-ended-task limitations.

  • Conclusion: 4DThinker integrates annotation-free 4D data generation, Dynamic-Imagery Fine-Tuning, and 4D reinforcement learning into one training recipe.The framework enables VLMs to reason about dynamic scenes through latent visual imagery.
  • Conclusion: Consistent improvements across multiple benchmarks support grounding chain-of-thought in continuous 4D imagery.
  • Limitation: The data pipeline relies on off-the-shelf geometric estimators whose errors may propagate into training data.The authors state that more robust geometric priors could improve data quality, while 4DRL partially mitigates this noise.
  • Limitation: Evaluation focuses on multiple-choice dynamic-reasoning benchmarks, leaving open-ended generation tasks such as embodied planning for future work.

A Object Selection Rules

The pipeline selects representative static and dynamic landmarks, verifies their masks, generates camera- and object-motion questions, and synthesizes reasoning traces interleaving text with imagery.

  • Object selection: Predefined rules guide Mhigh to select one representative static object and one dynamic object from each video.Selections are formatted with object names and brief visual descriptions for subsequent mask extraction.
  • Video preprocessing: Static-mask verification checks identity, consistency, quality, and visibility across overlay frames, while dynamic verification permits partial frame acceptance.The dynamic prompt returns per-frame validity indices rather than a single binary decision.
  • Question generation: Camera-motion prompts generate constrained natural-language multiple-choice questions, while object-motion prompts analyze direction and speed before question generation.Object-motion questions cover direction, bounding-box-grounded description, distance change, and speed variation.
  • CoT synthesis and format: CoT synthesis prompts interleave textual reasoning with dynamic mental imagery and add camera compensation for object-motion analysis.The system instruction fixes the output format during training and inference, and 4DRL checks adherence to it with a format reward.
  • Question design: The pipeline organizes candidate question types, target objects, descriptions, and answer choices for generated motion questions.The accompanying tables summarize question types and their corresponding answer choices.

D Training and Evaluation Datasets

DIFT training uses synthesized data from raw SpatialVID videos, while 4DRL uses compound-motion QA pairs; evaluation spans DSR-Bench and Dyn-Bench.

  • Training data: DIFT uses ∼38K synthesized samples from SpatialVID, a 2.7M-clip video collection with MegaSaM-estimated camera poses.The authors use raw videos and geometric annotations without human labels.
  • Training data: 4DRL uses ∼37K DSR-Train QA pairs covering compound camera-object motions in in-the-wild videos.These QA pairs do not provide explicit reasoning traces.
  • Benchmarks: DSR-Bench evaluates 13 fine-grained dynamic-spatial subtasks, while Dyn-Bench contains 1K videos and 7K VQA pairs for dynamic 4D-scene understanding.Dyn-Bench evaluates perception, tracking, and reasoning without using its mask data.
  • Data generation: The camera- and object-motion CoT synthesis prompts support the construction of training reasoning traces from video inputs.

E Benchmark Subtask Descriptions

The benchmarks organize dynamic spatial reasoning by viewpoint mobility, spatial attributes, and semantic relations among objects, scenes, and cameras.

  • Benchmark notation: The benchmark descriptions clarify the subtask abbreviations used in the reported tables.
  • DSR-Bench: DSR-Bench separates subtasks by absolute versus relative viewpoint mobility and by spatial attribute type.Absolute uses a fixed timestamp viewpoint, whereas relative follows the observing agent over time.
  • DSR-Bench: Its attributes include distance, direction, orientation, speed, speed comparison, and direction prediction.
  • DSR-Bench: N-Temp adds free-form questions generated by a language model to probe spatial-temporal understanding beyond fixed templates.
  • Dyn-Bench: Dyn-Bench evaluates inter-object interactions, object motion within scenes, and camera-object effects on perceived geometry and temporal consistency.

F Automated CoT Validation

A rule-based validator enforces structural completeness, placeholder consistency, and exact answer validity. Samples that fail are regenerated up to three times before persistent failures are discarded.

  • The validator checks paired reasoning and answer tags, matching placeholder and overlay-image counts, and exact agreement with predefined answer options.
  • Failed samples are regenerated up to three times, while persistent failures are discarded.

G Qualitative Examples

Qualitative examples show 4DThinker using latent visual tokens to articulate internal 4D imagery before answering dynamic spatial questions. The examples cover temporal distance changes and global movement patterns that baseline models misinterpret.

  • Qualitative Examples: 4DThinker first produces latent visual tokens representing its internal 4D mental imagery, then derives the answer.The tokens appear between <|latent_start|> and <|latent_end|>.
  • Qualitative Examples: On Dyn-Bench, it captures a player’s diagonal traversal across the court, while comparison models incorrectly infer that he remains in one half.The cited passage attributes the baseline error to local frame-level heuristics.
  • Qualitative Examples: On DSR-Bench, it correctly decomposes a guinea pig’s motion into a growing phase followed by a constant-distance phase, unlike Gemini-3 and Qwen2.5-VL-3B.

H Implementation Details

Implementation uses frozen visual encoders, model-specific DIFT and 4DRL configurations, and substantial H200 compute. Additional qualitative examples illustrate the same latent-4D reasoning process across movement, scale, and scene transitions.

  • Implementation Details: The visual encoder remains frozen throughout training, while 4DThinker can be built on base VLMs including Qwen2.5-VL, Qwen3-VL, and InternVL3.5.
  • Implementation Details: For Qwen2.5-VL-3B DIFT training, videos are sampled at 1 FPS and optimization uses AdamW, learning rate 1 × 10^-5, latent size 4, and 8 H200 GPUs.The configuration uses one epoch, batch size 1, λce = 0.1, and λsim = 1.0.
  • Implementation Details: Training scales to up to 64 NVIDIA H200 141GB GPUs per model, and the paper notes that RL hyperparameters strongly influence the overall training process.
  • Implementation Details: 4DRL uses modified GRPO with group size G=8, reward weights λacc = 1.0 and λfmt = 0.2, learning rate 1 × 10^-6, and KL coefficient β = 0.01.The reported setup also uses maximum completion length 8192, batch size 8, and two gradient-accumulation steps.
  • Qualitative Examples: Qualitative cases show latent 4D tracking of diagonal motion, stable apparent size despite posture change, and transitions from open fields to dense forest.
  • Implementation Details: Mask overlays use opacity α = 0.6 and red highlight color c = [255, 0, 0] for both static and dynamic objects.
Loading 2605.05997v2…