PaperlayerView exact revision on arXiv

Source-linked AI summary

Dyn-3D: Unveiling and Resolving Ego-Motion Ambiguity in Vision-Language Models

Jiayu Ding, Zhuodong Liu, Lei Zhang, Manyu Xiong, Hongbo Jin, Haoran Tang, Hongbo Zhang, Changen Zhu, Wenbo Xing

arXiv:2609.01059v1cs.CV

TL;DR

VLMs struggle to infer camera motion because monocular appearance is scale-ambiguous and training data encourages smooth-trajectory shortcuts. The paper introduces Dyn-3D and TempoVista with Kinematic-GSPO, whose physical supervision improves kinematic perception and broader 3D spatial reasoning. The approach assumes reliable camera extrinsics when using its SE(3) frame-selection procedure.

  • Problem

    VLMs rarely receive direct supervision for camera motion and therefore face visual-kinematic ambiguity when inferring metric 3D structure from monocular video.

  • Method

    The paper introduces Dyn-3D counterfactual rendering and TempoVista with Kinematic-GSPO, which embeds physical ground truth into policy optimization.

  • Results

    TempoVista outperforms leading open and proprietary models on Dyn-3D and generalizes to diverse spatial tasks in VSI-Bench.

  • Takeaways & Limitations

    Explicit kinematic perception is essential for robust 3D spatial understanding within the evaluated tasks.

  • Takeaways & Limitations

    The SE(3) frame-selection procedure assumes reliable camera extrinsics from odometry or a 3D Gaussian Splatting pipeline.

Abstract

from arXiv · show

As Vision-Language Models (VLMs) tackle dynamic 3D spatial reasoning, ego-motion perception becomes essential to resolve monocular scale ambiguity. However, current models often overfit to smooth trajectory priors rather than genuinely understanding physical motion. Consequently, their spatial reasoning degrades severely under large displacements, a phenomenon we term Kinematic Collapse. This failure stems from spurious visual-motion correlations in natural videos and a lack of explicit physical supervision. To evaluate this, we introduce Dyn-3D, a benchmark using counterfactual 3D rendering to rigorously decouple visual changes from true kinematic properties. Furthermore, we propose the TempoVista framework, featuring the Kinematic-GSPO algorithm. By embedding metric physical ground truth into policy optimization, TempoVista explicitly grounds visual representations in 3D space. Experiments demonstrate that our approach significantly improves both motion estimation and robust spatial reasoning by utilizing camera dynamics as an effective geometric calibration signal.

1 Introduction

Dynamic 3D spatial understanding requires ego-motion perception because monocular visual signals cannot resolve absolute depth without a metric reference. The paper introduces Dyn-3D and physical-grounded training to expose and address models’ reliance on superficial motion cues.

  • Motivation: Ego-motion perception links 2D visual changes to metric 3D space, helping distinguish object depth from camera-induced parallax and scaling.Monocular visual signals alone have inherent scale ambiguity, so camera trajectory and displacement provide essential geometric constraints.
  • Problem: Current models overfit smooth trajectory priors, causing spatial cognition to fail under large displacements or non-smooth trajectories, termed Kinematic Collapse.In a Qwen3.5-9B pilot, increasing inter-frame displacement from 0.1 to 5.0 meters reduced overall accuracy by 25.30 percentage points, from 63.97% to 38.67%.
  • Benchmark: Dyn-3D uses counterfactual 3D rendering to decouple visual paths from kinematic properties and evaluate VLM ego-motion perception.The benchmark contains 16,063 test samples and finds advanced models near chance when estimating motion magnitude under non-smooth or counter-intuitive trajectories.
  • Method: Kinematic-GSPO applies physical-ground-truth process supervision to penalize unphysical motion inferences and separate true kinematics from visual appearance.The approach is paired with the Dyn-3D-Instruct dataset, which contains 33.6K samples.
  • Results: Improved kinematic perception significantly boosts accuracy on 3D spatial benchmarks including Dyn-3D and VSI-Bench.The paper presents this as evidence that explicit physical-motion understanding supports broader spatial reasoning.

2 Related Work

Prior video-language benchmarks emphasize event and temporal reasoning, while spatial methods and classical geometry address only parts of metric 3D understanding. Dyn-3D targets the resulting visual-kinematic ambiguity with controllable rendering and physically consistent process supervision.

  • Video and spatial reasoning in MLLMs: Mainstream video benchmarks assess event recognition, temporal localization, and commonsense reasoning more than metric spatial inference.Spatial benchmarks such as ScanQA, SpatialVLM, OpenEQA, and VSI-Bench move closer to embodied 3D understanding.
  • Ego-motion and geometric ambiguity: Monocular video remains scale-ambiguous without metric cues, while VLMs rarely receive direct supervision for camera extrinsics, velocity, or displacement.This creates a blind spot in which appearance changes may be mistaken for translation and stable appearance may conceal physical displacement.
  • Controllable 3D rendering for diagnostic evaluation: Controllable neural rendering enables counterfactual evaluation that preserves visual content or paths while changing trajectory dynamics and rotation-translation relationships.Dyn-3D builds on posed ScanNet++ scenes and 3D Gaussian Splatting to make ego-motion an explicit evaluation factor.
  • Reasoning optimization for multimodal models: Kinematic-GSPO extends reasoning optimization with process supervision requiring fluent motion interpretations to remain physically consistent with camera trajectories.The method addresses the limitation that a plausible reasoning trace need not be physically valid.

3 Pilot Experiments

Pilot experiments show that increasing physical displacement exposes severe kinematic and spatial-reasoning failures, while counterfactual rotation-translation tests reveal systematic confusion between visual change and true motion.

  • Kinematic Collapse Under High Dynamics: 25.30 percentage points: Qwen3.5-9B overall accuracy falls from 63.97% to 38.67% as displacement increases from 0.1m to 5.0m.The evaluation contains 16,063 samples spanning spatial understanding, kinematic perception, and implicit trajectory reasoning.
  • Kinematic Collapse Under High Dynamics: Kinematic perception accuracy plunges from 69.71% to 9.66%, while implicit trajectory reasoning stays below the 25% chance level across all displacement magnitudes.The results indicate heavy reliance on continuous visual cues from smooth training data.
  • Visual-Kinematic Misalignment: Option-shuffled evaluation uses 3,340 balanced instances from 835 B17 motion-type pairs to avoid answer-position bias.These instances are derived from Dyn-3D rather than added as separate benchmark questions.
  • Visual-Kinematic Misalignment: Qwen3.5-9B scores 28.74% on pure rotation and 5.69% on pure translation, for a 17.22% decoupled average.Qwen3-VL-8B-Instruct and Qwen-VL-Max obtain decoupled averages of 13.70% and 19.84%, respectively.
  • Visual-Kinematic Misalignment: The displacement buckets partition the 16,063-question Dyn-3D benchmark, with accuracy reported in percent and 25.0% as the four-option chance level.The table evaluates Qwen3.5-9B under increasing inter-frame physical displacement ∆d.

4 Dataset

Dyn-3D is constructed from counterfactual 3D-rendered videos that decouple camera dynamics from visual appearance, enabling controlled evaluation of ego-motion and spatial understanding. Its metadata-derived, quality-controlled questions cover kinematic perception, spatial understanding, and implicit trajectory reasoning across balanced trajectory types.

  • Data generation: Dyn-3D generates counterfactual videos with identical spatial paths but varying dynamics to isolate visual variables and mitigate shortcut learning.The benchmark uses 3D Gaussian Splatting to decouple rotation, translation, and speed.
  • Data generation: The rendering pipeline uses 451 indoor ScanNet++ scenes, with each scene reconstructed from an average of 261 images.Scenes are rendered after fisheye undistortion, resolution standardization, and 3D Gaussian Splatting reconstruction.
  • Data generation: Each scene produces 30 FPS videos spanning smooth, fast, slow, rotation-only, and translation-only motion characteristics.The control group varies frame sampling rate along a smooth path, while additional trajectories decouple rotation and physical displacement.
  • Question design: Questions are deterministically generated from 3D metadata across 19 multiple-choice types covering kinematic, spatial, and implicit trajectory reasoning.Distractors use type-specific strategies such as range partitioning and sampled alternative directions.
  • Evaluation set: Three quality-control stages validate geometry, filter invalid or ambiguous samples, and manually review representative questions.Automated cross-validation marks all 16,063 aligned questions as geometry-recomputed and validation-passed.
  • Evaluation set: The benchmark contains 16,063 questions from 167 scenes and 835 videos, with 72.4% assessing spatial understanding and 18.8% assessing kinematic perception.Questions are balanced across five trajectory types, and answers are independently recomputed from structured 3D metadata.

5 Method

TempoVista combines motion-aware keyframe selection with Kinematic-GSPO policy optimization to ground video reasoning in camera kinematics and physical 3D annotations. The method allocates frames by trajectory variation and rewards reasoning traces that jointly satisfy answer, format, and motion-consistency requirements.

  • Framework overview: TempoVista consists of kinematic-adaptive frame selection and Kinematic-GSPO training for implicit kinematic perception.The framework uses explicit policy optimization rather than requiring VLMs to memorize 3D data.
  • Frame selection: Kinematic-adaptive selection allocates more frames to intervals with rapid camera-motion changes than to near-static intervals.Uniform sampling can underrepresent sharp turns or fast translations because it assumes information is linearly distributed over time.
  • Frame selection: SE(3) frame selection uses camera-to-world poses and a normalized geodesic-inspired distance balancing translation and rotation.The distance normalizes translation by a video-level constant and rotation to [0, 1], with α balancing the two terms.
  • Frame selection: Greedy FPS initializes with the first and last frames, then repeatedly selects the frame farthest from the selected set to preserve global trajectory coverage.This produces a geometric trajectory skeleton while preserving multi-object visibility.
  • Kinematic-GSPO: Kinematic-GSPO generates structured traces containing five camera-motion quantities alongside a final answer and applies physical ground-truth supervision.The quantities are total path length, displacement magnitude and vector, accumulated rotation angle, and average speed.
  • Kinematic-GSPO: The composite reward combines answer correctness, format compliance, and kinematic consistency, while group-relative advantages support clipped policy optimization.Kinematic consistency evaluates scalar motion ranges and both magnitude and direction for displacement vectors.

6 Experiments

Experiments evaluate TempoVista on Dyn-3D and VSI-Bench, compare multiple VLM baselines, and ablate its training and sampling strategies. TempoVista improves both base models on Dyn-3D, generally transfers to external spatial reasoning, and benefits from kinematic-aware optimization and oracle frame selection.

  • Experimental setup: Dyn-3D evaluation uses unseen scenes and counterfactual trajectories, while VSI-Bench tests relative distance, direction, appearance order, and route planning.The baselines include proprietary and open-source VLMs, and TempoVista is instantiated with InternVL-3.5-8B and Qwen3-VL-8B-Instruct.
  • Dyn-3D results: InternVL-3.5-8B + TempoVista improves Dyn-3D Overall accuracy from 50.6% to 57.1%, a gain of 6.5 percentage points.Its Kinematic, Spatial, and Trajectory accuracies also rise from 52.0%, 47.8%, and 70.4% to 64.5%, 52.7%, and 77.7%.
  • Dyn-3D results: Qwen3-VL-8B-Instruct + TempoVista improves Dyn-3D Overall accuracy from 50.3% to 61.2%, a gain of 10.9 points.This variant achieves the best open-source Spatial, Trajectory, and Overall results and surpasses Qwen-VL-Max by 11.4 points overall.
  • Transfer evaluation: TempoVista improves VSI-Bench overall accuracy from 51.3% to 55.6% for Qwen3-VL-8B-Instruct, while InternVL-3.5-8B rises from 50.1% to 50.7%.InternVL’s relative-direction accuracy decreases slightly from 48.6% to 48.1%, showing that transfer varies by model and task.
  • Training ablation: Kinematic-GSPO reaches 57.1% and 61.2% overall accuracy on the two base models, adding 1.1–1.6 points beyond Base GSPO.The results indicate that policy optimization provides most of the improvement, while the kinematic reward contributes an additional gain across both model families.
  • Sampling ablation: Oracle SE(3)-based sampling achieves 57.1% and 61.2%, exceeding uniform sampling by 1.5 points for both models.The flow-proxy variant underperforms uniform sampling by 0.5 and 0.7 points, respectively.

7 Conclusion

The paper identifies kinematic collapse as a failure of spatial cognition under nonsmooth trajectories and addresses it with counterfactual evaluation and explicit metric-motion optimization. TempoVista improves Dyn-3D performance and transfers to diverse VSI-Bench spatial tasks.

  • Conclusion: Kinematic collapse is a spatial cognition failure under nonsmooth trajectories caused by reliance on 2D visual shortcuts.The Dyn-3D benchmark uses counterfactual rendering to decouple visual changes from physical motion.
  • Conclusion: TempoVista combines kinematic-adaptive frame selection with Kinematic-GSPO, embedding metric motion constraints into policy optimization.The framework is intended to ground visual cues in 3D space.
  • Conclusion: Experiments show that TempoVista outperforms leading open and proprietary models on Dyn-3D and generalizes to diverse VSI-Bench spatial tasks.The conclusion states that explicit kinematic perception is essential for robust 3D spatial understanding.

Authors

The paper lists its authors and identifies equal contributions and project leadership.

  • Authors: Jiayu Ding, Zhuodong Liu, and Lei Zhang are listed as equal-contribution authors.The equal-contribution marker applies to these three authors.
  • Authors: Wenbo Xing is identified as the project leader and corresponding author.The dagger marker denotes this role.

A Dataset Construction Details

Dyn-3D constructs counterfactual multi-trajectory videos from filtered ScanNet++ scenes, reconstructed with 3D Gaussian Splatting and paired with metric camera and object metadata. The design separates visual variation from physical motion using shared paths, in-place rotation, and translation trajectories.

  • Scene Filtering: Scene filtering retains 451 of 499 raw ScanNet++ scenes with complete semantic annotations before rendering-quality inspection.The annotations provide object semantics, 3D bounding boxes, visibility, and object-level spatial relations.
  • Scene Filtering: Manual inspection removes scenes with divergent 3DGS training, broken trajectory renders, or insufficient frame quality.The second filtering stage evaluates reconstruction and video reliability across candidate scenes.
  • Reconstruction and Rendering: Each retained scene is undistorted, standardized to 1752×1168 resolution, converted to the required camera-pose format, and reconstructed with 3D Gaussian Splatting.Scenes retain an average of 261 valid images and are trained for 30,000 iterations on dual RTX 4090 GPUs.
  • Counterfactual Trajectory Design: Five 30-FPS trajectory types are rendered per scene, including fast, smooth, and slow videos sharing one physical path but differing in temporal sampling.The shared path is generated from clustered camera poses with cubic spline interpolation.
  • Counterfactual Trajectory Design: The rotation trajectory creates strong visual change with almost zero displacement, whereas the translation trajectory creates physical displacement with a relatively stable viewpoint.These diagnostic trajectories test whether models confuse pixel variation with metric motion.
  • 3D Metadata Extraction: Metadata extraction maps reconstructed scenes back to metric coordinates and records camera path length, displacement, rotation, positions, speed, and object-level properties.Recorded 3DGS transformations are used for the coordinate mapping.

B Benchmark Details

Dyn-3D organizes 19 deterministic multiple-choice question types across kinematic perception, object-level spatial understanding, and trajectory reasoning. Its diagnostic traps separate physically grounded motion interpretation from reliance on visual change.

  • Dyn-3D contains 19 multiple-choice question types spanning kinematic perception, spatial understanding, and implicit trajectory reasoning.
  • Kinematic perception: Kinematic tasks assess displacement, direction, rotation, speed, path length, and rotation-versus-translation discrimination.
  • Spatial understanding: Spatial tasks evaluate object directions, distance relations, nearest-object identification, coordinate re-anchoring, counting, size, visibility, and cross-frame direction.
  • Trajectory reasoning: Trajectory tasks require recognizing whether motion is rotational, translational, mixed, straight, curved, or circular.
  • Diagnostic traps: B4 and B5 test whether models distinguish strong visual changes from rotation and weak visual changes from true translation.
  • The benchmark separates physically grounded motion perception from superficial video recognition and probes Kinematic Collapse under inconsistent visual-motion cues.

C Method Details

The method combines five-field kinematic rewards with answer and formatting rewards during policy optimization. Implementation fixes an eight-frame input budget and structured reasoning output while evaluating scalar and vector motion predictions.

  • Kinematic reward: Kinematic-GSPO evaluates total path length, displacement magnitude, displacement vector, accumulated rotation angle, and average speed.
  • Reward design: The total reward combines answer correctness, XML-format compliance, and kinematic consistency, with γ = 0.1 and λ = 0.1.
  • Metric scoring: Scalar rewards use four bins, assigning +1 for exact matches, 0 for one-bin errors, and −1 for larger deviations or missing outputs.
  • Metric scoring: The displacement-vector reward averages magnitude-bin and direction scores based on cosine similarity thresholds.
  • Input budget: Each Dyn-3D video supplies k = 8 oracle-selected frames, while SFT and RL examples also contain eight frames.
  • Training: Training uses answer-only SFT before GSPO optimization, with LoRA adaptation and a frozen vision encoder.
  • Structured output: RL prompts require an XML reasoning block containing five motion quantities with units followed by an outer answer tag.

D Error Analysis

TempoVista improves Dyn-3D performance across both base models and all three evaluation dimensions, with transfer to VSI-Bench also positive overall. Remaining errors involve egocentric alignment, coordinate transformation, and fine-grained motion estimation.

  • Dyn-3D results: TempoVista improves both base models across kinematic perception, spatial understanding, and trajectory reasoning on Dyn-3D.Qwen3-VL rises from 51.0% to 62.6%, 47.6% to 58.3%, and 71.4% to 81.8%; InternVL rises from 52.0% to 64.5%, 47.8% to 52.7%, and 70.4% to 77.7%.
  • Egocentric alignment: Single-frame front-rear mistakes reflect egocentric depth-axis confusion, whereas cross-frame errors may also reflect incorrect coordinate-frame propagation.
  • Coordinate transformation: Coordinate re-anchoring remains difficult because object grounding, frame construction, and position transformation must be composed.
  • Metric motion estimation: Fine-grained motion errors persist under subtle or compound motion, including missed lateral displacement and overestimated speed.
  • Transfer results: On VSI-Bench, Qwen3-VL overall accuracy increases from 51.3% to 55.6%, while appearance-order reasoning gains 7.6 percentage points.
  • Transfer results: InternVL transfer is smaller, increasing overall accuracy from 50.1% to 50.7%, and varies by base model and spatial task.
  • Remaining limitations: The authors identify explicit coordinate re-anchoring, egocentric depth disambiguation, and fine-grained metric estimation as future improvement directions.
Loading 2609.01059v1…