Source-linked AI summary
V-DPM: 4D Video Reconstruction with Dynamic Point Maps
Edgar Sucar, Eldar Insafutdinov, Zihang Lai, Andrea Vedaldi
TL;DR
Existing Dynamic Point Maps are limited to image pairs and require optimization for additional views, motivating a video formulation. V-DPM extends DPMs on top of pretrained VGGT, and the resulting model achieves strong 3D and 4D reconstruction while recovering full 3D motion.
Problem
Dynamic Point Maps capture dynamic 3D information but remain limited to image pairs, requiring optimization when more than two views are processed.
Method
V-DPM extends DPMs to video using time-varying backbone predictions and decoders that produce viewpoint- and time-invariant point maps at selected reference times.
Results
V-DPM achieves state-of-the-art dynamic-scene reconstruction, with more than half the error rate of analogous feed-forward reconstructors on standard benchmarks.
Takeaways & Limitations
A static VGGT reconstruction network can be fine-tuned into a one-shot 4D video reconstructor using relatively little training data and modest compute.
Takeaways & Limitations
The evaluation scale is constrained by available resources, and V-DPM uses shorter fine-tuning snippets while longer sequences require sliding-window fusion and optimization.
Abstract
from arXiv · showhide
Powerful 3D representations such as DUSt3R invariant point maps, which encode 3D shape and camera parameters, have significantly advanced feed forward 3D reconstruction. While point maps assume static scenes, Dynamic Point Maps (DPMs) extend this concept to dynamic 3D content by additionally representing scene motion. However, existing DPMs are limited to image pairs and, like DUSt3R, require post processing via optimization when more than two views are involved. We argue that DPMs are more useful when applied to videos and introduce V-DPM to demonstrate this. First, we show how to formulate DPMs for video input in a way that maximizes representational power, facilitates neural prediction, and enables reuse of pretrained models. Second, we implement these ideas on top of VGGT, a recent and powerful 3D reconstructor. Although VGGT was trained on static scenes, we show that a modest amount of synthetic data is sufficient to adapt it into an effective V-DPM predictor. Our approach achieves state of the art performance in 3D and 4D reconstruction for dynamic scenes. In particular, unlike recent dynamic extensions of VGGT such as P3, DPMs recover not only dynamic depth but also the full 3D motion of every point in the scene.
1. Introduction
V-DPM extends Dynamic Point Maps from image pairs to video, enabling feed-forward reconstruction of dynamic 3D scenes while reusing static reconstruction architectures. Built on VGGT, it achieves strong 4D reconstruction after fine-tuning with limited dynamic data.
- Dynamic Point Maps represent 3D shape, 3D motion, camera intrinsics, and camera motion in a viewpoint- and time-invariant form.
- Existing dynamic point-map methods remain pairwise, so processing more than two images requires optimization-based post-processing.
- V-DPM predicts time-varying point maps per input image, then decodes viewpoint- and time-invariant maps at a chosen reference viewpoint and time.
- Varying the reference time reconstructs the scene at different instants and recovers scene flow by pooling information across input images.
- V-DPM preserves the static backbone architecture, allowing a pretrained VGGT model to be fine-tuned for dynamic reconstruction instead of trained from scratch.The design reduces training cost and the need for 4D annotated data.
- The approach more than halves the error rate versus analogous feed-forward reconstructors such as DPM, MonST3R, and St4rTrack on standard benchmarks.
2. Related Work
Related work progresses from pairwise static point-map reconstruction toward multi-view and dynamic feed-forward models. Dynamic systems vary in whether they intrinsically recover motion or require auxiliary tracking and optimization.
- Feed-forward static reconstruction: DUSt3R and MASt3R estimate 3D shape and camera parameters from image pairs but require test-time optimization to fuse additional views.
- Feed-forward static reconstruction: MV-DUST3R, Fast3R, Flare, MapAnything, and VGGT extend point-map reconstruction to multiple views, with VGGT improving feed-forward performance over optimization-dependent methods.
- Feed-forward dynamic reconstruction: MonST3R extends DUSt3R to dynamic reconstruction but requires a 2D tracker to recover 4D motion intrinsically.
- Feed-forward dynamic reconstruction: DPMs provide viewpoint- and time-invariant representations, while other models recover dynamic depth without scene motion unless augmented with auxiliary components.
- Other dynamic reconstruction approaches: MegaSAM combines feed-forward depth prediction with optimization-based non-rigid reconstruction for monocular dynamic 3D reconstruction.
3. Method
V-DPM extends Dynamic Point Maps from image pairs to multi-view video by predicting time-varying and time-invariant point maps in a feed-forward architecture. Its design reuses static-scene reconstruction models while enabling reconstruction at arbitrary reference timestamps.
- Multi-view DPMs: V-DPM extends Dynamic Point Maps to reconstruct dynamic 3D scenes from several images or a video in a feed-forward pass.The method is designed as a multi-view extension of DPMs rather than a pairwise system requiring optimization-based fusion.
- Dynamic Point Maps: The representation associates each image pixel with a 3D point expressed relative to a chosen viewpoint and evaluated at a specified timestamp.The image index identifies the corresponding pixels, while the viewpoint and timestamp can differ from those of the input image.
- Multi-view DPMs: Using a common viewpoint removes redundant viewpoint-indexed maps, reducing the representation from N 3 possible maps to N 2.Maps differing only by viewpoint are related by rigid transformations, so the remaining maps can be inferred after camera recovery.
- Multi-view DPMs: The backbone predicts N time-varying point maps, while decoders produce N time-invariant maps at a shared reference timestamp, yielding 2N −1 maps in one pass.The time-varying maps correspond to input timestamps; the additional maps share timestamp tj and viewpoint π0.
- Multi-view DPMs: Re-running only the decoder for different tj yields scene versions at different timestamps while reusing the backbone and most earlier computations.This design supports reconstructing the full 3D scene at a selected timestamp without recomputing the entire network.
- Implementation: V-DPM builds on pretrained VGGT, fine-tuning its static reconstruction backbone and adding a time-conditioned transformer decoder for shared-timestamp maps.The decoder uses frame and global attention and conditions its transformer blocks on the target-time token through adaptive LayerNorm.
4. Experiments
The experiments evaluate V-DPM on dynamic 3D reconstruction, dense tracking, video depth, and camera pose. Across these tasks, V-DPM maintains strong video-level tracking and generally competitive reconstruction, with π3 remaining stronger on some static and pose metrics.
- V-DPM is evaluated on dynamic 3D reconstruction, dynamic depth prediction, and camera pose estimation.
- 4.1. 4D Reconstruction: V-DPM maintains performance similar to two-view reconstruction when tracking dense 3D points across 10-frame snippets.The comparison attributes this to reasoning about temporal dynamics over the whole video snippet.
- 4.1. 4D Reconstruction: V-DPM produces smoother, more coherent motion trajectories and avoids failure cases observed for DPM and St4RTrack in qualitative 10-frame reconstructions.On the DAVIS examples, only V-DPM plausibly reconstructs the tennis player’s end-frame body pose.
- 4.2. Video Depth and Camera Pose: V-DPM outperforms prior methods by a substantial margin on video-depth estimation except π3, which benefits from larger-scale training and a stronger backbone.The benchmark evaluates time-variant point-map reconstruction on Sintel and Bonn.
- 4.2. Video Depth and Camera Pose: V-DPM demonstrates competitive camera-pose performance and is only outperformed by π3, which also exceeds the VGGT backbone.The reported pose metrics are ATE, RPE trans, and RPE rot.
5. Conclusions
V-DPM extends Dynamic Point Maps to one-shot 4D reconstruction from multiframe monocular videos and adapts static VGGT using modest compute and synthetic data. It predicts time- and viewpoint-invariant point clouds that support motion recovery and deformation-aware fusion, while evaluation scale remains resource-constrained.
- V-DPM supports one-shot 4D reconstruction from multiframe monocular videos.
- V-DPM is built by extending static VGGT with modest compute and synthetic data.
- The model predicts time- and viewpoint-invariant 3D point clouds for each image.
- These point clouds support dense point-motion tracking and fusion across images captured at different times to undo scene deformations.
- The evaluation scale is constrained by available resources.
6. Training details
Training uses randomly sampled frame windows, mixed static and dynamic data, and a dynamic point-map loss designed to prevent dense static annotations from dominating sparse dynamic supervision.
- Training samples 5-, 9-, 13-, or 19-frame windows, using the central frame as the VGGT reference view.
- The final model is trained for 60 epochs on 16 GH200 GPUs with AdamW and cosine learning-rate decay.The base learning rate is 1.5 × 10^-4.
- The dataset mixture combines static ScanNet++ and BlendedMVS data with dynamic Kubric-F, Kubric-G, PointOdyssey, and Waymo data.
- The reconstruction loss is defined per pixel and frame, addressing the risk that numerous static annotations overwhelm sparse dynamic point tracks.
7. Network design ablation
The ablation compares four network-design variants and finds that the complete model’s design elements are important for dynamic point-map reconstruction.
- Four variants are compared: Original, Decoder depth 2, Addition conditioning, and DPT decoder.The Original model uses four transformer blocks for decoding time-invariant point maps; Decoder depth 2 reduces this to two.
- Addition conditioning replaces adaLN time conditioning with a time token added to the input tokens.
- DPT decoder removes extra transformer layers for time-invariant decoding and instead conditions a copied DPT head directly through adaLN.
- The variants are evaluated on two views from Kubric-G with a margin of 8, and the results verify each design element’s importance for full model performance.