PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficiently Reconstructing Dynamic Scenes One D4RT at a Time

Chuhan Zhang, Guillaume Le Moing, Skanda Koppula, Ignacio Rocco, Liliane Momeni, Junyu Xie, Shuyang Sun, Rahul Sukthankar, Joëlle K. Barral, Raia Hadsell, Zoubin Ghahramani, Andrew Zisserman, Junlin Zhang, Mehdi S. M. Sajjadi

arXiv:2512.08924v2cs.CV

TL;DR

Dynamic-scene reconstruction from video requires recovering geometry and motion across time, but existing approaches use fragmented components or dense decoding. D4RT uses a unified feedforward encoder-decoder with independently queried 3D points, achieving state-of-the-art results across diverse 4D reconstruction tasks.

  • Problem

    Dynamic scenes require joint reconstruction of geometry and motion, while existing approaches rely on fragmented components and may miss dynamic correspondences.

  • Method

    D4RT encodes a video into a global scene representation and independently queries a lightweight decoder for 3D point positions across space and time.

  • Results

    D4RT achieves state-of-the-art results across depth, point-cloud estimation, and 3D point-tracking tasks while outperforming existing approaches in speed and accuracy.

  • Takeaways & Limitations

    A unified query-based interface supports diverse 4D outputs and efficient dense dynamic correspondence reconstruction.

Abstract

from arXiv · show

Understanding and reconstructing the complex geometry and motion of dynamic scenes from video remains a formidable challenge in computer vision. This paper introduces D4RT, a simple yet powerful feedforward model designed to efficiently solve this task. D4RT utilizes a unified transformer architecture to jointly infer depth, spatio-temporal correspondence, and full camera parameters from a single video. Its core innovation is a novel querying mechanism that sidesteps the heavy computation of dense, per-frame decoding and the complexity of managing multiple, task-specific decoders. Our decoding interface allows the model to independently and flexibly probe the 3D position of any point in space and time. The result is a lightweight and highly scalable method that enables remarkably efficient training and inference. We demonstrate that our approach sets a new state of the art, outperforming previous methods across a wide spectrum of 4D reconstruction tasks. We refer to the project webpage for animated results: https://d4rt-paper.github.io/.

1. Introduction

Dynamic-scene reconstruction is poorly served by fragmented, task-specific pipelines that struggle with dynamic correspondences. D4RT instead uses a unified feedforward architecture with on-demand queries to support multiple 4D outputs through one interface.

  • 1. Introduction: Traditional reconstruction approaches are ill-suited to dynamic scenes, while prior systems often divide unified 4D understanding into task-specific components.Existing methods may combine separate models, specialized decoders, or costly refinement, and some cannot establish dynamic correspondences.
  • 1. Introduction: D4RT replaces fragmented frame-level decoding with efficient, on-demand querying of spatiotemporal points.The model encodes the video into a latent scene representation and independently decodes arbitrary point queries.
  • 1. Introduction: D4RT is designed for efficient training and inference while remaining flexible enough to decode any number of spatiotemporal point queries.Its latent scene representation is queried independently for requested points rather than decoded densely for every frame.
  • 1. Introduction: A single D4RT interface provides 4D correspondence, point clouds, depth maps, and camera parameters for static and dynamic scenes.The unified interface supports multiple reconstruction outputs without separate task-specific decoders.
  • 1. Introduction: Experiments report state-of-the-art dynamic 4D reconstruction and tracking while outperforming existing approaches in speed and accuracy.The contribution list also highlights dense, holistic tracking of all video pixels.

2. Method

D4RT encodes a video into a global scene representation and independently decodes spatiotemporal queries into 3D points. This unified interface supports diverse reconstruction, tracking, camera, and depth tasks while enabling efficient training and inference.

  • D4RT Framework: D4RT encodes the video into a global scene representation, then uses a lightweight decoder to query 3D points independently.The encoder captures dense correspondence and temporal information, while the decoder cross-attends to the fixed representation.
  • D4RT Framework: Independent queries disentangle source space, target time, and camera coordinates, enabling flexible sparse or dense decoding.Queries contain normalized source coordinates and temporal indices for the source, target, and reference camera frames.
  • D4RT Framework: Varying query indices yields point tracks, full point clouds, depth-related outputs, and camera parameters through one interface.Camera extrinsics are recovered from corresponding 3D points with Umeyama’s algorithm, while intrinsics use a pinhole model and median aggregation.
  • Model Architecture: The pointwise decoder embeds coordinates and timesteps, with a local 9×9 RGB patch substantially improving performance.Queries are decoded independently through cross-attention, avoiding query interactions and supporting trivial parallelism.
  • Efficient dense dynamic correspondence: Dense correspondence reconstruction avoids O(T^2HW) naive queries by starting tracks only from unvisited pixels, achieving a 5–15× adaptive speedup.An occupancy grid marks visible track pixels as visited, exploiting spatiotemporal redundancy.

3. Related Work

Prior reconstruction and tracking methods enforce geometry, process videos pairwise or in multiple stages, or use separate task-specific components. D4RT instead presents a unified architecture that supports holistic dynamic-scene reconstruction with strong speed and accuracy.

  • 3D Reconstruction: Classical SfM and MVS methods enforce geometric consistency but are computationally intensive and brittle.COLMAP-style pipelines incrementally estimate sparse geometry and camera poses.
  • Comparison: D4RT achieves 200+ FPS pose estimation, 9× faster than VGGT and 100× faster than MegaSaM, while delivering superior accuracy.Pose accuracy averages 1 − error over ATE/RTE/RPE on Sintel and ScanNet, with throughput measured on an A100 GPU.
  • Feedforward Reconstruction: Feedforward models extended transformer-based reconstruction to videos, but existing approaches often retain fragmented processing or limited dynamic correspondence.MegaSaM uses multiple models and test-time optimization, while VGGT uses separate specialized decoders.
  • Comparison: D4RT is reported as the only compared method that reconstructs a full 4D scene representation including all video pixels in dynamic examples.MegaSaM repeats moving entities, π3 fails on the flower, and SpatialTrackerV2 leaves occlusion gaps.
  • 2D to 3D Point Tracking: Tracking methods progressed toward dense 3D tracks, yet pairwise models prevent holistic video processing and multi-stage models rely on slow iterative refinement.L4P also uses different heads for sparse and dense outputs of the same nature.

4. Experiments

D4RT is evaluated across qualitative and quantitative 4D reconstruction, tracking, depth, point-cloud, camera-pose, and efficiency tasks. It achieves strong reconstruction and tracking results while substantially improving throughput over prior methods.

  • 4.2. 4D Reconstruction and Tracking: D4RT achieves state-of-the-art 3D tracking in local camera coordinates with and without known ground-truth intrinsics.Evaluation uses APD3D, Occlusion Accuracy, and 3D Average Jaccard on TAPVid-3D.
  • 4.2. 4D Reconstruction and Tracking: D4RT also excels at world-coordinate tracking, showing strong improvements across APD3D and L1 track distance.This setting evaluates tracks in one consistent world coordinate system and measures implicit reference-frame changes.
  • 4.2. 4D Reconstruction and Tracking: 18–300× faster than prior methods for 3D tracking throughput.The comparison measures the maximum number of full-video 3D point tracks at a target FPS on one A100 GPU.
  • 4.3. 3D Reconstruction: D4RT outperforms recent state-of-the-art models on point-cloud reconstruction across dynamic MPI Sintel and static ScanNet.The evaluation reports mean L1 distance after mean-shifting alignment.
  • 4.3. 3D Reconstruction: D4RT achieves top-tier depth performance across Sintel, ScanNet, KITTI, and Bonn, with particularly strong results on dynamic Sintel.Depth is evaluated under scale-only and scale-and-shift alignments.
  • 4.3. 3D Reconstruction: D4RT surpasses existing methods in camera-pose accuracy and efficiency, including two-orders-of-magnitude higher throughput than MegaSaM.The comparison uses adjusted baselines with unrelated decoding heads removed.

5. Conclusion

D4RT is a scalable feedforward network for reconstructing dynamic 4D scenes with temporal correspondence. Its query-based decoder supports efficient inference while achieving state-of-the-art results across depth, point-cloud, and 3D tracking tasks.

  • 5. Conclusion: D4RT reconstructs dynamic 4D scenes with temporal correspondence through a simple, scalable feedforward network.The method is presented as a unified framework for complex dynamic environments.
  • 5. Conclusion: Independent point queries avoid dense per-frame decoding bottlenecks and make inference scale linearly with the number of reconstructed points.The decoder predicts any point’s 3D position in space and time.
  • 5. Conclusion: D4RT achieves state-of-the-art results across depth, point-cloud estimation, and 3D point tracking.The conclusion frames these results as evidence that dynamic-scene scaling need not sacrifice precision.

A. Model Overview & Training Details

D4RT combines a unified model overview with auxiliary losses, confidence weighting, and extensive video augmentations. Training queries emphasize challenging regions and selected temporal relationships.

  • Model Overview & Training Details: D4RT training combines 3D, 2D, visibility, displacement, confidence, and normal losses with specified weighting and confidence-based 3D error scaling.The loss uses confidence c to weight 3D point error, alongside the listed auxiliary objectives.
  • Model Overview & Training Details: Training applies temporally consistent color jitter, random color drop, Gaussian blur, and random crops to improve video generalization.Color drop has probability 0.2, Gaussian blur probability 0.4, and crop scale ranges from 0.3 to 1.0 of the original size.
  • Model Overview & Training Details: 30% of training queries target depth discontinuities or motion boundaries, while ttgt = tcam is enforced with probability 0.4.Challenging regions are identified using a Sobel filter on depth maps.

B. Generalization to Long Videos

D4RT processes long videos by stitching overlapping segment predictions with confidence-based Sim(3) alignment. On KITTI sequences, it performs better than VGG-T consistently and better than π3 on sequence “00”.

  • Generalization to Long Videos: Long-sequence processing partitions videos into overlapping segments and aligns them using Sim(3) transformations estimated from the highest-confidence overlapping points.The method uses the top 85% of points by confidence and omits loop detection and global optimization.
  • Generalization to Long Videos: D4RT consistently outperforms VGG-T on long KITTI sequences and significantly outperforms π3 on sequence “00”.The comparison uses predictions from overlapping chunks aligned and stitched with the Umeyama algorithm.

C. High-Resolution Decoding with Subpixel Precision

D4RT decouples global scene encoding from continuous point-wise decoding, enabling arbitrary-resolution queries. High-resolution local RGB patches improve recovery of edges and fine details without increasing overall computational or memory requirements.

  • High-Resolution Decoding with Subpixel Precision: Continuous normalized query coordinates let D4RT decode arbitrary resolutions independently of the Global Scene Representation F.This decouples global scene encoding from point-wise decoding.
  • High-Resolution Decoding with Subpixel Precision: Config 4 extracts local RGB patches at original source-frame resolution, enabling finer detail recovery beyond the encoder’s native output resolution.The progression from native-resolution output to local patches and continuous queries is evaluated across four configurations.
  • High-Resolution Decoding with Subpixel Precision: Table 10 evaluates query density and patch fidelity using PDBE and standard depth metrics, with high-resolution RGB patches producing sharper depth edges.PDBE measures preservation of high-frequency depth details.
  • High-Resolution Decoding with Subpixel Precision: Config 4 achieves the highest visual fidelity, preserving sharp edges and recovering fine details such as hair without increasing overall computation or memory.Figure 9 highlights improved recovery of hair strands and object boundaries.

D. Further Ablations

Further ablations show that pretrained initialization improves performance broadly, while local RGB patch size and encoder patch size affect depth and camera-pose accuracy. The strongest reported local RGB patch setting is 9×9.

  • Further Ablations: Initializing the video encoder with pretrained VideoMAE weights yields significant improvements across the evaluated tasks.The comparison is against random-weight initialization.
  • Further Ablations: A 9×9 local RGB patch provides the best overall performance across camera pose and depth estimation tasks.The result comes from the local RGB patch-size ablation.
  • Further Ablations: An RGB patch size between 9 and 12 gives the best depth and pose estimation performance on Sintel.The figure reports average error across the two tasks.

E. Further Qualitative Results

Additional visualizations show D4RT reconstructing static and dynamic scenes, while comparisons highlight its complete 4D representation and detailed dense depth estimation under motion blur.

  • The section adds reconstruction examples and baseline comparisons through Figures 11–13.
  • D4RT reconstructs accurate static environments and dynamic sequences in additional visualizations.
  • D4RT reconstructs a complete 4D scene representation by tracking all pixels throughout the video, unlike methods visualized as point clouds or sparse tracks.
  • D4RT produces dense depth estimates with finer details than current state-of-the-art methods while preserving geometric accuracy under large motion blur.
Loading 2512.08924v2…