Source-linked AI summary
Driv3R: Learning Dense 4D Reconstruction for Autonomous Driving
Xin Fei, Wenzhao Zheng, Yueqi Duan, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, Jiwen Lu
TL;DR
Dense dynamic-scene reconstruction is challenging for autonomous-driving perception, especially when camera-based depth and moving objects must be represented accurately. Driv3R uses temporal-spatial memory, flow-guided dynamic-region modeling, and optimization-free alignment to produce world-coordinate point maps. On nuScenes, it outperforms existing methods in depth estimation and scene reconstruction, with 15× higher inference speed than methods relying on global alignment.
Problem
Camera-only dense reconstruction of dynamic scenes requires accurate depth without 3D ground-truth supervision and precise representations of moving objects.
Source: §1 · p. 1Method
Results
15× higher inference speed than methods relying on global alignment, with Driv3R outperforming existing methods in depth estimation and scene reconstruction on nuScenes.
Source: §5 · p. 8Takeaways & Limitations
Driv3R provides a dense 4D reconstruction approach for large-scale dynamic autonomous-driving scenes without global alignment optimization.
Source: §5 · p. 8Point predictions retain edge floaters and some fast-moving-object and background failures, associated with R3D3 [35] prediction accuracy and the lack of high-quality continuous sequences rich in fast-moving objects.
Source: C. Additional Discussion and Future… · p. 9
Abstract
from arXiv · showhide
Realtime 4D reconstruction for dynamic scenes remains a crucial challenge for autonomous driving perception. Most existing methods rely on depth estimation through self-supervision or multi-modality sensor fusion. In this paper, we propose Driv3R, a DUSt3R-based framework that directly regresses per-frame point maps from multi-view image sequences. To achieve streaming dense reconstruction, we maintain a memory pool to reason both spatial relationships across sensors and dynamic temporal contexts to enhance multi-view 3D consistency and temporal integration. Furthermore, we employ a 4D flow predictor to identify moving objects within the scene to direct our network focus more on reconstructing these dynamic regions. Finally, we align all per-frame pointmaps consistently to the world coordinate system in an optimization-free manner. We conduct extensive experiments on the large-scale nuScenes dataset to evaluate the effectiveness of our method. Driv3R outperforms previous frameworks in 4D dynamic scene reconstruction, achieving 15x faster inference speed compared to methods requiring global alignment. Code: https://github.com/Barrybarry-Smith/Driv3R.
1. Introduction
Dense, real-time reconstruction of dynamic scenes matters for autonomous driving, but camera-only reconstruction must contend with limited depth supervision and dynamic-object representation. Driv3R combines temporal-spatial memory, dynamic-object guidance, and optimization-free alignment, and reports stronger 4D reconstruction with faster inference than methods requiring global alignment.
1. Introduction
- Camera-only dense reconstruction is computationally efficient and low-cost, but accurate depth without 3D ground-truth supervision and precise dynamic-object representation remains challenging.Source: §1 · p. 1
- DUSt3R [43] and MonST3R [52] use point-map representations, but global alignment is costly, while Spann3R [41] spatial memory does not adequately fuse temporal information for dynamic driving scenes.Source: §1 · p. 2
- Driv3R combines temporal-spatial memory with a RAFT-based 4D flow predictor and R3D3 [35] supervision on moving regions, then aligns point maps to world coordinates without optimization.Source: §1 · p. 2
- 15× faster inference speed than methods relying on global alignment accompanies Driv3R’s reported gains in 4D dynamic-scene reconstruction.The paper also reports comparable results to state-of-the-art multi-view depth estimation frameworks on nuScenes.Source: §1 · p. 2
2. Related Work
Related work spans camera-based depth estimation, static 3D representations, optical flow, and dynamic-scene reconstruction. The paper positions Driv3R against methods that lack efficient, consistent reconstruction of dynamic driving scenes.
Depth Estimation For Autonomous Driving
Static 3D Reconstruction
Dynamic 4D Reconstruction
Optical Flow
- RAFT [38] is used as the 4D flow predictor’s core component, and Driv3R targets real-time autonomous-driving reconstruction without optimization.Source: §2 · p. 3
3. Proposed Approach
Driv3R regresses dense point maps from multi-view sequences and uses temporal-spatial memory, dynamic-object masks, and multi-view alignment to produce world-coordinate 4D reconstructions. Its training combines temporal and cross-view interactions while addressing memory limits.
3.1. Problem Formulation
- Given multi-view images and camera poses across timestamps, Driv3R learns pixel-wise point clouds in global coordinates, using R3D3 [35] depth inference as supervision.Source: §3.1 · p. 3
3.2. Temporal-Spatial Memory Pool
- A sensor-specific memory stores timestamp-labeled key-value pairs and uses cosine similarity to select relevant spatial and temporal matches, reducing unnecessary attention computation and interference from irrelevant frames.The memory supports feature interaction across prior timestamps and viewpoints.Source: §3.2 · pp. 3–4, Figure 2 · p. 3
3.3. 4D Flow Predictor
- The RAFT-based 4D flow predictor combines adjacent-frame optical flow with motion induced by sensor movement to form coarse dynamic masks, then uses SAM2 segmentation to fill missing object regions.Source: §3.3 · p. 4
3.4. Multi-view Aligner
- The multi-view aligner estimates camera parameters, projects point maps to depth maps, then uses ground-truth camera parameters to unproject them into world coordinates without additional alignment.Source: §3.4 · pp. 4–5
3.5. Training of Driv3R
- Training uses R3D3 [35] depth supervision and, after warm-up, calculates confidence-aware loss only on points in predicted dynamic regions; two stages separate single-sensor temporal learning from cross-sensor spatial learning to address memory limits.Source: §3.5 · p. 5
4. Experiments
Experiments on nuScenes evaluate Driv3R for depth estimation and dense 4D reconstruction, including dynamic scenes. Driv3R performs strongly on dynamic-scene reconstruction while avoiding global alignment optimization.
4.1. Datasets; 4.2. Implementation Details
- NuScenes provides 1,000 driving scenes split into 700 training, 150 validation, and 150 test scenes; experiments use consecutive five-frame sequences and camera inputs.Images are divided into virtual cameras at 224x224 resolution; training proceeds in two stages for 30 and 20 epochs.Source: §4.1 · p. 5, §4.2 · p. 5
4.3. Evaluation Metrics
- Reconstruction is evaluated by accuracy, completion, and normal consistency; depth uses Abs Rel, Sq Rel, RMSE, and three δ-threshold metrics.Point maps and LiDAR clouds are transformed to global coordinates for reconstruction evaluation and sensor coordinates for depth evaluation.Source: §4.3 · p. 5
4.4. Results and Analysis
- Driv3R achieves competitive nuScenes depth-estimation results against multi-view frameworks and methods that regress per-frame pointmaps.Visualizations describe less blur in fast-moving scenes than Spann3R, with R3D3 mitigating the blur.Source: §4.4 · p. 5, Table 1 · p. 6, Figure 4 · p. 7
- On 3,508 dynamic nuScenes validation sequences, Driv3R outperforms previous methods in reconstruction and depth estimation, with slightly better reconstruction than MonST3R.The comparison evaluates sequences selected according to the 4D flow predictor's dynamic masks.Source: §4.4 · p. 6, Table 2 · p. 6
- Driv3R surpasses DUSt3R and MonST3R in inference efficiency, while its efficiency is comparable to Spann3R and its additional memory storage raises inference memory use.The efficiency evaluation includes global-alignment time for DUSt3R and MonST3R.Source: §4.4 · pp. 6–7, Table 3 · p. 8
- Ablations show temporal interactions improve reconstruction over the vanilla architecture, spatial interactions further boost performance, and dynamic masks mitigate inaccurate R3D3 predictions.The combined design leverages DUSt3R priors for static scenes and R3D3 predictions for moving objects.Source: §4.4 · p. 8, Table 4 · p. 8
5. Conclusion
Driv3R combines temporal-spatial memory, moving-object guidance, and optimization-free alignment to produce consistent global-coordinate 4D point maps. On nuScenes it outperforms existing methods, while memory use and supervision quality remain limitations.
5. Conclusion
- On nuScenes, Driv3R outperforms existing methods in depth estimation and scene reconstruction, with 15× higher inference speed than methods relying on global alignment.The approach uses an optimization-free multi-view aligner to generate consistent 4D point maps in global coordinates.Source: §5 · p. 8
- A temporal-spatial memory pool reasons across sequences and viewpoints, while a 4D Flow Predictor identifies moving objects to guide attention toward dynamic regions.An optimization-free multi-view aligner produces consistent 4D point maps in the global coordinate system.Source: §5 · p. 8
- Input length is constrained by memory; a five-frame sequence from six cameras requires about 10 GB during training, mainly for memory-pool storage.The authors identify improving memory storage as future work.Source: §5 · p. 8
- Sparse LiDAR supervision for dynamic objects does not yield optimal results, and predictions from the pretrained R3D3 model can sometimes be inaccurate.The authors propose adapting training to be fully self-supervised as future work.Source: §5 · p. 8
A.1. Training Configuration
Driv3R uses two training stages, moving from temporal-only sequences to multi-sensor temporal sequences. Baselines not trained on nuScenes receive the same training configuration for comparison.
A.1. Training Configuration
- Driv3R is trained for 30 epochs on temporal sequences, then fine-tuned for 20 more epochs at a smaller learning rate.The two-stage configuration is for the large-scale nuScenes dataset.Source: §A.1 · p. 9, Table 5 · p. 9
- The second training stage uses temporal sequences from multiple sensors so the memory pool can further refine its reasoning about spatial contexts.This stage follows the temporal-only training stage.Source: §A.1 · p. 9
- Baselines not previously trained on nuScenes are fine-tuned with the same training configuration for a fair comparison.Source: §A.1 · p. 9
A.2. Training Loss
Training combines confidence loss and scale loss. The confidence loss follows DUSt3R, while the scale loss encourages dynamic-object predictions to remain within a narrower range than R3D3 predictions.
A.2. Training Loss
- The training objective combines confidence loss with scale loss.The confidence loss follows DUSt3R [43].Source: §A.2 · p. 9
- The dynamic mask M_t,c is generated by the 4D Flow Predictor, and α controls the confidence-score range at 0.5 during training.Source: §A.2 · p. 9
- Scale loss encourages dynamic-object point predictions to stay within a narrower range than the corresponding R3D3 predictions, helping prevent unstable predictions for fast-moving objects.Source: §A.2 · p. 9
A.3. Model Architecture
Driv3R maintains chronological key-value memories separately for each sensor. During spatial training, cross-attention uses nearby overlapping frames, while working and long-term memories manage the retained context.
A.3. Model Architecture
- Driv3R keeps a separate memory pool for each sensor, storing key-value pairs in chronological timestamp order.During the first training stage, each input sequence updates features using only its corresponding sensor pool.Source: §A.3 · p. 9
- During spatial training, cross-attention updates the current frame using only the four previous timestamps that overlap its field of view.The four timestamps are selected as the most closely related frames to address training-memory limitations.Source: §A.3 · p. 9
- Working memory retains five recent frames in the first stage and five most-similar frames in the second; long-term memory prunes frames by accumulated attention weights.Pruning prevents memory from exceeding its limit during large-scale training and inference.Source: §A.3 · p. 9
B. Additional Results and Visualization
On nuScenes, Driv3R combines dynamic-object prediction with static-background modeling, achieving comparable depth prediction and faster inference than MonST3R. Its visualizations also show stronger fast-moving-object representations and temporal and spatial consistency.
B. Additional Results and Visualization
- 15x faster inference than MonST3R [52] with comparable depth prediction on nuScenes.Driv3R also outperforms MonST3R in several scenarios, attributed to robustness to inaccurate dynamic masks.Source: B. Additional Results and Visualization · p. 9
- Driv3R integrates R3D3's [35] dynamic-object prediction strengths with Spann3R's [41] static-background modeling strengths.Source: B. Additional Results and Visualization · p. 9
- Driv3R outperforms MonST3R [52] in several scenarios because it is robust to inaccurate dynamic masks.Source: B. Additional Results and Visualization · p. 9
- Figure 7 visualizations show better representations of fast-moving objects and temporal and spatial consistency in local and global 4D reconstruction.Source: B. Additional Results and Visualization · p. 9
C. Additional Discussion and Future works
Driv3R's point predictions retain edge artifacts and failures involving fast-moving objects and background. The discussion identifies point-prediction accuracy and limited dynamic-rich continuous data as constraints, and proposes self-supervision and improved data strategies for future work.
C. Additional Discussion and Future works
- Driv3R's point predictions still show floating edge artifacts and failure cases involving fast-moving objects and background.The authors attribute these limitations primarily to R3D3 [35] point-prediction accuracy constraints and a lack of high-quality continuous sequences with many dynamic, fast-moving objects.Source: C. Additional Discussion and Future… · p. 9
- Future work could adapt Driv3R to efficient self-supervision using reprojection and geometry-consistency losses.Source: C. Additional Discussion and Future… · p. 9
- The authors propose more efficient data sampling than temporal-order sequence splitting and joint training across multiple large-scale datasets, including non-driving data.Source: C. Additional Discussion and Future… · p. 9
- In some scenarios, MonST3R [52] has more refined depth prediction, while Driv3R remains comparable and 15x faster.Source: Figure 6 · p. 10