Source-linked AI summary
DepthSplat: Connecting Gaussian Splatting and Depth
Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, Marc Pollefeys
TL;DR
DepthSplat addresses limitations that affect Gaussian splatting and depth estimation in challenging scenes by connecting the two tasks. It integrates pre-trained monocular depth features into multi-view depth estimation and uses differentiable Gaussian splatting for unsupervised depth pre-training, achieving state-of-the-art depth and view-synthesis results across three datasets.
Problem
Gaussian splatting and depth estimation each face limitations in challenging regions such as occlusions, texture-less areas, and reflective surfaces, motivating their integration.
Method
DepthSplat augments multi-view cost volumes with pre-trained monocular depth features and uses Gaussian splatting rendering loss for unsupervised depth pre-training.
Results
DepthSplat achieves state-of-the-art performance on ScanNet, RealEstate10K, and DL3DV for both depth estimation and novel view synthesis.
Takeaways & Limitations
Gaussian splatting and depth estimation provide mutual benefits, enabling more robust multi-view depth models and improved novel view synthesis.
Takeaways & Limitations
The model requires camera poses and predicts pixel-aligned Gaussians, which can challenge extremely sparse inputs and scalability to many views.
Abstract
from arXiv · showhide
Gaussian splatting and single-view depth estimation are typically studied in isolation. In this paper, we present DepthSplat to connect Gaussian splatting and depth estimation and study their interactions. More specifically, we first contribute a robust multi-view depth model by leveraging pre-trained monocular depth features, leading to high-quality feed-forward 3D Gaussian splatting reconstructions. We also show that Gaussian splatting can serve as an unsupervised pre-training objective for learning powerful depth models from large-scale multi-view posed datasets. We validate the synergy between Gaussian splatting and depth estimation through extensive ablation and cross-task transfer experiments. Our DepthSplat achieves state-of-the-art performance on ScanNet, RealEstate10K and DL3DV datasets in terms of both depth estimation and novel view synthesis, demonstrating the mutual benefits of connecting both tasks. In addition, DepthSplat enables feed-forward reconstruction from 12 input views (512x960 resolutions) in 0.6 seconds.
1. Introduction
DepthSplat connects feed-forward Gaussian splatting with monocular and multi-view depth estimation to address their complementary limitations. It integrates monocular features into multi-view depth prediction, supports Gaussian reconstruction and unsupervised depth pre-training, and reports strong results across datasets and tasks.
- Novel view synthesis and depth prediction are fundamental computer-vision tasks supporting applications including augmented reality, robotics, and autonomous driving.
- Feature-matching-based Gaussian splatting and multi-view depth estimation struggle with occlusions, texture-less regions, and reflective surfaces, while monocular depths typically lack consistent multi-view scale.
- DepthSplat connects sparse-view feed-forward 3DGS and robust monocular depth estimation to improve both tasks through cross-task transfer.
- DepthSplat integrates pre-trained monocular depth features into a multi-view feature-matching branch, improving robustness while preserving multi-view consistency.The predicted depth maps provide Gaussian centers, while an additional lightweight network predicts the remaining Gaussian parameters for novel-view synthesis.
- Photometric rendering loss enables unsupervised depth pre-training on large-scale posed multi-view datasets, after which the depth model can be fine-tuned for specific tasks.The paper reports that this pre-training improves performance over training from scratch and enhances Gaussian-splatting view synthesis.
- 0.6 seconds enables feed-forward reconstruction from 12 input views at 512 × 960 resolution, while experiments report state-of-the-art results across multiple datasets and tasks.The reported evaluations include TartanAir, ScanNet, RealEstate10K, and DL3DV.
2. Related Work
Prior work developed multi-view depth estimation and monocular depth estimation separately, while DepthSplat’s shared architecture connects depth estimation with Gaussian splatting.
- Multi-View Depth Estimation: Multi-view depth estimation uses photometric consistency across images for feature matching and reference-image depth prediction.
- Multi-View Depth Estimation: Learning-based multi-view methods improve depth quality but remain challenged by occlusions, low-textured areas, and non-Lambertian surfaces.
- Monocular Depth Estimation: Monocular depth methods can produce accurate predictions on diverse in-the-wild data, but their depths typically lack consistent scale across views.
- DepthSplat: DepthSplat’s shared architecture combines multi-view cost volumes with monocular features and connects depth estimation to Gaussian splatting.
3. DepthSplat
DepthSplat combines multi-view feature matching with pre-trained monocular depth features for robust depth prediction and feed-forward Gaussian reconstruction. It unprojects predicted depths into Gaussian centers and predicts remaining Gaussian parameters for differentiable novel-view rendering.
- Multi-view feature extraction: A weight-sharing ResNet and multi-view Swin Transformer produce multi-view-aware features, with cross-attention restricted to each reference view’s top-2 camera-nearest neighbors when N > 2.This design makes computation tractable with many input views.
- Feature matching: Plane-sweep stereo warps neighboring-view features across sampled depth candidates and stacks dot-product correlations into per-view cost volumes.For more than two views, correlations use the top-2 nearest views and are averaged for a speed-accuracy trade-off.
- Monocular feature augmentation: DepthSplat combines cost-volume feature matching with pre-trained monocular features to improve depth prediction in difficult matching conditions.The targeted conditions include occlusions, texture-less regions, and reflective surfaces.
- Feature fusion and depth regression: Concatenated cost volumes and monocular features are processed by a 2D U-Net and softmax-weighted depth regression, followed by hierarchical higher-resolution refinement.The refinement searches a smaller depth range around an upsampled coarse prediction.
- Gaussian splatting and training: Per-pixel depths are unprojected with camera parameters into Gaussian centers, while a DPT head predicts opacity, covariance, and color for novel-view rendering.The model trains depth with inverse-depth L1 and gradient losses, and trains view synthesis with MSE plus LPIPS rendering losses.
4. Experiments
DepthSplat’s experiments show that monocular features and multi-view matching jointly improve depth estimation and Gaussian-splatting view synthesis. Across datasets, ablations, transfer tests, and varying input views, the method achieves strong performance while retaining fast feed-forward reconstruction.
- Model Variants: Larger monocular backbones and 2-scale hierarchical matching consistently improve both depth estimation and novel view synthesis.The model variants compare ViT-S, ViT-B, and ViT-L monocular backbones with 1-scale and 2-scale multi-view models.
- Ablation and Analysis: Monocular features improve depth in texture-less and reflective regions and reduce novel-view misalignment where multi-view correspondences are unavailable.The improvement is illustrated on ScanNet depth predictions and RealEstate10K Gaussian-splatting renderings.
- Ablation and Analysis: Removing either the monocular feature branch or cost volume causes large performance drops, showing that the two branches are complementary.A pure monocular backbone struggles to produce scale- and multi-view-consistent predictions, while multi-view matching alone fails in difficult correspondence regions.
- Ablation and Analysis: Simple concatenation of monocular features performs better than single-branch, depth-map-fusion, and attention-based alternatives in the reported comparisons.The two-branch design separates feature matching from monocular priors, while concatenation remains simpler than adaptive fusion.
- Unsupervised Depth Pre-Training with Gaussian Splatting: Gaussian-splatting pre-training on posed RealEstate10K improves subsequent depth predictions after fine-tuning, especially on challenging TartanAir and KITTI datasets.The pre-training uses only Gaussian-splatting rendering loss, without direct depth supervision.
- Benchmark Comparisons: DepthSplat achieves state-of-the-art depth and view-synthesis results on ScanNet and RealEstate10K and consistently outperforms MVSplat on DL3DV and cross-dataset tests.Its lightweight local feature matching also scales more efficiently to additional input views than MVSplat’s global pair-wise matching.
- Benchmark Comparisons: DepthSplat reconstructs larger-scale or 360° scenes from 12 input views at 512 × 960 resolution in 0.6 seconds.High-resolution results are also reported for different numbers of input views, including 6 and 12.
5. Conclusion
DepthSplat connects Gaussian splatting and depth estimation to achieve strong performance on both depth and view synthesis. It also uses Gaussian-splatting rendering loss for unsupervised depth pre-training, while retaining limitations in pose requirements and scalability to many views.
- DepthSplat achieves state-of-the-art results on ScanNet, RealEstate10K, and DL3DV for both depth estimation and view synthesis.
- The model requires camera poses and predicts pixel-aligned Gaussians, limiting use with extremely sparse views and scalability to many input views.The authors identify pose-free models and improved geometry representations as future directions.
A. Depth Pre-Training for Gaussian Splatting
DepthSplat’s depth-model initialization directly affects Gaussian-splatting view synthesis, with full depth pre-training outperforming partial feature initialization.
- Better depth initialization leads to improved view synthesis results with Gaussian splatting.The evaluated variants include monocular-only, monocular-plus-multi-view, and full depth-model initialization.
- Full depth-model pre-training achieves the best view synthesis results compared with initializing only monocular features or both monocular and multi-view features.The comparison evaluates three depth-model initialization variants during full DepthSplat training for view synthesis.
B.1. Unsupervised Depth Pre-Training with Gaussian Splatting
Gaussian splatting pre-training improves subsequent depth estimation, particularly in texture-less regions, where it may provide useful regularization.
- Gaussian splatting pre-training produces better depth-estimation results in texture-less regions.The comparison is made against depth estimation without Gaussian-splatting pre-training.
- The authors hypothesize that pre-training regularizes challenging scenarios and thereby improves performance.
B.2. Cross-Dataset Generalization
DepthSplat transfers more robustly to unseen DL3DV scenes than MVSplat and better preserves scene structures in four-view visual comparisons.
- B.2. Cross-Dataset Generalization: DepthSplat generalizes more robustly than MVSplat on unseen DL3DV scenes.The models are evaluated using RealEstate10K-pre-trained models.
- B.3. Visual Comparisons on DL3DV: With four input views, DepthSplat better preserves scene structures than MVSplat in visual synthesis comparisons.
B.4. High-Resolution Results
At 512 × 960 resolution, DepthSplat remains usable with six or twelve input views while improving efficiency through lower-resolution features and local cross-view attention.
- At 512 × 960 resolution, MVSplat runs out of memory with six or twelve input views, whereas DepthSplat improves efficiency.
- DepthSplat uses 1/8-resolution lowest features and local cross-view attention instead of MVSplat’s 1/4-resolution features and pair-wise global attention.These two technical components are identified as the sources of the efficiency improvement.
C. More Implementation Details
The high-resolution experiments use a small model trained and fine-tuned at progressively larger image resolutions. Additional figures examine unsupervised pre-training, cross-dataset generalization, and view synthesis with varying numbers of input views.
- High-resolution experiments: The high-resolution setup uses a small model with ViT-S monocular and single-scale multi-view branches.It is first trained on RealEstate10K with two input views at 256 × 448 resolutions, then fine-tuned on mixed RealEstate10K and DL3DV data at 448 × 768 resolutions.
- Unsupervised depth pre-training: Unsupervised depth pre-training improves performance in texture-less regions.
- Cross-dataset generalization: The appendix evaluates generalization from RealEstate10K to DL3DV.
- View synthesis and depth prediction: Additional experiments visualize view synthesis from 4 and 6 input views at 512 × 960 resolutions, alongside depth predictions from 12 input views.