PaperlayerView exact revision on arXiv

Source-linked AI summary

DepthSplat: Connecting Gaussian Splatting and Depth

Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, Marc Pollefeys

arXiv:2410.13862v3cs.CV

TL;DR

Gaussian splatting and single-view depth estimation are typically studied separately. DepthSplat connects them through multi-view depth modeling and Gaussian-splatting-based unsupervised pre-training, achieving state-of-the-art depth estimation and novel view synthesis on ScanNet, RealEstate10K, and DL3DV.

  • Problem

    Gaussian splatting and single-view depth estimation have typically been studied separately, leaving their interaction to be investigated.

    Source: §1 · pp. 1–2

  • Method

    DepthSplat augments multi-view depth matching with pre-trained monocular features and uses differentiable Gaussian-splatting rendering loss for unsupervised depth pre-training on posed multi-view data.

    Source: §1 · p. 2

  • Results

    DepthSplat achieves state-of-the-art performance on ScanNet, RealEstate10K, and DL3DV for both depth estimation and novel view synthesis.

    Source: §4.4 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    Gaussian-splatting rendering loss provides a way to use large-scale posed multi-view image datasets to train more robust and generalizable multi-view depth models.

    Source: §5 · p. 8

    The model requires camera poses as input, and its pixel-aligned Gaussians grow significantly in number when processing many input views.

    Source: §5 · p. 8

Abstract

from arXiv · show

Gaussian splatting and single-view depth estimation are typically studied in isolation. In this paper, we present DepthSplat to connect Gaussian splatting and depth estimation and study their interactions. More specifically, we first contribute a robust multi-view depth model by leveraging pre-trained monocular depth features, leading to high-quality feed-forward 3D Gaussian splatting reconstructions. We also show that Gaussian splatting can serve as an unsupervised pre-training objective for learning powerful depth models from large-scale multi-view posed datasets. We validate the synergy between Gaussian splatting and depth estimation through extensive ablation and cross-task transfer experiments. Our DepthSplat achieves state-of-the-art performance on ScanNet, RealEstate10K and DL3DV datasets in terms of both depth estimation and novel view synthesis, demonstrating the mutual benefits of connecting both tasks. In addition, DepthSplat enables feed-forward reconstruction from 12 input views (512x960 resolutions) in 0.6 seconds.

1. Introduction

DepthSplat connects sparse-view feed-forward Gaussian splatting with robust monocular depth estimation to improve both depth prediction and novel view synthesis. Its differentiable splatting module also enables photometric-only pretraining of depth models on posed multi-view datasets.

Introduction

  • Sparse-view feed-forward 3DGS methods such as MVSplat [9] rely on feature-matching multi-view depth and inherit its difficulties with occlusions, texture-less regions, and reflective surfaces, while monocular depths lack consistent scales across views.These limitations motivate connecting multi-view depth with robust monocular depth estimation.Source: §1 · p. 1
  • DepthSplat augments multi-view cost volumes with pre-trained monocular depth features, improving depth robustness in hard-to-match regions and using the predicted maps as Gaussian centers.A lightweight network predicts the remaining Gaussian parameters; unlike prior fusion methods [11] [31], the approach uses off-the-shelf monocular depth models for a simpler model with stronger performance.Source: §1 · p. 2
  • The fully differentiable splatting module uses only photometric supervision to pre-train depth models on large-scale posed multi-view datasets, and this pretraining improves results over training from scratch after task-specific fine-tuning.The approach requires no ground-truth geometry information.Source: §1 · p. 2
  • 0.6 seconds suffices for feed-forward reconstruction of large-scale or 360° scenes from 12 input views at 512 × 960, and DepthSplat achieves state-of-the-art results across the reported tasks and datasets.Experiments cover TartanAir, ScanNet, RealEstate10K, and DL3DV.Source: §1 · p. 2

2. Related Work

DepthSplat connects multi-view depth estimation and Gaussian splatting by combining monocular depth features with multi-view feature matching. This addresses weaknesses of each approach and supports both robust reconstruction and depth pre-training without ground-truth depth.

Multi-View Depth Estimation

  • Multi-view depth methods [15] [18] [25] [57] improve on traditional methods, but photometric consistency can fail with occlusions, low texture, and non-Lambertian surfaces.Classical multi-view stereo pipelines use photometric consistency for feature matching and reference-image depth prediction.Source: §2 · p. 2

Monocular Depth Estimation

  • Monocular depth methods [2] [28] [40] perform well on diverse in-the-wild data but have scale ambiguities; DepthSplat uses pre-trained features to improve robustness while maintaining multi-view consistency.The added features improve robustness in low-texture regions and on reflective surfaces.Source: §2 · p. 2

Feed-Forward Gaussian Splatting

  • Feed-forward Gaussian splatting methods [7] [8] [9] [10] [48] [51] handle sparse views; MVSplat [9] uses cost volumes for better geometry but remains vulnerable to textureless regions and reflective surfaces.DepthSplat integrates pre-trained monocular depth features for more robust depth prediction and Gaussian reconstruction.Source: §2 · p. 2
  • Data-driven methods LGM [51], GRM, and GS-LRM rely on training data and compute; GS-LRM uses 64 A100 GPUs for 2 days, while DepthSplat trains in 2 days with 4 GPUs.DepthSplat’s Gaussian splatting module also enables depth-model pre-training on large-scale posed multi-view data without ground-truth depth.Source: §2 · p. 2

Depth and Gaussian Splatting

  • Unlike TranSplat, which refines cost-volume depth with monocular features and is vulnerable to coarse-stage errors, DepthSplat combines both feature types early and performs significantly better; per-scene depth-loss methods [53] are orthogonal to feed-forward approaches.Flash3D [49] uses pre-trained monocular depth for single-image reconstruction, whereas DepthSplat targets large-scale scenes from multiple views.Source: §2 · p. 2

3. DepthSplat

DepthSplat combines multi-view matching with pre-trained monocular depth features to predict per-view depth and Gaussian parameters for novel-view rendering. The model is trained for depth estimation with ground-truth depth and for view synthesis with rendered-image losses.

3. DepthSplat

  • DepthSplat predicts per-pixel depth and Gaussian parameters from posed input images, unprojects depth to obtain Gaussian centers, and predicts remaining parameters with an additional head.The parameters include opacity, covariance, and color.Source: §3 · p. 3

3.1. Multi-View Feature Matching

  • A weight-sharing ResNet and six-layer multi-view Swin Transformer extract view-aware features; plane-sweep warping and correlations form cost volumes, using each view’s top-2 camera-nearest neighbors when N > 2.For more than two views, the selected views’ correlation values are averaged to balance speed and accuracy.Source: §3.1 · p. 3

3.2. Monocular Depth Feature Extraction

  • To address occlusions, texture-less regions, and reflective surfaces, DepthSplat adds interpolated features from the pre-trained Depth Anything V2 monocular backbone to the cost volume.The monocular backbone is a ViT with patch size 14; its features are bilinearly resized to cost-volume resolution.Source: §3.2 · p. 4

3.3. Feature Fusion and Depth Regression

  • The model concatenates monocular features and cost volumes, regresses depth with a 2D U-Net, and applies softmax-weighted averaging over depth candidates.A second matching stage searches a smaller depth range on features at 2× higher resolution, then a DPT head upsamples predictions to full resolution.Source: §3.3 · p. 4

3.4. Gaussian Parameter Prediction

  • For Gaussian splatting, DepthSplat unprojects per-pixel depth into 3D centers and uses a DPT head to predict opacity, covariance, and color from image, depth, and feature inputs.The predicted Gaussians are used to render novel views.Source: §3.4 · p. 4

3.5. Training Loss

  • Depth training uses inverse-depth L1 and gradient losses, while view-synthesis training combines rendered-image MSE and LPIPS with λ = 0.05.The depth model is trained with ground-truth depth; the full model is trained on rendered and ground-truth novel-view colors.Source: §3.5 · p. 4

4. Experiments

Experiments show that DepthSplat performs strongly across depth estimation and novel view synthesis benchmarks, while its monocular and multi-view components complement each other. They also show that Gaussian-splatting training can pre-train depth models without direct depth supervision.

4.1. Model Variants

  • Larger monocular backbones and two-scale hierarchical matching improve depth and view synthesis, while better depth-network architecture and initialization improve novel-view rendering.The evaluated model sizes pair ViT-S with one-scale matching, ViT-B with two-scale matching, and ViT-L with two-scale matching.Source: Table 1 · p. 5, §4.1 · p. 5

4.2. Ablation and Analysis

  • Removing either the monocular feature branch or the cost volume degrades performance: monocular priors help in textureless, reflective, and single-view regions, while the cost volume supports scale- and multi-view-consistent predictions.Without reliable depth for regions lacking correspondences, Gaussian-splatting renders can become misaligned.Source: §4.2 · pp. 5–6, Figure 3 · p. 6, Figure 4 · p. 6
  • Depth Anything V2 features achieve the best view-synthesis results among the tested monocular features, and simple feature concatenation performs better than single-branch or depth-map fusion while matching attention-based fusion.The alternatives include ConvNeXt-T features used in AFNet, MiDaS, and DINOv2 features.Source: Table 2 · p. 5, §4.2 · pp. 6–7, Table 3 · p. 6

4.3. Unsupervised Depth Pre-Training with Gaussian Splatting

  • Training on RealEstate10K using only Gaussian-splatting rendering loss provides unsupervised depth pre-training, which improves later depth predictions, especially on challenging TartanAir and KITTI evaluations.Fine-tuning the pre-trained model on TartanAir and VKITTI2 with ground-truth depth can further improve performance, including zero-shot evaluation on ScanNet and KITTI.Source: Table 4 · p. 7, §4.3 · p. 7

4.4. Benchmark Comparisons

  • DepthSplat achieves state-of-the-art results on ScanNet depth estimation and RealEstate10K novel view synthesis, and ScanNet depth benefits from Gaussian-splatting pre-training.The reported ScanNet depth models use ViT-B and two-scale feature matching, with and without pre-training.Source: Table 5 · p. 7, Table 6 · p. 7, §4.4 · pp. 7–8
  • On DL3DV, DepthSplat outperforms MVSplat [9] across metrics and 2, 4, and 6 input views, with larger gains for sparse views and more efficient scaling to additional views.The paper attributes the scaling advantage to lightweight local feature matching rather than MVSplat’s expensive global pair-wise matching.Source: Table 7 · p. 8, §4.4 · p. 8, Figure 6 · p. 8
  • RealEstate10K-pretrained DepthSplat generalizes better than MVSplat to DL3DV and ACID, especially DL3DV; qualitative results also cover 512 × 960 reconstructions from 6 or 12 views.The paper reports high-resolution examples in the appendix and video results on its project page.Source: Table 8 · p. 8, §4.4 · p. 8

5. Conclusion · Appendix

DepthSplat connects Gaussian splatting and depth estimation, achieving state-of-the-art results across ScanNet, RealEstate10K, and DL3DV for depth and view synthesis. Its Gaussian-splatting rendering loss also enables unsupervised depth pre-training, while the current model depends on camera poses and uses a pixel-aligned representation that can scale poorly with many views.

5. Conclusion

  • DepthSplat achieves state-of-the-art results on ScanNet, RealEstate10K, and DL3DV for both depth estimation and view synthesis.Source: §5 · p. 8
  • Gaussian-splatting rendering loss enables unsupervised depth pre-training using large-scale posed multi-view image datasets.The authors present this as a way to train more robust and generalizable multi-view depth models.Source: §5 · p. 8
  • The model requires camera poses alongside multi-view images, which may be challenging to obtain when input views are extremely sparse.Exploring pose-free models is identified as future work.Source: §5 · p. 8
  • Pixel-aligned Gaussians make the number of Gaussians grow significantly with many input views, motivating work on geometry representation and scalability.Source: §5 · p. 8

A. Depth Pre-Training for Gaussian Splatting

DepthSplat examines whether stronger depth-model initialization improves Gaussian-splatting view synthesis. Its full depth model pre-trained on TartanAir achieves the best view-synthesis results among the three compared initialization strategies.

A. Depth Pre-Training for Gaussian Splatting

B. More Visual Results · B.1. Unsupervised Depth Pre-Training with Gaussian Splatting

Gaussian splatting pre-training improves depth estimation in texture-less regions, which the authors hypothesize results from regularization in challenging scenarios.

B.1. Unsupervised Depth Pre-Training with Gaussian Splatting

  • Pre-training improves depth estimates in texture-less regions, potentially by regularizing challenging scenarios as observed in.The figure compares depth estimation with and without Gaussian splatting pre-training.Source: §B.1 · p. 12

B.2. Cross-Dataset Generalization · B.3. Visual Comparisons on DL3DV

DepthSplat shows stronger cross-dataset generalization than MVSplat on unseen DL3DV scenes using RealEstate10K-pretrained models, and its four-view visualizations better preserve scene structure.

B.2. Cross-Dataset Generalization

  • DepthSplat generalizes more robustly than MVSplat on unseen DL3DV scenes.The comparison uses models pretrained on RealEstate10K.Source: §B.2 · p. 12
  • The cross-dataset results evaluate RealEstate10K-pretrained models on DL3DV.These results are shown in Fig. B.2.Source: §B.2 · p. 12

B.3. Visual Comparisons on DL3DV

  • The DL3DV visual comparison is conducted with four input views and uses MVSplat as the comparator.It presents visual synthesis results in Fig. B.3.Source: §B.3 · p. 12
  • In four-input-view visual synthesis comparisons, DepthSplat better preserves scene structures than MVSplat.The comparison is shown in Fig. B.3.Source: §B.3 · p. 12

B.4. High-Resolution Results

At 512 × 960 resolution, MVSplat runs out of memory with 6 or 12 input views, while DepthSplat improves efficiency through lower-resolution features and local cross-view attention.

B.4. High-Resolution Results

  • At 512 × 960 resolution, MVSplat runs out of memory with 6 or 12 input views, while DepthSplat significantly improves efficiency.The comparison concerns view synthesis and multi-view depth estimation results.Source: §B.4 · p. 12
  • DepthSplat’s lowest feature resolution is 1/8 of the image resolution, compared with MVSplat’s 1/4.Source: §B.4 · p. 12
  • DepthSplat uses local cross-view attention instead of MVSplat’s pair-wise global attention.Source: §B.4 · p. 12

C. More Implementation Details

High-resolution experiments use a small DepthSplat model trained first on RealEstate10K and then fine-tuned on mixed RealEstate10K and DL3DV data. The supplied figures show texture-less-region gains from unsupervised depth pre-training, cross-dataset generalization, and view-synthesis or depth-prediction examples.

C. More Implementation Details

  • The final small model uses a ViT-S monocular branch and single-scale multi-view branch, trained on RealEstate10K before fine-tuning on mixed RealEstate10K and DL3DV data.Training uses two views at 256 × 448 for 150K iterations with batch size 256, then 448 × 768 for 200K iterations with batch size 64 and 4–10 randomly sampled views; it is used for predictions at resolutions such as 512 × 960 and 6 or 12 views.Source: C. More Implementation Details · p. 12
  • Unsupervised depth pre-training improves performance in texture-less regions.Source: Figure B.1 · p. 13
  • The supplied figure illustrates generalization from RealEstate10K to DL3DV.Source: Figure B.2 · p. 13
  • The appendix shows view synthesis on DL3DV from 4 input views.Source: Figure B.3 · p. 14
  • Examples include view synthesis at 512 × 960 from 6 input views and depth predictions on DL3DV at 512 × 960 from 12 input views.Source: Figure B.4 · p. 14, Figure B.5 · p. 15

Related papers

Loading 2410.13862v3…