PaperlayerView exact revision on arXiv

Source-linked AI summary

DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing

Matias Turkulainen, Xuqian Ren, Iaroslav Melekhov, Otto Seiskari, Esa Rahtu, Juho Kannala

arXiv:2403.17822v3cs.CV

TL;DR

Indoor reconstruction remains challenging for 3D Gaussian splatting because optimization lacks 3D cues and surface constraints. DN-Splatter adds depth and normal priors, and the authors report enhanced novel-view metrics and significantly improved extracted surface quality.

  • Problem

    3D Gaussian splatting optimization lacks 3D cues and surface constraints, leaving artifacts and poor surface reconstruction in indoor scenes.

    Source: §1 · p. 1

  • Method

    DN-Splatter regularizes 3D Gaussian splatting with edge-aware depth constraints and monocular normal priors to align Gaussians with scene geometry.

    Source: §1 · pp. 1–2

  • Results

    DN-Splatter enhances novel-view metrics and significantly improves the surface quality extracted from Gaussian scenes.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Prior regularization is essential for more geometrically valid and consistent reconstructions in challenging indoor scenes.

    Source: §6 · p. 8

    Reconstruction with monocular depth supervision remains inferior to using iPhone depths, and further research is needed to improve it.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

High-fidelity 3D reconstruction of common indoor scenes is crucial for VR and AR applications. 3D Gaussian splatting, a novel differentiable rendering technique, has achieved state-of-the-art novel view synthesis results with high rendering speeds and relatively low training times. However, its performance on scenes commonly seen in indoor datasets is poor due to the lack of geometric constraints during optimization. In this work, we explore the use of readily accessible geometric cues to enhance Gaussian splatting optimization in challenging, ill-posed, and textureless scenes. We extend 3D Gaussian splatting with depth and normal cues to tackle challenging indoor datasets and showcase techniques for efficient mesh extraction. Specifically, we regularize the optimization procedure with depth information, enforce local smoothness of nearby Gaussians, and use off-the-shelf monocular networks to achieve better alignment with the true scene geometry. We propose an adaptive depth loss based on the gradient of color images, improving depth estimation and novel view synthesis results over various baselines. Our simple yet effective regularization technique enables direct mesh extraction from the Gaussian representation, yielding more physically accurate reconstructions of indoor scenes.

1. Introduction

Accurate indoor-scene reconstruction remains difficult in textureless or poorly observed regions, while existing methods often trade off geometry, photorealism, and speed. DN-Splatter regularizes Gaussian splatting with depth and normal priors to improve reconstruction and enable mesh extraction.

Introduction

  • Textureless and less-observed regions make indoor-scene geometry ambiguous, and existing methods often struggle to achieve both photorealism and geometric accuracy efficiently.Neural implicit methods [2] [25] [30] [50] can also incur long training and rendering times.Source: §1 · p. 1
  • 3D Gaussian splatting [19] enables real-time rendering through explicit, optimizable Gaussian primitives, but photometric-only optimization without surface constraints can produce floaters and poor meshes.Its Gaussian properties are modified by gradient descent using photometric losses, motivating stronger regularization for plausible geometry.Source: §1 · p. 1
  • DN-Splatter uses an edge-aware depth constraint to regularize Gaussian positions and monocular normal priors to align Gaussian orientations with estimated scene boundaries.The priors draw on depth sensors and depth and normal estimation networks [1] [7] [53]; the depth loss is designed for imperfect depth estimates.Source: §1 · pp. 1–2
  • Depth and normal regularization improves novel-view synthesis over baselines and enables direct mesh extraction from Gaussian scenes with smoother, more geometrically accurate reconstructions.The overview describes regularizing Gaussian positions, local smoothness, and orientations across challenging indoor room datasets.Source: §1 · p. 2, Figure 1 · p. 2

2. Related work

The related work moves from image-based rendering and neural implicit reconstruction to geometric regularization and mesh extraction. DN-Splatter applies depth and normal priors to Gaussian splatting and investigates direct surface extraction from its representation.

Traditional IBR

  • SfM and MVS reconstruct geometry from sparse multi-view-consistent points, while learning-based methods replace parts of the pipeline with learnable modules.Related work also includes normal estimation and triangle-mesh construction.Source: §2 · p. 2

Neural implicit IBR

  • NeRF achieves photorealistic novel views, but extracted geometry often has artifacts and floaters; later work explores regularization and SDF scene representations.Source: §2 · p. 2

Prior regularization

  • Prior NeRF work regularizes ray termination, rendered-depth smoothness, or multi-view consistency, while Manhattan-SDF [12] and MonoSDF [59] target indoor geometry.This work investigates depth and normal priors for Gaussian splatting to enhance photometric and geometric reconstruction.Source: §2 · p. 2

Meshable implicit representations

  • NeRF mesh extraction can require expensive post-refinement, while SDF- or occupancy-based methods use marching cubes and dense volume queries, making training generally slow.This work investigates extracting meshable surfaces directly from Gaussian scenes.Source: §2 · p. 2

Meshable 3D Gaussians

  • Back-projected depth maps with analytically estimated normals yield noisy point clouds that are difficult to mesh without regularization; NeuSG [4] aligns Gaussians with SDF-estimated normals but takes over 16 hours on high-end GPUs.NeuSG jointly trains a dense SDF neural implicit model [50] alongside a Gaussian scene.Source: §2 · p. 3
  • SuGaR produces noisy reconstructions without geometric priors and costly refinement, while 2DGS [16] performs poorly on outward-facing indoor scenes without prior regularization.SuGaR uses a signed-distance loss to move Gaussian positions toward a surface; 2DGS explicitly models planar surfaces with Gaussian surfels.Source: §2 · p. 3

3. Preliminaries

3DGS represents scenes with differentiable Gaussian primitives and renders views by projecting and compositing them; scenes are typically initialized from sparse SfM points and optimized with Adaptive Density Control.

3. Preliminaries

  • 3DGS represents a scene with differentiable Gaussian primitives parameterized by position, covariance, opacity, and spherical-harmonic color.Covariance is decomposed into a scaling vector and rotation quaternion.Source: §3 · p. 3
  • Rendering projects Gaussians into camera-space 2D Gaussians, sorts them by depth, and alpha-composites their colors using accumulated transmittance.The blending coefficient depends on Gaussian opacity and its screen-space footprint at the pixel.Source: §3 · p. 3
  • Scenes are typically initialized from sparse SfM points, while this work also explores sensor-depth initialization; ADC culls, splits, and duplicates Gaussians at fixed intervals [19].ADC bases these operations on opacity, screen-space size, and the gradient magnitude of Gaussian means, respectively.Source: §3 · p. 3

4. Method

DN-Splatter regularizes Gaussian splatting with depth and normal cues, initializes scenes from dense sensor depth when available, combines the resulting losses with photometric optimization, and extracts meshes from the optimized scene.

4.1. Leveraging depth cues

  • An RGB-gradient-aware logarithmic depth loss emphasizes smooth, textureless regions and reduces regularization near image edges; monocular priors are scale-aligned to sparse SfM depths when sensor data are unavailable.The method uses off-the-shelf ZoeDepth and DepthAnything [53] priors, while sensor depths are directly regularized; depth and color are rasterized together in one CUDA pass.Source: §4.1 · pp. 3–4

4.2. Leveraging normal cues

  • Gaussian normals come from each flattened Gaussian’s smallest scaling axis and rotation, while Omnidata [7] normal supervision and a smoothness prior regularize rendered normals.The normal loss combines L1 supervision against monocular normals with total variation; deriving normals from Gaussian geometry lets scale and rotation updates affect them without additional learnable parameters.Source: §4.2 · p. 4

4.3. Gaussian initialization

  • When sensor depth is available, dense back-projected points initialize the scene, and point-cloud normals set Gaussian orientations and flattened scales to aid normal-estimation convergence.The orientation initialization uses normal directions estimated from the initial point cloud [61].Source: §4.3 · p. 4

4.4. Optimization

  • The reported objective combines the original photometric loss [19] with depth, scale, and normal losses; in experiments, λd = 0.2, λn = 0.1, and λs = 0.1.The normal loss consists of λn times the predicted-normal loss plus λs times the smoothness loss.Source: §4.4 · p. 5

4.5. Meshing

  • After optimization, rendered depth and normal maps from training views are back-projected into an oriented point set for Poisson surface reconstruction.The authors attribute the suitability of Gaussian positions for meshing to their regularization strategy, which aligns and distributes them along scene surfaces.Source: §4.5 · p. 5

5. Experiments

Experiments on MuSHRoom and ScanNet++ evaluate DN-Splatter’s mesh reconstruction, novel-view synthesis, and depth estimation, along with geometric-supervision and initialization strategies. The results examine how depth and normal cues affect reconstruction in challenging indoor scenes.

5. Experiments

  • The experiments use indoor MuSHRoom and ScanNet++ data, evaluate color, depth, and mesh metrics, and compare NeRF-, SDF-, and Gaussian-based baselines.Baselines include Nerfacto, Depth-Nerfacto, NeuS-facto, MonoSDF, Splatfacto, SuGaR, 2DGS, and Poisson meshing with back-projected sensor depth.Source: §5 · p. 5

5.1. Mesh evaluation

  • Geometric cues make direct mesh extraction competitive with more computationally expensive SDF methods on challenging indoor scenes, without additional refinement.NeRF and Gaussian-based baselines struggle, and depth supervision can hinder NeRF mesh performance.Source: §5.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6

5.2. Novel-view synthesis and depth estimation

  • DN-Splatter achieves the highest novel-view synthesis results among NeRF, SDF, and Gaussian methods, while geometric supervision improves depth and RGB metrics.Reported evaluations include two distinct MuSHRoom test sets: one sampled from training trajectories and one from a separate, non-overlapping trajectory.Source: §5.2 · p. 6, Figure 4 · p. 7, Table 3 · p. 7

5.3. Ablation studies

  • The normal loss aligns Gaussians with scene geometry, the depth loss improves reconstruction and novel-view synthesis in ambiguous textureless regions, and normal smoothing aids Poisson-mesh normal completeness.The smoothing prior has minimal impact on other quantitative metrics, though its benefit is evident in qualitative renders.Source: §5.3 · p. 7, Table 4 · p. 7, Figure 6 · p. 8
  • Dense sensor-depth initialization improves mesh and novel-view synthesis metrics over sparse COLMAP SfM initialization.The comparison uses 1M initialization points and averages results over six MuSHRoom scenes.Source: §5.3 · p. 7, Table 5 · p. 7
  • Among tested depth losses, L1 and LLogL1 generally yield the best color metrics, while the proposed edge-aware loss produces the smoothest reconstructions.The study compares LMSE, L1, LLogL1, and edge-aware depth losses on ScanNet++; qualitative comparisons highlight reduced edge uncertainty with the proposed loss.Source: §5.3 · p. 8, Table 6 · p. 8, Figure 5 · p. 8
  • 2DGS also struggles with textureless indoor regions and planar surfaces, and the study compares its depth-regularized variant with DN-Splatter using monocular and sensor-depth supervision.The comparison uses the same loss and initialization strategy for both methods.Source: §5.3 · pp. 7–8, Table 7 · p. 8
  • Monocular and multi-view depth alternatives, including patch-based Pearson supervision, remain inferior to iPhone sensor depths, despite Pearson outperforming naive monocular supervision.The sensor depths are low resolution and inaccurate mainly at object edges, but the passage describes them as practical for real-world indoor scenes.Source: §5.3 · p. 8, Table 8 · p. 8

6. Conclusion

DN-Splatter uses depth and normal regularization of 3D Gaussian splatting to improve photorealistic and geometrically accurate reconstruction of challenging indoor scenes. The conclusion reports enhanced novel-view metrics, better extracted surface quality, and more geometrically valid, consistent reconstructions.

6. Conclusion

  • DN-Splatter applies depth and normal regularization to 3D Gaussian splatting for challenging indoor-scene reconstruction.The method targets both photorealistic and geometrically accurate reconstruction.Source: §6 · p. 8
  • The regularization strategy enhances novel-view metrics and significantly improves the surface quality extracted from a Gaussian scene.Source: §6 · p. 8
  • Prior regularization is essential for more geometrically valid and consistent reconstructions in challenging indoor scenes.Source: §6 · p. 8

1 ETH Zurich, 2 Tampere University, 3 Aalto University, 4 Spectacular AI · A. Implementation details · A.1. Baselines

The supplementary material details baseline implementations and points to appendices for evaluation definitions and further quantitative and qualitative results.

1 ETH Zurich, 2 Tampere University, 3 Aalto University, 4 Spectacular AI

  • The supplementary material directs readers to Appendix A for baseline and dataset details, Appendix B for evaluation definitions, and Appendices C–D for further quantitative and qualitative results.Source: §1 · p. 12

A.1. Baselines

  • Nerfacto [45] uses default settings without pose optimization and Ref-NeRF [48] normals; Depth-Nerfacto adds DS-NeRF [6] sensor-depth supervision on ray termination distributions.Both use rendered normal and depth maps for Poisson reconstruction.Source: §A.1 · p. 12
  • Neusfacto uses default settings and marching cubes, while MonoSDF [59] uses sensor-depth and Omnidata [7] normal supervision with loss multipliers 0.1 and 0.05.Source: §A.1 · p. 12
  • Splatfacto, using Nerfstudio 1.1.3 and gsplat 1.0.0, serves as the default-settings baseline and faithfully reimplements 3DGS [19].Source: §A.1 · p. 12
  • SuGaR uses its SDF-trained variant with coarse and refined meshes; code changes support JSON camera poses, and experiments correct its outward-facing indoor normal directions.The coarse and refined meshes differ only slightly in geometry metrics.Source: §A.1 · p. 12
  • SuGaR’s code was modified to support depth rendering by replacing its CUDA backend with a variant that includes depth rendering.Source: §A.1 · p. 12
  • 2DGS [16] uses its official code, default training settings, and TSDF-fusion meshing, with JSON camera-pose support added for non-COLMAP datasets.Source: §A.1 · p. 12
  • The 2DGS [16] depth-regularized variant adds sensor or monocular depth supervision and rasterizer depth gradients, optimizing L = Lrgb + λdLd with λd = 0.2.Source: §A.1 · p. 12

A.2. Datasets

The experiments use MuSHRoom and two ScanNet++ scenes, with evaluation images held out from camera trajectories or image sequences. MuSHRoom mesh extraction uses 5 million points.

MuSHRoom

  • MuSHRoom evaluation uses official splits and images sampled every 10 frames from the training trajectory plus images from a different trajectory; mesh extraction uses 5 million points.Both evaluation sequences use globally optimized COLMAP.Source: §A.2 · p. 12

ScanNet++

  • Experiments use ScanNet++ scenes b20a261fdf and 8b5caf3398, iPhone sequences with COLMAP registered poses, uniformly load every fifth frame, and reserve every tenth frame from those sequences for evaluation.The sequences contain 358 and 705 registered images, respectively.Source: §A.2 · p. 12

Depth Evaluation Metrics

  • Table 9 defines depth evaluation metrics using predicted and ground-truth depths for each pixel and gives example threshold factors δ < 1.25, δ < 1.252, and δ < 1.253.Source: Table 9 · p. 13

B. Definitions for metrics and losses … B.3. Depth losses

The paper defines depth and mesh evaluation measures and compares depth-supervision loss variants. Its mesh evaluation uses a 5 cm threshold and is limited to regions visible from training cameras.

B.1. Depth evaluation metrics

  • Depth evaluation on ScanNet++ and MuSHRoom reports Abs Rel, Sq Rel, RMSE, RMSE log, and threshold accuracy (δ < t).The paper follows [23] [28] [32] [42] [47] [51] for these metrics.Source: §B.1 · p. 13
  • Abs Rel measures average relative error, while Sq Rel measures squared relative error between predicted and ground-truth depth.Source: §B.1 · p. 13
  • RMSE measures the magnitude of depth errors, RMSE log applies it in the logarithmic domain, and threshold accuracy counts predictions within a ground-truth factor δ.The logarithmic variant can be useful for very large depth values.Source: §B.1 · p. 13

B.2. Mesh evaluation metrics

  • Mesh precision, recall, and F-scores use a 5 cm threshold, and quality is evaluated only within the visibility of training camera views.Source: §B.2 · p. 13

B.3. Depth losses

  • The paper compares depth-supervision loss variants defined in Table 11 and evaluates their performance in Table 15.Table 11 specifies δ = 0.2 max(∥D − ˆD∥1), g_rgb = exp(−∇I), and ground-truth and rendered depth and RGB notation.Source: §B.3 · p. 13, Table 11 · p. 13

B.2. Mesh evaluation metrics

  • Table 10 defines mesh evaluation using point clouds sampled from predicted and ground-truth meshes, with n_p denoting the normal at point p.Source: Table 10 · p. 13

C. Additional quantitative results … C.3. Additional depth comparisons

The additional quantitative results examine Poisson meshing, depth estimation against Faro scanner data, sparse-view supervision, and depth-loss variants.

C. Additional quantitative results

  • The section reports additional quantitative results on meshing strategies, Faro-ground-truth depth quality, and depth-loss ablations.These results extend the paper’s analyses with further meshing comparisons, scanner-based depth evaluation, and loss-variant experiments.Source: C. Additional quantitative results · p. 13

C.1. Mesh extraction techniques

  • Back-projection of optimized depth and normal maps produces smoother, more realistic Replica reconstructions than the other tested Poisson meshing strategies.The compared strategies include Gaussian means and normals, SuGaR surface-density extraction, and back-projection; all models use depth and normal regularization, with 500k extracted points for the latter two methods.Source: §C.1 · p. 13, Table 12 · p. 14

C.2. Depth estimation compared to Faro scanner ground truth

  • Against Faro lidar ground truth on MuSHRoom, DN-Splatter still outperforms the baseline methods on depth estimation.This evaluation uses laser-scanner depths rather than the low-resolution iPhone depth captures used for the corresponding main-paper comparison.Source: §C.2 · p. 14

C.3. Additional depth comparisons

  • In sparse-view Tanks & Temples training, monocular depth supervision yields notable increases in novel-view synthesis metrics, unlike its minimal gains with dense captures.The comparison evaluates training with dense and sparse captures.Source: §C.3 · p. 14
  • LLogL1 outperforms LL1 and LMSE on depth estimation and RGB synthesis.The comparison evaluates depth-loss variants on depth estimation and novel-view synthesis.Source: §C.3 · p. 14
  • The gradient-aware logarithmic depth variant L ˆ D outperforms its simpler variant on the reported comparisons.The paper attributes its advantage to mitigating inaccurate sensor-depth readings at edges and sharp boundaries.Source: §C.3 · p. 14
  • Figure 7 visualizes normal estimates with and without the Lsmooth smoothing prior on the MuSHRoom “VR Room” scene.The supplied caption identifies the visual comparison but does not state a quantitative result.Source: Figure 7 · p. 14

D. Additional qualitative results … D.3. Mesh and NVS renders

The additional results show smoother normal estimates with the smoothing prior, compare DN-Splatter with baselines in renderings, and report depth, monocular-supervision, and depth-loss evaluations.

D.1. Normal smoothing loss

  • Applying the Lsmooth prior produces smoother rendered normal estimates.The effect is visualized in Figure 7.Source: §D.1 · p. 14

D.3. Mesh and NVS renders

  • On MuSHRoom, low-resolution iPhone depth regularization greatly outperforms other baselines against Faro scanner ground truth, averaged over 10 scenes.The evaluation uses more accurate Faro-reconstructed depth maps rather than noisy captured iPhone depth maps.Source: Table 13 · p. 15
  • With sparse views on the Tanks & Temples advanced Courtroom sequence, monocular depth supervision aids novel-view synthesis.The ablation compares Zoe-Depth [3] and DepthAnything [53].Source: Table 14 · p. 15
  • On MuSHRoom, the proposed edge-aware L̂_D loss achieves the best performance among the evaluated depth losses.The ablation considers depth-estimation and novel-view-synthesis performance.Source: Table 15 · p. 15

D.2. 2DGS vs. DN-Splatter renders

  • On Honka and Kokko, DN-Splatter obtains higher novel-view and mesh-reconstruction metrics than 2DGS, while 2DGS produces smoother depth renders.Both methods use the L̂_D regularization loss in the qualitative comparison.Source: §D.2 · p. 14, Figure 8 · p. 16

D.3. Mesh and NVS renders

  • Additional MuSHRoom results show qualitative baseline comparisons for mesh reconstruction and rendered depth and RGB images across sauna, classroom, coffee room, and koivu sequences.The mesh comparison is in Figure 9; rendered depth and RGB comparisons appear in Figures 10–12.Source: Figure 9 · p. 17, Figure 10 · p. 18, Figure 11 · p. 19

Related papers

Loading 2403.17822v3…