PaperlayerView exact revision on arXiv

Source-linked AI summary

DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing

Matias Turkulainen, Xuqian Ren, Iaroslav Melekhov, Otto Seiskari, Esa Rahtu, Juho Kannala

arXiv:2403.17822v3cs.CV

TL;DR

Common indoor scenes remain difficult to reconstruct accurately because textureless and under-observed regions provide weak geometric constraints, limiting 3D Gaussian splatting despite its efficient rendering. DN-Splatter adds depth and normal regularization, including an adaptive edge-aware depth loss and monocular priors, improving novel-view synthesis and mesh quality while exposing a limitation of monocular depth supervision.

  • Problem

    Textureless and poorly observed indoor regions create ambiguities and insufficient geometric constraints, while existing methods often struggle to achieve photorealism and accurate geometry together.

  • Method

    DN-Splatter regularizes 3D Gaussian splatting with depth and normal cues, local smoothness, an edge-aware depth loss, and off-the-shelf monocular networks.

  • Results

    The regularization improves novel-view metrics and produces smoother, more geometrically accurate surfaces that can be extracted directly as meshes.

  • Takeaways & Limitations

    Depth and normal priors provide a simple strategy for obtaining more geometrically valid and consistent Gaussian reconstructions of challenging indoor scenes.

  • Takeaways & Limitations

    Monocular depth supervision remains inferior to iPhone depths, so further research is needed to improve its performance.

Abstract

from arXiv · show

High-fidelity 3D reconstruction of common indoor scenes is crucial for VR and AR applications. 3D Gaussian splatting, a novel differentiable rendering technique, has achieved state-of-the-art novel view synthesis results with high rendering speeds and relatively low training times. However, its performance on scenes commonly seen in indoor datasets is poor due to the lack of geometric constraints during optimization. In this work, we explore the use of readily accessible geometric cues to enhance Gaussian splatting optimization in challenging, ill-posed, and textureless scenes. We extend 3D Gaussian splatting with depth and normal cues to tackle challenging indoor datasets and showcase techniques for efficient mesh extraction. Specifically, we regularize the optimization procedure with depth information, enforce local smoothness of nearby Gaussians, and use off-the-shelf monocular networks to achieve better alignment with the true scene geometry. We propose an adaptive depth loss based on the gradient of color images, improving depth estimation and novel view synthesis results over various baselines. Our simple yet effective regularization technique enables direct mesh extraction from the Gaussian representation, yielding more physically accurate reconstructions of indoor scenes.

1. Introduction

DN-Splatter addresses the difficulty of reconstructing photorealistic and geometrically accurate indoor scenes by regularizing 3D Gaussian splatting with accessible depth and normal cues. Its strategy improves novel-view synthesis and supports more accurate mesh extraction from challenging scenes.

  • Indoor reconstruction remains difficult because textureless and poorly observed regions create ambiguities and insufficient constraints for valid geometry.
  • 3D Gaussian splatting renders efficiently and represents appearance and geometry explicitly, but photometric-only optimization can produce floaters and poor surfaces.
  • DN-Splatter regularizes Gaussian positions, local smoothness, and orientations using depth and normal priors from handheld sensors and general-purpose networks.
  • The method introduces an edge-aware depth loss for indoor scenes with imperfect depth estimates and uses monocular normal priors to align Gaussians with scene geometry.
  • Depth and normal regularization improves novel-view synthesis and enables efficient direct mesh extraction with smoother, more geometrically accurate reconstructions.

2. Related work

Prior work spans traditional image-based reconstruction, neural implicit representations, and geometry-aware regularization. DN-Splatter extends this line by applying depth and normal priors directly to 3D Gaussian splatting while targeting meshable reconstructions.

  • Traditional IBR: Traditional SfM and MVS reconstruct geometry from sparse multi-view-consistent 3D points obtained by triangulating image features.
  • Neural implicit IBR: Neural implicit methods such as NeRF achieve strong novel-view results through volumetric scene representations, while later work improves geometry with regularization and alternative representations.
  • Prior regularization: Existing regularization approaches supervise depth, impose smoothness, enforce multi-view consistency, or apply planar and monocular geometric constraints.
  • Meshable implicit representations: NeuSG aligns Gaussians with SDF-estimated normals but requires over 16 hours on high-end GPUs, reducing the appeal of 3D Gaussian splatting.
  • Meshable implicit representations: SuGaR optimizes Gaussian positions toward a signed-distance surface, yet its reconstructions remain noisy and mesh refinement requires additional optimization.

3. Preliminaries

3D Gaussian splatting represents scenes with differentiable Gaussian primitives and renders views by projecting, depth-sorting, and alpha-compositing them. Scenes are optimized with adaptive density control after initialization from sparse SfM points or sensor depth.

  • 3D Gaussian representation: Each Gaussian is parameterized by a mean, covariance, scale, rotation quaternion, opacity, and spherical-harmonic color.
  • Rasterization: Rendering projects 3D Gaussians into camera-space 2D Gaussians, depth-sorts them, and alpha-composites them to produce pixel colors.
  • Rasterization: The blending coefficient depends on Gaussian opacity and screen-space distance from the projected Gaussian center, while accumulated transmittance accounts for preceding Gaussians.
  • Optimization: The scene is initialized from sparse SfM points or sensor depth readings and optimized with Adaptive Density Control that culls, splits, and duplicates Gaussians.

4. Method

The method regularizes Gaussian splatting with sensor or monocular depth, Gaussian-derived and monocular normal cues, and smoothness priors. These constraints align Gaussians with scene geometry and support direct mesh extraction through Poisson reconstruction.

  • 4.1. Leveraging depth cues: Sensor depth is rendered with approximate Gaussian compositing, normalized by accumulated transmittance, and computed alongside color in one CUDA pass.The single forward pass improves inference and training speed compared with separate rendering steps.
  • 4.1. Leveraging depth cues: The adaptive depth loss weights errors by g_rgb = exp(−∇I), using the gradient of the aligned RGB image to account for scene geometry and texture complexity.The loss uses a robust logarithmic L1 depth discrepancy.
  • 4.1. Leveraging depth cues: Without sensor depth, scale-aligned monocular predictions from ZoeDepth or DepthAnything provide dense depth priors through sparse-depth alignment.The monocular prediction is aligned using an affine scale-and-shift fit before applying the depth loss.
  • 4.2. Leveraging normal cues: Gaussian normals use the rotation of the minimum scaling axis, while scale regularization encourages disc-like surfels whose orientations approximate surface normals.Normals are rendered by transforming and alpha-compositing Gaussian normals in camera space.
  • 4.2. Leveraging normal cues: Monocular normal supervision is combined with total-variation smoothing to encourage neighboring pixels to have smooth predicted normals; pretrained Omnidata estimates are less noisy than depth-gradient pseudo-normals.The method also initializes Gaussian positions and orientations from sensor-depth point clouds when available.
  • 4.4. Optimization: The optimization combines photometric, depth, scale, normal, and normal-smoothness losses, with λd = 0.2, λn = 0.1, and λs = 0.1.The normal term combines monocular normal supervision with total-variation smoothing.
  • 4.3. Gaussian initialization: After optimization, rendered depth and normal maps are backprojected into an oriented point set, and Poisson surface reconstruction extracts a mesh from the aligned Gaussians.The regularization is intended to distribute Gaussian positions along the scene surface for more accurate meshing.

5. Experiments

Experiments on MuSHRoom and ScanNet++ evaluate geometric supervision for mesh extraction, novel-view synthesis, and depth estimation in challenging indoor scenes. DN-Splatter’s depth and normal regularization improves reconstruction quality, while sensor-depth initialization and edge-aware depth losses provide additional benefits.

  • Experimental setup: Experiments use MuSHRoom and ScanNet++ indoor datasets, evaluating color, depth, mesh, and training-time performance against NeRF, SDF, Gaussian, and Poisson baselines.Mesh metrics include Accuracy, Completion, Chamfer-L1, Normal Consistency, and F1; color and depth metrics include PSNR, SSIM, LPIPS, and standard depth measures.
  • 5.1. Mesh evaluation: DN-Splatter extracts meshes directly after optimization without refinement, improving geometry over baseline NeRF and Gaussian methods on challenging indoor scenes.The method is competitive with more computationally expensive SDF approaches, while NeRF depth supervision can sometimes hinder mesh performance.
  • 5.2. Novel-view synthesis and depth estimation: Depth and normal supervision improves novel-view synthesis and depth quality, with DN-Splatter achieving the highest qualitative novel-view synthesis among the compared NeRF, SDF, and Gaussian methods.The qualitative comparisons also show reduced floaters and artifacts.
  • 5.3. Ablation studies: The normal loss aligns Gaussians with scene geometry, while the depth loss improves reconstruction and novel-view synthesis in ambiguous, textureless regions.Normal smoothing further improves normal completeness for Poisson meshing, with minimal quantitative impact but visible qualitative benefits.
  • 5.3. Ablation studies: Dense sensor-depth initialization improves mesh and novel-view synthesis metrics compared with sparse COLMAP SfM initialization.The experiments initialize Gaussian scenes with 1M back-projected sensor-depth points.
  • 5.3. Ablation studies: The gradient-aware depth regularizer produces the best qualitative depth-loss results by mitigating edge uncertainty in raw iPhone depth captures.Relative-depth supervision performs better than naive monocular supervision but remains inferior to iPhone depths, motivating further work on monocular depth supervision.

6. Conclusion

The conclusion presents DN-Splatter as a depth- and normal-regularized 3DGS method for photorealistic and geometrically accurate reconstruction of challenging indoor scenes. The authors report improved novel-view metrics and surface quality, with direct mesh extraction from the Gaussian scene.

  • DN-Splatter regularizes 3DGS with depth and normal cues for photorealistic and geometrically accurate reconstruction of challenging indoor datasets.
  • The strategy enhances novel-view metrics and significantly improves the surface quality extracted from Gaussian scenes.
  • Prior regularization is presented as essential for geometrically valid and consistent reconstructions in challenging indoor scenes.

Supplementary Material

The supplied supplementary-material passage contains author information rather than substantive supplementary findings.

  • The passage lists the paper’s authors and affiliations-related identifiers but provides no supplementary experimental or methodological content.

1 ETH Zurich, 2 Tampere University, 3 Aalto University, 4 Spectacular AI

The supplementary material documents baseline implementations and code modifications used to compare novel-view synthesis, depth estimation, and mesh reconstruction methods.

  • The supplementary material provides further details on baseline methods, datasets, evaluation metrics, losses, and quantitative and qualitative results.
  • The evaluated baselines include Nerfacto, Depth-Nerfacto, Neusfacto, MonoSDF, and Splatfacto, with method-specific settings and reconstruction procedures.
  • SuGaR’s normal directions were corrected for outward-facing indoor datasets because of a small inconsistency found in its implementation.
  • The original SuGaR code was modified to support depth rendering by replacing its CUDA backend with a depth-capable variant.
  • The official 2DGS implementation was modified to import camera information and poses directly from pre-made JSON files instead of relying only on COLMAP-based formats.
  • For 2DGS with depth regularization, the optimization loss is L = Lrgb + λdLd with λd set to 0.2.

A.2. Datasets

The supplementary material specifies the MuSHRoom and ScanNet++ evaluation setups, mesh and depth metrics, and the depth-regularization objectives compared in experiments.

  • MuSHRoom evaluation uses official training and evaluation splits, uniformly sampled camera trajectories, globally optimized COLMAP poses, and 5 million points for Poisson mesh extraction.
  • ScanNet++ experiments use two iPhone sequences with COLMAP-registered poses containing 358 and 705 registered images, reserving every 10th frame for evaluation after loading every fifth frame.
  • Depth evaluation uses Abs Rel, Sq Rel, RMSE, RMSE log, and threshold-accuracy metrics such as δ < 1.25, δ < 1.252, and δ < 1.253.
  • Mesh quality is evaluated with a 5cm threshold for precision, recall, and F-scores, and only within the visibility of training camera views.
  • The experiments compare multiple depth regularization objectives, including terms based on depth differences and RGB-image gradients.

C. Additional quantitative results

Additional experiments analyze mesh extraction, depth accuracy, sparse-view monocular supervision, depth-loss variants, and qualitative reconstruction behavior.

  • Back-projecting optimized depth and normal maps produces smoother and more realistic Poisson reconstructions than alternative meshing techniques.The comparison includes direct Gaussian-based meshing, SuGaR’s strategy, and the proposed back-projection approach.
  • Ground-truth Faro scanner evaluation shows that DN-Splatter’s depth regularization still outperforms other baseline methods on MuSHRoom depth estimation.The evaluation uses ten scenes and more accurate Faro lidar depths rather than noisy iPhone depth maps.
  • Under sparse-view settings, monocular depth supervision produces a notable increase in novel-view synthesis metrics, whereas gains are minimal for dense captures.
  • The logarithmic depth loss LLogL1 outperforms LL1 and LMSE on depth and RGB synthesis, while the gradient-aware logarithmic L̂D variant performs better than the simpler variant.The results support using edge-aware weighting because captured sensor depths can be inaccurate at edges or sharp boundaries.
  • The Lsmooth prior yields smoother rendered normal predictions in the VR Room scene.
  • Compared with 2DGS, DN-Splatter achieves higher novel-view and mesh-reconstruction metrics, while 2DGS produces smoother depth renders.
  • Additional figures provide qualitative mesh comparisons and rendered depth and RGB comparisons across MuSHRoom sequences including sauna, classroom, coffee room, and koivu.
Loading 2403.17822v3…