PaperlayerView exact revision on arXiv

Source-linked AI summary

DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization

Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, Lin Gu

arXiv:2403.06912v3cs.CV

TL;DR

Sparse-view radiance fields must balance novel-view quality with training, memory, and rendering costs, while 3D Gaussian Splatting can lose geometry as input views decrease. DNGaussian uses Hard and Soft Depth Regularization with Global-Local Depth Normalization to reshape Gaussian geometry under coarse depth supervision. Across LLFF, DTU, and Blender, it reports competitive or superior detail quality with lower memory costs, 25× reduced training time, and over 3000× faster rendering.

  • Problem

    Sparse-view radiance-field methods face high training costs, slow processing, and geometry degradation when few input views constrain 3D Gaussian Splatting.

  • Method

    DNGaussian freezes Gaussian shapes, encourages primitive movement through Hard and Soft Depth Regularization, and applies Global-Local Depth Normalization to emphasize local depth changes.

  • Results

    DNGaussian achieves competitive quality and superior details across LLFF, Blender, and DTU, with 25× lower training time and over 3000× faster rendering.

  • Takeaways & Limitations

    The framework provides fast, high-quality few-shot novel view synthesis with significantly lower training costs and real-time rendering.

  • Takeaways & Limitations

    On LLFF, DNGaussian improves quality with 6 input views but does not work well with 9 input views.

Abstract

from arXiv · show

Radiance fields have demonstrated impressive performance in synthesizing novel views from sparse input views, yet prevailing methods suffer from high training costs and slow inference speed. This paper introduces DNGaussian, a depth-regularized framework based on 3D Gaussian radiance fields, offering real-time and high-quality few-shot novel view synthesis at low costs. Our motivation stems from the highly efficient representation and surprising quality of the recent 3D Gaussian Splatting, despite it will encounter a geometry degradation when input views decrease. In the Gaussian radiance fields, we find this degradation in scene geometry primarily lined to the positioning of Gaussian primitives and can be mitigated by depth constraint. Consequently, we propose a Hard and Soft Depth Regularization to restore accurate scene geometry under coarse monocular depth supervision while maintaining a fine-grained color appearance. To further refine detailed geometry reshaping, we introduce Global-Local Depth Normalization, enhancing the focus on small local depth changes. Extensive experiments on LLFF, DTU, and Blender datasets demonstrate that DNGaussian outperforms state-of-the-art methods, achieving comparable or better results with significantly reduced memory cost, a $25 \times$ reduction in training time, and over $3000 \times$ faster rendering speed.

1. Introduction

Sparse-view radiance fields face a tension between reconstruction quality and computational efficiency, while 3D Gaussian Splatting retains detail but suffers geometry degradation as views decrease. DNGaussian addresses this with depth-based spatial reshaping and global-local normalization, achieving competitive quality with substantially lower costs.

  • Motivation: Sparse-view NeRF methods often require substantial memory and processing time, while faster grid-based methods trade training cost against rendering quality.These constraints restrict practical applications.
  • Motivation: 3D Gaussian Splatting preserves some detailed local features from sparse views, but reduced view constraints can cause incorrect geometry and failed novel-view synthesis.Figure 2 illustrates both retained fine details and geometry failures.
  • Method: Hard and Soft Depth Regularization freezes Gaussian shapes and encourages primitive movement to reshape spatial geometry while preserving fine-grained color appearance.Separate depth renderings independently adjust Gaussian centers and opacities.
  • Method: Global-Local Depth Normalization emphasizes small local depth changes while retaining information about the global scale.The design addresses noisy primitive distributions caused by overlooked local errors.
  • Results: 25× lower training time and over 3000× faster rendering accompany competitive quality and superior details across LLFF, Blender, and DTU sparse-view settings.The framework also reports substantially lower memory costs.

2. Related Work

Related work pursues sparse-view novel view synthesis through radiance-field representations, efficiency improvements, generative or pretrained regularization, and depth supervision. The passages identify unresolved challenges in jointly achieving quality and efficiency and in using monocular depth without losing local geometric detail.

  • Radiance Fields for Novel View Synthesis: NeRF represents scenes with large MLPs and volume rendering, but improvements commonly prioritize either quality or efficiency rather than both.The related work frames speed as a limitation of the original representation.
  • Few-shot Novel View Synthesis: Few-shot novel view synthesis methods use NeRF regularization, generative models trained on large datasets, or pretrained models as loss functions.These approaches incorporate priors or learned knowledge to handle sparse inputs.
  • Depth Supervision in Sparse-view Neural Fields: Sparse-view neural fields use depth from reliable point clouds, monocular estimators, and scale-invariant losses to supervise reconstruction.Prior work also includes depth ranking losses and related depth-supervision strategies.
  • Depth Supervision in Sparse-view Neural Fields: Fixed global-scale alignment can overlook minor local depth changes, producing noisy primitive distributions in intricate-texture regions.Flexible Gaussians are described as especially sensitive to incorrect depth cues.

3. Method

DNGaussian combines Gaussian-splatting color supervision with hard and soft depth regularization, using Gaussian position and opacity adjustments to improve geometry while preserving appearance. Global-local depth normalization addresses both small local depth changes and overall scene shape.

  • 3.1. Preliminary for 3D Gaussian Splatting: 3D Gaussian Splatting represents scenes with Gaussian primitives defined by centers, scales, rotations, opacities, and color features.The primitives are gathered by an optimized rasterizer for point-based rendering.
  • 3.1. Preliminary for 3D Gaussian Splatting: DNGaussian starts from randomly initialized Gaussians and combines inherited color optimization with a Neural Color Renderer.The renderer uses a grid encoder and MLP to predict primitive colors and accelerates inference by caching intermediate results.
  • 3.2. Depth Regularization for Gaussians: Hard Depth Regularization freezes Gaussian shape and encourages incorrectly positioned primitives to move toward monocular-depth-supported locations.A large opacity value makes hard depth emphasize the nearest Gaussians, while similarity loss aligns it with the target depth.
  • 3.2. Depth Regularization for Gaussians: Soft Depth Regularization freezes Gaussian centers and tunes opacity so the rendered soft depth remains accurate and surfaces avoid semitransparency and hollowness.Together, hard and soft regularization constrain nearby Gaussians to suitable positions with high opacity, composing complete surfaces.
  • 3.3. Global-Local Depth Normalization: Global-Local Depth Normalization combines patch-wise local normalization with global image statistics to emphasize small depth changes while retaining overall shape.The local component refocuses on small errors, while the global component supplies awareness of global scale; patch-wise normalization also reduces long-distance depth errors.
  • 3.4. Training Details: The full objective combines color reconstruction loss with hard and soft depth regularization, each containing separate local and global terms.Color supervision follows 3D Gaussian Splatting with L1 and D-SSIM components, while depth similarity uses L2 loss with an error tolerance.

4. Experiments

Experiments evaluate DNGaussian on LLFF, DTU, and Blender under sparse-view settings, comparing reconstruction quality, efficiency, and the effects of depth regularization and parameter freezing.

  • Setups: Experiments use LLFF and DTU with 3 input views, and Blender with 8 input views, following established dataset splits and evaluation protocols.Metrics include PSNR, SSIM, LPIPS, and Average Error; known camera poses and dataset-specific preprocessing are used.
  • LLFF: DNGaussian achieves the best LPIPS on LLFF while maintaining comparable PSNR, SSIM, and Average Error to the strongest methods.Qualitative results show finer details and more precise geometry than competing sparse-view methods.
  • DTU: DNGaussian achieves the best LPIPS and SSIM and the second-best Average Error on DTU 3-view evaluation.Its qualitative results show more complete geometry and detailed rendering in plush and reflective regions, despite lower PSNR.
  • Blender: DNGaussian achieves the best PSNR, SSIM, and LPIPS scores on Blender with 8 input views.It produces correct geometry, fewer floaters, and improved detail for complex objects with transparent and reflective materials.
  • Ablation Study: Ablations show that Hard and Soft Depth Regularization, Global-Local Depth Normalization, and parameter freezing improve geometry, detail, and overall quality.Hard regularization improves high-frequency details but can cause hollows; Soft Depth addresses this, while freezing prevents unexpected primitive movement.
  • Efficiency: 25× faster training and over 3000× faster rendering are achieved under limited-resource comparisons while retaining competitive novel-view quality.The efficiency study uses RTX 3090 Ti GPUs with 24GB/12GB memory limits and 1.0h/0.5h training-time settings.

5. Conclusion

The conclusion presents DNGaussian as a 3DGS-based framework for few-shot novel view synthesis that uses depth regularization.

  • DNGaussian introduces 3D Gaussian Splatting into few-shot novel view synthesis through depth regularization.

Overview

The supplemental document adds studies, implementation details, dataset settings, and limitations with future work.

  • The supplement reports additional ablations and backbone studies, describes implementation and dataset settings, and discusses limitations and future work.

A.1. Ablation Study on Depth Normalization

Additional ablations separate the roles of global and local depth normalization and show that their combination provides the strongest quality.

  • Global normalization supports overall scene learning but is weak on minor errors, whereas local normalization cannot stand alone.
  • Combined Global-Local Depth Normalization achieves the best quality in the additional depth-normalization ablation.

A.2. Ablation Study on Neural Color Renderer

The neural color renderer addresses sparse-view color inconsistencies that affect spherical harmonics while preserving fast inference. Comparisons across backbones show that existing NeRF regularization strategies transfer poorly to efficient representations, whereas DNGaussian is designed for 3DGS.

  • Neural Color Renderer: The neural color renderer improves rendering quality while maintaining an equally fast inference speed.
  • Neural Color Renderer: Sparse-view spherical harmonics can overfit and produce inconsistent colors in unseen views, while the neural color renderer relieves this problem.The paper attributes the issue partly to independent primitives lacking regional consistency.
  • Transfer of Previous Strategies: SparseNeRF and FreeNeRF weakly improve quality when transferred from implicit Mip-NeRF to Instant-NGP, TensoRF, and 3DGS.The comparison evaluates these strategies across current fast backbones rather than only their original implicit setting.
  • Transfer of Previous Strategies: The transfer experiment reports training time, GPU memory cost, and inference FPS for each backbone and method.
  • Transfer of Previous Strategies: On 3DGS, SparseNeRF and FreeNeRF cannot effectively regularize the efficient backbone, while FreeNeRF provides only limited improvement.The paper links FreeNeRF’s limited effect to the absence of frequency positional encoding and describes SparseNeRF as producing blur.

A.4. Comparison with Grid-based Methods

DNGaussian is compared with grid-based few-shot methods on LLFF and with FreeNeRF on Blender, emphasizing quality, detail, and efficiency across sparse-view settings.

  • Comparison with Grid-based Methods: On Blender with 8 input views, FreeNeRF trades accurate geometry for blurry details by masking high-frequency signals.
  • Comparison with Grid-based Methods: DNGaussian does not explicitly constrain high-frequency content and performs better in fine-grained details on Blender.
  • Comparison with Grid-based Methods: DNGaussian outperforms the grid-based methods VGOS and DiffusioNeRF on LLFF with 3 training views.
  • Comparison with Grid-based Methods: DNGaussian achieves the best efficiency, with much lower cost and faster inference than the compared grid-based methods.
  • Comparison with Grid-based Methods: Additional qualitative results are shown for DNGaussian with 3 training views on the DTU and LLFF datasets.

B.1. Implementations

The implementation uses pretrained monocular depth estimators, patch-based depth normalization, and standardized dataset protocols for LLFF, DTU, and Blender evaluations.

  • Implementations: DNGaussian uses pretrained DPT depth estimators, with DPT Hybrid 384 for LLFF and DPT Large 384 for DTU and Blender.
  • Implementations: Performance gaps across different monocular depth estimators are slight, indicating robustness to the chosen depth model.
  • Implementations: Patch sizes for Global-Local Depth Normalization are sampled from [5, 17] for LLFF and Blender and for DTU.
  • Implementations: LLFF evaluation uses 8 forward-facing scenes, every 8th image for testing, evenly sampled inputs, and images downsampled 8× to 378×504.
  • Implementations: Blender evaluation uses 8 input views and 25 evenly sampled test views, with images downsampled 2× to 400 ×

C. Discussions and Limitations

The discussion identifies boundaries involving more input views and difficult scene structures, including solid-color planes, specular regions, and gaps between Gaussian primitives.

  • More Input Views: With 9 LLFF input views, DNGaussian does not work well, possibly because monocular-depth errors negatively influence optimization.
  • More Input Views: DNGaussian can still improve synthesized-view quality with 6 input views, but its benefit weakens when color constraints become nearly sufficient at 9 views.
  • Solid Color Planes: Solid-color planes can produce ray-like noises and hollows because sparse views poorly constrain Gaussian primitives by both color and depth.
  • Specular Regions: Specular regions remain challenging because 3DGS can produce inconsistent appearances despite depth supervision and Local Depth Normalization.
  • Hollows and Cracks: Empty spaces between projected Gaussian primitives can create hollows and cracks when camera pose changes.
Loading 2403.06912v3…