Source-linked AI summary
DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models
Jamie Wynn, Daniyar Turmukhambetov
TL;DR
NeRFs are under-constrained with limited views, motivating a learned prior that improves scene geometry and color. The paper uses a denoising diffusion model trained on RGBD patches to provide regularizing score gradients during NeRF training. Experiments report improved novel-view synthesis and 3D reconstruction, while future work remains on replacing heuristic gradient scheduling.
Problem
NeRF color and geometry fields are severely under-constrained, allowing multiple scene explanations and artifacts such as floaters, especially with few views.
Method
A denoising diffusion model trained on RGBD patches approximates the score function and backpropagates its gradients to regularize NeRF color and density fields.
Results
The proposed regularization scheme improves performance on novel view synthesis and 3D reconstruction.
Takeaways & Limitations
The framework can extend beyond RGBD patches to regularize 3D voxel grids or density weights sampled along rays.
Takeaways & Limitations
A principled combination of DDM gradients with the NeRF objective remains future work because current scheduling of τ and gradient weights is heuristic.
Abstract
from arXiv · showhide
Under good conditions, Neural Radiance Fields (NeRFs) have shown impressive results on novel view synthesis tasks. NeRFs learn a scene's color and density fields by minimizing the photometric discrepancy between training views and differentiable renderings of the scene. Once trained from a sufficient set of views, NeRFs can generate novel views from arbitrary camera positions. However, the scene geometry and color fields are severely under-constrained, which can lead to artifacts, especially when trained with few input views. To alleviate this problem we learn a prior over scene geometry and color, using a denoising diffusion model (DDM). Our DDM is trained on RGBD patches of the synthetic Hypersim dataset and can be used to predict the gradient of the logarithm of a joint probability distribution of color and depth patches. We show that, these gradients of logarithms of RGBD patch priors serve to regularize geometry and color of a scene. During NeRF training, random RGBD patches are rendered and the estimated gradient of the log-likelihood is backpropagated to the color and density fields. Evaluations on LLFF, the most relevant dataset, show that our learned prior achieves improved quality in the reconstructed geometry and improved generalization to novel views. Evaluations on DTU show improved reconstruction quality among NeRF methods.
1. Introduction
NeRFs can synthesize novel views but their color and geometry fields remain under-constrained, producing physically implausible artifacts. The paper addresses this by using denoising diffusion models as a joint learned prior over RGBD patches.
- NeRFs learn scene appearance with an MLP that predicts density and color from 3D points and viewing directions.
- The method encourages flat TV and table surfaces in the density field while modeling view-dependent color changes through the color field.
- Differentiable volumetric rendering lets NeRFs reconstruct training images and generate views from arbitrary camera positions.
- An infinite number of NeRFs can explain the training views, leaving color and geometry fields under-constrained and enabling floaters and implausible novel views.
- Existing methods use hand-crafted geometry constraints, learned rendering priors, or monocular depth and normal estimates, but do not learn a joint scene geometry-and-color distribution.
- The proposed DDM learns an RGBD-patch prior whose log-probability gradients can be backpropagated during NeRF training as a regularizer.
2. Related work
Related work spans alternative geometry and field representations, learned priors, and generative models. This paper instead regularizes rendered color and depth jointly, using RGBD patches and a diffusion-based prior within NeRF optimization.
- Geometry modeling: Scene geometry can be represented as a density, occupancy, or signed distance field and rendered with differentiable surface or volumetric rendering.
- Field representation: Sparse voxel and hash-table representations can reduce MLP size or accelerate NeRF training, while the proposed prior remains agnostic to the underlying geometry representation.
- Learned priors: The method regularizes geometry directly because poor geometry is identified as the cause of severe few-view degradation, rather than constraining it indirectly through RGB patches.
- Learned priors: Joint RGBD modeling retains information from the correlation between color and depth that separate regularizers would discard.
- Method comparison: The approach samples random image patches, renders color and depth, and uses diffusion-model gradients to regularize NeRF color and density fields.
- Learned priors: Unlike RegNeRF’s 8 × 8 prior patches, Instant NGP enables priors over 48 × 48 patches through faster rendering.
- Diffusion-based methods: Unlike Dreamfusion, which uses diffusion models to guide text-conditioned NeRF optimization, this work uses them to regularize NeRFs from training images.
3. Method
DiffusioNeRF combines differentiable NeRF rendering and geometric regularizers with a denoising diffusion model that supplies score-based priors over rendered RGBD patches. The method trains the diffusion model on noisy RGBD patches and backpropagates its estimated score gradients into NeRF color and density fields.
- NeRFs: NeRF represents a scene with density and color fields that are optimized by differentiable rendering and photometric reconstruction loss.Rendered ray colors are formed from weighted samples along each ray, with transmittance determining the contribution of each sample.
- Geometric regularization: The geometric baseline combines photometric, foreground, frustum, and distortion losses, with λ coefficients controlling their contributions.The frustum term penalizes density contained in fewer than two training view frustums, while the foreground term encourages ray weights to sum to unity.
- Score functions and DDMs: Diffusion models progressively add Gaussian noise and learn a reverse process by predicting noise from a noised sample and its noise level.The predicted noise is proportional to the negative score, so moving against it leads toward modes of the learned data distribution.
- Score functions and DDMs: The method models a prior over NeRF color and density through the score of rendered RGBD patches sampled from random 48 × 48 pixel regions.The DDM output is normalized to control gradient magnitude before being used as a regularization function.
4. Experiments
Experiments on LLFF and DTU evaluate novel-view synthesis and geometry reconstruction across few-view and all-view settings. DiffusioNeRF improves geometry quality, while its results reveal trade-offs involving unseen regions, thin structures, patch size, and regularization scheduling.
- Datasets and metrics: Evaluations use LLFF and DTU to measure novel-view synthesis and geometry reconstruction under few-view and all-view settings.LLFF provides held-out views for image synthesis, while DTU provides scans and ground-truth geometry for reconstruction evaluation.
- Few-view evaluation: With few input views, the geometric baseline has higher LLFF metrics, while DiffusioNeRF produces more plausible depths but can over-smooth thin structures.The comparison covers models trained with 3, 6, and 9 views and notes that unseen scene parts can substantially affect reconstruction scores.
- All-view geometry evaluation: With all DTU views, DiffusioNeRF generalizes from indoor scenes to object reconstruction and performs adequately against occupancy- and signed-distance-field methods.The priors are less important when many training views constrain the scene, but they can help ambiguous regions such as textureless surfaces.
- All-view geometry evaluation: Qualitative DTU results show density-based methods struggle with shiny objects, while DiffusioNeRF yields fewer holes and smoother surfaces on some diffuse or textured objects.The DDM provides a complementary signal to geometric regularizers in the reported house-completion example.
- Few-view evaluation: DiffusioNeRF improves DTU geometry scores, although novel-view synthesis metrics are hampered by details introduced in regions absent from training views.The geometric baseline can score favorably on LLFF while still producing unrealistic geometry, whereas the DDM-based approach improves DTU results.
- Ablation studies: A 24 × 24 patch model outperforms a 48 × 48 model on LLFF but underperforms it on DTU, and feeding input-image patches 25% of the time is significant.The ablations also indicate that DDM training-data quantity, gradient-weight scheduling, and λDDM affect reconstruction behavior and scores.
5. Conclusions
The paper uses denoising diffusion models as RGBD-based regularizers for NeRF optimization and reports improved novel view synthesis and 3D reconstruction. The framework is presented as versatile, though combining its gradients with the NeRF objective still relies on heuristic scheduling.
- The proposed DDM regularization scheme improves performance on novel view synthesis and 3D reconstruction.
- The framework can regularize different representations, including rendered RGBD patches, 3D voxel grids, and density weights sampled along rays.
- Early results indicate that training a DDM on 3D voxel blocks can regularize the density field directly during NeRF optimization.
- A future direction is to combine the DDM gradient with the NeRF objective more principledly instead of using heuristic τ and gradient scheduling.
- The general DDM-regularization approach could potentially extend to gradient-descent tasks such as self-supervised monocular depth estimation and stereo matching.