ReconFusion: 3D Reconstruction with Diffusion Priors

Rundi Wu; Ben Mildenhall; Philipp Henzler; Keunhong Park; Ruiqi Gao; Daniel Watson; Pratul P. Srinivasan; Dor Verbin; Jonathan T. Barron; Ben Poole; Aleksander Holynski

Source-linked AI summary

ReconFusion: 3D Reconstruction with Diffusion Priors

Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, Aleksander Holynski

arXiv:2312.02981v1Dec 5, 2023cs.CV

TL;DR

NeRFs require tens to hundreds of images for high-quality reconstruction, creating a demanding capture process. ReconFusion integrates a multiview-conditioned diffusion prior into NeRF optimization, and outperforms existing baselines across in-distribution and out-of-distribution forward-facing and 360° datasets.

Problem
NeRF reconstruction requires tens to hundreds of images because under-observed viewpoints produce artifact-prone renderings and existing regularizers degrade at novel viewpoints.
Method
ReconFusion uses a diffusion model trained for multiview novel-view synthesis to regularize NeRF renderings at random unobserved poses.
Results
ReconFusion outperforms all baselines on in-distribution and out-of-distribution datasets, achieving state-of-the-art few-view NeRF reconstruction performance.
Takeaways & Limitations
The diffusion prior provides plausible geometry and appearance in few-view settings while reducing fog and floater artifacts in denser captures and preserving well-sampled regions.
Takeaways & Limitations
Existing few-view regularization methods still show considerable degradation at novel viewpoints compared with denser captures.

Abstract

from arXiv · show

3D reconstruction methods such as Neural Radiance Fields (NeRFs) excel at rendering photorealistic novel views of complex scenes. However, recovering a high-quality NeRF typically requires tens to hundreds of input images, resulting in a time-consuming capture process. We present ReconFusion to reconstruct real-world scenes using only a few photos. Our approach leverages a diffusion prior for novel view synthesis, trained on synthetic and multiview datasets, which regularizes a NeRF-based 3D reconstruction pipeline at novel camera poses beyond those captured by the set of input images. Our method synthesizes realistic geometry and texture in underconstrained regions while preserving the appearance of observed regions. We perform an extensive evaluation across various real-world datasets, including forward-facing and 360-degree scenes, demonstrating significant performance improvements over previous few-view NeRF reconstruction approaches.

1. Introduction

ReconFusion addresses the heavy image-capture requirements of NeRF reconstruction by combining a multiview-conditioned diffusion prior with NeRF optimization. It improves reconstruction across forward-facing and 360° scenes, supporting plausible geometry and appearance in under-observed regions while preserving well-sampled regions.

1. Introduction: NeRF reconstruction typically requires tens to hundreds of images because under-observed viewpoints produce artifact-prone renderings.Existing low-level regularizers still degrade considerably at novel viewpoints compared with dense captures.
1. Introduction: ReconFusion derives a novel-view image prior from a diffusion model trained on real-world and synthetic multiview datasets.The model is finetuned from a pretrained latent diffusion model using RealEstate10K, CO3D, MVImgNet, and Objaverse.
1. Introduction: ReconFusion outperforms existing baselines across forward-facing and unbounded 360° scene datasets.The method provides a strong prior for plausible geometry and appearance in few-view settings.
1. Introduction: In denser capture settings, the diffusion prior reduces fog and floater artifacts while preserving well-sampled regions.This demonstrates that the prior functions as a drop-in regularizer across capture settings.
1. Introduction: The system integrates multiview-conditioned diffusion with an optimization-based NeRF pipeline to reduce the need for rigorous scene capture.The diffusion model guides reconstruction while the NeRF representation maintains multiview consistency.

2. Related Work

Prior work reduces few-view reconstruction artifacts through geometric, appearance, frequency, and diffusion-based regularization, while view-synthesis models generate novel images from posed inputs. ReconFusion combines these directions by using multiview-conditioned diffusion to guide a consistent NeRF.

Few-view NeRF: Few-view NeRF methods regularize geometry and appearance using sparse depth, visibility, frequency, normalizing-flow color, or RGB-D patch distributions.Examples include DS-NeRF, DDP-NeRF, FreeNeRF, RegNeRF, and DiffusioNeRF.
Regression models for view synthesis: Feed-forward view-synthesis networks learn from large posed multiview collections but extrapolate poorly to ambiguous viewpoints.Most lift input images into a 3D representation before predicting novel views.
Generative models for view synthesis: Generative view-synthesis methods use GANs or diffusion models to generate unseen scene content beyond observed inputs.Examples include pose-conditioned diffusion models trained on synthetic ShapeNet data and models trained on real-world multiview data.
ReconFusion: ReconFusion optimizes a NeRF with reconstruction and sample losses, using diffusion predictions at random poses to regularize novel-view renderings.Its sample loss compares NeRF renderings with diffusion-generated images for sampled viewpoints.
Lifting 2D diffusion models for 3D generation: 2D diffusion models have also been lifted into 3D generation, including score distillation sampling and multistep sampling strategies.These approaches use image-generation priors to supervise or iteratively update 3D representations.

3. ReconFusion

ReconFusion combines a diffusion model for plausible novel views with a NeRF reconstruction procedure that uses those views as a prior. The pipeline preserves observed-image reconstruction while regularizing unobserved viewpoints through sampled diffusion targets.

ReconFusion: ReconFusion combines a novel-view diffusion model with a 3D reconstruction procedure that uses the diffusion model as a prior.The method is described as two components: diffusion-model training and diffusion-prior-guided 3D reconstruction.
Diffusion model for novel view synthesis: The diffusion model estimates scene appearance at novel viewpoints from multiple posed images and is trained by finetuning a latent diffusion model on real-world and synthetic multiview datasets.The conditioning design uses input-image embeddings and rendered feature maps that encode relative camera transforms.
3D reconstruction with diffusion priors: A photometric loss also trains PixelNeRF to reconstruct the RGB target image, helping avoid local minima where the diffusion model cannot leverage PixelNeRF inputs.The loss compares the PixelNeRF output with the downsampled target image at the feature-map resolution.
3D reconstruction with diffusion priors: NeRF reconstruction minimizes error on observed posed images, but this reconstruction loss alone never evaluates the model from novel views.ReconFusion adds a diffusion-derived regularization loss to address this limitation.
3D reconstruction with diffusion priors: At each iteration, ReconFusion samples a perturbed novel camera pose, renders the current model, and uses intermediate-noise diffusion sampling to generate a target image for supervision.The decoded diffusion sample is used to supervise the rendering, with an L1 image term and LPIPS perceptual distance weighted by noise level.
Novel view selection: Novel-view poses are sampled from paths fitted to the capture pattern, using elliptical paths for structured captures and B-splines for more unstructured trajectories.Each path pose is perturbed in position, up vector, and look-at point within a specified range.

4. Experiments

ReconFusion is evaluated across diverse real-world datasets, baseline comparisons, ablations, and capture densities. The experiments show improved reconstruction quality, reduced artifacts, and benefits from the proposed diffusion-prior design choices.

Experiment Setup: The evaluation covers five real-world datasets with 3, 6, and 9 input views, including in-distribution and out-of-distribution forward-facing and 360-degree scenes.Baselines include Zip-NeRF, DiffusioNeRF, FreeNeRF, SimpleNeRF, ZeroNVS*, and SparseFusion on CO3D.
Comparison Results: Qualitative comparisons show higher appearance and geometry quality than baselines, which exhibit floaters, color artifacts, blurry geometry, and misplaced textures.These failures are especially visible in minimally observed regions and depth maps.
Comparison Results: ReconFusion outperforms all baselines across in-distribution and out-of-distribution datasets, achieving state-of-the-art few-view NeRF reconstruction performance.Few-view regularizers help on forward-facing scenes but fall short on 360-degree scenes, while ZeroNVS often reconstructs complete scenes with limited visual fidelity.
Ablation Studies: The ablations evaluate pretrained diffusion weights and conditioning, comparing pose conditioning and PixelNeRF conditioning across in-domain and out-of-domain datasets.The full model uses pretrained weights and PixelNeRF conditioning; the alternatives include training from scratch and a ZeroNVS-like pose mechanism.
Ablation Studies: Standard SDS produces more artifacts, whereas the multistep diffusion loss mitigates them; annealing tmin further improves background detail.The SDS comparison specifically shows blotchy texture, while annealing improves details in the background.
Scaling to More Views: The diffusion prior consistently outperforms Zip-NeRF across capture densities, including up to 81 input views on the kitchenlego scene.The diffusion-loss weighting is set inversely proportional to input-view count as additional views reduce ambiguity.

5. Discussion

ReconFusion combines optimization-based 3D reconstruction with multiview-conditioned diffusion to reduce capture requirements, while exposing important computational and reconstruction limitations.

5. Discussion: ReconFusion combines a multiview-consistent 3D reconstruction pipeline with a multiview-conditioned diffusion model that guides reconstruction in underconstrained regions.The diffusion model generates plausible novel views, while the 3D representation maintains multiview consistency.
5. Discussion: The authors identify high diffusion-model cost, limited 3D outpainting, and tedious reconstruction-loss balancing as current limitations.
5. Discussion: The system produces compelling results across varied scene types using significantly fewer views.
5. Discussion: The authors view ReconFusion as a possible template for sparse reconstruction with dramatically reduced capture requirements.

A. Diffusion Model Details

The diffusion model is adapted from a pretrained text-to-image latent diffusion model and augmented with image and PixelNeRF conditioning for novel-view synthesis.

A. Diffusion Model Details: The model starts from a pretrained text-to-image latent diffusion model and modifies cross-attention to use conditioning derived from input images.The modified model maps 512 × 512 × 3 inputs into a 64 × 64 × 8 latent representation.
A. Diffusion Model Details: PixelNeRF outputs are concatenated with the input noise to provide additional spatial and feature conditioning to the diffusion U-Net.
A. Diffusion Model Details: The model independently drops CLIP and PixelNeRF conditioning during training to enable separate classifier-free guidance pathways.The model uses 10% conditioning dropout, 250,000 training iterations, a 10^-4 learning rate, and batch size 128.

B. PixelNeRF Details

PixelNeRF converts posed input images into target-view RGB and feature conditioning for diffusion, while additional visualizations examine how view count and conditioning affect generated samples.

B. PixelNeRF Details: PixelNeRF takes posed input images and a target camera pose, producing an approximate target-view rendering with RGB and feature channels.Its output is 64 × 64 × 131, matching the diffusion model’s latent resolution.
B. PixelNeRF Details: PixelNeRF extracts 128-channel image features, reprojects sampled target rays into input cameras, and aggregates the gathered features across inputs.
B. PixelNeRF Details: Increasing known scene observations decreases the expected distance to the nearest training view and increases diffusion-sample fidelity.The diffusion model is given the three nearest images to the novel viewpoint in each case.
B. PixelNeRF Details: PixelNeRF RGB outputs are visualized alongside corresponding diffusion samples conditioned on those outputs.

C. Dataset Details

The dataset protocol uses established or designed train/test splits, including uniformly distributed views for mip-NeRF 360 scenes.

C. Dataset Details: For mip-NeRF 360, the authors sample 106 nine-view splits and select the split minimizing a heuristic for uniform hemisphere coverage toward the central object.

D. Baselines Details

The evaluation uses adjusted baselines and specified procedures to compare ReconFusion against competing few-view reconstruction methods. The Zip-NeRF baseline disables only the diffusion regularizer, while other baselines use author-provided results, code, or retrained models.

Zip-NeRF uses modified hyperparameters intended to improve few-view reconstruction, including distortion loss, weight decay, and a smaller view-dependence network.The setup also removes hexagonal spiral control points and downweights density outside the unit sphere.
ReconFusion and Zip-NeRF share identical hyperparameters, with their only difference being the diffusion regularizer weight.The diffusion regularizer weight is set to 0 for results labeled “Zip-NeRF.”
The adjusted Zip-NeRF baseline performs competitively on LLFF and RealEstate10K but often produces floaters or fails to reconstruct meaningful geometry on harder datasets.The authors frame this model as a baseline with the diffusion regularizer disabled rather than a state-of-the-art few-view method.
RegNeRF and FreeNeRF use shared author results for LLFF and DTU, while FreeNeRF code runs on the other three datasets.These procedures combine published outputs with code-based evaluation across datasets.
DiffusioNeRF uses author results on DTU and code on the other four datasets, while SimpleNeRF is evaluated with author code on all five datasets.SimpleNeRF required rerunning because its LLFF and DTU train/test splits differed.
SparseFusion is retrained on unmasked CO3D images category-specifically before its 3D distillation pipeline produces final renderings.The original model was trained using masked foreground-object images.
ZeroNVS is adapted for multiview evaluation by conditioning on the input view closest to each sampled random view.New viewpoints are sampled using dataset-specific strategies, including spline and circle paths fitted to input views.