PaperlayerView exact revision on arXiv

Source-linked AI summary

NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion

Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, Ravi Ramamoorthi

arXiv:2302.10109v1cs.CVcs.LG

TL;DR

Single-image novel view synthesis must infer unseen regions while maintaining consistency, but projection-based NeRFs become blurry under severe occlusion and 2D diffusion views can be inconsistent. NerfDiff distills a 3D-aware conditional diffusion model into an image-conditioned NeRF through NeRF-guided virtual-view generation and test-time finetuning, achieving state-of-the-art results on multiple challenging benchmarks.

  • Problem

    Single-image NeRFs cannot resolve multiple plausible occluded-region completions, while 2D diffusion models produce only partially 3D-consistent views.

  • Method

    NerfDiff jointly learns an image-conditioned triplane NeRF and 3D-aware CDM, then alternates NeRF finetuning with diffusion-guided generation of virtual views.

  • Results

    NerfDiff achieves state-of-the-art results on multiple challenging benchmarks and significantly outperforms existing baselines in reported evaluations.

  • Takeaways & Limitations

    NeRF-guided distillation improves multi-view consistency while enabling high-quality completions of unseen regions in single-image view synthesis.

  • Takeaways & Limitations

    The method requires at least two scene views during training, and its finetuning process is time-expensive for real-time applications.

Abstract

from arXiv · show

Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input. Existing approaches condition neural radiance fields (NeRF) on local image features, projecting points to the input image plane, and aggregating 2D features to perform volume rendering. However, under severe occlusion, this projection fails to resolve uncertainty, resulting in blurry renderings that lack details. In this work, we propose NerfDiff, which addresses this issue by distilling the knowledge of a 3D-aware conditional diffusion model (CDM) into NeRF through synthesizing and refining a set of virtual views at test time. We further propose a novel NeRF-guided distillation algorithm that simultaneously generates 3D consistent virtual views from the CDM samples, and finetunes the NeRF based on the improved virtual views. Our approach significantly outperforms existing NeRF-based and geometry-free approaches on challenging datasets, including ShapeNet, ABO, and Clevr3D.

1. Introduction

Single-image novel view synthesis must infer occluded and unseen regions while preserving consistency with the input. NerfDiff addresses projection uncertainty by combining an image-conditioned NeRF with a 3D-aware diffusion model and NeRF-guided distillation.

  • Motivation: Novel view synthesis supports virtual and augmented reality, immersive photography, and digital replicas, but must account for occlusions and unseen regions.The task seeks new views from other viewing directions given a few input views.
  • Problem: Pixel-aligned single-image NeRFs perform well near the input camera but become blurry for target views far from the input.Projection to the input image cannot resolve uncertainty in large unseen regions.
  • Related approaches: 2D generative models can predict novel views conditioned on the input view but synthesize only partially 3D-consistent images.
  • NerfDiff: NerfDiff jointly trains a camera-space triplane NeRF and a 3D-aware conditional diffusion model, then uses diffusion-predicted virtual images to finetune the NeRF.NeRF-guided distillation alternates NeRF updates with guidance of the multi-view diffusion process.
  • Results: NerfDiff significantly outperforms existing baselines across three challenging benchmarks while generating high-quality, multi-view-consistent images.
  • Contributions: NerfDiff jointly learns an image-conditioned NeRF and CDM, uses camera-aligned triplanes, and integrates volume rendering into 2D diffusion.These components support efficient rendering, finetuning, and generalization to novel views.

2. Related Work

Prior work spans diffusion models, direct 3D generation, single-view reconstruction, and neural-field methods. These approaches face requirements for 3D supervision, limited generalization, camera-pose dependence, or difficulty capturing details under occlusion.

  • Diffusion models: Diffusion models have become state-of-the-art for image synthesis and have been applied to under-constrained text-to-image and text-to-video generation.
  • 3D diffusion: Diffusion methods operating directly in 3D representations require 3D ground truth to learn the diffusion process, which is difficult to acquire in real environments.
  • Single-view reconstruction: Early single-view 3D reconstruction methods estimated object geometry from images using depth or direct 3D-shape prediction, often with ground-truth training data.
  • NerfDiff: NerfDiff combines a single-image NeRF and a 2D conditional diffusion model in a training and finetuning pipeline.The learned parameters initialize an NeRF that is subsequently supervised by NeRF-guided denoised images from the frozen CDM.
  • Neural fields: Neural-field methods use pseudo geometry, global latent codes, or test-time tuning, but may require estimated camera poses and struggle with fine details or occlusions.

3. Background

Single-image NeRF view synthesis is under-constrained: projection-based methods struggle with unseen regions, while diffusion models address uncertainty but lack full multi-view consistency. NerfDiff combines a triplane-conditioned NeRF with a 3D-aware diffusion process to improve this trade-off.

  • Neural radiance fields: NeRF represents scenes as an implicit function mapping spatial location and ray direction to color and density for volume rendering.Rendering marches camera rays through sampled points and computes pixel colors using volume rendering.
  • Neural radiance fields: Triplane encodings capture local structure while providing faster inference without quality loss compared with sinusoidal positional encodings.The representation encodes spatial location and ray direction to capture high-frequency details.
  • Image-conditioned NeRF: Single-image NeRFs use projected image features to generalize across scenes, but this projection cannot resolve uncertainty in large unseen regions and produces blurry distant views.These approaches perform well for cameras near the input but degrade when target views are far away.
  • NerfDiff overview: NerfDiff uses a camera-aligned triplane NeRF to produce an initial target-view rendering that conditions a diffusion process for consistent denoising.The architecture maps the input image to a triplane-based representation, renders from the target view, and feeds that rendering to the CDM.
  • Image-conditioned NeRF: Under severe occlusion, MSE averages across multiple possible solutions, yielding inaccurate and blurry predictions.The single-image reconstruction problem is under-constrained because occluded regions can have multiple valid appearances.
  • Geometry-free view synthesis: Conditional diffusion models iteratively denoise target-view images to model view uncertainty, but geometry-free methods suffer from alignment and multi-view inconsistency.Independent 2D denoising can produce views that are not mutually consistent, especially for complex scenes and out-of-distribution cameras.

4. NerfDiff

NerfDiff combines an image-conditioned NeRF, a 3D-aware conditional diffusion model, and test-time NeRF-guided distillation to synthesize sharper, multi-view-consistent novel views from one image.

  • 4. NerfDiff: NerfDiff uses a two-stage training-finetuning approach that incorporates diffusion models into image-conditioned NeRFs for single-image view synthesis.The components are jointly optimized during training, followed by test-time finetuning with NeRF-Guided Distillation.
  • 4.1. Single-image NeRF with Local Triplanes: Camera-aligned triplanes provide an efficient 3D representation extracted from the input image, assigning unique features to 3D points without additional positional encoding.Depth information in the orthogonal planes permits a shallow MLP, efficient high-resolution rendering, and fast NeRF finetuning.
  • 4.2. 3D-aware CDM: The 3D-aware CDM iteratively refines single-image NeRF renderings to resolve uncertainty and produce sharper novel views.It conditions diffusion on target-view images rendered by the NeRF rather than directly on the input-view image, addressing the alignment problem described for geometry-free methods.
  • 4.5. Experiments: NerfDiff is evaluated on ShapeNet Cars and Chairs, ABO, and Clevr3D, with comparisons spanning NeRF-based, geometry-free, and internal variants.The reported evaluation covers single-category, category-agnostic, and multi-object settings; Table 1 compares ShapeNet and ABO baselines, while Table 2 reports Clevr3D results.
  • 4.4. Fine-tuning Phase: NGD alternates diffusion sampling with NeRF distillation, using NeRF-based multi-view constraints to generate improved virtual views and finetune NeRF parameters.The procedure optimizes the NeRF toward denoised images across virtual views, avoiding conflicts caused by inconsistent independent CDM predictions.
  • 4.4. Fine-tuning Phase: Unlike SDS, NerfDiff completes the diffusion trajectory for each virtual view while alternating diffusion refinement with NeRF updates under a decreasing noise schedule.This distinction reflects NerfDiff’s use of NeRF to guide each view’s diffusion process rather than only denoising randomly noised NeRF outputs.

5. Experiments

NerfDiff is evaluated across three benchmarks and consistently improves detail, perceptual quality, and multi-view consistency over established baselines and ablations.

  • Experimental Settings: NerfDiff is evaluated on SRN-ShapeNet, ABO, and Clevr3D, covering single-category, category-agnostic, and multi-object settings.
  • Quantitative Evaluation: NerfDiff significantly outperforms existing baselines in PSNR and SSIM across the three datasets.It also surpasses previous approaches in LPIPS, indicating stronger perceptual completions behind occlusions.
  • Quantitative Evaluation: NGD finetuning improves sharpness and FID and LPIPS scores, although it slightly reduces PSNR in some cases.Larger model variants further improve perceptual quality.
  • Qualitative Evaluation: On ShapeNet and ABO, NerfDiff produces more detailed renderings than VisionNeRF and the single-image NeRF ablation, which regress blurry occluded regions.The CDM-only results are less aligned and inconsistent across views.
  • Qualitative Evaluation: On Clevr3D, NerfDiff provides consistent, high-quality renderings, whereas VisionNeRF overfits the camera distribution and fails on viewpoints close to the input.
  • Ablation Study: The ablation shows that only NGD sampling achieves state-of-the-art conditional SSIM and LPIPS among the tested finetuning strategies.CDM finetuning improves unconditional FID.

6. Discussion

The paper identifies two practical limitations: training requires multiple scene views, and test-time finetuning is too time-consuming for real-time applications.

  • Limitations: Training requires at least two views of a scene.
  • Limitations: The finetuning process is expensive in time, limiting application in real-time domains.
  • Future Work: Future work may address these limitations and explore NGD for text-to-3D, more complex datasets, and 3D GANs.

7. Conclusion

NerfDiff distills a 3D-aware conditional diffusion model into a triplane-based image-conditioned NeRF, using NeRF-guided distillation to improve virtual-view synthesis and NeRF renderings.

  • Conclusion: NerfDiff distills a 3D-aware CDM into a triplane-based image-conditioned NeRF for single-image view synthesis.
  • Conclusion: NeRF-guided distillation samples multiple views from the CDM while simultaneously improving NeRF renderings.
  • Conclusion: NerfDiff achieves state-of-the-art results on multiple challenging benchmarks.

A. Datasets

The evaluation uses three datasets spanning object categories, category-agnostic objects, and multi-object scenes, with distinct scene counts, views, and camera configurations.

  • ShapeNet: SRN-ShapeNet contains Cars and Chairs categories, with 3,514 car scenes and 6,591 chair scenes.Training scenes contain 50 posed images, while testing scenes contain 250 posed images.
  • ABO: ABO contains 6,743 training, 396 validation, and 794 testing scenes across 64 object categories.Each scene has 30 physically based renders on a white background.
  • Clevr3D: Clevr3D uses 70,000 training scenes and 1,000 held-out test scenes for multi-object learning.Each scene has three posed images rendered at 120-degree rotations around the z-axis.
  • Implementation: NerfDiff uses base and large configurations of approximately 400 million and 1 billion parameters, respectively.
  • Implementation: The image encoder reshapes its final output into a camera-aligned triplane with the input image’s spatial resolution and feature dimension 48.

B.2. Training phase

The training and fine-tuning setup uses dataset-specific virtual-camera sampling, diffusion refinement, and iterative NeRF updates. Clevr3D additionally requires noisy camera locations to avoid degenerate geometry.

  • Training setup: Base models train with AdamW for 500K iterations on 8 A100 GPUs using batch size 32 and a 2e−5 learning rate.The cosine noise schedule uses αt = cos(0.5πt), with λIC = λDM = 1 and EMA decay 0.9999.
  • Clevr3D stabilization: Clevr3D training adds camera-location noise with variance 0.3 to prevent input-view overfitting that creates a degenerate density plane.The adjustment fixed the issue for NerfDiff but not for VisionNeRF, even after increasing the noise.
  • Virtual-view sampling: NerfDiff samples K virtual views near the test trajectory, using K = 50 for ShapeNet and Clevr3D and K = 30 for ABO by default.The sampling approximates the camera distribution Π relative to the input image.
  • Dataset-specific cameras: ShapeNet samples every fifth camera from its 251-view Archimedean spiral, while ABO uses all 30 testing cameras.These procedures yield K = 50 and K = 30 cameras, respectively.
  • Dataset-specific cameras: Clevr3D constructs virtual-camera look-at points by intersecting relative camera optical axes because each scene has only three cameras.Its cameras also have similar heights relative to the ground plane.

D. Details of Baseline Methods

The baseline comparison assembles published or reproduced results for ShapeNet, while Table 6 isolates the contribution of 3D-aware diffusion through alternative CDM architectures.

  • Baseline sources: ShapeNet baseline metrics combine results gathered from prior papers with FID calculations using renderings supplied by PixelNeRF and VisionNeRF authors.The listed baselines include LFN, 3DiM, SRN, PixelNeRF, CodeNeRF, FE-NVS, and VisionNeRF.
  • CDM architectures: Table 6 compares Concat and Cross-attention conditioning against a 3D-aware CDM architecture.Concat directly concatenates the input and noisy images, whereas Cross-attention follows X-UNet-style conditioning.
  • CDM architectures: The 3D-aware diffusion model consistently achieves better results and more coherent views than the 3D-unaware alternatives.The compared alternatives omit volume rendering during encoding.

E.2. Fine-tuning Strategies

The fine-tuning study contrasts direct distillation, SDS, stochastic conditioning, and NerfDiff’s NeRF-guided strategy. The alternatives can suffer from inconsistent supervision, noisy artifacts, or accumulated errors.

  • Direct distillation: Direct distillation independently samples CDM virtual views from initial pixelNeRF renderings and uses an L2 loss to fine-tune NeRF.Independent denoising makes the virtual views unlikely to be multiview consistent, producing inconsistent renderings and NeRF learning conflicts.
  • Score distillation sampling: SDS adds random noise to NeRF renderings and denoises them, but inconsistent noise levels can yield blurry results that regress the mean of supervision.NerfDiff instead samples once per view while progressively increasing NeRF guidance.
  • NeRF guidance: NerfDiff uses NeRF to guide CDM sampling rather than directly seeking a mode of the learned score distribution.This design addresses the possibility that score-function modes do not correspond to sharp dataset images.
  • NeRF guidance: NerfDiff refines NeRF during sampling, keeping CDM inputs close to the data manifold instead of producing out-of-distribution low-noise denoiser inputs.The contrast is with SDS, where noisy NeRF renderings may not resemble real images with comparable noise.
  • Stochastic conditioning: Naive stochastic conditioning can generate noisy floating artifacts because inconsistent CDM predictions conflict during NeRF learning.Its autoregressive dependencies do not guarantee multiview consistency, and errors can accumulate over long sequences without stable geometry.

F. Additional Qualitative Results

Additional qualitative results compare single-image novel-view synthesis with VisionNeRF across ShapeNet Cars, ShapeNet Chairs, and ABO scenes at specified viewpoints.

  • Qualitative comparisons: The supplementary qualitative comparisons render views from single-image inputs on ShapeNet Cars, ShapeNet Chairs, and ABO.The examples are presented in Figures 7–10, with additional video results referenced in the supplementary materials.
Loading 2302.10109v1…