PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffRF: Rendering-Guided 3D Radiance Field Diffusion

Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner

arXiv:2212.01206v2cs.CV

TL;DR

Generalizable radiance-field generation remains limited by photorealism, geometric accuracy, and ambiguous training representations. DiffRF directly denoises voxel-grid radiance fields with a rendering loss, learning multi-view-consistent priors that support high-quality unconditional and conditional 3D synthesis. On PhotoShape Chairs, it improves FID from 16.54 to 15.95 and MMD from 5.62 to 4.42 over GAN-based approaches.

  • Problem

    Generalizable neural field methods still have significant gaps in photorealism and geometric accuracy, while posed-image radiance fields are ambiguous and may contain fitting artifacts.

  • Method

    DiffRF directly applies a 3D denoising diffusion model to explicit voxel-grid radiance fields and adds volumetric rendering loss to favor image quality over fitting artifacts.

  • Results

    15.95 FID versus 16.54 and 4.42 MMD versus 5.62 improve image quality and geometry synthesis over GAN-based approaches on PhotoShape Chairs.

  • Takeaways & Limitations

    DiffRF learns multi-view-consistent priors enabling free-view synthesis, accurate shape generation, and conditional masked completion without task-specific model adaptation.

  • Takeaways & Limitations

    DiffRF requires sufficient posed views for good training samples, has lower sampling times than GAN-based approaches, and is constrained in maximum grid resolution by training-time memory.

Abstract

from arXiv · show

We introduce DiffRF, a novel approach for 3D radiance field synthesis based on denoising diffusion probabilistic models. While existing diffusion-based methods operate on images, latent codes, or point cloud data, we are the first to directly generate volumetric radiance fields. To this end, we propose a 3D denoising model which directly operates on an explicit voxel grid representation. However, as radiance fields generated from a set of posed images can be ambiguous and contain artifacts, obtaining ground truth radiance field samples is non-trivial. We address this challenge by pairing the denoising formulation with a rendering loss, enabling our model to learn a deviated prior that favours good image quality instead of trying to replicate fitting errors like floating artifacts. In contrast to 2D-diffusion models, our model learns multi-view consistent priors, enabling free-view synthesis and accurate shape generation. Compared to 3D GANs, our diffusion-based approach naturally enables conditional generation such as masked completion or single-view 3D synthesis at inference time.

1. Introduction

DiffRF addresses the limited generalization and geometric accuracy of neural radiance field generation by directly diffusing over volumetric radiance fields. A rendering-guided objective favors realistic, multi-view-consistent synthesis and supports conditional 3D generation.

  • Existing NeRF extensions primarily fit individual scenes, limiting generalization beyond their input training data.
  • Generalizable neural field methods remain limited in photorealism and geometric accuracy across object categories and datasets.
  • DiffRF introduces the first diffusion-based generative model that directly synthesizes 3D radiance fields from collections of posed RGB images.
  • Its 3D denoising model operates on an explicit voxel grid and adds volumetric rendering loss to reduce fitting artifacts and noise accumulation.
  • DiffRF learns multi-view-consistent priors for free-view synthesis, accurate shapes, and inference-time masked completion without task-specific adaptation.
  • 15.95 FID versus 16.54 and 4.42 MMD versus 5.62 improve image quality and geometry synthesis over GAN-based approaches on PhotoShape Chairs.

2. Related work

Prior 3D generative work includes GANs, point-cloud diffusion, and radiance-field methods driven by 2D diffusion or latent representations. DiffRF differs by directly applying denoising diffusion to volumetric radiance fields.

  • Diffusion models have achieved strong results in 2D synthesis, including high-resolution and conditional image generation, but are less explored for 3D synthesis.
  • Earlier 3D adversarial methods generate meshes, textures, voxels, or neural radiance fields, often using collections of 2D images without 3D supervision.
  • Point-cloud diffusion methods support conditional synthesis and completion, while radiance-field approaches use pretrained 2D diffusion losses or latent-space diffusion.
  • DiffRF directly synthesizes volumetric radiance fields rather than generating point clouds or diffusing over factorized or learned latent representations.

3. Method

DiffRF generates 3D object radiance fields by reversing a diffusion process over explicit voxel-grid tensors. It combines radiance-field denoising and rendering objectives so generated fields better match observed images and avoid rendering artifacts.

  • 3.1. Radiance Fields: Radiance fields represent 3D objects with density and RGB color fields, rendered into novel views by casting rays from a camera.The method uses an explicit voxel grid, queried continuously by bilinear interpolation, yielding a 4D tensor of spatial and density/color channels.
  • 3.2. Generating Radiance Fields: The model samples a noisy radiance field at a diffusion time step and uses a time-conditioned 3D-UNet to predict the applied noise for iterative denoising.Generation starts from a standard multivariate normal radiance field and iteratively produces the final object field through learned Gaussian reverse transitions.
  • 3.1. Radiance Fields: DiffRF uses explicit voxel-grid radiance fields because they provide good rendering quality with faster training and inference than the alternatives considered.The grid is queried at continuous positions through bilinear interpolation of voxel vertices.
  • 3.2. Generating Radiance Fields: The forward diffusion process gradually corrupts data radiance fields with scheduled Gaussian noise, while the reverse process learns to recover radiance fields from random states.The noise variance schedule uses predefined β_t coefficients, with α_t := 1 − β_t.
  • 3.3. Training Objective: Training combines a radiance-field generation loss with an RGB rendering loss because representation-space similarity alone may not prevent rendered artifacts.The rendering loss compares views rendered from denoised fields with the ground-truth images used to obtain the original fields, using a computationally simpler approximation.
  • 3.3. Training Objective: The radiance-field loss follows a variational upper bound on negative log-likelihood, while the rendering loss biases denoising toward higher-quality images.The denoised estimate is formed from the noisy field and the difference between actual and predicted noise.

4. Experiments

Experiments evaluate DiffRF on unconditional and conditional radiance-field generation using PhotoShape Chairs and ABO Tables, with image, geometry, diversity, and completion measures. DiffRF improves quality and diversity over GAN-based methods, while rendering-guided denoising supports masked completion and single-view synthesis.

  • Experimental setup: Experiments cover unconditional and conditional radiance-field generation on PhotoShape Chairs and ABO Tables, using FID, KID, Coverage Score, and MMD.The datasets contain 15,576 chairs and 1,676 tables, represented at voxel resolution 32^3.
  • Unconditional synthesis: DiffRF yields better image quality with significant improvements in geometric quality and diversity than π-GAN and EG3D.EG3D can produce view-dependent artifacts and inaccurate shapes, whereas DiffRF generates fine photometric and geometric details.
  • Rendering-loss ablation: Removing the volumetric rendering loss increases FID by approximately 2.3 on PhotoShape and 8.8 on ABO Tables.The ablation indicates that 2D rendering supervision improves image quality, while its guidance can reduce Coverage Score by suppressing spurious shapes.
  • Unconditional synthesis: Table 1 reports better image and geometric quality than state-of-the-art GAN-based approaches on PhotoShape Chairs, with rendering loss improving image quality.MMD and KID scores in the table are multiplied by 10^3.
  • Unconditional synthesis: Table 2 reports better image and geometric quality than GAN-based approaches on ABO Tables, with the 2D rendering loss being important.MMD and KID scores in the table are multiplied by 10^3.
  • Conditional generation: For masked radiance-field completion, DiffRF maintains unmasked regions and synthesizes coherent masked-region completions, unlike EG3D’s weaker reconstruction of known regions.The task evaluates FID from 10 random views and unmasked-region photometric accuracy using mPSNR at different masking levels.
  • Conditional generation: Masked completion fuses known input regions with iterative denoising estimates, while single-view synthesis steers denoising using volumetric rendering against a posed RGB image and object mask.Qualitative results show diverse masked-chair proposals and meaningful single-view reconstructions on ScanNet chairs.
  • Limitations: DiffRF requires sufficient posed views for good training samples, has lower sampling times than GAN approaches, and is constrained in maximum grid resolution.The authors suggest faster sampling methods as a future direction.

5. Conclusions

DiffRF is presented as a diffusion-based approach that directly synthesizes volumetric radiance fields and learns multi-view consistent priors from posed images. It supports free-view synthesis, accurate shape generation, and conditional and unconditional 3D generation.

  • DiffRF directly operates on volumetric radiance fields using denoising diffusion probabilistic models.
  • The model learns multi-view consistent priors from collections of posed images, enabling free-view image synthesis and accurate shape generation.
  • DiffRF is evaluated across several object classes against state-of-the-art GAN-based approaches for conditional and unconditional 3D generation.

Appendix

The supplementary document provides additional methodological, training-data, evaluation, and qualitative-result details. It also presents further qualitative results beyond the main paper.

  • The supplement discusses additional details about the method.
  • It covers the data used for training and evaluation.
  • It includes further qualitative results and refers to a comprehensive overview.

A. Additional qualitative results

Additional qualitative results are reported for PhotoShape Chairs and ABO Tables, alongside comparisons showing the effect of removing the rendering loss.

  • Additional qualitative results are provided on PhotoShape Chairs.
  • Additional qualitative results are provided on ABO Tables.
  • The supplement qualitatively compares DiffRF with and without the rendering loss.

B. Implementation detail

DiffRF uses a 3D U-Net-style architecture, rendering supervision during diffusion training, and explicit voxel grids. Sampling uses 1000 DDPM iterations, while the explicit representation supports fast rendering afterward.

  • Architecture: The architecture replaces 2D convolutions and pooling in a U-Net with corresponding 3D operations.The U-Net has four scaling blocks, two ResNet blocks per scale, and feature channels increasing from 64 to 256.
  • Training details: Training uses batch size 8, Adam optimization, linear beta scheduling, and rendering supervision from four random 128 × 128 views.The rendering supervision samples 8192 random pixels with 92 z-steps for volumetric rendering.
  • Training details: Models train for 3.0m iterations with learning rates decaying from 10^-4 to 10^-6 at voxel grid resolution 32 on two GPUs.
  • Sampling time: 1000 DDPM sampling iterations require 48.6s per sample, while the synthesized explicit representation renders at over 380 FPS at 128 × 128 resolution.

C. Data

The experiments construct radiance-field datasets for PhotoShape Chairs and ABO Tables, then evaluate image quality, geometric quality, and masked completion with defined protocols.

  • Dataset construction: Both datasets begin with explicit 32^3 voxel grids fitted through volumetric rendering and spherical harmonics, then fine-tune degree-0 features for sharper geometry.The feature space is also bounded to [−1, 1].
  • Evaluation: Image quality is measured with FID and IS from 10k rendered views generated by sampling 1,000 objects from 10 random views at 128 × 128 resolution.
  • Evaluation: Geometric quality uses Coverage Score and Minimum Matching Distance with Chamfer Distance on test and generated shape sets.Meshes are extracted with marching cubes and represented by 2,048 sampled surface points.
  • Evaluation: Masked completion is additionally evaluated with mPSNR, averaging PSNR over non-masked pixels across 10 randomly selected evaluation views.Non-masked image regions are obtained by depth-based projection of the input radiance-field mask.

D. Conditional sampling

Conditional sampling supports masked radiance-field completion and image-to-volume synthesis by steering diffusion sampling with rendered-image errors.

  • Masked completion: Masked completion optimizes a GAN latent code for 200 iterations using eight rendered views, projected masks, and photometric errors on non-masked regions.The latent update uses Adam with learning rate 10^-2 and L2 regularization weighted by 5 × 10^-2.
  • Qualitative results: Figure 8 provides additional qualitative sampling results for PhotoShape Chairs, while Figure 9 provides corresponding results for ABO Tables.
  • Image-to-volume synthesis: Image-to-volume synthesis conditions diffusion sampling on a posed, segmented image by progressively reducing the photometric error between rendered and foreground-masked target images.
  • Image-to-volume synthesis: The denoised radiance field is guided using the photometric-error gradient computed through volumetric rendering, with a small guidance weight λ.

E. CLIP conditioning

DiffRF extends conditioning to text and single-image embeddings from CLIP, using cross-attention during denoising without training the image encoder or text–radiance-field pairs.

  • CLIP conditioning: The model conditions on text or single-image CLIP ViT-B/32 embeddings through cross-attention layers adapted for the 3D U-Net.Training uses random single views encoded by the frozen CLIP model.
  • CLIP conditioning: The image encoder remains frozen to preserve CLIP’s image–text correspondence, with examples shown for PhotoShape and real-world Pix3D images.
  • Text conditioning: CLIP’s text correspondence enables text-prompt guidance during sampling without training on text–radiance-field pairs.Figure 12 illustrates text-conditional inference on PhotoShape Chairs.
  • Ablation: Figures 10 and 11 qualitatively compare results on PhotoShape Chairs and ABO Tables after removing the 2D rendering loss.
  • Image conditioning: Figures 13 and 14 show image-conditional inference on Pix3D and single-view PhotoShape Chairs, respectively.
Loading 2212.01206v2…