PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Novel View Synthesis with 3D-Aware Diffusion Models

Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, Gordon Wetzstein

arXiv:2304.02602v1cs.CVcs.AIcs.GR

TL;DR

Single-image novel view synthesis must generate plausible unseen content despite ambiguity while maintaining 3D consistency across distant views. The paper conditions 2D diffusion on a latent 3D feature field, enabling diverse novel views and autoregressive sequences; it reports state-of-the-art performance across synthetic, room-scale, and real-world settings.

  • Problem

    Existing regression methods blur ambiguous extrapolations, while generative methods without strong geometry priors struggle to maintain geometrically consistent sequences.

  • Method

    The method extends 2D diffusion with a latent 3D feature field conditioned on features extracted from input images.

  • Results

    The model generates realistic single-image novel views across object-level, room-level, and real-world datasets, including long multi-view-consistent trajectories without regression blurring or pure-generative drift.

  • Takeaways & Limitations

    Combining explicit geometry priors with diffusion provides realistic, 3D-aware novel views and supports long-range sequence synthesis across diverse scenes.

  • Takeaways & Limitations

    Output resolution is limited to 128^2, sampling is too slow for interactive visualization, and challenging real-world data can show minor inconsistencies and drift; training requires multi-view supervision.

Abstract

from arXiv · show

We present a diffusion-based model for 3D-aware generative novel view synthesis from as few as a single input image. Our model samples from the distribution of possible renderings consistent with the input and, even in the presence of ambiguity, is capable of rendering diverse and plausible novel views. To achieve this, our method makes use of existing 2D diffusion backbones but, crucially, incorporates geometry priors in the form of a 3D feature volume. This latent feature field captures the distribution over possible scene representations and improves our method's ability to generate view-consistent novel renderings. In addition to generating novel views, our method has the ability to autoregressively synthesize 3D-consistent sequences. We demonstrate state-of-the-art results on synthetic renderings and room-scale scenes; we also show compelling results for challenging, real-world objects.

1. Introduction

The paper targets single-image novel view synthesis that can handle ambiguous long-range extrapolation, individual objects, and complex scenes. It combines 2D diffusion with a latent 3D feature field to generate diverse, realistic, and view-consistent renderings and sequences.

  • Motivation: Single-image NVS must support long-range sequences, individual objects, and complex scenes while completing unseen content under ambiguity.Existing few-shot regression methods can preserve geometric consistency near input views but struggle with extrapolation and unbounded scenes.
  • Motivation: Regression objectives approximate ambiguous multi-modal renderings with a mean estimator, producing blurry views when scene regions are occluded.This limitation becomes pronounced when all input views hide parts of the scene.
  • Motivation: Generative approaches sample plausible renderings from the conditional distribution, but methods without strong geometry priors often lose coherent rigid scene structure across frames.Diffusion models handle ambiguity and autoregressive extrapolation, yet image diffusion alone struggles with 3D-consistent views.
  • Approach: A latent 3D feature field represents a distribution of possible scene representations, whose renderings are distilled into scene realizations during diffusion sampling.The formulation is designed to address long-range ambiguity while preserving geometrically consistent sequences.
  • Approach: The method extends 2D diffusion models by conditioning them on 3D neural features extracted from input images.This is the paper’s central 3D-aware diffusion formulation.
  • Results: The model generates realistic novel views from a single image across object-level, room-level, and challenging real-world datasets, including long multi-view-consistent trajectories.The authors report avoiding both regression-model blurring and pure-generative-model drift.

2. Related work

Prior NVS methods span geometry-based, regression-based, and generative approaches, with different trade-offs between reconstruction quality, ambiguity handling, extrapolation, and view consistency. The paper builds on 2D diffusion and 3D generative-model ideas while emphasizing stronger geometry priors.

  • Geometry-based NVS: Geometry-based NVS uses camera estimation, multi-view stereo, and neural rendering, but explicit geometry can fail to produce photorealistic complete views in occluded regions.Deep-learning methods combine recovered geometry with learned view synthesis, though quality can depend on the geometry pipeline.
  • Regression-based NVS: Regression-based few-shot methods generalize across 3D-scene classes from one or a few input images, often using 3D representations and differentiable neural rendering.Their primary setting is inference from sparse views across a scene category.
  • Generative models for NVS: Generative NVS methods target long-range extrapolation by sampling unseen scene content and autoregressively conditioning on previous generations.They commonly use weak or absent geometry priors and therefore achieve only approximate long-range view consistency.
  • 3D GANs: 3D GANs learn generative priors over 3D objects with implicit neural scene representations and can be inverted for detailed-scene NVS.Their typical task is unconditional 3D-object synthesis rather than conditional novel view synthesis.
  • 2D diffusion models: 2D diffusion models provide mode coverage and stable training while modeling conditional image distributions from labels, text, or other images.These properties motivate adapting diffusion backbones for NVS.
  • Recent 3D diffusion works: Recent 3D diffusion work applies 2D image diffusion to 3D generation or NVS, but 3DiM does not employ explicit geometry priors.The paper distinguishes its approach by incorporating geometry priors into diffusion-based NVS.

3. Method

The method conditions a 2D diffusion denoiser on geometry-aware feature renderings derived from one or more input views and camera parameters. It supports stochastic novel-view sampling, efficient inference, and autoregressive generation of view-consistent sequences.

  • 3.1. 3D-aware diffusion model architecture: Input-view features are unprojected into a latent 3D feature volume, aggregated across views, and rendered into a target-view feature image.A feature extractor T produces the volume, while an MLP decodes sampled features into density and feature vectors for volume rendering.
  • 3.1. 3D-aware diffusion model architecture: The U-Net denoiser receives the rendered feature image, a noisy target image, and the noise level to produce the denoised target view.The architecture uses a DDPM++ denoiser; the model contains 90M trainable parameters.
  • 3.1. 3D-aware diffusion model architecture: The model targets the conditional distribution of query views given input images and their camera parameters, rather than a single deterministic prediction.Diffusion sampling iteratively denoises an initial noisy image across a sequence of noise levels.
  • 3.2. Incorporating multiple views: Multiple conditioning views are processed into separate feature volumes and averaged, narrowing the distribution of possible renderings as additional information reduces uncertainty.Training samples one, two, or three conditioning views uniformly, and multi-view conditioning also supports loop-consistent video synthesis.
  • 3.1. 3D-aware diffusion model architecture: Inference uses deterministic second-order diffusion sampling with 25 or fewer denoising steps, while the rendered feature image is computed once and reused.This decoupling avoids rerunning the feature-rendering module at every sampling step.
  • 3.5. Autoregressive generation: Autoregressive synthesis conditions each new frame on input views and selected previously generated frames to balance local and long-range consistency.The reported default uses the most recent generated image plus five randomly selected earlier frames; the model effectively suppresses error accumulation in extended sequences.

4. Experiments

Experiments evaluate the method across synthetic objects, room-scale scenes, and unconstrained real-world objects, measuring image quality, view fidelity, and geometric consistency. Results show sharp, diverse, and geometrically consistent novel views, including long autoregressive trajectories and single-shot CO3D scenes.

  • Experimental Setup: The evaluation spans ShapeNet cars, Matterport3D rooms, and challenging CO3D hydrant scenes, with comparisons against geometry-based, transformer, and diffusion baselines.CO3D evaluation uses complete backgrounds and, according to the paper, is the first single-shot NVS attempt on this dataset without object masks.
  • Experimental Setup: The evaluation measures image distribution quality, similarity to ground-truth views, and geometric consistency of generated sequences.Metrics include FID and KID for distribution quality, LPIPS and DISTS for perceptual similarity, and Chamfer distance for geometric consistency.
  • 4.1. ShapeNet: Our method generates sharp realizations, whereas PixelNeRF predicts a blurry conditional mean; ViewFormer preserves fewer small input details such as headlight shape.The comparison is qualitative on ShapeNet single-view novel view synthesis.
  • 4.1. ShapeNet: The method produces sharp, diverse outputs with stronger FID, LPIPS, and DISTS performance than regression baselines, while PixelNeRF remains stronger on PSNR and SSIM.The one-step denoised prediction matches PixelNeRF’s state-of-the-art PSNR and SSIM, while autoregressive conditioning uses 90M parameters and 25 diffusion steps versus 471M and 512 steps for the compared model.
  • 4.1. ShapeNet: Sampling produces multiple plausible views whose variation increases in unseen or occluded regions, unlike deterministic regression outputs.Pixel-wise standard deviation is computed over 50 samples, with darker pixels indicating higher ambiguity.
  • 4.2. Matterport3D: Autoregressive generation yields geometrically consistent sequences, supported by dense point-cloud reconstructions and Chamfer-distance comparisons to ground truth.On Matterport3D, the approach outperforms baselines in prediction quality and consistency, and cyclic trajectories show better loop closure than LOTR.
  • 4.3. Common Objects in 3D (CO3D): On CO3D, the method produces plausible sharp foregrounds and backgrounds that do not deteriorate with increasing distance from the source pose.The paper reports quantitative comparison against PixelNeRF and uses real-world hydrant videos with complete backgrounds.
  • 4.4. Ablation Studies: A geometry-free intermediate representation performs worse than variants using geometry priors, while the explicit representation is slightly faster but generally less effective.One-step inference achieves strong PSNR and SSIM but lacks plausible results far from the input; autoregressive conditioning is necessary for coherent multiview sequences despite a slight image-quality trade-off.

5. Discussion

The paper combines geometry-based priors with diffusion models for realistic, 3D-aware novel views from a single image, while acknowledging practical limitations in resolution, speed, consistency, supervision, and ethical use.

  • The hybrid method combines explicit 3D representations with diffusion models for realistic, 3D-aware novel views.
  • The approach demonstrates state-of-the-art performance on object-scale and room-scale scenes, with results on challenging real-world CO3D data.
  • Output resolution is limited to 128^2, and diffusion-based sampling is not fast enough for interactive visualization.
  • The method can exhibit minor inconsistencies and drift on challenging real-world datasets.
  • Inference can use a single view, but training requires multi-view supervision with accurate camera poses.
  • The authors caution that extending diffusion models for DeepFakes could create societal threats and oppose misinformation-oriented use.

Supplementary Material

The supplementary material adds experiments, implementation and training details, experimental details, artifact and limitation discussion, and supplemental videos with further visual results.

  • The supplement begins with additional experiments and then details the model architecture, training process, and hyperparameters.
  • It follows with experimental details and a discussion of artifacts and limitations as possible future-work targets.
  • Supplemental videos provide additional visual results for readers to inspect.

A. Additional experiments & ablations

Additional experiments examine out-of-distribution camera poses, finding that performance deteriorates with large deviations while the 3D prior tolerates small extrapolations.

  • ShapeNet training uses cameras placed on a sphere, aimed at object centers, with a shared up direction.
  • The out-of-distribution pose test generates zooms, one-dimensional camera translations, and camera rolls from a fixed pose.
  • Novel views deteriorate with large deviations from the training pose distribution, but the 3D prior reasonably tolerates small extrapolations.

A.2. Percentile results based on LPIPS

The supplementary experiments use LPIPS percentiles to order synthesized views and analyze how input-view count and camera distance affect uncertainty, diversity, and consistency.

  • A.2. Percentile results based on LPIPS: Synthesized ShapeNet results are ordered by LPIPS percentile, with the best-scoring examples at the top and worst-scoring examples at the bottom.
  • A.2. Percentile results based on LPIPS: Nine realizations per input are evaluated, and the median image and score are used for ordering to reduce randomness effects.
  • Effect of varying the number of input views: With one rear-view input, unseen fronts admit multiple plausible completions and produce high standard deviation.
  • Effect of varying the number of input views: Adding two or three input views reduces uncertainty and yields novel views compatible with multiple inputs.
  • Input-view distance and output variance: Nearby input views reduce output variance because they provide more valuable information than distant views.
  • Input-view distance and output variance: Autoregressive conditioning on nearby views narrows the output distribution and improves geometric consistency over non-autoregressive conditioning.

A.5. Classifier-free guidance

Classifier-free guidance trades diversity against sample quality in independent novel-view synthesis, but can amplify artifacts and error accumulation in autoregressive sequences. The method therefore uses moderate guidance for isolated views and defaults to CFG=0 for long sequences.

  • Independent predictions: CFG increases conditioning strength, improving independent-view sample quality while narrowing diversity.At guidance=0, views vary more; higher guidance improves fidelity but reduces variation.
  • Independent predictions: Excessively high guidance introduces saturation, color artifacts, and visual artifacts.
  • Independent predictions: Moderate CFG benefits independent novel-view predictions, whereas negative guidance produces unconditional samples without input-image influence.
  • Autoregressive generation: In autoregressive generation, stronger guidance carries visual errors forward, causing saturation and artifact accumulation over long sequences.
  • Autoregressive conditioning: The default autoregressive strategy conditions on input, recent, and randomly selected previous renderings to balance short- and long-term consistency.

A.9. Additional Common Objects in 3D results

Additional results evaluate single-view synthesis on real-world CO3D hydrants and ShapeNet cars. The method plausibly completes complex scenes, produces sharper details than PixelNeRF, and transfers input details better than competing baselines.

  • Autoregressive comparison: Feature aggregation is at least on par with stochastic conditioning in geometric consistency while using substantially fewer diffusion steps.
  • CO3D Hydrants: The CO3D hydrant comparison tests single-view synthesis on real-world objects with complex backgrounds.The comparison includes ViewFormer and PixelNeRF, with differences in masks and train/test splits noted for ViewFormer.
  • CO3D Hydrants: Our method plausibly completes complex CO3D scenes despite the comparison disadvantages noted for ViewFormer.
  • ShapeNet: On ShapeNet, our method renders sharper details and higher perceived quality than PixelNeRF.
  • ShapeNet: On ShapeNet, our method transfers details from the input image better than ViewFormer and EG3D.The reported renderings are selected from autoregressively generated sequences.

B. Implementation details

The implementation combines a feature-volume encoder, neural feature rendering, and a diffusion UNet for 3D-aware novel-view synthesis. Inference reuses rendered features across denoising steps and typically uses 25 steps, with more expensive CO3D settings for quality and temporal consistency.

  • Architecture: The feature encoder reshapes its output into a 16×64×128×128 latent feature volume.It processes a 3×128×128 image and produces a (16×64)×128×128 feature image before reshaping.
  • Architecture: Multiple input views are fused by projecting query points into independently predicted feature volumes and mean-pooling the sampled vectors.
  • Architecture: Neural volume rendering converts the feature field into feature images using 64 stratified depth samples per ray at half spatial resolution.
  • Architecture: The diffusion UNet takes 19 channels—noisy RGB plus 16 feature channels—and outputs a 3-channel 128×128 denoised image.
  • Inference: Inference uses a deterministic second-order sampler with 25 default denoising timesteps and reuses the rendered feature image throughout sampling.Rendering from scratch takes approximately 1.8 seconds per image on an RTX 3090.
  • Inference: CO3D production settings use 256 denoising steps, 128 samples per ray, and two-pass autoregressive conditioning for improved quality and temporal consistency.

C.2. Baselines

The study compares against established baselines across ShapeNet, Matterport3D, and CO3D, using dataset-specific protocols and evaluation settings. Reported comparisons favor the proposed method qualitatively and quantitatively, while CO3D remains noisy and unconstrained.

  • ShapeNet baselines: PixelNeRF and ViewFormer provide single-image ShapeNet baselines under matched evaluation splits and established view-synthesis protocols.
  • Matterport3D baselines: LOTR is the primary Matterport3D and RealEstate10K comparison, evaluated with matched data preparation and train/test methodology.
  • Matterport3D baselines: GeoGPT and SynSin are additional Matterport3D baselines, although prior results had already shown them to underperform LOTR.
  • Matterport3D results: The proposed novel-view renderings are reported as significantly more desirable both qualitatively and quantitatively than the additional Matterport3D baselines.
  • CO3D scope: CO3D hydrants contain unconstrained real-world videos with variable scene properties, miscategorized objects, and short or low-quality recordings.
  • CO3D protocol: Depth-coordinate input augmentation helps resolve CO3D’s highly variable scale and provides information parity with PixelNeRF’s global reference frame.

D.2. Limitations

The method has two acknowledged limitations: rendered sequences are not inherently 3D or temporally consistent, and input details may not always transfer accurately.

  • Rendered sequences can show noticeable flicker and other artifacts because the method is not inherently 3D or temporally consistent.
  • The model may fail to transfer details such as car-tire angles, window styles across symmetry, and fine details in complex scenes.The authors suggest that a more powerful attention-based encoder could help resolve these issues.
  • The lightweight ResNet-backed Deeplab feature encoder may limit long-range information flow needed for accurate detail transfer.
Loading 2304.02602v1…