Source-linked AI summary
Novel View Synthesis with Diffusion Models
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, Mohammad Norouzi
TL;DR
Novel view synthesis must infer unseen viewpoints from limited images despite ambiguity, while prior regressors can be blurry and scene-specific models do not scale across scenes. 3DiM uses pose-conditional image-to-image diffusion with stochastic conditioning to autoregressively generate views, and reports sharp, approximately 3D-consistent completions from a single image. The paper also introduces 3D consistency scoring, while noting limitations from out-of-distribution poses and real-world data challenges.
Problem
Novel view synthesis requires inferring unseen viewpoints from few images despite multiple plausible completions and limitations of blurry regressive models.
Method
3DiM trains a geometry-free pose-conditional image-to-image diffusion model and uses stochastic conditioning for autoregressive multi-view generation.
Results
3DiM generates sharp, approximately 3D-consistent views from as few as a single image, while 3D consistency scoring evaluates generated outputs using neural fields.
Takeaways & Limitations
A single geometry-free model can produce high-quality, approximately consistent novel views while scaling to many scenes without hyper-networks or test-time optimization.
Takeaways & Limitations
3DiM performs poorly on SRN chairs with test poses at unseen scales, and real-world deployment still requires handling noisy poses and varying focal lengths.
Abstract
from arXiv · showhide
We present 3DiM, a diffusion model for 3D novel view synthesis, which is able to translate a single input view into consistent and sharp completions across many views. The core component of 3DiM is a pose-conditional image-to-image diffusion model, which takes a source view and its pose as inputs, and generates a novel view for a target pose as output. 3DiM can generate multiple views that are 3D consistent using a novel technique called stochastic conditioning. The output views are generated autoregressively, and during the generation of each novel view, one selects a random conditioning view from the set of available views at each denoising step. We demonstrate that stochastic conditioning significantly improves the 3D consistency of a naive sampler for an image-to-image diffusion model, which involves conditioning on a single fixed view. We compare 3DiM to prior work on the SRN ShapeNet dataset, demonstrating that 3DiM's generated completions from a single view achieve much higher fidelity, while being approximately 3D consistent. We also introduce a new evaluation methodology, 3D consistency scoring, to measure the 3D consistency of a generated object by training a neural field on the model's output views. 3DiM is geometry free, does not rely on hyper-networks or test-time optimization for novel view synthesis, and allows a single model to easily scale to a large number of scenes.
1 INTRODUCTION
3DiM applies pose-conditional diffusion to novel view synthesis, addressing the blur and limited generative diversity of prior few-view approaches while enabling approximately 3D-consistent outputs.
- Novel view synthesis infers how a 3D scene appears from viewpoints not present among the input images.
- Few-view NeRF variants can artifact with sparse inputs, require scene-specific models, and regressively produce blurry rather than diverse plausible views.
- 3DiM trains a geometry-free image-to-image diffusion model on paired same-scene images with known poses.
- Stochastic conditioning enables autoregressive generation of entire sets of approximately 3D-consistent frames from the pose-conditional model.
- The paper introduces X-UNet and 3D consistency scoring alongside 3DiM and stochastic conditioning.The scoring method measures consistency by training neural fields on generated model outputs.
2 POSE-CONDITIONAL DIFFUSION MODELS
3DiM uses pose-conditional diffusion to generate novel views, then improves multi-view consistency through stochastic conditioning and an X-UNet architecture that shares processing and mixes view information.
- Pose-conditional diffusion models: 3DiM learns a conditional generative model of one scene view given another view and both poses.
- Pose-conditional diffusion models: The diffusion process adds increasing Gaussian noise as the log signal-to-noise ratio decreases, then learns to reverse that process by denoising one frame from another clean frame.
- Stochastic conditioning: The ideal autoregressive factorization conditions each generated frame on all previous frames, but memory limits and declining sample quality motivate using the minimum k = 2 conditioning frames.
- Stochastic conditioning: Stochastic conditioning resamples a random conditioning view at every denoising step, adds each completed frame to the available set, and repeats to generate more views.
- Stochastic conditioning: Stochastic conditioning approximates autoregressive sampling while avoiding the multi-view training data required by a score model conditioned on all previous views.
- X-UNet: X-UNet shares parameters across the clean conditioning and noisy target views and uses cross-attention to mix their information.
3 EXPERIMENTS
Experiments benchmark 3DiM on SRN ShapeNet and compare its diffusion sampler and X-UNet architecture with prior methods and ablations. The results emphasize sharp, high-fidelity samples and expose limitations of standard reconstruction metrics for geometry-free synthesis.
- Benchmark setup: 3DiM is benchmarked on 128x128 car and chair views from the SRN ShapeNet dataset against geometry-free and geometry-aware novel-view methods.Compared methods include LFN, ENR, SRN, PixelNeRF, and VisionNeRF, using PSNR, SSIM, and FID.
- Benchmark setup: The original SRN chairs test split uses out-of-distribution camera scales, causing poor geometry-free performance; merging, shuffling, and re-splitting fixes the issue.Results on the original split remain in Table 2, while the corrected split is labeled SRN chairs*.
- State-of-the-art comparisons: 3DiM produces qualitatively sharper and more faithful videos, although it does not necessarily achieve superior PSNR or SSIM reconstruction errors.Diffusion sampling captures sharp modes, whereas regression approaches can appear blurry despite higher reconstruction scores.
- Ablation studies: Both stochastic conditioning and multi-step diffusion are crucial: removing them yields worse consistency or blurry samples, while standardized metrics can favor these weaker models.Naive image-to-image sampling improves FID but severely worsens shape and texture consistency; regression improves PSNR and SSIM despite severe blurriness.
- UNet architecture comparisons: X-UNet produces substantially less 3D inconsistency and better conditioning-frame alignment than Concat-UNet, with all reported metrics worsening for the latter.The authors hypothesize that weight sharing exploits frame and pose symmetries, while cross-attention improves alignment.
4 EVALUATING 3D CONSISTENCY IN GEOMETRY-FREE VIEW SYNTHESIS
The paper proposes 3D consistency scoring because standard metrics can miss inconsistency in geometry-free view synthesis. It trains neural fields on generated views and evaluates held-out renders to distinguish coherent outputs from inconsistent ones.
- Motivation and desiderata: Standard metrics can fail for geometry-free view synthesis by missing 3D inconsistency and improving for qualitatively worse models.The evaluation should instead tolerate consistent outputs that differ from ground truth while penalizing inconsistency and conditioning misalignment.
- Evaluation scheme: 3D consistency scoring trains a NeRF-like neural field on some generated views and compares its renders with held-out views from the same source.This avoids directly comparing outputs with ground-truth views, which could penalize plausible but different generations.
- Evaluation scheme: The evaluation samples equal numbers of views at identical poses and conditioning views across real views, 3DiM outputs, and naive image-to-image samples.Ten percent of the test views, 25 of 251, are held out from neural-field training.
- Results: Neural fields trained on stochastic-conditioning outputs score better across all metrics than fields trained on naive image-to-image samples, confirming the sampler’s consistency benefit.The scoring also avoids penalizing coherent outputs that differ from target views and agrees with qualitative observations.
5 CONCLUSION AND FUTURE WORK
3DiM combines X-UNet and stochastic conditioning to generate sharp, approximately 3D-consistent views from a single image. It also introduces 3D consistency scoring and identifies real-world pose and camera variation as future challenges.
- 3DiM combines X-UNet with stochastic conditioning to generate sharp, approximately 3D-consistent views from as few as one image.Stochastic conditioning enables autoregressive generation over frames.
- 3D consistency scoring evaluates geometry-free generative models by training neural fields on their output views.The method reflects that inconsistent training views hinder neural-field performance.
- Classical metrics fail to capture both sharp modes and 3D inconsistency, motivating quantitative and visual evaluation.
- Applying 3DiM to large real-world 3D datasets requires handling noisy poses, varying focal lengths, and other dataset-specific challenges.
6 ARCHITECTURE DETAILS
The architecture processes noisy and conditioning frames jointly with pose and noise-level embeddings in a multi-resolution X-UNet. Frame-wise normalization, attention, residual blocks, and skip connections support the network’s image-to-image processing.
- The architecture applies learned or fixed embeddings to represent noise levels, image positions, and frame identity.These embeddings are integrated into the conditioning pathway and residual-block modulation.
- X-UNet uses residual blocks with GroupNorm, FiLM-style scale-and-shift modulation, optional attention, downsampling, and upsampling with skip connections.Attention can operate within each frame or cross-attend between the two frames.
- ConditioningProcessor embeds log signal-to-noise ratios and camera poses, including ray positions and directions, at multiple UNet resolutions.Pose embeddings can be masked to enable classifier-free guidance, while optional spatial and reference-pose embeddings distinguish frames.
- X-UNet stacks the conditioning image and noisy target image as separate frames before convolutional processing.The implementation preserves a frame dimension throughout the network.
7 HYPERPARAMETERS
The experiments use a 471M-parameter main model, cosine noise scheduling, classifier-free guidance, and fixed denoising settings. Larger channel widths improve chair consistency but are comparable on cars, while neural-field implementation choices affect scoring.
- The main experiments use ch=256, corresponding to approximately 471M parameters, while larger experiments use ch=448, corresponding to approximately 1.3B parameters.The architecture uses channel multipliers (1, 2, 2, 4), with the lowest UNet resolution at 8x8.
- The ∼1.3B model is comparable to the smaller model on cars but significantly improves 3D consistency on chairs.The larger model increases the base channel size rather than blocks, attention heads, or cross-attention layers.
- The noise schedule uses a cosine-shaped log signal-to-noise ratio that decreases monotonically from 20 to -20.
- All samples use 256 denoising steps, and classifier-free guidance is trained with unconditional examples at a 10% probability.Small guidance weights are used because they encourage 3D consistency further.
- The 3D consistency scoring neural field uses a simplified instant-NGP-based design without view-dependent components, and its implementation details affect the metrics considerably.Each scene receives 1000 training steps without camera-pose optimization.