Source-linked AI summary
RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, Paul Guerrero
TL;DR
Image diffusion models have limited support for 3D generation and understanding, especially when explicit 3D supervision is scarce. RenderDiffusion embeds a triplane-based 3D representation into the denoiser while training on posed 2D images, achieving 3D-consistent generation and inference across several datasets. Its main scope boundaries are camera-parameter requirements, blur, reduced 3D fidelity from regularization, and restricted generalization across object placements.
Problem
3D diffusion is limited by explicit-representation memory demands and the scarcity of 3D supervision relative to image data.
Method
RenderDiffusion incorporates a triplane representation and volumetric renderer into an image denoiser trained directly on posed 2D images.
Results
RenderDiffusion produces plausible, diverse, 3D-consistent scenes and supports monocular reconstruction and 3D inpainting, with competitive quality and diversity across sampling and inference tasks.
Takeaways & Limitations
The learned 3D representation can be rendered from novel viewpoints and used for 3D scene inference, generation, and editing through 2D inpainting.
Takeaways & Limitations
The method requires camera extrinsics during training, and global-coordinate triplanes restrict generalization across object placements.
Abstract
from arXiv · showhide
Diffusion models currently achieve state-of-the-art performance for both conditional and unconditional image generation. However, so far, image diffusion models do not support tasks required for 3D understanding, such as view-consistent 3D generation or single-view object reconstruction. In this paper, we present RenderDiffusion, the first diffusion model for 3D generation and inference, trained using only monocular 2D supervision. Central to our method is a novel image denoising architecture that generates and renders an intermediate three-dimensional representation of a scene in each denoising step. This enforces a strong inductive structure within the diffusion process, providing a 3D consistent representation while only requiring 2D supervision. The resulting 3D representation can be rendered from any view. We evaluate RenderDiffusion on FFHQ, AFHQ, ShapeNet and CLEVR datasets, showing competitive performance for generation of 3D scenes and inference of 3D scenes from 2D images. Additionally, our diffusion-based approach allows us to use 2D inpainting to edit 3D scenes.
1. Introduction
RenderDiffusion addresses limited 3D diffusion success by embedding a latent 3D representation into a denoiser trained only on posed 2D images. It produces 3D-consistent generation and supports reconstruction and inpainting tasks.
- 3D diffusion methods face memory costs from explicit representations and limited 3D supervision compared with abundant image data.These constraints are especially problematic for data-hungry diffusion models.
- RenderDiffusion is a diffusion method for 3D content trained using only 2D images and an explicit latent triplane representation.The representation is created from noisy images and rendered back into denoised 2D images.
- The triplane representation avoids cubic memory growth, while direct 2D training avoids explicit 3D supervision and supports sharper results than pretrained latent-space diffusion.
- The method assumes intrinsic and extrinsic camera parameters are available during training.
- RenderDiffusion generates plausible, diverse, 3D-consistent scenes and performs monocular reconstruction and masked-image 3D inpainting without task-specific training.It also improves reconstruction accuracy over a state-of-the-art monocularly supervised method.
2. Related Work
Related work spans diffusion, neural-field, and 3D generative models, but RenderDiffusion targets direct 3D diffusion from single-view 2D images without pretrained 3D latent spaces.
- Diffusion models achieve state-of-the-art image generation, while earlier 3D diffusion methods commonly rely on point, voxel, or SDF representations with 3D supervision.
- Unlike shape-focused implicit models, RenderDiffusion combines diffusion with neural fields to generate both 3D shape and surface appearance.
- Single-view training enables inference quality comparable to methods trained with multi-view data, while 2D diffusion-prior approaches cannot synthesize new images or scenes a priori.
- DreamFusion and Latent-NeRF use pretrained 2D diffusion priors to optimize NeRFs for text-driven 3D generation.
- RenderDiffusion performs direct 3D generation by sampling, using single-view 2D images and end-to-end diffusion without pretraining a 3D latent space.This contrasts with GAUDI's multi-view-trained triplane latent space and DiffDreamer's repeated RGB-D inpainting.
3. Method
RenderDiffusion modifies image diffusion so each denoising step predicts a 3D triplane scene and renders it into an image. The resulting representation supports view-consistent reconstruction while exposing a fidelity–generalization trade-off.
- 3. Method: The model preserves iterative diffusion denoising but encodes each noisy image into a 3D representation that is volumetrically rendered into the denoised output.
- 3.1. Image Diffusion Models: Diffusion generation starts from Gaussian noise and progressively denoises through intermediate images toward the data distribution.
- 3.1. Image Diffusion Models: Training adds Gaussian noise through a forward process controlled by a variance schedule, with a closed form allowing direct sampling of a noisy image at any timestep.The schedule increases from β0 = 0 to βT = 1; experiments use a cosine schedule.
- 3.1. Image Diffusion Models: Because the target image is unknown, the denoiser is trained to predict x0 and then approximates the reverse-process posterior mean during generation.The training objective is an L1 loss between the denoiser prediction and x0.
- 3.2. 3D-Aware Denoiser: The 3D-aware denoiser uses a triplane encoder and renderer: the encoder maps a posed noisy image to triplanes, and the renderer maps them back to a same-view denoised image.The input viewpoint is assumed available because the output must be rendered from the same viewpoint.
- 3.2. 3D-Aware Denoiser: Triplanes compactly factorize a 3D feature grid into three 2D feature maps, whose sampled features are converted by an MLP into densities and colors for volume rendering.
- 3.3. Regularization: Score-distillation regularization encourages plausible random-view renderings on FFHQ and AFHQ, reducing trivial geometry but causing some loss of 3D-model fidelity.
- 3.4. 3D Reconstruction: Reconstruction noise level tr controls a trade-off: tr = 0 maximizes fidelity, whereas larger tr handles increasingly out-of-distribution images with reduced fidelity.
4. Experiments
RenderDiffusion is evaluated on monocular reconstruction, unconditional generation, and 3D-aware inpainting across real-world and synthetic datasets. It achieves strong reconstruction and competitive, diverse 3D-consistent generation while supporting inpainting and OOD reconstruction trade-offs.
- Experiment setup: The experiments cover monocular 3D reconstruction, unconditional 3D generation, and 3D-aware inpainting.Evaluation uses FFHQ, AFHQv2, CLEVR1, and three ShapeNet categories.
- Baselines and metrics: RenderDiffusion is compared with EG3D and PixelNeRF for reconstruction, while generation comparisons include EG3D and GIRAFFE.PixelNeRF receives multi-view supervision, unlike RenderDiffusion and EG3D.
- Baselines and metrics: Reconstruction is measured across held-out views using PSNR and SSIM, evaluating both shape and surface color.The evaluation averages results over all test images.
- Monocular 3D Reconstruction: 26.1 average PSNR on ShapeNet is achieved by RenderDiffusion, versus 24.1 for EG3D and 28.9 for multi-view-trained PixelNeRF.RenderDiffusion outperforms EG3D across all evaluated datasets, while PixelNeRF remains higher but is not directly comparable.
- Monocular 3D Reconstruction: RenderDiffusion reconstructs FFHQ and AFHQ inputs in under 0.03 seconds per scene, compared with approximately 3 minutes for EG3D inversion.The reconstructions include plausible depth maps and novel-view renderings.
- Out-of-Distribution Reconstruction: Adding noise improves generalization to increasingly out-of-distribution images at the cost of reconstruction fidelity.The OOD reconstructions use multiple reverse-process steps and fill information lost to noise with generated content.
- Unconditional Generation: Generation produces realistic, 3D-consistent scenes with somewhat higher color and shape diversity than EG3D, while achieving better coverage on two of three ShapeNet classes.Both models generate complex structures and physically plausible ground-plane shadows; FFHQ and AFHQ samples show some artifacts at larger azimuth angles.
- 3D-Aware Inpainting: 3D-aware inpainting generates diverse, plausible masked regions matching observed image content, despite no explicit training for the task.The method conditions denoising on known pixels while sampling unknown regions, producing a latent 3D structure consistent with both.
5. Conclusion
RenderDiffusion uses a triplane rendering denoiser to generate, infer, and edit 3D scenes from posed 2D images. It achieves competitive quality and diversity while retaining several stated limitations.
- RenderDiffusion incorporates a triplane rendering model that enforces a strong inductive bias and produces 3D-consistent generations.
- The model supports 3D scene inference from images, 2D-inpainting-based 3D editing, and 3D scene generation.
- The method delivers competitive performance on sampling and inference tasks in both quality and diversity.
- Generated images can be blurrier than GAN outputs, score distillation reduces generated 3D-model fidelity, and training requires camera extrinsics.
S7. Overview of Supplementary
The supplementary material expands the paper with architecture details, generation and inpainting results, multi-category generation, and reconstruction experiments.
- Section S8 provides additional architecture details.
- Sections S9 and S10 provide additional results for unconditional generation and 3D-aware inpainting.
- Sections S11 and S12 cover multi-category generation with one model and additional reconstruction results.
S8. Architecture Details
The denoiser uses a triplane encoder and volumetric renderer to convert image inputs into renderable 3D representations.
- The triplane encoder transforms an M × M × 3 input image into an N × N × 3n_f triplane representation using a UNet.
- The renderer samples points along each ray and uses a two-layer fully connected network to predict color and density from triplane features.
- The renderer additionally uses positional embeddings of 3D sample positions to represent parts of the ground plane extending beyond the visible object.
S9. Additional Unconditional Generation Results
Additional unconditional-generation results evaluate distribution quality, coverage, density, diversity, and multi-category generation across synthetic and real datasets.
- FID_r and FID_t compare generated training and test views, while coverage and density measure distribution coverage and sample proximity.Coverage uses 3-nearest-neighbor neighborhoods; density counts real samples whose neighborhoods contain each generated sample.
- Six random seeds produce diverse, randomly selected 3D reconstructions from masked car images, with quantitative evaluation using the best-performing reconstruction.
- On synthetic datasets, RenderDiffusion tends to outperform baselines on coverage and density, while EG3D performs well on FID.
- RenderDiffusion samples are slightly blurrier than EG3D but show more variety and similar shape quality apart from blurriness.
- The results suggest better data-distribution coverage at comparable sample quality, although the models were not fully converged during measurement.
- A single RenderDiffusion model generates scenes across both chair and airplane ShapeNet categories.
S10. Additional Inpainting Results
The inpainting evaluation treats masked regions as occlusions and samples multiple plausible reconstructions, finding masked and unmasked performance generally comparable.
- RenderDiffusion evaluates 3D inpainting by comparing reconstructed test-view renders with ground-truth renders using PSNR and SSIM.Because inpainting is ambiguous, the evaluation samples multiple outputs and selects the best match.
- Table S5 reports 3D reconstruction performance when part of the input image is masked alongside copied unmasked results.
- For ambiguous inpainting, the model samples K outputs, using K = 25 for CLEVR and K = 10 for ShapeNet.CLEVR uses more samples because masks often hide most of the object.
- Masked and unmasked reconstruction performance is comparable in most cases, indicating that plausible ground-truth-like scenes occur in the output distribution.
S11. Multi-Category Generation Results
A single RenderDiffusion model trained jointly on chair and plane images captures both categories, producing plausible 3D-consistent samples with realistic novel views.
- Multi-Category Generation Results: The multi-category experiment trains one RenderDiffusion model on the union of ShapeNet chair and plane categories.The architecture and training protocol otherwise remain unchanged from the main experiments.
- Multi-Category Generation Results: The joint model successfully captures both dataset modes, sampling plausible chairs and airplanes.
- Multi-Category Generation Results: The generated samples are 3D-consistent, have plausible depth maps, and look realistic from novel test viewpoints.
- Multi-Category Generation Results: Additional reconstructions vary the amount of added noise while keeping the random seed fixed, exposing a trade-off between reconstruction quality and input fidelity.
S12. Additional Reconstruction Results
Additional reconstruction results show that added noise makes reconstructions more generic and less faithful to the input while increasing shape quality.
- Additional Reconstruction Results: Reconstructions from out-of-distribution images are shown with noise levels ranging from t = 0 to t = 50.
- Additional Reconstruction Results: More added noise produces increasingly generic reconstructions that diverge from the input image as the model fills in noisy details.
- Additional Reconstruction Results: On ShapeNet chair and CLEVR1, RenderDiffusion better preserves shape identity than EG3D and produces sharper, more detailed results than PixelNeRF.
- Additional Reconstruction Results: Higher noise levels also produce increasingly higher-quality shapes, creating a trade-off between reconstruction fidelity and shape quality.