PaperlayerView exact revision on arXiv

Source-linked AI summary

HoloDiffusion: Training a 3D Diffusion Model using 2D Images

Animesh Karnewar, Andrea Vedaldi, David Novotny, Niloy Mitra

arXiv:2303.16509v2cs.CVcs.GR

TL;DR

3D diffusion is constrained by the scarcity of 3D training data and the challenges of representation. HOLODIFFUSION learns an unconditional 3D generator from posed 2D images, and the authors report competitive results on Co3Dv2, including in few-image settings.

  • Problem

    3D diffusion lacks sufficient training data, and selecting a suitable 3D representation is also challenging.

    Source: §1 · p. 1

  • Method

    HOLODIFFUSION learns an unconditional 3D diffusion model from posed 2D images using a hybrid feature grid and photometric supervision of rendered views.

    Source: §1 · p. 2

  • Results

    On Co3Dv2, HOLODIFFUSION outperforms existing alternatives qualitatively and quantitatively, and balances quality and diversity even with few images.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The method can be trained on raw posed image sets, including in the few-image setting.

    Source: §5 · p. 8

    In limited-data settings, the models may recover original training samples, and the severity of this risk remains for future study.

    Source: §6 · p. 8

Abstract

from arXiv · show

Diffusion models have emerged as the best approach for generative modeling of 2D images. Part of their success is due to the possibility of training them on millions if not billions of images with a stable learning objective. However, extending these models to 3D remains difficult for two reasons. First, finding a large quantity of 3D training data is much more complex than for 2D images. Second, while it is conceptually trivial to extend the models to operate on 3D rather than 2D grids, the associated cubic growth in memory and compute complexity makes this infeasible. We address the first challenge by introducing a new diffusion setup that can be trained, end-to-end, with only posed 2D images for supervision; and the second challenge by proposing an image formation model that decouples model memory from spatial memory. We evaluate our method on real-world data, using the CO3D dataset which has not been used to train 3D generative models before. We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling.

1. Introduction

The paper targets 3D diffusion’s data and representation challenges, introducing HOLODIFFUSION, an unconditional 3D generator trained from posed real 2D images. It reports qualitative and quantitative gains on Co3Dv2.

1. Introduction

  • 3D models offer view consistency and direct content manipulation, but 3D diffusion is hindered by limited training data and the choice of 3D representation.Prior work had not provided a diffusion-based 3D generator trained using only 2D image supervision.Source: §1 · pp. 1–2
  • HOLODIFFUSION is presented as the first unconditional 3D diffusion model trainable using only real posed 2D images, without 3D ground truth.Posed images are different views of the same object with known cameras.Source: §1 · p. 2
  • The method learns distributions over hybrid explicit-implicit 3D feature grids and uses a denoising 3D UNet trained with photometric error between rendered and ground-truth images.Intermediate 3D-aware features are generated from posed images, noised, and then denoised.Source: §1 · p. 2
  • HOLODIFFUSION outperforms existing alternatives qualitatively and quantitatively on Co3Dv2.The method is evaluated on this dataset, which the paper describes as not previously used to train 3D generative models.Source: §1 · p. 2

2. Related Work

Prior work develops image-conditioned reconstruction and 3D generative models using varied neural rendering methods and representations. The paper distinguishes its setting through category videos with multiple views that provide camera-pose constraints.

2.1. Image-conditioned 3D Reconstruction

  • Neural rendering methods use neural networks to approximate light transport, with 2D variants mapping feature images to photorealistic RGB and 3D NeRF methods modeling scenes for rendering.NeRF frames reconstruction as recovering shape and appearance from multiple 2D images.Source: §2.1 · p. 2
  • Few-view reconstruction methods use category-level priors or sampled image features to reconstruct scenes, while ViewFormer instead learns implicit new-view synthesis.The cited approaches include mesh prediction, NeRF fitting, and transformer-based processing of projected features.Source: §2.1 · p. 2

2.2. 3D Generative Models

  • Early 3D GANs generated shapes from uncurated category images, but HoloGAN’s feature-rendering pipeline produced results inconsistent with camera motion.PlatonicGAN renders voxel grids from random viewpoints; HoloGAN adds a 2D decoder to address voxel memory costs.Source: §2.2 · p. 2
  • Later generative approaches use neural radiance fields and feature-rendering pipelines, including PiGAN, StyleNeRF, EG3D, EpiGRAF, and GAUDI.EG3D and related methods use tri-plane representations, with EG3D rendering a radiance field supported by the tri-plane.Source: §2.2 · pp. 2–3
  • Other 3D generators explore sparse voxels, signed distance fields, warped voxel grids, and differentiably rendered meshes.GET3D converts an initial tri-plane representation into a colored mesh.Source: §2.2 · p. 3
  • Uncurated-image generators can struggle when camera motion makes rendering non-smooth; category videos provide multiple views that support consistent camera-pose estimation.The paper’s overview uses category-specific video frames and posed-image supervision without 3D ground truth.Source: §2.2 · p. 3, Figure 2 · p. 3
  • Prior 3D diffusion work used full 3D supervision for point clouds, while concurrent new-view synthesis lacked an explicit image-formation model that may help avoid geometric inconsistencies.HOLODIFFUSION generates view-consistent results from novel camera locations.Source: §2.2 · p. 3, Figure 2 · p. 3

3. HOLODIFFUSION

HOLODIFFUSION learns a distribution over renderable 3D feature grids from posed video frames, despite having no ground-truth grids for diffusion training. Its bootstrapped photometric objectives train the denoiser through image rendering.

3.1. Diffusion Models

  • The diffusion backbone predicts clean x0 from noised inputs with squared loss, then samples by iteratively refining Gaussian noise.The model uses the x0-formulation for denoising and starts sampling from pure noise.Source: §3.1 · p. 4

3.2. Learning 3D Categories by Watching Videos

  • The target is a distribution over 3D feature voxel grids learned from 2D videos; each grid renders observed frames using their camera poses.Ground-truth feature grids are unavailable, so the usual diffusion objective cannot be applied directly.Source: §3.2 · p. 4

3.3. Bootstrapped Latent Diffusion Model

  • BLDM replaces the unavailable target-grid diffusion loss with image-space photometric supervision, then bootstraps denoising on its own clean-grid outputs to narrow train/test mismatch.Its photometric loss compares a rendered denoised auxiliary grid with the known image at its camera pose.Source: §3.3 · pp. 4–5
  • Video-derived auxiliary grids aggregate projected image features across frames into a 3D grid using a trainable encoder.The paper uses a ResNet-32 encoder for the 2D image features.Source: §3.3 · pp. 4–5
  • The bootstrap assumes photometrically optimized denoising of auxiliary grids follows the clean-grid distribution, allowing those outputs to serve as data for a second denoising pass.The second pass noises and denoises the clean-grid samples recurrently under a photometric loss.Source: §3.3 · p. 5

3.4. Implementation Details

  • Each iteration samples 10 source views, renders against three different target views, and applies the bootstrapped loss with 50% probability as lazy regularization.This avoids running two rendering passes in every training iteration.Source: §3.4 · p. 5
  • For rendering, an MLP decodes sampled grid features into density and color, and emission-absorption ray marching combines sampled colors into pixels.The color depends on ray direction, while density does not.Source: §3.4 · p. 5

4. Experiments

HOLODIFFUSION outperforms EG3D and GET3D on reported generative-quality scores, though pi-GAN scores better on some categories. Qualitative comparisons also assess sample realism and consistency across viewpoints.

Qualitative evaluation

  • Across camera changes, HOLODIFFUSION remains view-consistent, whereas pi-GAN's shape renders show significant appearance variation.Figure 3 compares the methods under camera motion.Source: Figure 3 · p. 6, §4 · p. 7

Datasets and baselines

  • Experiments use four CO3Dv2 categories, training each on 500 videos selected for high camera-cloud quality, and compare against pi-GAN, EG3D, and GET3D.The dataset provides camera poses and foreground masks; GET3D uses the available masks for training.Source: §4 · pp. 6–7

Qualitative evaluation

  • HOLODIFFUSION produces the most appealing, consistent, and realistic samples in the visual comparison with pi-GAN, EG3D, and GET3D.Figure 4 shows generated samples from the compared methods.Source: Figure 4 · p. 7, §4 · p. 7

Quantitative evaluation

  • HOLODIFFUSION produces better FID and KID scores than EG3D and GET3D, while pi-GAN scores better on some categories.The authors caution that pi-GAN's 3D-agnostic training cannot recover proper 3D structure for unaligned CO3Dv2 shapes.Source: §4 · p. 7

5. Conclusion

HOLODIFFUSION uses a learnable rendering module, a feature-space diffusion denoiser, and a pretrained feature encoder to decouple volumetric memory from final image resolution. The conclusion reports a quality–diversity balance even with few posed images, while noting that training currently requires camera information.

5. Conclusion

  • A learnable rendering module trains with the feature-space denoiser, while a pretrained feature encoder decouples cubic volumetric memory from final rendering resolution.Source: §5 · p. 7
  • HOLODIFFUSION trains on raw posed-image sets, including few-image settings, while balancing result quality and diversity.Source: §5 · p. 8
  • Training currently requires camera information; jointly learning a viewpoint estimator and extending to conditional generation, editing, composition, and multi-class training remain future directions.Source: §5 · p. 8

6. Societal Impact

The societal-impact discussion identifies potential misuse of generated 3D media and possible recovery of training samples. It notes that the released models were trained only on Co3D and that memorization risk warrants further study.

6. Societal Impact

  • Generated 3D assets could be misused to create fake view-consistent GIFs or videos, although Co3D-only training means released models cannot directly infer potentially malicious samples.Source: §6 · p. 8
  • In limited-data settings, the models may memorize and recover original training samples; assessing the severity and extent is left for future work.Source: §6 · p. 8

Supplementary material

The Views2Voxel-grid mechanism projects each 3D grid coordinate into the video frames, samples image features, and aggregates frame-wise information into voxel features.

A. Views2Voxel-grid Unprojection Mechanism

B. Implementation Details

This supplementary section introduces itself as providing implementation details for the proposed method.

B. Implementation Details

B.1. Network Architectures

The pipeline combines an encoder, a diffusion U-Net, and a renderer. Its 3D U-Net uses residual blocks with downsampling, upsampling, and self-attention.

B.1. Network Architectures

  • The pipeline has three neural components: an Encoder, Diffusion UNet, and Renderer.The Encoder is a ResNet32 model.Source: §B.1 · p. 12
  • The diffusion network is a 3D variant of the UNet used by Dhariwal and Nichol [9].Source: §B.1 · p. 12
  • Its residual blocks contain downsampling, upsampling, and self-attention blocks with additive residual connections.Source: §B.1 · p. 12

B.2. Renderer

The RenderMLP decodes generated voxel-grid features into density and radiance fields. It uses a four-layer MLP with a skip connection and incorporates view directions at a later layer.

B.2. Renderer

  • The NeRF-like MLP has four layers of 256 hidden units and concatenates input features with intermediate features at its third-hidden-layer skip connection.Source: §B.2 · p. 12
  • The RenderMLP decodes generated voxel-grid features into density and radiance fields.Source: Figure I · p. 12, §B.2 · p. 12
  • The MLP adds view directions at a later layer, sinusoidally encodes them with maximum frequency level L = 4, and uses LeakyReLU, Softplus, and Sigmoid activations.These activations are used for hidden layers, the density output head, and the radiance output head, respectively.Source: §B.2 · p. 12

B.3. Training Details

Training uses multiple source views to build an initial voxel grid and a reserved novel view to enforce 3D structure. The diffusion module uses a 1,000-step DDPM schedule and clipped voxel features.

B.3. Training Details

  • 11 source views form the initial voxel grid, while one reserved target novel view supplies an L2 photometric-consistency loss against its ground-truth view.The full pipeline is trained for 1,000 epochs on object-centric videos.Source: §B.3 · p. 12

B.4. Diffusion Details

  • The bootstrap latent diffusion module uses DDPM with 1,000 time steps and a linearly interpolated β_t schedule from β_0 = 0.0001 to β_999 = 0.02.Source: §B.4 · p. 13
  • Applying tanh constrains voxel features to [-1, 1], allowing [-1, 1] clipping during sampling.The passage attributes this conditioning approach to Karras et al. [30].Source: §B.4 · p. 13
Loading 2303.16509v2…