PaperlayerView exact revision on arXiv

Source-linked AI summary

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, Greg Shakhnarovich

arXiv:2212.00774v1cs.CVcs.LG

TL;DR

Pretrained 2D diffusion models must be adapted to guide 3D generation despite a distribution mismatch between denoising-trained inputs and rendered non-noisy images. The paper applies chain-rule score backpropagation through a voxel radiance-field renderer and introduces Perturb-and-Average Scoring, demonstrating compelling 3D models with Stable Diffusion and effectiveness for 3D text-driven generation.

  • Problem

    Applying pretrained 2D diffusion scores to rendered 3D images creates an out-of-distribution problem because denoisers were trained only on noisy inputs.

  • Method

    The method lifts 2D diffusion models to 3D by back-propagating multiview image scores through a differentiable voxel radiance-field renderer and estimating non-noisy-image scores with Perturb-and-Average Scoring.

  • Results

    The approach generates compelling 3D models with Stable Diffusion, demonstrates effectiveness for 3D text-driven generation, and is evaluated across pretrained image diffusion models.

  • Takeaways & Limitations

    The work repurposes pretrained 2D diffusion models for 3D asset generation without requiring access to 3D data.

  • Takeaways & Limitations

    The 3D asset distribution is assumed proportional to the expected densities of its multiview 2D renderings, and the paper identifies open problems in applying Perturb-and-Average Scoring as an optimization gradient.

Abstract

from arXiv · show

A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.

1. Introduction

The paper lifts pretrained 2D diffusion models into 3D radiance-field generators without 3D data by chaining image scores through differentiable rendering. It addresses the resulting OOD mismatch with Perturb-and-Average Scoring and demonstrates SJC for text-driven 3D generation.

  • Method: The renderer aggregates 2D image gradients across camera viewpoints into a 3D asset gradient for a voxel radiance field.The 3D asset is parameterized by voxels, and volume rendering serves as the differentiable function.
  • Technical challenge: Directly evaluating the denoiser on rendered, non-noisy images creates an out-of-distribution problem because training used noisy inputs.Perturb-and-Average Scoring estimates scores for these non-noisy images by adding noise and averaging update directions.
  • Method: Score Jacobian Chaining applies the chain rule to learned diffusion scores, converting image-space gradients into gradients for 3D parameters.The method back-propagates scores through a differentiable renderer.
  • Technical challenge: Perturb-and-Average Scoring is proposed to resolve the mismatch between denoising-trained diffusion models and rendered images encountered during 3D optimization.The paper identifies this issue as a central technical challenge of lifting 2D scores to 3D.
  • Results and limitations: The paper demonstrates SJC for 3D text-driven generation using pretrained diffusion models, including Stable Diffusion trained on LAION.It also reports open problems when applying Perturb-and-Average Scoring as an optimization gradient.

2. Related Works

Prior work trains 3D generators from 2D images, optimizes 3D assets with CLIP guidance, or uses diffusion-model parameterizations. SJC instead uses diffusion models as direct 2D generative guidance for realistic 3D content.

  • Radiance-field representations: Neural radiance fields support multiview reconstruction, while voxel radiance fields provide faster volume queries without observed end-task performance loss.The paper uses a customized voxel radiance field based on DvGO.
  • 2D-supervised 3D generation: 2D-supervised 3D GANs use unstructured 2D images to supervise renderings of generated 3D assets, but training such models from scratch remains challenging.Evaluation has also remained concentrated mostly on human and animal faces.
  • CLIP-guided optimization: CLIP-guided methods optimize 3D assets through rendered images using a pretrained image-text matching model rather than a 2D generative model.DreamFields and PureClipNeRF use NeRF as differentiable renderers.
  • Diffusion-guided generation: SJC uses diffusion models as proper 2D generative models to create realistic-looking 3D content.This distinguishes its guidance source from CLIP-based image-text matching.
  • Concurrent work: DreamFusion is concurrent and similar at the pseudocode level, but SJC is motivated by applying the chain rule to the 2D score.The paper compares against DreamFusion while using available diffusion resources instead of the closed-source Imagen model.

3. Preliminaries

The preliminaries frame diffusion models as denoisers that approximate scores of noise-perturbed data distributions. Sampling repeatedly updates images across decreasing noise levels, with a mean-shift interpretation toward data samples.

  • Denoising score matching: Denoising score matching trains a denoiser to map noisy samples y + σn back toward clean samples y across noise levels.For images, the denoiser is commonly parameterized by a ConvNet.
  • Denoising score matching: Pretrained diffusion models are treated as denoisers, with implementation-level conversion when a model instead predicts a noise residual.This provides a common interface for the paper’s score-based formulation.
  • Score from denoiser: A trained denoiser approximates the denoising score of the Gaussian-perturbed data distribution pσ(x).The score is the gradient of the log-density with respect to the data.
  • Sampling: Diffusion sampling repeatedly updates a sample through noise levels σT > ··· > σ0 = 0, moving from highly noisy toward less noisy states.The noise schedule is chosen empirically; DDPM commonly uses a range of [0.01, 157].
  • Score as mean-shift: The score can be interpreted as a mean-shift direction toward weighted nearest data samples under a Gaussian-kernel view of the perturbed distribution.This interpretation describes the score as a guide for updating inputs toward nearby data.

4. Score Jacobian Chaining for 3D Generation

SJC lifts a pretrained 2D diffusion model into a 3D radiance-field generator by chaining image scores through differentiable rendering across viewpoints. PAAS addresses the OOD mismatch from applying denoisers to rendered images, while voxel regularization promotes coherent geometry during optimization.

  • Score Jacobian Chaining for 3D Generation: A 3D asset distribution is modeled from the expected densities of its multiview 2D renderings over camera poses.This assumption connects the available image model to 3D scene generation without 3D data.
  • Score Jacobian Chaining for 3D Generation: SJC computes 3D gradients by chaining 2D rendering scores through the Jacobian of a differentiable volume renderer.The asset is represented as a voxel radiance field, whose rendered images are differentiated with respect to voxel parameters.
  • Computing 2D Score on Non-Noisy Images: Direct denoiser evaluation on rendered images creates an OOD problem because training inputs were noisy data samples, unlike generally inconsistent rendered images.The denoiser can produce artifacts when its score is applied directly to non-noisy rendered views.
  • Computing 2D Score on Non-Noisy Images: PAAS adds Gaussian noise to each rendered image, evaluates the denoiser on perturbed inputs, and averages the resulting score directions.The expectation is estimated with multiple sampled noises, yielding an update direction for the original rendered image.
  • Inverse Rendering on Voxel Radiance Field: Emptiness loss penalizes small ray weights to suppress low-density noise, while its staged weighting avoids hindering early geometry learning.The schedule increases the emptiness-loss weight after K iterations; β is set to 10.
  • Inverse Rendering on Voxel Radiance Field: With Stable Diffusion, SJC generates detailed prompted 3D models with sharp depth structure, and it produces better image quality and more sensible structure than Stable-DreamFusion in many cases.The method uses Stable Diffusion pretrained on the web-scale LAION dataset and generates models across diverse prompts.

5. Experiments

Experiments evaluate SJC on unconditioned and text-conditioned diffusion models, then test Stable Diffusion for 3D generation and compare its outputs with Stable-DreamFusion. Results show schedule-dependent behavior, competitive qualitative generation, and benefits from the proposed emptiness-loss schedule.

  • Experimental setup: Experiments cover both unconditioned diffusion models and the language-conditioned Stable Diffusion model.The unconditioned models are trained on FFHQ and LSUN Bedroom, while Stable Diffusion is trained on LAION5B.
  • 2D experiments: Annealed σ outperforms Random σ for unconditioned FFHQ models, producing better pose variation and image quality.Randomized σ exhibits severe mode-seeking behavior that converges toward average faces.
  • 2D experiments: At high language guidance, Random σ produces crisper Stable Diffusion images than Annealed σ, although neither matches standard diffusion inference quality.At guidance scale 10.0, Random σ outperforms Annealed σ; the authors identify the appropriate PAAS gradient use for optimization as open.
  • 3D generation: For Stable Diffusion 3D generation, the method uses a Random σ schedule with high language guidance because tuning Annealed σ in 3D was difficult in practice.The implementation renders latent-space feature images from a voxel-grid feature field.
  • 3D generation: SJC generates diverse text-prompted 3D models and often produces better image quality and more sensible structure than Stable-DreamFusion.Both systems show quality fluctuations across trials, so the comparison is intended to establish overall pipeline competitiveness.
  • Ablations: The complete emptiness-loss method improves 3D model quality, including fewer floating artifacts and better geometry.The study compares no emptiness loss, constant weighting, and the proposed weight schedule.

6. Conclusion

The paper presents SJC as an optimization-based route from pretrained 2D diffusion models to 3D assets without 3D training data. It addresses the denoising-versus-rendered-image gap, improves scene quality with regularization, and demonstrates compelling Stable Diffusion results while identifying future work.

  • The method generates 3D assets from pretrained 2D image diffusion models without requiring 3D data.
  • Perturb-and-Average Scoring bridges denoising-trained diffusion models and the non-noisy rendered images used during 3D optimization.
  • A new regularization loss is proposed to improve the quality of generated 3D scenes.
  • Using Stable Diffusion, the approach generates compelling 3D models and compares favorably with concurrent work.
  • The paper identifies the effect of noise-scheduling regimes in unconditional and text-conditional diffusion models as an avenue for future work.

Appendix

The appendix provides supplementary material on score-based diffusion models, experiments, and implementation details. Its algorithm card summarizes deterministic sampling under a specific scaling and noise schedule.

  • The appendix supplies a score-based perspective on diffusion models.
  • Additional appendix experiments include ablations, qualitative results, and video results.
  • Implementation details are documented in the appendix.
  • With s(t) = 1 and σ(t) = t, the sampling update is a weighted average of the current iterate and denoiser output, enabling large steps and as few as 80 network evaluations.

A1. Diffusion Models from Score-Based Perspective

This section presents diffusion models through their score-based formulation, forward noising process, and deterministic probability-flow ODE. The s(t) = 1, σ(t) = t schedule avoids additional trajectory curvature and supports large inference steps.

  • The appendix recaps diffusion models from the score-based perspective and summarizes deterministic and stochastic sampling algorithms.
  • The forward process scales clean data and adds noise, with σ(t) representing the noise-to-signal ratio.
  • Different diffusion families use distinct scaling schedules, with DDPM maintaining unit variance through a decreasing s(t).
  • The infinitesimal noising process is represented by a stochastic differential equation whose marginal density follows a Fokker–Planck equation.
  • The corresponding deterministic ordinary differential equation enables sampling back toward the initial clean-data distribution using the score function.
  • The s(t) = 1, σ(t) = t schedule permits large inference steps because it adds no curvature beyond that induced by the score function.

A2. Additional Experiments

Additional experiments examine optimization behavior, qualitative generation, and video results for SJC. The method produces varied 3D assets and supports view-consistent scene effects in qualitative examples.

  • Ablation on center depth loss: A moderate center depth loss places objects at the scene-box center and reduces viewpoint-dependent drifting.Without this loss, some objects, including French Fries, are placed far from the center and drift when camera viewpoints change.
  • Additional qualitative results: SJC qualitatively generates volumetric effects including shadows, water surfaces, grasslands, and traffic lanes from volume densities.Examples include a horse, Sydney Opera House, duck, zebra, and school bus.
  • Video results: The supplemental materials include numerous videos named after their text prompts, including ablation videos for Fig. 7 and Fig. A2.

A3. Implementation Details

The implementation uses voxel-grid scene optimization with randomized camera and field-of-view settings, Adamax updates, and view-dependent text prompts. Additional results are included for text-prompted 3D generation with SJC.

  • 3D scene setup: Voxel grids use size 100^3 at the world origin, while cameras are sampled uniformly on a hemisphere around the voxel cube.The camera radius is 1.5, field of view is jittered from 40° to 70° during optimization, and fixed at 60° during testing.
  • Optimization: Optimization uses Adamax with learning rate 0.05 for 10,000 steps, with longer schedules for some prompts.NEScore relies implicitly on the optimizer’s momentum state for averaging, and the language-guidance scale is 100.
  • View-dependent prompting: View-dependent prompting prepends overhead, front, backside, or side-view text according to camera elevation and azimuth.The technique is intended to reduce degeneracy from painting multiple frontal faces around an object.
  • Additional qualitative results: Additional SJC results show text-prompted generation of 3D models.
Loading 2212.00774v1…