PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional 3D Scene Generation using Locally Conditioned Diffusion

Ryan Po, Gordon Wetzstein

arXiv:2303.12218v2cs.CV

TL;DR

Text-to-3D methods have largely focused on object-centric generation, while designing composed 3D scenes has required expertise and manual work. This paper introduces locally conditioned diffusion and applies it in an SDS-based 3D pipeline using bounding boxes and text prompts. Its qualitative results show component capture and seamless blending, while SDS remains slow and relies on high guidance scales associated with lower diversity.

  • Problem

    Designing multi-component 3D scenes requires expertise and manual labor, and existing text-to-3D methods mostly specialize in object-centric generations.

    Source: §7 · p. 8

  • Method

    Locally conditioned diffusion is applied in an SDS-based text-to-3D pipeline that uses 3D bounding boxes and text prompts to control scene components.

    Source: §1 · p. 1, §5 · p. 5

  • Results

    Our method captures individual scene components and blends them into a coherent scene, while SJC fails to capture some components when composing multiple components into one prompt.

    Source: §6.2 · p. 7

  • Takeaways & Limitations

    Defining bounding boxes and text prompts provides a way to generate coherent 3D scenes that fit input specifications without 3D modeling expertise.

    Source: §7 · p. 8

    SDS-based methods require thousands of iterations to learn a scene, and unusually high guidance scales promote mode-seeking and low diversity.

    Source: §7 · p. 8

Abstract

from arXiv · show

Designing complex 3D scenes has been a tedious, manual process requiring domain expertise. Emerging text-to-3D generative models show great promise for making this task more intuitive, but existing approaches are limited to object-level generation. We introduce \textbf{locally conditioned diffusion} as an approach to compositional scene diffusion, providing control over semantic parts using text prompts and bounding boxes while ensuring seamless transitions between these parts. We demonstrate a score distillation sampling--based text-to-3D synthesis pipeline that enables compositional 3D scene generation at a higher fidelity than relevant baselines.

1. Introduction

The introduction frames 3D scene creation as labor-intensive and difficult to control, while existing text-to-3D methods struggle with scenes containing multiple distinct elements. It presents locally conditioned diffusion and camera-pose sampling as components of a compositional 3D generation approach.

1. Introduction

  • The paper identifies 3D scene modeling as a time-consuming task requiring expertise, with designers sometimes spending hours to days creating and composing assets.A large public bank of assets rarely contains a scene matching a user's exact specifications.Source: §1 · p. 1
  • Existing text-to-3D methods can produce high-quality assets but struggle with scenes containing multiple distinct elements and provide no layout control from a single global prompt.The introduction contrasts these methods with object-category-specific GANs that limit result diversity.Source: §1 · p. 1
  • Locally conditioned diffusion applies text prompts to specified regions, while the 3D pipeline takes bounding boxes and prompts to control asset size and position.The method is also presented as a compositional text-to-image approach using diffusion models.Source: §1 · p. 1
  • The authors identify camera-pose sampling strategies as crucial for compositional 3D generation.Source: §1 · p. 1

2. Related work

Related work explores text-conditioned diffusion, spatial composition, and score-distillation approaches to 3D synthesis. The paper positions locally conditioned diffusion as a way to add region-specific composition control while using pretrained text-conditioned diffusion models.

2D diffusion models

  • Large-scale text-conditioned diffusion models can generate high-quality images from complex prompts, while compositional information is often ignored.Source: §2 · p. 1
  • Bounding-box-conditioned diffusion offers composition control but requires annotated image data, often from smaller datasets that restrict result diversity.The proposed approach instead leverages pretrained text-conditioned 2D diffusion models.Source: §2 · p. 2

Compositional image generation

  • Existing compositional diffusion methods combine denoiser gradients across the whole image, whereas locally conditioned diffusion applies denoising selectively within user-defined regions.This provides control over concept position and size while aiming for seamless transitions.Source: §2 · p. 2

Text-to-3D diffusion models

  • The method applies locally conditioned diffusion within an SDS-based 3D generation pipeline, using a latent diffusion prior conditioned on a bounding-box rendering.The overview describes per-prompt noise estimation and denoising steps applied according to the rendered segmentation mask.Source: Figure 2 · p. 3
  • Text-to-3D methods based on SDS can generate high-quality assets but do not provide explicit control over scene-component size and position.Prior work applies 2D diffusion priors to rendered views of differentiable 3D representations.Source: §2 · pp. 2–3

3. Diffusion preliminaries

The preliminaries describe diffusion models as learned denoising processes and explain classifier-free guidance for text-conditioned noise prediction. At inference, guidance combines conditional and unconditional predictions.

Diffusion preliminaries

  • DDPMs begin with sampled Gaussian noise and apply successive denoising steps to obtain an image.Their forward process gradually adds Gaussian noise according to a predetermined variance schedule.Source: §3 · p. 3
  • A diffusion model is trained to predict noise in a noisy image, using a function approximator with learned parameters.The paper describes training with a simplified variant of the variational lower bound on the data distribution.Source: §3 · p. 3
  • The diffusion process obtains a sample at step t−1 from the sample at step t through an update step.Source: §3 · p. 3
  • Classifier-free guidance represents inference-time noise prediction using both conditioned and unconditioned predictions, with s controlling guidance strength.The conditional input is typically a text prompt; conditioning is randomly dropped during training.Source: §3 · p. 3

4. Locally conditioned diffusion

Locally conditioned diffusion assigns text-conditioned denoising to user-labeled image regions. The resulting images adhere to coarse semantic layouts while maintaining smooth transitions between regions.

4. Locally conditioned diffusion

  • Locally conditioned diffusion selectively applies denoising steps conditioned on different text prompts to specific image regions.Source: §4 · p. 3
  • A user-defined semantic mask labels image pixels, and the overall noise prediction applies each prompt's prediction to the corresponding labeled patches.The method uses classifier-free guidance to form predictions for the different prompts.Source: §4 · pp. 3–4
  • Only one diffusion model is queried in practice, and calls for distinct text conditions can be batched for efficiency.Source: §4 · p. 4
  • The method produces high-fidelity images that follow the input masks without visible seams, with smooth transitions between differently labeled regions.The figure examples include results from GLIDE and StableDiffusion.Source: §4 · p. 4, Figure 3 · p. 4

5. Compositional 3D synthesis

The method adapts score distillation sampling (SDS) to generate 3D scenes from text prompts and bounding boxes. It selectively applies prompt-specific denoising using rendered box masks, while sampling camera views to address empty off-center regions.

Bounding-box-guided text-to-3D synthesis

  • Figure 4 compares single-prompt, independently generated, and Composable-Diffusion baselines with the proposed method, which composes objects while following input boxes.Source: Figure 4 · p. 5

Text-to-3D with Score Distillation Sampling

  • The pipeline builds on SDS text-to-3D methods, using a Voxel NeRF scene representation, volumetric rendering, and diffusion-prior gradients to update scene parameters.It follows SJC’s pipeline; the scene is rendered from sampled camera poses.Source: §5 · pp. 4–5

Bounding-box-guided text-to-3D synthesis

  • Rendered 3D views are conditioned on visible bounding-box regions, and their masks selectively apply denoising for the corresponding text prompts before gradients update the scene.The mask assigns each pixel to a user prompt; camera pose, scene rendering, and box rendering use the same sampled viewpoint.Source: §5 · p. 5
  • The method aims for all rendered views to align with the rendered bounding-box segmentation masks, in addition to fitting the diffusion prior.Source: §5 · p. 6

Object-centric camera pose sampling

  • Object-centric camera sampling points views at each box center, giving every region a chance to appear centrally and addressing empty off-center boxes under high guidance.The issue arises because high guidance can favor centered objects even when semantic masks apply gradients to off-center regions.Source: §5 · p. 6

Locally conditioned diffusion with latent diffusion models

  • Locally conditioned diffusion can also be applied in SJC’s latent space because the latent space is described as a downsampled version of image space.SJC uses StableDiffusion as a latent diffusion prior.Source: §5 · p. 6

6. Experiments

Experiments show locally conditioned diffusion supports compositional generation in 2D and 3D, with user-controlled scene layouts and seamless blending. In 3D, it outperforms the single-prompt SJC baseline across evaluation methods.

6.1. Compositional 2D results

  • For 2D generation, the method produces high-fidelity images that follow semantic masks and prompts on both GLIDE and StableDiffusion.It targets seamless transitions between regions rather than exact adherence to mask boundaries.Source: §6.1 · p. 6

6.2. Compositional 3D results

  • The 3D implementation builds on SJC with a Voxel NeRF scene representation and StableDiffusion as the SDS diffusion prior.The scene representation uses resolution 1003 and about 10 GB of GPU memory.Source: §6.2 · p. 6
  • User-defined bounding boxes let the method adapt scene composition to changes in component size and position.Figure 5 shows generations of the same scene prompts with differing box sizes and positions.Source: §6.2 · p. 7, Figure 5 · p. 7
  • Unlike SJC's single-prompt compositions, the method captures individual scene components and blends them into coherent scenes.SJC can generate high-quality object-centric results, but scene-centric prompts such as “a desert” or “a beach” produce dense volumes without reasonable geometry.Source: §6.2 · p. 7
  • Components in separate boxes can interact through scene-level detail, including a cherry blossom tree reflected in a lake.The passages contrast this with generating components individually and manually combining them, which would not produce such effects.Source: §6.2 · p. 7, Figure 6 · p. 7
  • The method outperforms SJC with a single prompt across all evaluation methods for CLIP R-Precision on rendered scene views.Table 1 reports the comparison; no numeric values are provided in the supplied passages.Source: Table 1 · p. 8, §6.2 · p. 8
  • Object-centric camera pose sampling is essential for composing multiple components, especially when their boxes are far from the origin.Without it, the method tends to ignore some components and fails to fully capture off-centered ones.Source: §6.2 · p. 8, Figure 7 · p. 8
  • For the same number of prompts, the method generates scenes faster than individually generating components with SJC because diffusion calls can be batched.The speed comparison uses 3000 denoising iterations; scenes with more distinct prompts take longer to generate.Source: §6.2 · p. 8, Table 2 · p. 8

7. Discussion and Conclusions

The paper presents bounding boxes and text prompts as a simple interface for generating coherent 3D scenes without 3D-design expertise. Its scope remains constrained by SDS speed and guidance requirements, and by risks associated with the StableDiffusion prior.

Limitations and future work

  • SDS-based methods require thousands of iterations, and unusually high guidance scales can reduce generation diversity.The authors identify speed and low diversity as limitations of current SDS-based methods.Source: §7 · p. 8

Ethical considerations

  • Because the method uses StableDiffusion as its image prior, it may inherit that model’s biases and limitations; generative-model misuse could also spread disinformation.Source: §7 · p. 8

Conclusion

  • Bounding boxes and text prompts let users specify scene components and generate coherent scenes without 3D-design expertise.The paper frames this as an interface for matching user specifications.Source: §7 · p. 8

Supplementary Materials

The supplementary materials are associated with the paper’s arXiv version dated 23 March 2023.

Supplementary Materials

1. Implementation details

The implementation combines an SDS-based 3D baseline and object-focused camera sampling with view-dependent prompts. The authors also evaluate CLIP R-precision and test concept negation, which improves 2D mask adherence but shows no notable 3D difference.

CLIP R-precision

  • CLIP R-precision measures whether CLIP ranks the correct caption above distractors; evaluation averages results across four compositional scenes.The setup uses 267 random DreamFusion-gallery prompts plus the composed prompt and two component prompts for each scene.Source: §1 · p. 12

3D implementation of Composable-Diffusion

  • The Composable-Diffusion implementation on SJC did not converge to coherent whole-scene prompts, though it worked best for object-centric prompts.Additional examples of the implementation appear in Supplementary Fig. 1.Source: §1 · p. 12, Figure 1 · p. 12

Object-centric camera pose sampling

  • Off-center scene components often require higher sampling probability, and camera poses are redirected toward the sampled component’s bounding-box center.The sampling probabilities are hyperparameters; SJC instead samples views facing the scene origin.Source: §1 · p. 12

View-dependent prompting

  • View-dependent prompts add azimuth- and elevation-based view prefixes; “a zoomed out photo of” and “highly detailed” sometimes improve generations.The view prefixes are intended to alleviate the Janus problem.Source: §1 · pp. 12–13

Concept negation for locally conditioned diffusion

  • Concept negation improves adherence to input semantic maps in 2D, but produces no notable difference in 3D; the paper’s 3D results omit it.For negation, the other text prompt in the semantic map is used as the negated concept.Source: §1 · p. 13, Figure 2 · p. 13

2. Additional results

The supplementary results include depth maps corresponding to the main figure generations.

2. Additional results

Loading 2303.12218v2…