PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled 3D Scene Generation with Layout Learning

Dave Epstein, Ben Poole, Ben Mildenhall, Alexei A. Efros, Aleksander Holynski

arXiv:2402.16936v1cs.CVcs.LG

TL;DR

Text-to-3D systems can generate 3D representations from image priors, but existing methods often focus on isolated objects and do not readily separate scenes into their constituent entities. This paper jointly optimizes multiple NeRFs and layouts to generate scenes decomposed into objects, reporting meaningful disentanglement and object-level control, with occasional segmentation and geometry failures.

  • Problem

    Diffusion-guided text-to-3D methods often produce isolated objects in a single representation that is difficult to split into constituent entities.

    Source: §2.2 · p. 3

  • Method

    Layout learning jointly optimizes multiple NeRFs and 3D affine layouts so independently manipulated objects form prompt-conditioned, in-distribution scenes.

    Source: §1 · p. 1

  • Results

    Layout learning yields meaningful object-level decompositions and outperforms baselines; its final variant is 0.1 CLIP-score points from supervised per-object rendering with the largest CLIP model as oracle.

    Source: §1 · p. 1, §4.2 · p. 7

  • Takeaways & Limitations

    Decomposed scenes provide users of text-to-3D systems with more granular, local control over complex outputs.

    Source: §6 · p. 8

    Outputs sometimes show over- or under-segmentation, Janus artifacts, or other undesirable geometry, and layouts can converge to near-identical values.

    Source: §6 · p. 8

Abstract

from arXiv · show

We introduce a method to generate 3D scenes that are disentangled into their component objects. This disentanglement is unsupervised, relying only on the knowledge of a large pretrained text-to-image model. Our key insight is that objects can be discovered by finding parts of a 3D scene that, when rearranged spatially, still produce valid configurations of the same scene. Concretely, our method jointly optimizes multiple NeRFs from scratch - each representing its own object - along with a set of layouts that composite these objects into scenes. We then encourage these composited scenes to be in-distribution according to the image generator. We show that despite its simplicity, our approach successfully generates 3D scenes decomposed into individual objects, enabling new capabilities in text-to-3D content creation. For results and an interactive demo, see our project page at https://dave.ml/layoutlearning/

1. Introduction

The introduction motivates discovering scene objects as useful for parsing the 3D world, while noting that image generators lack explicit 3D entities and prior text-to-3D work often depicts only a few isolated objects. Layout learning jointly optimizes multiple NeRFs and layouts so independently rearranged objects still form prompt-consistent scenes, enabling unsupervised object-level generation and editing.

1. Introduction

  • Autonomous parsing of 3D scenes into component objects is motivated by its potential to help agents navigate and interact with their surroundings.The introduction relates this goal to object individuation in humans and other creatures.Source: §1 · p. 1
  • Prior text-to-3D methods use diffusion priors without 3D supervision but often focus on prompts depicting only one or two isolated objects.Image generators can create high-quality samples but do not explicitly represent their distinct 3D entities.Source: §1 · p. 1
  • Objects are defined as scene portions that remain valid when manipulated independently, implemented through learned layouts that yield prompt-conditioned, in-distribution images.Each layout applies 3D affine transformations to the NeRFs.Source: §1 · pp. 1, 3
  • Layout learning produces meaningful object-level decompositions, enables scene manipulation, and outperforms baselines without auxiliary models or per-example annotation.The paper applies it to building scenes around an asset, sampling plausible arrangements, and parsing a provided NeRF.Source: §1 · p. 1
  • The architecture jointly optimizes multiple NeRFs and layouts, using each NeRF to represent a separate 3D entity.The approach is applied to novel scene-generation and editing tasks without object labels, bounding boxes, fine-tuning, external models, or other additional supervision.Source: §1 · p. 3

2. Background

The background describes NeRFs as 3D representations that can be differentiably rendered, then explains how 2D diffusion priors guide text-to-3D optimization. It also notes that existing setups often favor isolated objects and produce single representations that are difficult to separate.

2.1. Neural 3D representations

  • An MLP-based NeRF maps a 3D point to density and albedo, representing a volume for three-dimensional scene generation.A ray is differentiably rendered by alpha-compositing sampled densities and colors; generation scores rendered images from random camera poses with a generative model.Source: §2.1 · p. 3

2.2. Text-to-3D using 2D diffusion models

  • Text-to-3D optimization uses a pretrained diffusion model to score rendered images conditioned on text and update the 3D representation through score distillation.Each iteration samples a camera, renders an image, and evaluates a noised version; the update uses a noise-level-dependent weighting.Source: §2.2 · p. 3
  • Existing diffusion-guided 3D setups often favor high-quality individual objects, while their single representations are difficult to split into constituent entities.Gaussian density initialization at the origin and bounding spheres help favor centered objects and prevent background density, but often fail to produce interesting scenes.Source: §2.2 · p. 3

3. Method

The method represents scenes with multiple composited NeRFs and learns layouts that rearrange them while keeping the composite scene valid. It defines objects by their ability to be rearranged into different valid compositions, with per-NeRF regularization discouraging degenerate components.

3.1. Compositing multiple volumes

  • The naive approach jointly renders K NeRFs by summing their densities and taking a density-weighted average of their albedos, but often yields random point-cloud-like subsets instead of meaningful objects.Multiple initial density spheres may make it easier to optimize for more objects, and local smoothness biases may favor spatially smooth entities.Source: §3.1 · pp. 3–4

3.2. Layout learning

  • Each NeRF receives a learnable affine transform, and the transformed rays are used to query the NeRFs before their outputs are composited in a shared coordinate space.The transforms encode rotation, translation, and scale.Source: §3.2 · pp. 4–5
  • Only multiple layouts produce strong disentanglement; K NeRFs alone scatter object parts across models, while learning one layout alleviates some of this problem.Even with layout learning, NeRFs can still be adjoined in undesirable ways.Source: Figure 3 · p. 5, §3.2 · p. 5
  • Layout learning builds on multiple NeRFs by learning multiple valid arrangements, with a randomly selected layout applied at each training step.The layouts are affine transformations of each NeRF.Source: §3 · p. 3, §3.2 · p. 5
  • Objects are defined as scene parts that can be arranged in different ways to form valid compositions, implemented by learning and sampling multiple layouts.The method adds 8NK parameters and requires no fine-tuning or manual annotation.Source: §3.2 · p. 5
  • Per-NeRF regularization inherits orientation, distortion, and accumulation losses and penalizes empty NeRFs by requiring their soft-binarized accumulated density to occupy at least 10% of the canvas.The method also applies score distillation sampling and per-NeRF regularizations to the rendered composite.Source: Figure 2 · p. 4, §3.2 · p. 5

4. Experiments

Experiments assess layout learning through qualitative and CLIP-based evaluation, then demonstrate scene editing, layout variation, and decomposition of existing NeRFs.

4.1. Qualitative evaluation

  • Qualitative examples show prompt-matching composite scenes whose NeRFs correspond to objects, including some entities not named in the prompt.Examples of unmentioned entities include a basket of Easter eggs, a chef’s hat, and a picnic table.Source: §4.1 · p. 6

4.2. Quantitative evaluation

  • Layout learning’s final variant is only 0.1 points away from supervised per-object rendering when the largest CLIP model serves as oracle.The evaluation matches NeRFs to object descriptions across 30 three-object prompts; individually optimized per-object NeRFs provide the supervised comparison.Source: §4.2 · pp. 6–7
  • Adding regularization improves CLIP scores somewhat, but the largest ablation gains come from layout learning and co-learning multiple arrangements.A collection of NeRFs alone separates some content, but can distribute object parts randomly and scores well below the upper bound.Source: §4.2 · p. 7

4.3. Applications of layout learning

  • Freezing an object’s NeRF while learning layouts incorporates provided assets into prompted scenes and arranges frozen assets into reasonable configurations.The applications include learning a scene around a specified cat or motorbike, and arranging off-the-shelf assets such as a duck and tub.Source: Figure 4 · p. 6, §4.3 · p. 7
  • A single run discovers meaningfully different layouts, letting users explore alternative arrangements of the same scene content.Examples include flamingos wading inside or beside a pond and cats in different poses around a snooker table.Source: §4.3 · p. 7, Figure 5 · p. 7
  • Layout learning also produces reasonable decompositions when reconstructing a pre-existing NeRF containing multiple entities, without per-object supervision.The first layout is constrained to reconstruct the target NeRF in RGB space while the other layouts vary freely.Source: §4.3 · p. 7

5. Related work

Related work includes supervised segmentation and unsupervised object discovery, while compositional 3D generation has used annotations or auxiliary models to specify objects and layouts.

Object recognition and discovery

  • Image segmentation commonly identifies scene objects using extensive manual annotation, which creates challenges and scales poorly to 3D data [Kirillov et al., 2023].Source: §5 · p. 7
  • Unsupervised object-discovery methods use inductive biases, but are largely restricted to 2D images or constrained 3D data [Russell et al., 2006; Rubinstein et al., 2013; Oktay et al., 2018; Hénaff et al., 2022; Smith et al., 2022; Ye et al., 2022; Monnier et al., 2023; Locatello et al., 2019; Yu et al., 2021; Sajjadi et al., 2022].Source: §5 · p. 7

Compositional 3D generation

6. Discussion

Layout learning generates object-decomposed 3D scenes without additional supervision or auxiliary models, offering more local control over text-to-3D outputs. Its simple definition of objects and ill-posed task leave room for undesirable decompositions and generation failures.

6. Discussion

  • The method optimizes multiple NeRFs across layouts to form valid scenes, requiring no additional supervision or auxiliary models.The resulting object decomposition gives users more granular, local control over complex text-to-3D creations.Source: §6 · p. 8
  • Because 3D disentanglement is ill-posed and the object definition is simple, many undesirable solutions can satisfy the method’s constraints.Source: §6 · p. 8
  • Failures include bad geometry, over- or under-segmentation, coupled objects, and near-identical layouts despite different initializations; failures generally accompany lower visual quality.For prompts with many small objects, choosing K correctly can also be difficult and hurt disentanglement.Source: §6 · p. 8, Figure 7 · p. 8

Impact statement

The impact statement notes risks tied to generative models and the possibility that more capable 3D creation could affect creative work, while also describing potential benefits for artists.

Impact statement

  • Generative models raise concerns about data attribution, nefarious applications, and longer-term societal effects.The authors note that undesirable depictions in popular datasets may leak into model weights despite filtering of the model's training data.Source: Impact statement · p. 9
  • Distilling an image generator may inherit its negative use-cases, and more compositional 3D scenes may expand associated risks.The authors stress the importance of ethically sourced, well-curated data to minimize harmful deployment.Source: Impact statement · p. 9
  • The method may contribute to displacement of creative workers such as video game asset designers, while also potentially giving artists more control over outputs.The authors describe both automation-related displacement and the possibility of creating new forms of content.Source: Impact statement · p. 9

A.1. Implementation details

The implementation uses Mip-NeRF 360 and Imagen, composites multiple NeRFs through density-weighted albedo and density merging, and specifies regularization, camera sampling, and optimization settings.

A.1. Implementation details

  • Multiple NeRFs are composited by merging albedos and densities at each point, with final albedo weighted by per-NeRF density.This operation is applied to proposal MLP outputs as well as final RGB-outputting NeRFs.Source: §A.1 · p. 12
  • The 3D backbone is Mip-NeRF 360, while most experiments use Imagen, a 128px pixelspace diffusion model, and render at 512px.Source: §A.1 · p. 12
  • The empty loss penalizes a NeRF's mean accumulated ray density when it falls below 10% of the image canvas, using a scaled sigmoid to binarize it for robustness to noise.The method uses λdist = 0.001, λacc = 0.01, λori = 0.01, and λempty = 0.05.Source: §A.1 · p. 12
  • SDS uses classifier-free guidance strength 200 and textureless shading probability 0.1, with view-dependent prompting disabled; Shampoo optimization runs for 15000 steps.The learning rate starts at 10^-9, peaks at 10^-4 after 3000 steps, and decays to 10^-6; Instant NGP experiments unlock grid settings above 64 × 64 only after 2000 steps to prevent density from concentrating in one NGP.Source: §A.1 · p. 12

A.2. Pseudo-code for layout learning

The appendix provides NumPy-like pseudocode for the core layout-learning logic, covering camera-ray transformations, volume compositing, and regularization.

A.2. Pseudo-code for layout learning

  • The pseudocode covers transforming camera rays as part of layout learning's core logic.Source: §A.2 · p. 12
  • The pseudocode includes compositing multiple 3D volumes.Source: §A.2 · p. 12
  • The pseudocode also includes regularizing the volumes.Source: §A.2 · p. 12

A.3. CLIP evaluation

The evaluation measures NeRF-to-prompt similarity with CLIP across prompts containing three objects. It averages scores over views and selects the best object-to-NeRF assignment across three seeds.

A.3. CLIP evaluation

  • The reported metric is raw cosine similarity, scaled by 100, rather than retrieval-based precision or recall.The authors say this provides a more granular metric and avoids dependence on test-set size and difficulty.Source: §A.3 · p. 13
  • CLIP evaluation uses 30 prompts, each listing three objects, to test whether three NeRFs contain different prompted objects.For each NeRF, CLIP scores use the query “a DSLR photo of [A/B/C]” and form a 3 × 3 score matrix.Source: §A.3 · p. 13
  • Each evaluation prompt is rendered as “a DSLR photo of {prompt}, plain solid color background”; its three objects are also separated into prompts for independent optimization.Source: Figure 11
  • NeRF-prompt scores average similarity across 12 views spaced 30 degrees apart at −30° elevation, then select the best of six assignments.The process is repeated across three seeds, with the seed having the highest mean NeRF-prompt score selected.Source: §A.3 · p. 13
Loading 2402.16936v1…