PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis

Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, Matthias Nießner

arXiv:2303.14207v2cs.CV

TL;DR

Realistic, diverse, and semantically meaningful indoor 3D scene synthesis remains challenging because existing optimization and generative approaches face labor, diversity, or fidelity limitations. DiffuScene uses denoising diffusion over unordered object sets containing spatial, semantic, and geometry attributes, with shape retrieval for object geometry. It produces more plausible and diverse scenes than prior state-of-the-art methods and supports completion, rearrangement, and text-conditioned synthesis.

  • Problem

    Indoor scene synthesis seeks realistic, semantically meaningful, and diverse 3D scenes, while traditional optimization is laborious and existing generative models trade off diversity and fidelity.

  • Method

    DiffuScene denoises unordered sets of object attributes—including location, size, orientation, class, and shape features—to learn holistic 3D scene configuration priors.

  • Results

    DiffuScene synthesizes more plausible and diverse indoor scenes than prior state-of-the-art methods, as measured by multiple metrics and a user study.

  • Takeaways & Limitations

    The model supports scene completion, object rearrangement, and text-conditioned scene synthesis, while geometry diffusion improves symmetric and semantically coherent arrangements.

  • Takeaways & Limitations

    The method is limited to single-room generation for a specific room type, relies on 3D-labeled scenes, and retrieves shapes and textures from CAD datasets.

Abstract

from arXiv · show

We present DiffuScene for indoor 3D scene synthesis based on a novel scene configuration denoising diffusion model. It generates 3D instance properties stored in an unordered object set and retrieves the most similar geometry for each object configuration, which is characterized as a concatenation of different attributes, including location, size, orientation, semantics, and geometry features. We introduce a diffusion network to synthesize a collection of 3D indoor objects by denoising a set of unordered object attributes. Unordered parametrization simplifies and eases the joint distribution approximation. The shape feature diffusion facilitates natural object placements, including symmetries. Our method enables many downstream applications, including scene completion, scene arrangement, and text-conditioned scene synthesis. Experiments on the 3D-FRONT dataset show that our method can synthesize more physically plausible and diverse indoor scenes than state-of-the-art methods. Extensive ablation studies verify the effectiveness of our design choice in scene diffusion models.

1 Technical University of Munich 2 Sony Europe RDC Stuttgart 3 Technical University of Darmstadt https://tangjiapeng.github.io/projects/DiffuScene

DiffuScene supports diverse indoor scene synthesis and three downstream uses: completing partial scenes, arranging given objects, and generating scenes from partial text-described configurations.

  • DiffuScene supports scene completion from partial scenes, scene arrangement of given objects, and text-conditioned generation from partial configurations.

1. Introduction

Indoor scene synthesis seeks realistic, semantically meaningful, and diverse 3D scenes, but traditional optimization is laborious and rule-based while existing generative models face diversity or fidelity limitations. DiffuScene addresses this with unordered object-set diffusion over comprehensive scene attributes and supports multiple synthesis applications.

  • Motivation: Realistic, semantically meaningful, and diverse indoor scene synthesis remains a long-standing computer-graphics problem with applications in games, films, virtual reality, and interior design.
  • Motivation: Traditional methods rely on hand-defined design rules and iterative optimization, which demand artistic expertise, computational effort, and can limit compositional diversity.
  • Existing limitations: GANs often lack diversity through limited mode coverage, whereas VAEs offer better diversity but lower-fidelity results.
  • Approach: DiffuScene represents scenes as unordered object sets whose attributes include location, size, orientation, semantics, and geometry features.
  • Approach: Shape-code diffusion supports natural object arrangements, including symmetric relations commonly found in real scenes.
  • Applications: The method enables completion from partial scenes, object rearrangement in existing scenes, and text-conditioned scene synthesis.

2. Related work

Prior scene-synthesis methods use optimization or learned generative models, while DiffuScene models joint object compositions through unordered-set diffusion and predicts comprehensive 3D instance properties.

  • Traditional methods: Traditional scene synthesis uses data-driven optimization constrained by interior-design guidelines, object frequencies, affordances, or arrangement examples.
  • Learned generative models: Learned scene-synthesis models include feed-forward networks, VAEs, GANs, and autoregressive models, with trade-offs between quality, diversity, and mode coverage.
  • DiffuScene: DiffuScene parameterizes each object with location, size, orientation, class label, and latent shape code, then applies denoising diffusion to the scene set.
  • DiffuScene: Unlike approaches that predict objects sequentially, DiffuScene jointly models unordered object compositions and denoises multiple properties synchronously.
  • 3D diffusion models: Compared with prior 3D diffusion work, DiffuScene predicts 3D locations, orientations, semantics, and geometries and supports completion and multimodal conditional synthesis.

3. DiffuScene

DiffuScene represents indoor scenes as unordered sets of object attributes and learns to denoise these configurations into complete scenes. Its diffusion framework supports geometry-aware synthesis, physically informed training, and downstream completion, rearrangement, and text-conditioned generation.

  • Object Set Diffusion: DiffuScene represents each scene as a fixed-size unordered object set containing class, size, location, rotation, and shape-code attributes.Variable object counts are handled by padding scenes with an additional empty object.
  • Diffusion Process: The forward process gradually adds Gaussian noise to scene configurations, while the learned reverse process denoises samples from Gaussian noise into clean scene configurations.The scene configuration is represented as a 2D tensor of concatenated object properties with a scheduled noise variance.
  • Denoising Network: The denoising network processes multiple object attributes with 1D convolutions, skip connections, and attention blocks that aggregate inter-object relationships and global scene context.The network predicts corrupted noise during training, supporting reverse diffusion toward plausible scenes.
  • Training Objective: Training combines a scene-distribution loss with an IoU-based regularizer that penalizes intersections between object bounding boxes.The scene loss is simplified into an objective for correctly predicting noise added to corrupted scene configurations.
  • Applications: DiffuScene supports scene completion, rearrangement, and text-conditioned synthesis by conditioning diffusion on known objects, fixed attributes, or language embeddings.Rearrangement updates locations and orientations, while text conditioning injects BERT-derived embeddings through cross-attention layers.

4. Experiments

Experiments on 3D-FRONT evaluate DiffuScene for unconditional synthesis, ablations, scene completion, rearrangement, and text-conditioned synthesis. Across these settings, the method produces diverse, plausible scenes with improved object relationships, while its main limitations concern retrieved shapes, textures, room scope, and labeled-data requirements.

  • Experimental Setup: The evaluation uses 3D-FRONT rooms, comparing DiffuScene with state-of-the-art generative scene-synthesis baselines across multiple tasks and metrics.The benchmark includes bedrooms, dining rooms, and living rooms, with FID, KID, SCA, and CKL measuring plausibility and diversity.
  • Unconditional Scene Synthesis: DiffuScene consistently outperforms competing methods on unconditional-synthesis metrics, generating more diverse and plausible scenes with fewer object intersections.Qualitative comparisons also report more symmetric pairs and better scene arrangements.
  • Ablation Studies: IoU loss penalizes object intersections, promotes more reasonable placements, and preserves symmetries, producing consistent improvement across metrics.This comparison evaluates the effect of IoU loss within the ablation study.
  • Ablation Studies: 0.72 (w/ shape diffusion) vs. 0.50 (w/o shape diffusion) for Sym demonstrates improved symmetric placements from geometry feature diffusion.The geometry-aware diffusion and shape retrieval also support semantically coherent arrangements and more plausible synthesis results.
  • Downstream Applications: DiffuScene improves downstream scene completion, rearrangement, and text-conditioned synthesis through more diverse, faithful, symmetric, and realistic outputs.In the user study, 62% preferred DiffuScene for realism and 55% for text matching against ATISS.
  • Limitations: The method is limited by same-semantics shape retrieval, dataset-derived textures, single-room and single-room-type training, and reliance on 3D-labeled scenes.The paper identifies texture diffusion, multi-room synthesis, and learning from 2D-labeled datasets as future directions.

5. Conclusion

DiffuScene is introduced as a denoising diffusion model for holistic indoor scene configuration, supporting completion, rearrangement, and text-conditioned synthesis. The method produces more plausible and diverse scenes than prior state-of-the-art methods, supported by metrics and user studies.

  • DiffuScene learns holistic indoor scene configurations through denoising diffusion over object semantics, placements, and geometries.
  • The method supports scene completion, scene rearrangement, and text-conditioned scene synthesis.
  • The authors report more plausible and diverse indoor scenes than prior state-of-the-art methods, confirmed by metrics and a user study.

A.1. Shape Auto-Encoder

The shape auto-encoder extracts latent codes from CAD-model point clouds, which DiffuScene scales for diffusion and later uses for geometry-aware retrieval. Generated instances retrieve the nearest same-class CAD geometry during inference.

  • A variational shape auto-encoder encodes each 2,048-point CAD-model cloud into a 512-dimensional global latent code.The encoder predicts the latent mean and variance.
  • Extracted shape codes are scaled to [−1, 1] before diffusion and rescaled during inference.
  • During inference, each generated instance retrieves the nearest CAD model with the same class label and closest geometry feature.

B. Dataset

The dataset pipeline filters problematic 3D-FRONT scenes, defines bedroom and dining/living-room object categories, and generates partial-scene text prompts from sampled objects and spatial relations.

  • Preprocessing: Preprocessing removes scenes with severe intersections, incorrect labels, unnatural room sizes, or unsuitable object counts.Natural rooms are constrained to floors within 6m × 6m and heights below 4m.
  • Object categories: The dataset includes separate bedroom and dining/living-room object-category lists.
  • Text prompt generation: Text prompts describe partial scene configurations using one to three sentences about sampled object categories and relations.
  • Text prompt generation: Valid object pairs require distances below 1.5, after which relative orientations and translations determine relation descriptions.

D. Ablation Studies

Ablations show that DiffuScene’s UNet-1D with attention and geometry feature diffusion improve scene-distribution modeling and symmetric, semantically coherent arrangements.

  • Network architecture: UNet-1D with attention layers aggregates object features and inter-object relationships to learn global scene context.Compared with a pure transformer, it produced marginally better FID, KID, SCA, and CKL metrics.
  • Geometry feature diffusion: Geometry feature diffusion improves symmetric placements, with Sym increasing from 0.50 without shape diffusion to 0.72 with it.
  • Geometry feature diffusion: Geometry feature diffusion also improves FID, KID, and SCA while lowering CKL.

E. Additional Results

Additional experiments evaluate DiffuScene across diversity, unconditional synthesis, scene arrangement, scene completion, real-world generalization, and text-conditioned synthesis. The results report diverse and plausible outputs, stronger completion quality, and text-consistent scene generation, while supporting text-guided editing.

  • Diversity Analysis: FID, KID, Precision, and Recall are used to quantify generated-scene diversity by comparing generated and reference scene distributions.The evaluation also compares mean and covariance and uses precision/recall as complementary generative-model metrics.
  • Unconditional Scene Synthesis: DiffuScene consistently outperforms competing methods across all listed metrics, indicating more diverse and plausible unconditional scenes.The comparisons use FID, KID, SCA, and CKL to assess plausibility and diversity.
  • Scene Arrangement: Compared with ATISS and LEGO, DiffuScene generates more varied and plausible object placements with more symmetries for scene arrangement.DiffuScene predicts full 3D instance properties, including locations, orientations, semantics, and geometries.
  • Scene Completion: Compared with ATISS, DiffuScene produces more diverse, higher-fidelity scene completions and consistently outperforms it on all listed metrics.The quantitative comparison covers 3D-FRONT bedrooms, dining rooms, and living rooms with only three objects provided as input.
  • Real-world Scene Generalization: Without finetuning on real-world scenes, DiffuScene produces a more favourable scene than ATISS for scene completion.The model is trained on a synthetic dataset and evaluated on real-world scenes.
  • Text-conditioned Scene Synthesis: In text-conditioned synthesis, DiffuScene avoids object intersections, follows specified object relations and counts, and supports text-guided object suggestion and scene editing.The editing operation does not change the attributes of other objects.

F. User Study

The user study compares DiffuScene and ATISS on text-conditioned scene synthesis using judgments of text matching and realism. It presents randomized result pairs to reduce comparison bias and collects responses across 225 scenes from 45 users.

  • Study Design: Users judge each synthesized-scene pair on which result better matches the text prompt and which is more reasonable and realistic.A reference scene is used to generate the text prompt shown in the study interface.
  • Study Design: The study collects responses for 225 scenes from 45 users.The compared methods are ATISS and DiffuScene, with results randomly shuffled in the interface.
  • Qualitative Comparisons: The additional qualitative examples cover unconditional synthesis, scene completion, scene arrangement, and text-conditioned scene synthesis.The examples are described as diverse, plausible, higher-fidelity, or better matched depending on the task.
  • Study Design: ATISS and DiffuScene outputs are randomly shuffled to avoid bias in the user comparisons.The same randomized presentation is used when users assess text matching and realism.
Loading 2303.14207v2…