PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixed Diffusion for 3D Indoor Scene Synthesis

Siyi Hu, Diego Martin Arroyo, Stephanie Debats, Fabian Manhardt, Luca Carlone, Federico Tombari

arXiv:2405.21066v2cs.CV

TL;DR

Floor-conditioned 3D scene synthesis requires predicting object categories and realistic geometric arrangements, a setting where labels are easier to predict than positions and orientations. MiDiffusion combines discrete and continuous diffusion for scene attributes and uses corruption-and-masking for partial constraints. The authors report that it outperforms state-of-the-art autoregressive and diffusion methods on 3D-FRONT, including with partial constraints.

  • Problem

    Floor-conditioned indoor scene synthesis must predict both object categories and realistic geometry, whose positions and orientations remain challenging.

    Source: §1 · p. 1

  • Method

    MiDiffusion jointly diffuses discrete object semantics and continuous geometry, with a time-variant transformer denoiser and corruption-and-masking for partial constraints without retraining.

    Source: §1 · pp. 1–2, §4 · p. 3, §4.2 · p. 4

  • Results

    On 3D-FRONT, MiDiffusion outperforms state-of-the-art autoregressive and diffusion methods, with reported advantages both with and without partial constraints.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Corruption-and-masking lets the model handle scene completion and furniture arrangement with partial object constraints without retraining.

    Source: §1 · p. 2, §7 · p. 9

    Objects are represented by bounding-box features and labels, which the authors describe as imprecise for 3D; the method also requires a model-retrieval strategy to compose the final scene.

    Source: §6 · p. 8

Abstract

from arXiv · show

Generating realistic 3D scenes is an area of growing interest in computer vision and robotics. However, creating high-quality, diverse synthetic 3D content often requires expert intervention, making it costly and complex. Recently, efforts to automate this process with learning techniques, particularly diffusion models, have shown significant improvements in tasks like furniture rearrangement. However, applying diffusion models to floor-conditioned indoor scene synthesis remains under-explored. This task is especially challenging as it requires arranging objects in continuous space while selecting from discrete object categories, posing unique difficulties for conventional diffusion methods. To bridge this gap, we present MiDiffusion, a novel mixed discrete-continuous diffusion model designed to synthesize plausible 3D indoor scenes given a floor plan and pre-arranged objects. We represent a scene layout by a 2D floor plan and a set of objects, each defined by category, location, size, and orientation. Our approach uniquely applies structured corruption across mixed discrete semantic and continuous geometric domains, resulting in a better-conditioned problem for denoising. Evaluated on the 3D-FRONT dataset, MiDiffusion outperforms state-of-the-art autoregressive and diffusion models in floor-conditioned 3D scene synthesis. Additionally, it effectively handles partial object constraints via a corruption-and-masking strategy without task-specific training, demonstrating advantages in scene completion and furniture arrangement tasks.

1. Introduction

Object-based 3D indoor scene synthesis predicts semantic labels and geometrically consistent object arrangements. While object labels are relatively easy to predict, realistic positions and orientations remain challenging, motivating joint denoising and mixed discrete-continuous diffusion.

1. Introduction

  • Realistic object positions and orientations remain challenging, and autoregressive models can diverge after an early bad prediction.The paper argues that jointly refining all objects through denoising is better suited to this problem.Source: §1 · p. 1
  • MiDiffusion combines discrete categorical corruption with continuous geometric diffusion because object attributes occupy both domains.The formulation is presented as a way to avoid representing semantic labels as continuous one-hot vectors.Source: §1 · pp. 1–2
  • Prior diffusion-based scene synthesis includes DiffuScene, which converts semantic labels to one-hot vectors before combining them with continuous geometric features.The paper argues that discrete diffusion captures categorical dependencies more naturally and effectively.Source: §1 · pp. 1–2
  • The paper introduces a floor-conditioned model, a compatible denoising network, and corruption-and-masking for partial object constraints without retraining.On 3D-FRONT, the authors report that MiDiffusion outperforms existing autoregressive and diffusion methods.Source: §1 · p. 2

2. Related Works

Prior 3D scene synthesis spans procedural, graph-based, and learned approaches. Diffusion has also been applied to layout synthesis, but MiDiffusion differs from prior scene-layout work by operating over mixed discrete and continuous domains.

2. Related Works

  • Learned 3D scene synthesis models capture inter-object relationships through feed-forward networks, VAEs, GANs, and autoregressive models, alongside procedural and graph-based approaches.Recent autoregressive models commonly use transformer backbones to model object interactions through attention.Source: §2 · p. 2
  • Diffusion models use a forward noise-injection process and a reverse denoising process; discrete variants model data in discrete state spaces.DDPMs typically use continuous latent variables, while D3PMs and VQ-Diffusion have shown results on character-level image generation and text-to-image synthesis.Source: §2 · p. 2
  • Layout diffusion includes document, scene, furniture-rearrangement, graph-conditioned, text-conditioned, and mesh-generation tasks.LayoutDM discretizes position and size, whereas DiffuScene uses continuous diffusion after converting semantic labels to one-hot encodings; MiDiffusion instead uses mixed domains.Source: §2 · p. 2

3. Preliminary: Diffusion Models

Diffusion models progressively corrupt data and learn a reverse process to recover it. The paper describes continuous Gaussian diffusion, conditional synthesis, and discrete categorical diffusion as preliminaries.

3. Preliminary: Diffusion Models

  • The forward process iteratively injects controlled noise, while a learned reverse process progressively denoises latent variables.Training minimizes a variational bound; with known forward noise, the approximate posterior has no learnable parameters, and efficient learning requires sampling from q(x_t|x_0) and tractable computation of q(x_{t-1}|x_t,x_0).Source: §3 · pp. 2–3
  • Conditional synthesis incorporates an input y into each reverse transition, replacing pθ(x_{t-1}|x_t) with pθ(x_{t-1}|x_t,y).The conditional reverse process is written as pθ(x_{0:T}|y).Source: §3 · p. 3
  • DDPM corrupts continuous variables with Gaussian noise under a fixed variance schedule, while discrete data can use categorical transition probabilities.For discrete variables, the denoising network predicts pθ(z_{t-1}|z_t); the described mask-and-replace strategy adds a special [MASK] token.Source: §3 · p. 3

4. 3D Scene Synthesis via Mixed Diffusion

MiDiffusion represents floor-conditioned indoor scenes as object categories and continuous geometry, then jointly denoises those domains. Its transformer denoiser supports object constraints through corruption and masking, without additional training.

4. 3D Scene Synthesis via Mixed Diffusion

  • Each scene contains a floor plan and up to N objects, each represented by a semantic label, centroid, size, and rotation; empty objects use a designated label and zero geometry.Rotation is represented by [cos(γ), sin(γ)]ᵀ to maintain continuity on SO(2).Source: §4 · p. 3

4.1. Diffusion in Mixed Discrete-Continuous Domains

  • MiDiffusion independently corrupts discrete semantic labels and continuous geometry, and its resulting loss is the exact sum of the D3PM and DDPM variational bounds.The reverse step predicts each domain conditioned on the other current latent and the floor plan.Source: §4.1 · pp. 3–4

4.2. Denoising Network

  • A transformer denoiser processes semantic and geometric features jointly, predicting a categorical distribution for labels and Gaussian means for geometry, conditioned on the floor plan.Object geometry is encoded with an MLP and combined with trainable semantic embeddings; the floor plan feature comes from sampled boundary points processed by PointNet.Source: §4.2 · p. 4

4.3. Partial Condition on Objects

  • For scene completion, known objects follow their precomputed corruption trajectories during reverse denoising, so their latents converge to the inputs without additional training.The corruption-and-masking strategy can also target different latent sections for furniture arrangement and label-conditioned scene synthesis.Source: §4.3 · p. 5

5. Experimental Evaluation

The evaluation compares MiDiffusion with ATISS and DiffuScene on 3D-FRONT floor-conditioned scene synthesis, then tests whether the same models handle scene completion and furniture arrangement without retraining.

5. Experimental Evaluation

  • Evaluation uses 3D-FRONT bedrooms, dining rooms, and living rooms, comparing against ATISS and DiffuScene with layout-image and object-placement metrics.The benchmark includes 18,797 rooms; 1,000 test-set floor plans are sampled for evaluation.Source: §5 · p. 5

5.1. Floor Plan Conditioned 3D Scene Synthesis

  • MiDiffusion remains superior in object placement and floor-boundary compatibility, with stronger layout-image metrics especially for dining and living rooms; DiffuScene has the lowest IoU% but more out-of-boundary objects.The reported trade-off is between keeping objects inside the boundary and minimizing overlap; the supplied passages do not provide table cell values.Source: §5.1 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • The two-stage discrete-label-then-geometry approach performs significantly worse across all metrics than joint MiDiffusion, while mixed diffusion also beats DDPM with the same floor-plan extractor.The complete Mixed+PointNet formulation gives the best results; PointNet helps dining and living rooms but is slightly worse than ResNet in bedrooms.Source: §5.1 · p. 6, Table 3 · p. 7
  • All three approaches have similar overall diversity; MiDiffusion gives the most diverse in-boundary positions but slightly less size variety than DiffuScene.Diversity is measured by average standard deviation of object centroid positions and bounding-box sizes; “IB” refers to in-boundary objects.Source: §5.1 · p. 7, Table 4 · p. 8

5.2. Floor Plan Conditioned 3D Scene Synthesis with Object Constraints

  • A corruption-and-masking strategy reuses MiDiffusion for partial-object constraints without task-specific retraining.The constrained attributes follow the reverse of a precomputed corruption process during denoising.Source: §5.2 · p. 7

5.2.1 Scene completion

  • Without retraining, MiDiffusion beats the baselines with one existing object and remains better for dining and living rooms with three, but DiffuScene-SC wins on bedrooms with three.The authors note that DiffuScene-SC is optimized for three objects and bedrooms average 5.22 objects.Source: §5.2.1 · pp. 7–8, Table 5 · p. 8

5.2.2 Furniture Arrangement

  • Without retraining, MiDiffusion outperforms DiffuScene-FA on all furniture-arrangement metrics.ATISS is not suited to this setup because replacing predicted labels with constraints breaks its autoregressive stopping condition.Source: §5.2.2 · p. 8, Table 6 · p. 8

6. Limitations

The paper identifies limits in its object representation and in assembling the final 3D scene.

6. Limitations

  • Bounding-box features and labels are not the most precise representation of objects in 3D.Source: §6 · p. 8
  • MiDiffusion still requires a model-retrieval strategy to compose the final 3D scene configuration.Source: §6 · p. 8
  • The authors suggest that stronger 3D shape priors and mesh and texture synthesis could enhance realism.Source: §6 · p. 8

7. Conclusion

The conclusion presents MiDiffusion as a mixed diffusion model for object-based 3D scene synthesis and reports advantages across evaluations with and without partial constraints.

7. Conclusion

  • MiDiffusion combines DDPM and D3PM for object-based 3D indoor scene synthesis and uses separate predictions for discrete semantic and continuous geometric attributes.Source: §7 · p. 8
  • MiDiffusion shows a clear advantage over state-of-the-art autoregressive and diffusion models with or without partial constraints.Source: §7 · p. 9

A.1. DDPM forward process re-parameterization, simplified loss

  • The DDPM appendix reparameterizes the forward process and uses a simplified variational-bound objective to compute LDDPM.Source: §A.1 · p. 12

A.2. D3PM reverse parameterization, auxiliary loss

  • The D3PM appendix describes an auxiliary denoising objective that encourages predictions of z0 at each time step and is implemented through the backward-step parameterization.Source: §A.2 · p. 12

B. MiDiffusion Loss Factorization

C. Network and Implementation Details

MiDiffusion uses a transformer decoder architecture with separate semantic and geometric prediction heads.

C.1. Network Hyper-parameters

  • The denoising network uses eight transformer blocks with hidden dimension 512, four heads, and feed-forward internal dimension 2048.Source: §C.1 · p. 12
  • Semantic attributes use learned 512-dimensional embeddings, while geometric attributes and floor-plan features are mapped to 512 and 64 dimensions, respectively.Source: §C.1 · p. 12
  • Separate MLP decoders produce a categorical distribution over semantic labels and an eight-dimensional Gaussian mean for geometric attributes.Source: §C.1 · pp. 12–13

C.2. Implementation

The implementation uses a linear noise schedule over 1,000 diffusion steps and trains on a single NVIDIA V100 GPU.

C.2. Implementation

  • Bedroom models train for 50k epochs with learning-rate decay every 10k epochs, while living- and dining-room models train for 100k epochs with decay every 15k epochs.Source: §C.2 · p. 13
  • The method uses Adam with fixed learning rate 2e−4 and dropout ratio 0.1 in transformer-block attention and feed-forward layers, plus a linear schedule over 1,000 diffusion steps for all forward-process noise parameters.Source: §C.2 · p. 13
  • Training uses a single NVIDIA V100 with under 8GB GPU RAM and takes about 20 hours for living and dining rooms or 32 hours for bedrooms.Source: §C.2 · p. 13

D. Example Synthesized Layout Images

The evaluation images show top-down layouts across bedroom, dining room, and living room scenes. MiDiffusion produces realistic arrangements on harder living-room and dining-room datasets while respecting floor boundaries.

D. Example Synthesized Layout Images

E. Additional Experimental Evaluations

The supplementary evaluations add ablations, partial-constraint comparisons and applications, plus an unconditioned scene-synthesis comparison with DiffuScene.

E. Additional Experimental Evaluations

E.1. Ablation Study

Supplementary experiments compare staged and mixed diffusion, masking strategies, additional constraint types, and unconditioned synthesis. They report MiDiffusion's advantages in the ablation and masking comparisons, while finding similar unconditioned results to DiffuScene.

E.1. Ablation Study

  • Even with ground-truth labels, the two-stage variant underperforms MiDiffusion in scene-layout realism.Bypassing the semantic diffusion stage improves the two-stage variant, but does not close the gap.Source: §E.1 · p. 14
  • Across all three datasets, OOB % improves as the floor-plan feature extractor and diffusion framework are changed toward MiDiffusion's final design.Source: §E.1 · p. 14

E.2. Masking Comparison

  • The proposed corruption-and-masking strategy yields better furniture-arrangement results than direct masking at each denoising step.Source: §E.2 · p. 14

E.3. Additional Applications

  • Label-constrained synthesis and scene completion are demonstrated, including a dining room conditioned on five chairs, an uncommon count.The examples combine label constraints with existing objects for scene completion.Source: §E.3 · p. 14, Figure 8 · p. 15, Figure 9 · p. 15

E.4. Unconditional 3D Scene Synthesis

  • In unconditioned synthesis, MiDiffusion achieves results similar to DiffuScene despite using a denoising transformer selected for floor-conditioned synthesis.The comparison uses the released DiffuScene weights and modifies MiDiffusion by removing floor-plan-related modules.Source: §E.4 · p. 15, Table 10 · p. 15, Table 11 · p. 15

F. Additional Layout Examples

Related papers

Loading 2405.21066v2…