PaperlayerView exact revision on arXiv

Source-linked AI summary

RoomDesigner: Encoding Anchor-latents for Style-consistent and Shape-compatible Indoor Scene Generation

Yiqun Zhao, Zibo Zhao, Jing Li, Sixun Dong, Shenghua Gao

arXiv:2310.10027v1cs.CVcs.GR

TL;DR

Existing indoor-scene methods often generate plausible furniture layouts without representing individual furniture shapes, leaving shape compatibility and style consistency insufficiently addressed. RoomDesigner encodes furniture with anchor-latents and predicts scenes autoregressively; it reports more consistent, compatible scenes and supports shape generation and object-level applications.

  • Problem

    Many existing approaches generate plausible furniture layouts without incorporating individual furniture-shape details, and retrieval can yield shape-incompatible or style-inconsistent arrangements.

    Source: §1 · p. 2

  • Method

    RoomDesigner encodes furniture geometry and structure as anchor-latents, then uses a transformer to autoregressively generate furniture location and shape information.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    The model reports improved style consistency and shape compatibility, with compelling room-mask-conditioned generation results against state-of-the-art scene-generation models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach supports furniture-shape generation and applications including scene completion, mismatch correction, and controllable object-level editing.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Indoor scene generation aims at creating shape-compatible, style-consistent furniture arrangements within a spatially reasonable layout. However, most existing approaches primarily focus on generating plausible furniture layouts without incorporating specific details related to individual furniture pieces. To address this limitation, we propose a two-stage model integrating shape priors into the indoor scene generation by encoding furniture as anchor latent representations. In the first stage, we employ discrete vector quantization to encode furniture pieces as anchor-latents. Based on the anchor-latents representation, the shape and location information of the furniture was characterized by a concatenation of location, size, orientation, class, and our anchor latent. In the second stage, we leverage a transformer model to predict indoor scenes autoregressively. Thanks to incorporating the proposed anchor-latents representations, our generative model produces shape-compatible and style-consistent furniture arrangements and synthesis furniture in diverse shapes. Furthermore, our method facilitates various human interaction applications, such as style-consistent scene completion, object mismatch correction, and controllable object-level editing. Experimental results on the 3D-Front dataset demonstrate that our approach can generate more consistent and compatible indoor scenes compared to existing methods, even without shape retrieval. Additionally, extensive ablation studies confirm the effectiveness of our design choices in the indoor scene generation model.

1. Introduction

Indoor scene methods often predict furniture boxes and retrieve shapes, which can leave furniture shape-incompatible or style-inconsistent and cannot produce novel shapes. RoomDesigner encodes furniture geometry and structure as anchor-latents and uses them in autoregressive scene generation, supporting compatible arrangements and object-level interaction tasks.

1. Introduction

  • Retrieval-based methods can produce shape-incompatible or style-inconsistent furniture and cannot generate novel shapes; part-level alternatives require annotations often unavailable in 3D-Front.Bounding-box parameters do not capture complex furniture geometry and structure.Source: §1 · p. 2
  • Anchor-latents combine global structure from sampled anchor points with local geometry features encoded by VQ-VAE and decoded into occupancy fields.The local features let anchors produce smooth surfaces with geometry detail.Source: §1 · p. 2
  • Furniture is parameterized by anchor-latents, location, size, rotation, and category, which a transformer models through autoregressive next-token prediction.The prediction is over an unordered set of furniture items.Source: §1 · p. 2
  • The model reports improved style consistency and shape compatibility, and supports scene completion, furniture mismatch correction, and controllable object-level editing.The anchor-latents also facilitate generating novel shapes for each furniture category.Source: §1 · p. 2
  • The paper presents anchor-latents as encoding furniture geometry and structure without part-level annotations, and pairs them with a geometry-aware scene-generation transformer.The stated applications include style-consistent and shape-compatible generation.Source: §1 · p. 2

2. Related Work

Prior indoor-scene synthesis spans rule-based methods and deep generative models, including VAEs, GANs, autoregressive models, and diffusion. Related approaches address furniture shape in different ways, but face limits in shape sensitivity, template topology, geometry quality, annotation needs, or scene-level computational cost.

Indoor scene generation

  • Traditional scene synthesis relies on human-defined constraints, while deep generative methods include VAEs, GANs, autoregressive models, and diffusion models.ATISS [35] uses an unordered-set representation but does not model furniture shape sensitivity.Source: §2 · p. 2
  • One approach predicts deformation for each furniture-template vertex using differentiable rendering [25] with multiview images, but limited template-mesh topology prevents reasonable shapes; volume-rendering methods [1, 50] cannot produce high-quality geometry for each furniture piece.Source: §2 · p. 3
  • SceneHGN [18] uses a hierarchical part-level VAE for furniture shapes but depends on part-level annotations.Source: §2 · p. 3

3D shape representation

  • Shape representations trade off detail and efficiency: global latent vectors capture high-level structure but miss geometry details, while local representations can be too costly to scale to scenes.The passage motivates representations combining global structure with local detail.Source: §2 · p. 3

3. Method

RoomDesigner encodes furniture geometry as anchor-latents and combines them with object attributes for autoregressive, room-mask-conditioned scene generation. Its method learns the furniture representation and scene and shape generation components through a two-stage model, with layout and shape generation jointly trained.

3. Method

  • The overall model combines a VQ-VAE furniture encoder with a geometry-aware transformer that autoregressively produces indoor scenes.The transformer is followed by a shape-generation branch based on the anchor-latents.Source: §3 · p. 3

3.1. Anchor-latents representation

  • Anchor-latents compress furniture geometry into sampled anchor points and quantized features, trained to predict occupancy fields.The encoding uses 512 anchor points, 32-neighbor local patches, and a codebook of 1,024 entries.Source: §3.1 · p. 3

3.2. Autoregressive Shape-aware Scene Generation

  • Furniture is represented by category, size, rotation, translation, and anchor-latents, and scene generation models objects conditioned on the floor plan and previously generated objects.The object sequence is trained with permutations to support order invariance.Source: §3.2 · p. 4
  • A scene-generation transformer uses room-mask, box, and anchor-latent embeddings to predict the next furniture token, followed by attribute and shape generation.The predicted box attributes condition autoregressive generation of the next object's anchor-latents.Source: §3.2 · p. 5

3.3. Training

  • The model is trained by randomly selecting and permuting prior objects, then predicting the next object's box attributes and anchor-latents from the room and object context.Layout and shape generation are jointly trained rather than trained in separate stages.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate room-mask-conditioned generation, shape consistency and diversity, ablations, and downstream interaction tasks. The reported results indicate improved scene consistency and compatibility, with anchor-latents supporting completion, mismatch correction, and object-level editing.

4. Experiments

  • Experiments use filtered 3D-Front bedrooms, dining rooms, and living rooms, with 3D-Future furniture; evaluation covers scene quality, collisions, shape consistency, and diversity.The comparison includes ATISS and Sync2Gen; Sync2Gen is adapted to condition on the room mask for a fair comparison.Source: §4 · pp. 5–6

4.1. Room mask conditioned scene generation.

  • Our method improves style consistency and shape compatibility over baselines, while anchor-latents capture both global structure and furniture geometry details.The comparisons include room-mask-conditioned generation and shape consistency; Table 1 reports quantitative room-mask comparisons.Source: §4.1 · p. 6, Table 1 · p. 6

4.2. Ablation studies

  • Fewer anchor points reduce shape awareness and can worsen collisions, FID, and SCA; two-stage training moderately declines FID and SCA and seriously increases CKL.The ablations examine anchor-point count and transformer training strategy on 3D-Front bedrooms.Source: §4.2 · pp. 6–7, Table 4 · p. 7

4.3. Downstream tasks.

  • The model supports shape-compatible scene completion, mismatch correction by resampling low-likelihood objects, and shape editing by mixing anchor points.The completion comparison is with ATISS; the paper states that previous scene-synthesis methods cannot perform the described mismatch correction or controllable editing.Source: §4.3 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

5. Conclusion

RoomDesigner jointly generates room layouts and furniture shapes using anchor-latents that capture furniture geometry and structure. The authors report style-consistent, shape-compatible generation and support for scene completion, mismatch correction, and furniture-level editing.

5. Conclusion

  • RoomDesigner jointly synthesizes room layouts and furniture shapes, then autoregressively predicts each furniture item's location and shape information.Its anchor-latent representation captures both global structure and local geometry for each furniture piece.Source: §5 · p. 8
  • The model supports shape-compatible, style-consistent scene completion, style-mismatch detection and correction, and controllable furniture-level editing.The authors also report compelling results on room-mask conditional generation.Source: §5 · p. 9

Supplementary Material

The supplementary material provides implementation details, controllable-editing results, ablation visuals, and additional generation visualizations.

Supplementary Material

A.1. Network Architecture and loss functions

The scene-generation transformer combines bounding-box and anchor-latent features to predict furniture tokens, while a separate transformer autoregressively generates furniture shape anchors.

A.1. Network Architecture and loss functions

  • The scene transformer has 4 layers and 16 attention heads, with 1024-dimensional furniture tokens combining 512-dimensional bounding-box and anchor-latent features.A learnable 64-dimensional object query predicts the output feature for the next furniture prediction.Source: §A.1 · p. 13
  • Category uses cross-entropy loss, while size, translation, and rotation are modeled with mixture-of-logits distributions and negative log-likelihood losses.The mixture count is K = 10, following ATISS [35].Source: §A.1 · p. 13
  • The 12-layer shape-generation transformer sums learned embeddings for anchor coordinates x, y, z and quantized codebook indices, then predicts those attributes sequentially.The first six layers predict x embeddings, followed by two layers each for y and subsequent features.Source: §A.1 · p. 13, Figure 7 · p. 13
  • A separate two-layer MLP decodes anchor points, whose coordinates are modeled with mixture-of-logits distributions.Source: §A.1 · p. 14
  • The shape branch uses negative log-likelihood losses for quantized-index classification and anchor-coordinate distributions.The stated coordinate loss expression includes y and z terms.Source: §A.1 · p. 14
  • At inference, the model starts from Cshape and autoregressively predicts anchor-latents until M = 512.Source: §A.1 · p. 14

A.2. Training Details

Training uses separate schedules for the shape VQVAE and scene-generation stage, with both models optimized using Adam.

A.2. Training Details

  • The shape VQVAE trains for 1000 epochs on 4 NVIDIA A40 GPUs, using batch size 16 per GPU and learning rate 1e-3.It uses N = 2048 sample points, M = 512 latent anchors, k = 32 for k-NN, and a 1024-entry codebook with latent dimension 256.Source: §A.2 · p. 14
  • The scene-generation stage trains for 500k iterations with batch size 128, learning rate 1e-3, and no weight decay on 8 NVIDIA A40 GPUs.Source: §A.2 · p. 14

A.3. Dataset processing details

The appendix describes 3D-Front preprocessing and evaluation procedures, then shows how mixed anchor-latents support furniture interpolation and scene-level editing.

A.3. Dataset processing details

  • Preprocessing removes mislabeled, unnatural-scale, invalid-count, and out-of-category scenes, leaving 4,041 bedrooms, 900 dining rooms, and 813 living rooms.Valid object counts are 3–13 for bedrooms and 3–21 for dining and living rooms; furniture meshes are made watertight and scaled to [−0.95, 0.95] for VQ-VAE training.Source: §A.3 · p. 14

A.4. Evaluation details

  • Evaluation measures collision rate, within-category shape consistency using Chamfer Distance and Open-Shape similarity, and shape diversity across multiple generated scenes per test mask.The procedures randomly generate 1,000 scenes for collision and consistency evaluations; diversity uses ten scenes per mask and selected furniture categories.Source: §A.4 · pp. 14–15

B. Controllable furniture-level to scene-level editing with Anchor-latents.

  • Mixing ordered anchor-latents from two furniture items yields an interpolated shape, which can replace repetitive furniture before shape-aware scene completion edits the scene.The described procedure replaces part of one item's anchor-latents with another's, decodes the mixture, then completes the scene after removing other repetitive items.Source: B. Controllable furniture-level to… · p. 15

C. Qualitative ablation studies

The qualitative ablation examines how anchor-point count affects generated furniture and layout. The appendix also describes three comparison baselines and their shape-retrieval setups.

C. Qualitative ablation studies

  • Fewer anchor-latents can produce incomplete, inconsistent furniture and do not significantly improve layout optimization because the transformer cannot learn geometric relations between furniture items.The authors attribute incomplete fine structures to point-cloud-density sampling: too few anchor points leave fewer latents for controlling those structures.Source: C. Qualitative ablation studies · p. 15

D. Baseline details

  • Sync2Gen is adapted from unconditional to room-mask-conditioned synthesis using a conditional VAE and the same ResNet-18 room-mask encoder as the proposed method.Its original approach uses Bayesian optimization after encoding scene arrangements with a VAE.Source: D. Baseline details · p. 15
  • ATISS predicts object bounding boxes and retrieves furniture by bounding-box size, whereas the proposed method directly generates each furniture item.ATISS uses a permutation-invariant transformer to predict attributes of the next object conditioned on previously combined object features.Source: D. Baseline details · p. 15
  • ATISS+OpenShape concatenates ATISS bounding-box information with pretrained OpenShape embeddings, then retrieves same-class furniture using embedding and size similarity.The stated nearest-neighbor search is over 3D-FUTURE furniture.Source: D. Baseline details · p. 15

E. Additional Visualization results

The additional visualizations cover living rooms, bedrooms, and dining rooms, and the qualitative ablation compares anchor-point counts under the same room mask.

E. Additional Visualization results

  • The room-mask-conditioned visualizations cover living rooms, bedrooms, and dining rooms, and are reported to show style-consistent, shape-compatible scenes with diverse furniture shapes.The paper notes that furniture shapes can remain diverse even when the bedroom layout is limited.Source: E. Additional Visualization results · pp. 17–18
  • With the same bedroom room mask, the 128-, 256-, and 512-anchor-point examples compare how anchor count affects generated furniture and layout.The caption identifies incomplete shapes, unreasonable layouts, and inconsistent shapes as outcomes fewer anchor-latents may lead to.Source: Figure 12 · p. 18
Loading 2310.10027v1…