PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneFormer: Indoor Scene Generation with Transformers

Xinpeng Wang, Chandan Yeshwanth, Matthias Nießner

arXiv:2012.09793v2cs.CV

TL;DR

SceneFormer addresses indoor scene generation by representing scenes as object sequences and autoregressively predicting object properties without visual information. Conditioned on room layouts or text, it generates realistic, diverse scenes, with layout-conditioned scenes preferred over FastSynth in user studies and fast inference reported.

  • Problem

    Existing indoor scene-generation methods use visual information and assumptions about possible object relations, motivating a method that avoids both.

  • Method

    SceneFormer uses transformers to represent scenes as object-property sequences, predict object categories and properties autoregressively, and implicitly learn object relations through self-attention.

  • Results

    SceneFormer generates realistic and diverse scenes without visual information, supports room-layout and text conditioning, and layout-conditioned scenes are preferred over FastSynth 53.9% for bedrooms and 56.7% for living rooms.

  • Takeaways & Limitations

    SceneFormer provides a flexible framework for indoor scene generation that can learn object relations from data and perform fast inference.

  • Takeaways & Limitations

    The model does not use an additional loss to enforce that generated objects remain within the input floor region.

Abstract

from arXiv · show

We address the task of indoor scene generation by generating a sequence of objects, along with their locations and orientations conditioned on a room layout. Large-scale indoor scene datasets allow us to extract patterns from user-designed indoor scenes, and generate new scenes based on these patterns. Existing methods rely on the 2D or 3D appearance of these scenes in addition to object positions, and make assumptions about the possible relations between objects. In contrast, we do not use any appearance information, and implicitly learn object relations using the self-attention mechanism of transformers. We show that our model design leads to faster scene generation with similar or improved levels of realism compared to previous methods. Our method is also flexible, as it can be conditioned not only on the room layout but also on text descriptions of the room, using only the cross-attention mechanism of transformers. Our user study shows that our generated scenes are preferred to the state-of-the-art FastSynth scenes 53.9% and 56.7% of the time for bedroom and living room scenes, respectively. At the same time, we generate a scene in 1.48 seconds on average, 20% faster than FastSynth.

1. Introduction

SceneFormer represents indoor scenes as object-property sequences and uses transformers to generate realistic scenes from layouts or text without visual inputs or manually annotated relations.

  • Motivation: Realistic indoor scene generation supports applications including virtual walkthroughs and interactive visualization for real estate and interior furnishing.These applications can reduce the need to physically rearrange objects when presenting furnished rooms.
  • Core approach: SceneFormer autoregressively predicts each object's category, location, orientation, and size as a sequence.The generated properties determine object placement, after which relevant CAD models are retrieved and inserted.
  • Design rationale: The approach avoids visual information such as 2D renderings and requires little domain knowledge or data preparation.It instead treats a scene as a sequence of objects and learns patterns from scene datasets.
  • Conditional generation: SceneFormer generates scenes conditioned on either room layouts or text descriptions through transformer attention mechanisms.Layout inputs include room shape and door and window positions; text inputs describe objects and spatial relations.
  • Core approach: The model uses self-attention to implicitly learn object relations directly from raw object locations and orientations, avoiding manual relation annotations.This contrasts with methods that assume a fixed set of possible relations.

2. Related Work

Prior indoor scene-generation methods use graphs, matrices, hierarchies, images, or interactive text workflows, whereas SceneFormer sequentially generates objects from layout information and supports complex text-conditioned scenes.

  • Graph-based generation: Graph-based methods represent objects and room features as nodes connected by semantic relations, then autoregressively expand the graph.Graph convolutional networks are a natural processing choice for this representation.
  • Alternative representations: Other approaches represent scenes as category matrices or object hierarchies, use probabilistic placement, or condition on full 3D scans.These alternatives may require complex optimization and post-processing to obtain realistic scenes.
  • Image-based generation: Image-based methods encode top-down objects, walls, and floors with binary images and use CNN architectures for scene generation.Their representation supports arbitrary normalized room shapes and sizes.
  • Text-conditioned generation: Text-conditioned prior methods often create partial scenes or rely on interactive user input for refinement.SceneFormer is described as generating high-quality complex scenes with many objects without requiring user input.
  • SceneFormer pipeline: Figure 2 depicts layout-conditioned SceneFormer taking room shape and door and window locations as input, then sequentially inserting generated objects.The final scene appears after the object-generation sequence is completed.

3. Method

SceneFormer represents indoor scenes as autoregressive sequences of object properties and uses separate transformer models to generate categories, orientations, locations, and dimensions. Layout and text conditioning are implemented through transformer attention, while inference and data preparation impose structured ordering and quantization.

  • Scene representation: SceneFormer treats each scene as an ordered sequence of objects and trains separate models to autoregressively predict each object's category, orientation, location, and size.Object ordering is based on class-category frequency, producing a unique representation up to ordering among same-class objects.
  • Autoregressive factorization: Each model predicts its property conditioned on previously generated objects and, where applicable, outputs from preceding property models.The category model uses previous objects; later models additionally receive already generated categories, orientations, locations, or dimensions as specified by the factorization.
  • Model design: A single transformer was difficult to optimize and produced unrealistic scenes, so the design uses separate property models; dimension inputs were omitted because they did not improve performance.The authors report that likely dimensions can instead be inferred from object categories and locations.
  • Inference: Inference generates category, orientation, location, and dimension tokens in sequence, running location and dimension models three times each for x, y, z and l, w, h.Nucleus sampling with p = 0.9 is used for categories, while the other models select maximum-score tokens; a stop token terminates generation.
  • Layout conditioning: The layout-conditioned model encodes a 512 × 512 floor image, inserts door and window locations as leading tokens, and applies decoder cross-attention to the resulting sequence.Walls are assumed to lie at the floor edges, and no additional loss enforces that generated objects remain within the input floor region.
  • Text conditioning and scene assembly: The text-conditioned variant embeds the first three description sentences and uses cross-attention for category and location generation without an auxiliary alignment loss.Orientation and dimension decoders are replaced by encoders because the descriptions specify object classes and spatial relations; generated CAD models are selected by dimension similarity with collision handling.

4. Results

SceneFormer generates diverse, realistic indoor scenes, supports scene completion and text conditioning, learns object relations from data, and provides faster inference than prior methods.

  • Qualitative generation: SceneFormer generates complex scenes with learned object categories, relations, and placements, including objects on walls and ceilings beyond floor-supported objects.The model captures relations such as television placement opposite beds or sofas.
  • Scene completion: Scene completion adds relevant missing objects sequentially, with intermediate stopping producing realistic scenes suitable for interactive control.Training on sorted sequences causes large and frequent objects to appear before small and infrequent ones.
  • Scene diversity: Multiple scenes conditioned on the same room shape vary in object sets and orientations while remaining consistent, indicating novel rather than memorized generation.Nearest-neighbor comparisons use room shape and object class.
  • Object relations: Heatmaps for bedroom and living-room scenes show that the model captures common relative-location patterns between object pairs.The comparisons include ground-truth and generated scenes.
  • Text conditioning: Text-conditioned scenes largely follow described object categories and relations while adding objects that improve scene completeness.The text-conditioned examples are paired with their input descriptions.
  • Perceptual evaluation: 53.5%–65% of perceptual-study comparisons preferred the model's scenes over other methods across studies with 30 users each.Users selected the more realistic image from paired comparisons after vigilance-test filtering.
  • Ablation studies: A 14.5-point accuracy improvement resulted from using multiple transformers, while rotation and jitter augmentation produced small accuracy gains.The ablations measure next-token prediction accuracy on a fixed bedroom validation set.
  • Timing: 20% faster than Fast & Flexible, the layout-conditioned model also runs an order of magnitude faster than PlanIT and Deep Synth.Inference time is evaluated against state-of-the-art methods.

5. Conclusion and Future Work

SceneFormer generates realistic indoor scenes with transformers, implicitly learning object relations and enabling fast, flexible conditioning. The framework supports interactive generation and suggests extensions to reconstruction, mesh-based style consistency, and joint conditioning.

  • SceneFormer combines transformer models to generate realistic indoor scenes while implicitly learning object relations from data.
  • Fast inference can enable interactive scene generation from partial scenes.
  • Changing the object properties or conditioning inputs can adapt the framework to different scene-generation tasks.
  • Future directions include mesh-based global style consistency, indoor-scene reconstruction from 3D scans, and joint room-layout and text conditioning.

Appendix

The appendix specifies shared transformer hyperparameters and a separate ResNet-based component for shape conditioning.

  • The models use common transformer hyperparameters unless otherwise specified.
  • The maximum scene length is 50 objects.
  • The embedding dimension is 256, increasing to 1024 for the location model.
  • Transformer activation dimensions are 256, or 1024 for the location model.
  • The transformers use 8 attention heads and 8 blocks.
  • Shape conditioning uses a ResNet.

B.1. Results

Additional bedroom and living room scenes from the shape-conditioned model are compared with DeepSynth under identical room-shape conditions.

  • Additional bedroom and living room scenes are generated using the shape-conditioned model.
  • The generated scenes are compared with DeepSynth scenes conditioned on the same room shapes.

B.2. Object Category Heatmaps

Heatmap comparisons indicate that SceneFormer captures ground-truth object-category distributions while introducing some novel relations that support diversity and help avoid overfitting.

  • SceneFormer captures the ground-truth distribution well in object-category heatmap comparisons.
  • The model adds novel relations in some cases, supporting diverse scenes and avoiding overfitting.
  • The user-preference survey interface compares generated scenes with DeepSynth, PlanIT, FastSynth, and dataset scenes.

C.1. Scene Description Generation

The method generates scene descriptions by first deriving object relations, then expressing objects and their spatial relationships in templated sentences.

  • Relations are generated between object pairs using bounding-box geometry, including relation type and center-to-center distance.Supported relation types include on, above, surrounding, inside, right of, left of, behind, and in front of.
  • The first sentence introduces the first two or three objects using a randomly selected starting phrase.Repeated objects are represented by a count followed by the object category.
  • Subsequent sentences probabilistically describe undescribed objects through relations to previously described objects within 2.5 meters.Each candidate object is selected with probability 0.7, and one qualifying relation is chosen uniformly.
  • Objects in the same category are not related to one another, and ordinal prefixes identify later instances of a category.This rejects constructions such as relating a second table to the first table.
  • Articles and relation-specific templates convert the selected objects and relations into sentences that are appended into the final scene description.Examples include templates expressing adjacency or relative horizontal position.

C.2. Results

For text-conditioned generation, the model places objects within a room-shape prior learned from ground-truth scenes because the room shape is not provided as input.

  • Text-conditioned scenes use a room-shape prior learned from ground-truth scenes when room shape is unavailable.

C.3. Survey Interface

The survey interface evaluates generated-scene quality through paired visual comparisons and 1–7 response scales, alongside figures showing generated layouts, relations, and text-conditioned examples.

  • Survey Interface: Each survey scene receives two questions, with responses recorded on a 1–7 scale.The interface introduction explains the task to participants.
  • Generated Scenes: Figure 12 pairs room-layout inputs with generated bedroom and living-room scenes, showing boundary-respecting placement and learned category relations.
  • Comparisons: Figure 13 compares DeepSynth and the proposed method on the same room, focusing on complexity and completeness of object placement.
  • Spatial Analysis: Figure 14 visualizes object-pair locations in ground-truth and generated bedroom and living-room scenes.
  • Survey Interface: Figure 15 presents paired top-down images from the proposed method, the dataset, or DeepSynth for realism judgments.Participants choose which image appears more realistic.
  • Text Conditioning: Figure 16 shows text-conditioned bedroom scenes with corresponding text inputs and relations such as table-chair, bed-television, and sofa-television.
Loading 2012.09793v2…