PaperlayerView exact revision on arXiv

Source-linked AI summary

ATISS: Autoregressive Transformers for Indoor Scene Synthesis

Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, Sanja Fidler

arXiv:2110.03675v1cs.CV

TL;DR

Indoor scene synthesis must produce realistic, diverse layouts while supporting automatic generation and partial user input without relying on costly structured annotations. ATISS treats rooms as unordered sets and autoregressively predicts objects from room geometry and partial scenes. Across four 3D-FRONT room types, it generates plausible layouts that are more realistic than existing methods, with fewer parameters, simpler training, and up to 8× faster runtime.

  • Problem

    Existing scene-synthesis approaches impose costly rules or annotations, and ordered object sequences constrain valid generation orders and interactive use.

  • Method

    ATISS uses an autoregressive transformer that generates indoor rooms as permutation-invariant unordered sets from room type, floor plan, and partial objects, trained with labeled 3D bounding boxes.

  • Results

    ATISS consistently generates plausible layouts across four 3D-FRONT room types, more realistic than existing methods, with up to 8× faster runtime and fewer parameters.

  • Takeaways & Limitations

    A single trained model supports automatic layout synthesis, general scene completion, partial rearrangement, and constrained object suggestions.

  • Takeaways & Limitations

    The model can learn biases from indoor-scene data, and problematic training arrangements are associated with overlapping objects and unnatural orientations in generated rooms.

Abstract

from arXiv · show

The ability to synthesize realistic and diverse indoor furniture layouts automatically or based on partial input, unlocks many applications, from better interactive 3D tools to data synthesis for training and simulation. In this paper, we present ATISS, a novel autoregressive transformer architecture for creating diverse and plausible synthetic indoor environments, given only the room type and its floor plan. In contrast to prior work, which poses scene synthesis as sequence generation, our model generates rooms as unordered sets of objects. We argue that this formulation is more natural, as it makes ATISS generally useful beyond fully automatic room layout synthesis. For example, the same trained model can be used in interactive applications for general scene completion, partial room re-arrangement with any objects specified by the user, as well as object suggestions for any partial room. To enable this, our model leverages the permutation equivariance of the transformer when conditioning on the partial scene, and is trained to be permutation-invariant across object orderings. Our model is trained end-to-end as an autoregressive generative model using only labeled 3D bounding boxes as supervision. Evaluations on four room types in the 3D-FRONT dataset demonstrate that our model consistently generates plausible room layouts that are more realistic than existing methods. In addition, it has fewer parameters, is simpler to implement and train and runs up to 8 times faster than existing methods.

1 Introduction

Indoor scene synthesis seeks realistic and diverse 3D environments, but existing procedural, graph-based, and ordered autoregressive approaches impose costly annotations or unnatural generation constraints. ATISS instead models scenes as unordered object sets, enabling broader user-controlled applications.

  • Realistic and diverse synthetic 3D environments support content creation, AR/VR, games, and computer-vision training data.
  • Procedural methods require time-consuming rules, while graph-based methods require laborious scene-graph annotations.
  • Ordered autoregressive methods impose spatial or frequency-based object sequences that can assign zero probability to otherwise valid orderings.
  • ATISS formulates scene synthesis as unordered set generation, using labeled object classes and 3D bounding boxes rather than costly support or scene-graph annotations.
  • The unordered formulation supports automatic synthesis, user-constrained scene completion, and object suggestions.

2 Related Work

Prior indoor scene-synthesis work includes procedural, graph-based, CNN-based, and transformer-based approaches, while transformer generative models commonly assume ordered sequences. ATISS applies transformers to unordered set generation and describes an encoder-based generation pipeline.

  • Procedural Modeling with Grammars: Procedural modeling recursively applies synthesis functions, while grammars formalize this approach for structures including indoor scenes.
  • The method overview encodes floor layout and existing objects before a transformer predicts the next object's attributes autoregressively.
  • Autoregressive Scene Synthesis: Autoregressive indoor-scene models sequentially predict and place objects, using CNN or transformer architectures and varying supervision requirements.
  • Transformers for Set Generation: Existing transformer generative models generally assume ordered sequences even when point clouds or scene objects have no natural order.
  • Transformers for Set Generation: ATISS enforces that adding an object is invariant to the order of previously generated elements and reports outperforming ordered-set transformers on every scene-synthesis metric.

3 Method

ATISS formulates indoor scene synthesis as autoregressive generation of unordered object sets conditioned on room layout. It uses permutation-aware training and a transformer architecture to predict plausible next-object attributes from partial scenes.

  • Autoregressive Set Generation: ATISS represents each scene as an unordered set of labeled 3D bounding boxes conditioned on the room’s floor layout.Each object includes category, size, location, and orientation.
  • Autoregressive Set Generation: The model maximizes likelihood across object permutations so arbitrary generation orders can support plausible completion of partial scenes.The exact permutation sum is approximated with Monte Carlo sampling during training.
  • Autoregressive Set Generation: Object likelihood is factorized autoregressively as category, position, orientation, and size, allowing later attributes to condition on previously predicted properties.The category is predicted first so the model can reason about object size and position conditionally.
  • Network Architecture: The network combines a floor-layout encoder, per-object structure encoder, transformer encoder, and attribute extractor to predict the next object.A ResNet-18 encodes the top-down floor projection into a 64-dimensional feature, while a learnable query produces next-object features.
  • Network Architecture: Removing positional encodings makes transformer features invariant to the order of existing object embeddings.This supports conditioning on partial scenes without imposing an ordering on their objects.
  • Training and Inference: Training randomly permutes scene objects, conditions on a randomly selected prefix, and maximizes the likelihood of the next object, with an end symbol terminating generation.The training objective uses the T+1 object from the permuted scene as the prediction target.

4 Experimental Evaluation

ATISS is evaluated on four room types using quantitative realism metrics, qualitative comparisons, diversity tests, and interactive applications. It consistently produces plausible layouts, generalizes to manually designed floor plans, and supports scene completion, correction, and object suggestion.

  • Evaluation Setup: ATISS is evaluated on bedrooms, living rooms, dining rooms, and libraries using 3D-FRONT data filtered for uncommon arrangements and unnatural rooms.The evaluation uses 5,996 bedrooms, 2,962 living rooms, 2,625 dining rooms, and 622 libraries after preprocessing.
  • Evaluation Setup: The evaluation compares ATISS with FastSynth and SceneFormer using FID, KL divergence, and real-versus-synthetic classification accuracy.Classification accuracy closer to 0.5 indicates greater similarity between synthesized and real scenes.
  • Scene Synthesis: Classification accuracy remains around 50%, while ATISS achieves lower FID and KL divergence than the baselines across room types.These metrics indicate synthesized scenes are close to the test-set distributions under the reported measures.
  • Scene Synthesis: ATISS generates valid and diverse arrangements for shared floor plans, unlike baselines that particularly struggle with living rooms and dining rooms.The paper attributes baseline difficulty to less training data and more objects per scene in those room types.
  • Scene Synthesis: ATISS also produces diverse bedroom layouts consistent with manually designed unconventional floor plans, while both baselines fail to generate valid scenes.The experiment probes layouts with shapes not represented in the training data.
  • Applications: The unordered-set formulation supports interactive scene completion, failure-case correction, and object suggestion under user-specified location constraints.The model can complete partial scenes, relocate low-likelihood objects, and sample objects satisfying positional constraints.
  • Perceptual Study: In paired perceptual studies, FastSynth scenes contained interpenetrating furniture 41.4% of the time, while ATISS was judged more realistic than FastSynth in 73.1% of cases.SceneFormer performed significantly worse in the reported comparison.

5 Conclusion

The paper concludes that ATISS advances indoor scene synthesis by representing rooms as unordered object sets. This formulation also supports interactive authoring applications, while future work targets broader attribute invariance, style, and cultural coverage.

  • Conclusion: ATISS synthesizes 3D rooms as unordered sets of objects and generates realistic scenes that advance the state of the art.The conclusion presents the architecture and its realism as the paper’s central contribution.
  • Conclusion: The formulation enables semi-automated scene authoring applications including scene completion, object suggestions, and anomaly detection.These applications are enabled by the model’s interactive use of partial or user-specified inputs.
  • Future Work and Scope: Future work will extend order invariance to object attributes, incorporate style information, and learn from less structured data across cultures and environments.The authors also identify learned biases for indoor scenes as a concern.

Abstract

The supplementary material documents ATISS’s network architecture, training procedure, preprocessing, ablations, transformer comparisons, and qualitative results.

  • Supplementary Contents: The supplementary document details the network architecture and training procedure, then describes preprocessing for filtering problematic 3D-FRONT rooms.It also reports component ablations and comparisons with transformer models that use ordering.
  • Supplementary Contents: The architecture section provides training protocols, metric-computation details, and additional information about the baselines.These details complement the main evaluation description.

A.1 Network Architecture

ATISS combines a floor-plan layout encoder, permutation-compatible object-context encoding, a transformer encoder, and autoregressive attribute prediction. The supplementary implementation also specifies training, retrieval, and evaluation procedures.

  • Layout Encoder: The layout encoder maps a top-down floor-plan projection to a 64-dimensional global feature using an ImageNet-un pretrained ResNet-18.The final fully connected layer is replaced with average pooling followed by a linear projection.
  • Structure Encoder: The structure encoder maps each object’s category, size, location, and orientation into a per-object context embedding.Categories use learnable embeddings, while geometric attributes use positional encodings applied separately by dimension.
  • Transformer Encoder: A four-layer, eight-head transformer without positional encoding combines the floor feature, object contexts, and a learnable query to predict the next-object features.Queries, keys, and values have 64 dimensions, and the transformer output feeds attribute prediction.
  • Attribute Extractor: The attribute extractor autoregressively predicts object categories and distribution parameters for location, orientation, and size.The category predictor outputs class probabilities, while geometric predictors use mixtures of K logistic distributions with K = 10.
  • Inference: During inference, ATISS retrieves a 3D-FUTURE model by nearest-neighbor search over predicted object dimensions.The implementation uses the predicted category, location, orientation, and size to place the retrieved model.
  • Training: Training uses Adam with learning rate 10^-4, batch size 128, 100k iterations, and random rotations between 0 and 360 degrees.Validation metrics are evaluated every 1,000 iterations to determine when to stop training.
  • Baselines and Evaluation: The baselines are retrained on 3D-FRONT and compared without rule-based post-processing, using the same object retrieval procedure.The comparison includes FastSynth and SceneFormer, whose original SUNCG evaluations could not be reproduced because SUNCG is unavailable.

B 3D-FRONT Dataset Filtering

The 3D-FRONT data were filtered to remove problematic rooms and objects before evaluation, with room-type-specific subsets and duplicate-aware splits.

  • Dataset scope: 3D-FRONT contains 6,813 houses and roughly 14,629 designed rooms populated with 3D-FUTURE furniture; evaluation focuses on four room types.The selected types are bedrooms, living rooms, dining rooms, and libraries.
  • Dataset scope: The dataset includes room types with widely varying counts, so types with very few rooms were excluded from evaluation.Listed counts range from 16 nannyrooms to 3,313 masterbedrooms.
  • Filtering: Preprocessing removes rooms with unnatural dimensions, invalid object placement, overlapping objects, unusual layouts, and unsuitable object counts.Filtering thresholds vary by room type, including different maximum floor sizes and object-count ranges.
  • Filtering: Bedroom, library, living-room, and dining-room subsets use room-type-specific source combinations and filtering procedures before training and testing.Dining-room preparation combines diningroom and livingdiningroom scenes because the standalone diningroom subset is small.
  • Splits: Rooms are split into 70% training, 20% testing, and 10% validation after preprocessing, while duplicate arrangements are handled to avoid train–test leakage.Splitting solely by house would allow the same room arrangement to appear in both partitions.

C Ablation Study

The ablations examine attribute distributions, layout encoders, object ordering, and interactive uses, showing benefits of the selected configuration and unordered formulation.

  • C.1 Mixture of Logistic distributions: Using one logistic distribution reduces performance, while more than 10 hurts FID and classification accuracy; the model therefore uses K = 10.The authors attribute the degradation beyond 10 distributions to possible overfitting.
  • C.2 Layout Encoder: Replacing ResNet-18 with AlexNet produces slightly worse performance, although the method is not particularly sensitive to the layout encoder.The comparison uses FID, classification accuracy, and KL-divergence.
  • C.3 Transformers with Ordering: The permutation-invariant model performs better than ordered variants in the ordering ablation.The authors conjecture that invariance becomes more important with larger datasets or scenes containing more objects.
  • Interactive applications: Interactive applications include failure correction, constrained object suggestion, scene completion, and placement of a user-specified object.These applications use partial scenes, location constraints, or fixed object categories while sampling remaining attributes.
  • Scene synthesis: On four 3D-FRONT room types, ATISS generates realistic arrangements, while some baselines produce objects outside boundaries, unnatural sizes, or functionally poor library rooms.The authors associate some baseline difficulties with smaller amounts of training data for challenging room types.

E.1 Object Co-occurrence

ATISS more closely reproduces real object co-occurrence and per-object frequencies than the baselines across room types, while all methods produce plausible co-occurrences.

  • Object co-occurrence: ATISS better captures object co-occurrences than FastSynth and SceneFormer in living rooms, dining rooms, and libraries.The comparison uses absolute differences between real and synthesized co-occurrence probabilities; lower differences are better.
  • Object co-occurrence: All methods generate plausible object co-occurrences, although reproducing categories is easier than learning object sizes and 3D positions.The authors explicitly distinguish category co-occurrence from the harder spatial attribute modeling problem.
  • Per-object frequencies: ATISS produces per-object frequencies comparable to real rooms across all four room types.For living rooms, dining rooms, and libraries, the discrepancy is smaller despite these being more challenging room types due to their smaller size.
  • Location distributions: The location-distribution analysis samples 5,000 scenes for each of six bedroom floor plans to visualize generated placements for chairs, desks, nightstands, and wardrobes.The analysis examines distributions conditioned on fixed floor plans.
  • Computational requirements: ATISS requires significantly less time to generate scenes than the compared methods, with cost varying by room type and average object count.Living and dining rooms typically require more object generation because they are larger.

F Perceptual Study

Two paired perceptual studies compare ATISS with FastSynth and SceneFormer using independently generated scenes, with users judging layout errors and realism.

  • Study design: The studies sampled 211 test-set floor plans and generated six scenes per floor plan for each method without filtering or post-processing.Samples were drawn randomly and independently across methods.
  • Study design: The rendering protocol used rotating animated rooms and instructed participants to focus on layout rather than individual furniture properties.Using identical furniture objects was rejected because dimension-based rescaling could deform them unrealistically.
  • Measures: Participants evaluated errors including furniture interpenetration and furniture outside the floor area, then researchers aggregated average error rates per layout.The study also measured realism through pairwise preferences between ATISS and baselines.
  • Results: FastSynth was judged more realistic than ATISS only 26.9% of the time, so ATISS was preferred 73.1% of the time in that comparison.The reported realism values represent the fraction of times users chose the baseline over ATISS.
  • Study protocol: Workers were paid $0.05 per question set, for a total study compensation of USD $106.The participation risks were described as the regular risks associated with computer use.

H Discussion and Limitations

The discussion highlights rare failure cases and remaining limitations in ATISS, illustrated through qualitative examples across several room types.

  • Qualitative comparisons: Figure 28 visualizes various model failure cases across room types.
  • Failure cases: Overlapping objects, especially chairs in living and dining rooms, remain a documented failure case.The authors hypothesize that problematic arrangements remained in the filtered 3D-FRONT training data.
  • Failure cases: Unnatural object orientations, such as chairs facing bookshelves or the opposite direction from tables, also occur in generated layouts.These failures are linked to problematic rooms in the training data and are described as quite rare.
  • Limitations: ATISS does not guarantee error-free layouts, leaving room for improvement despite the rarity of these failures.The paper connects their rarity to quantitative analysis and a perceptual study.
  • Qualitative comparisons: The qualitative results compare generated bedroom, living-room, dining-room, and library scenes from FastSynth, SceneFormer, and ATISS against nearby training examples.The figures include the closest training-set scene to showcase generalization abilities.
Loading 2110.03675v1…