PaperlayerView exact revision on arXiv

Source-linked AI summary

GRAINS: Generative Recursive Autoencoders for INdoor Scenes

Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, Hao Zhang

arXiv:1807.09193v5cs.GR

TL;DR

Generating realistic and diverse 3D indoor scenes requires handling hierarchical object relations, semantic composition, and variable layouts. GRAINS uses a recursive variational autoencoder to encode annotated scene hierarchies and decode sampled latent codes into new scenes, supporting efficient generation and downstream applications. Its main limitations are limited direct control over generated objects and occasional unnatural placements arising from imperfect, heuristically structured training data.

  • Problem

    Indoor scenes have structured but variable object layouts and semantic relations, creating a need for models that can generate large quantities of diverse 3D scenes.

  • Method

    GRAINS trains a recursive variational autoencoder on annotated scene hierarchies, encoding labeled OBBs and their relations into latent codes and decoding sampled codes hierarchically.

  • Results

    GRAINS generates plausible and diverse 3D indoor scenes efficiently and supports 2D-layout modeling, scene editing, and semantic scene segmentation with PointNet.

  • Takeaways & Limitations

    The method provides a large and varied source of generated 3D scenes for modeling, editing, and downstream semantic segmentation applications.

  • Takeaways & Limitations

    GRAINS cannot directly specify object counts or required object subsets, precisely control object shapes or fine-grained semantics, and can produce unnatural placements from imperfect training data and heuristic hierarchies.

Abstract

from arXiv · show

We present a generative neural network which enables us to generate plausible 3D indoor scenes in large quantities and varieties, easily and highly efficiently. Our key observation is that indoor scene structures are inherently hierarchical. Hence, our network is not convolutional; it is a recursive neural network or RvNN. Using a dataset of annotated scene hierarchies, we train a variational recursive autoencoder, or RvNN-VAE, which performs scene object grouping during its encoding phase and scene generation during decoding. Specifically, a set of encoders are recursively applied to group 3D objects based on support, surround, and co-occurrence relations in a scene, encoding information about object spatial properties, semantics, and their relative positioning with respect to other objects in the hierarchy. By training a variational autoencoder (VAE), the resulting fixed-length codes roughly follow a Gaussian distribution. A novel 3D scene can be generated hierarchically by the decoder from a randomly sampled code from the learned distribution. We coin our method GRAINS, for Generative Recursive Autoencoders for INdoor Scenes. We demonstrate the capability of GRAINS to generate plausible and diverse 3D indoor scenes and compare with existing methods for 3D scene synthesis. We show applications of GRAINS including 3D scene modeling from 2D layouts, scene editing, and semantic scene segmentation via PointNet whose performance is boosted by the large quantity and variety of 3D scenes generated by our method.

1 INTRODUCTION

GRAINS addresses the challenge of generating diverse indoor scenes by modeling their hierarchical object structure with a recursive variational autoencoder. It encodes semantic and spatial relations, then generates new scenes hierarchically from sampled latent codes.

  • Motivation: Indoor scenes combine clear semantic and spatial patterns with substantial layout variation, making their structure difficult to model using structure-oblivious operations.Object-object relations can be semantic or functional rather than purely geometric, and scene layouts lack a consistent global alignment.
  • Approach: GRAINS uses a recursive neural network because indoor scene structures are inherently hierarchical.The method targets large-scale generation of plausible and varied 3D indoor scenes.
  • Approach: The RvNN-VAE recursively groups scene objects bottom-up during encoding and generates a new scene top-down from a randomly sampled code.The fixed-length codes learned from annotated scene hierarchies roughly follow a Gaussian distribution.
  • Representation: Unlike object-shape models, GRAINS encodes both numerical spatial information and categorical object semantics using labeled oriented bounding boxes.Relative object positioning is used because indoor scenes lack a sensible global alignment.
  • Results and applications: GRAINS generates plausible and diverse scenes efficiently and supports 2D-layout modeling, hierarchy-based editing, and PointNet semantic segmentation.The paper reports that PointNet performance is boosted by the quantity and variety of generated scenes.

2 RELATED WORK

Prior work models indoor scenes with probabilistic, graphical, activity-based, convolutional, and deep generative approaches. GRAINS is situated among structure-aware generative models that represent hierarchical organization rather than only volumetric appearance.

  • Indoor scene synthesis: Indoor scene synthesis has used probabilistic models of object occurrence and placement, graphical models, 2D sketches, videos, RGB-D inputs, and human activity priors.These approaches address object composition and layout through different sources of guidance.
  • Indoor scene synthesis: A convolutional approach learns top-view depth representations and sequentially places objects using predicted 2D placement priors.This provides a learned scene-synthesis alternative based on image-like layout encodings.
  • Deep generative models: Generative 3D models based on volumetric representations can sample new objects or scenes but do not explicitly represent parts, topology, or structure.The passage identifies structure-oblivious generated volumes as a drawback of these representations.
  • Generative neural models for structures: Structure-aware neural models have explored graph representations, while recursive autoencoders support generation through invertible hierarchical encodings.The paper adapts recursive structural modeling to indoor scene generation with non-trivial extensions.
  • Scene representation: Hierarchical vectorized scene encodings store object sizes and labels at leaves and relative child positions at internal nodes.This representation makes scene structure explicit for recursive modeling.

3 OVERVIEW

GRAINS represents indoor scenes as hierarchies of labeled objects and trains a recursive VAE over those structures. Sampling a learned root-code distribution lets the decoder generate new hierarchies of oriented bounding boxes.

  • Structural scene representation: Scenes are organized into hierarchies whose leaves are objects and whose internal nodes group objects according to spatial relations.Walls and floors are included in the hierarchy, while object labels and sizes are encoded at the leaves.
  • Recursive VAE: The recursive VAE maps an OBB hierarchy bottom-up to a fixed-length root code, and its decoder performs the inverse operation.The recursive network repeatedly applies encoders after leaf vectors have been converted into fixed-length codes.
  • Scene generation: After training, a sampled root code is decoded into an OBB hierarchy containing sibling relations and leaf bounding boxes.This learned distribution provides the basis for generating novel scene structures.

4 STRUCTURAL SCENE REPRESENTATION

GRAINS represents indoor scenes as hierarchies of objects and object groups, using spatial relations and relative positioning to preserve subscene structure. Its representation combines semantic object information with geometric relations encoded through angles, offsets, and explicit attachment or alignment indicators.

  • Hierarchical scene structures: Indoor scenes are organized hierarchically because common features lie in subscenes defined by objects and their relative placements.The hierarchy contains objects at leaf nodes and object groups at internal nodes.
  • Hierarchical scene structures: Object relations are categorized as support, surround, and co-occurrence to construct scene subtrees despite variation in object count, geometry, and placement.Support links an object placed on another; surround groups similarly sized, similarly labeled objects around a central object; co-occurrence is a catch-all relation.
  • Hierarchical scene structures: Training hierarchies cluster objects by nearby walls, prioritize support over surround and co-occurrence, then merge clusters with walls and the floor into a root hierarchy.For example, nightstands and lamps are first merged by support, then merged with the bed by surround.
  • Hierarchical scene structures: Ordered children expose relation-specific structure: supported objects follow their supports, surround nodes place the central object first, and co-occurrence children are sorted by OBB size.These ordering rules are intended to make recurring dataset features easier for the network to learn.
  • Scene object representation: Relative object positioning uses a 28-dimensional representation combining 3 real values with 25 binary indicators for orientation, offsets, edge pairs, attachment, and alignment.The three real values are one rotation angle and two closest-edge offsets; binary codes explicitly reinforce strict attachment and alignment relations.

5 RECURSIVE MODEL OF INDOOR SCENES

The model represents indoor scenes as fixed-dimensional codes using a recursive autoencoder whose modules mirror hierarchical object relations. Encoders recursively merge objects, relations, walls, and floor into a root code, while decoders reconstruct the hierarchy and a classifier selects decoding modules.

  • Recursive scene representation: The scene representation extends recursive autoencoders from binary trees to indoor-scene hierarchies with non-binary surround and root nodes.The hierarchy encodes multiple object-object relations with distinct features, requiring multiple encoder-decoder pairs.
  • Encoder-decoder architecture: Six encoder-decoder pairs handle leaf boxes, support, co-occurrence, surround, wall, and root relations.The root encoder merges four wall nodes with the floor, while relation-specific modules merge object or object-group codes with relative positions.
  • Encoder-decoder architecture: BoxEnc maps object dimensions and one-hot semantic labels into n-D leaf codes, which BoxDec reconstructs into the original leaf vectors.The input leaf representation consists of object bounding-box information concatenated with label vectors.
  • Relation modules: Relation encoders merge child codes with relative OBB positions, preserving ordered structural roles such as support direction, larger-first co-occurrence, and wall-left ordering.Support places the supporting object first, co-occurrence places the object or group with the larger OBB first, and wall encoding places the wall code on the left.
  • Training and decoding: The root module uses larger hidden and output layers than other modules because it must accommodate more information.RootEnc and RootDec use dimensions 1,050 and 350, respectively, compared with 750 and 250 for the other modules.
  • Training and decoding: During training, objects are encoded bottom-up to a root code, approximated to a Gaussian distribution, sampled, and decoded recursively with reconstruction, classifier, and KL-divergence losses.The jointly trained node classifier predicts whether each decoded node is a box, support, co-occurrence, surround, or wall node, selecting the corresponding decoder.

6 RESULTS, EVALUATION, AND APPLICATIONS

GRAINS generates plausible, diverse indoor scenes efficiently through hierarchical scene generation, and its outputs support applications including layout-guided modeling and data augmentation. Evaluation examines generation quality, similarity, timing, co-occurrence statistics, perceptual plausibility, and downstream generalization.

  • Scene generation: Generated scenes preserve plausible object placements and frequent co-occurrences across bedroom samples with varying object-count complexity.Examples include TV–TV stand, bed–nightstands, and desk–chair–computer combinations.
  • Scene generation: Graph-kernel comparisons use nearest training scenes and nearest generated scenes to assess novelty and diversity rather than simple memorization.The evaluation compares generated bedrooms with both training-set neighbors and neighbors among 1,000 generated results.
  • Timing: 0.1027s per 3D bedroom scene is achieved on average, after generating 10K scene hierarchies in 94 seconds and converting them to scenes in 933 seconds.The reported generation timing is substantially below the 4-minute comparison cited for Wang et al. [2018].
  • Object co-occurrence: Strong similarities between training and generated conditional object-co-occurrence probabilities indicate that the learned RvNN reproduces scene co-occurrence patterns, except notably for living-room TV–floor-lamp pairs.The comparison uses 1,000 randomly generated scenes for bedrooms and living rooms.
  • Perceptual evaluation: Perceptual studies find generated scenes frequently selected as plausible, with ratings comparable to training scenes and variation across participant groups.Graphics/Vision researchers especially often selected the generated scenes, while non-specialists gave them higher scores despite favoring training scenes in a separate comparison.
  • Applications: Combining SUNCG with generated scenes improves generalization on mixed test data, supporting their use as a reserve set for training deep models.Training exclusively on either dataset leads to overfitting and weaker cross-dataset generalization.

7 DISCUSSION, LIMITATIONS, AND FUTURE WORK

GRAINS combines a recursive neural network with a variational autoencoder to learn hierarchical indoor-scene structure and generate plausible scenes. The approach improves plausibility through handcrafted design choices, but remains limited in object control, hierarchy generality, and data-driven reliability.

  • GRAINS integrates a recursive neural network with a variational autoencoder to learn hierarchical structures of 3D indoor scenes.
  • A scene can be generated from a random vector in less than a second using relative positioning, semantic encoding, and wall objects as initial references.
  • The design choices improve generated-scene plausibility but introduce handcrafted grouping operations, explicit semantics, and special-purpose relative-position encoding.
  • GRAINS lacks direct control over object counts, required object subsets, precise object shapes, and fine-grained semantics such as style compatibility.
  • Hierarchical representations may be unsuitable for messy scenes where object organization is ambiguous or no single best hierarchy exists.
  • Heuristic training hierarchies and imperfect scene data can produce unnatural object placements and orientations, motivating subscene-based generative models.

1 MORE SCENE GENERATION RESULTS

GRAINS generates room scenes that preserve common sub-scene patterns while combining multiple sub-scenes to provide diversity. The kitchen results remain less varied because the training data is biased toward kitchen cabinets.

  • GRAINS preserves common sub-scenes such as sofa-table pairs, desk-computer-chair groups, and attached kitchen cabinets while combining them to create diverse rooms.
  • Kitchen-scene variety is limited by training-set bias toward kitchen cabinets, although generated scenes reflect their characteristic attachment.

2 OBJECT CO-OCCURRENCE

The authors evaluate whether generated scenes reproduce object co-occurrence probabilities from the training scenes. The generated scenes show similar object occurrences to the training data.

  • Object co-occurrence is measured with the conditional probability P(c1|c2), based on scenes containing at least one object from each category.
  • Generated scenes reproduce object-occurrence patterns similar to those in the training scenes.

3 DISTRIBUTION OF THE RELATIVE POSITIONS

The relative-position analysis compares training scenes with scenes generated using three position representations. GRAINS better reproduces some learned spatial patterns, especially for nightstands near beds, while missing some chair-behind-desk configurations.

  • The analysis compares relative-position distributions from training scenes with scenes generated using the proposed, absolute-position, and box-transform representations.
  • For bed-nightstand pairs, the proposed representation places most nightstands to the left or right of beds and outperforms the other two representations.
  • For chair-desk pairs, generated scenes rarely place chairs behind desks, although some chairs appear beside them.
  • The scene-generation results cover living rooms, offices, and kitchens, while co-occurrence plots compare bedrooms, living rooms, offices, and kitchens.
Loading 1807.09193v5…