PaperlayerView exact revision on arXiv

Source-linked AI summary

GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations

Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, Ingmar Posner

arXiv:1907.13052v4cs.LGcs.CVcs.NEcs.ROstat.ML

TL;DR

Existing unsupervised object-centric models decompose scenes but do not model component interactions for principled novel-scene sampling. GENESIS combines an object-centric spatial GMM, sequential amortized inference, and an autoregressive prior, and produces coherent generated scenes while performing well on decomposition. Its formulation assumes conditionally independent components given spatial allocation and uses a fixed scalar image-likelihood standard deviation.

  • Problem

    MONet and IODINE decompose scenes without labels but use factorized priors that do not account for interactions between scene components, limiting principled novel-scene generation.

  • Method

    GENESIS models images as a spatial GMM decoded from object-centric latents inferred sequentially or sampled from an autoregressive prior.

  • Results

    GENESIS generates coherent scenes and performs well on scene decomposition across evaluations on publicly available datasets.

  • Takeaways & Limitations

    GENESIS provides an unsupervised object-centric model that both decomposes rendered 3D scenes into meaningful parts and generates coherent scenes component by component.

  • Takeaways & Limitations

    The model assumes scene components are conditionally independent given their spatial allocation and uses a fixed scalar standard deviation for image-likelihood Gaussian components.

Abstract

from arXiv · show

Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centric generative model of 3D visual scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESIS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.

1 INTRODUCTION

Robotics and reinforcement learning need unsupervised representations that capture discrete scene elements, but existing object-centric models do not model interactions needed for principled novel-scene generation. GENESIS addresses this gap with an autoregressive object-centric generative model that supports both decomposition and generation.

  • Robotics and reinforcement learning require accurate perception and reasoning about discrete environmental elements.
  • Unsupervised generative latent-variable modeling offers a way to discover objects and represent data compactly without exhaustive labels.
  • MONet and IODINE decompose scenes into objects without labels but use factorized priors that treat scene components as independent.
  • GENESIS models relationships between scene components with an autoregressive prior learned alongside sequential amortized inference.Sequential inference occurs in low-dimensional latent space, allowing convolutional encoders and decoders to run in parallel.
  • Experiments on Multi-dSprites, GQN, and ShapeStacks evaluate scene generation, decomposition, and object-centric representations for supervised tasks.ShapeStacks annotations support evaluating whether representations predict tower stability.

2 RELATED WORK

Prior work discovers objects through attention, spatial mixture models, or adversarial and inverse-graphics approaches, but important limitations remain in scene complexity, supervision, probabilistic modeling, or scalability. GENESIS is positioned as an object-centric model that combines decomposition and generation while modeling component relationships.

  • Structured Models: Structured models use spatial attention, spatial mixture models, or related mechanisms to discover objects without direct supervision.
  • Structured Models: An image-generation extension handles only binary images with uniform black backgrounds and assumes non-overlapping object bounding boxes.
  • Structured Models: GENESIS targets rendered 3D scenes with complex backgrounds and considerable occlusion for both decomposition and generation.
  • MONet & IODINE: MONet and IODINE do not capture relations between scene components, whereas GENESIS uses an autoregressive prior to enable unconditional generation of coherent novel scenes.
  • Adversarial Methods: Adversarial approaches may generate scenes or segment objects, while GENESIS additionally provides probabilistic inference and O(K) complexity rather than O(K^2) relational complexity in one comparison.
  • Inverse Graphics: Inverse-graphics methods rely on scene annotations for training and lack probabilistic formulations.

3 GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING

GENESIS represents images as spatial Gaussian mixture models whose components correspond to scene elements, models component dependencies autoregressively, and infers latents sequentially. GENESIS-S simplifies this design by using one latent per component, while GECO-based learning addresses prior–posterior mismatch affecting sample quality.

  • Generative model: GENESIS formulates image generation as a spatial GMM with up to K image-sized scene components and pixelwise mixing probabilities acting as attention masks.
  • Generative model: An autoregressive prior models dependencies between mask variables because scene components have strong spatial dependencies.
  • Generative model: Scene components are conditionally independent given their spatial allocation, and component variables encode the individual components.
  • Inference: GENESIS decodes mask latents into mixing probabilities and combines image and masks to infer component latents that decode into scene components.
  • GENESIS-S: GENESIS-S uses a single latent variable per component instead of separate mask and component-appearance latents.
  • Inference: The approximate posterior mirrors the generative structure and amortizes inference with parameters φ.
  • Learning: GECO minimizes KL divergence subject to a reconstruction constraint because standard ELBO training can increase posterior–prior mismatch and reduce sample quality.

4 EXPERIMENTS

GENESIS is evaluated on scene generation, decomposition, and representation utility. Across these tasks, it generates coherent component-wise scenes, segments scenes more accurately than MONet, and supports strong downstream prediction of scene state.

  • 4.1 COMPONENT-WISE SCENE GENERATION: GENESIS sequentially samples scene components from an autoregressive prior, conditioning each component on those generated previously.This models dependencies such as compatible background parts, unique sky components, and non-overlapping object locations.
  • 4.1 COMPONENT-WISE SCENE GENERATION: GENESIS generates coherent novel GQN scenes, whereas MONet produces random artefacts that do not form sensible scenes.GENESIS generates foreground objects or background parts at each step, and the components fit into semantically consistent scenes.
  • 4.1 COMPONENT-WISE SCENE GENERATION: GENESIS consistently generates floor and sky first, foreground objects next, and background walls last, while GENESIS-S generates foreground objects in one step.The authors conjecture that floor and sky are generated first because they are large, easy-to-model surfaces with strong reconstruction-loss impact; they also report artefacts near wall boundaries.
  • 4.2 INFERENCE OF SCENE COMPONENTS: GENESIS differentiates individual objects and captures fine-grained background patterns more clearly than MONet in qualitative GQN decomposition examples.MONet fails to disambiguate one foreground object and reconstructs less background detail in the compared examples.
  • 4.2 INFERENCE OF SCENE COMPONENTS: 0.73 ± 0.03 ARI for GENESIS exceeds 0.63 ± 0.07 for MONet on 300 ShapeStacks test images.The evaluation used five models trained with different random seeds; ARI does not penalise over-segmentation.
  • 4.2 INFERENCE OF SCENE COMPONENTS: GENESIS achieves higher segmentation covering than MONet: SC 0.64 ± 0.08 and mSC 0.60 ± 0.09 versus 0.52 ± 0.09 and 0.49 ± 0.09.These results use the same 300 ShapeStacks images and five random seeds.
  • 4.3 EVALUATION OF UNSUPERVISED REPRESENTATION UTILITY: GENESIS and GENESIS-S outperform the baselines on tower-stability prediction, while MONet alone exceeds them on tower-height prediction.All models achieve high camera-view accuracy, and the authors attribute MONet’s height advantage to its deterministic segmentation network.
  • 4.4 QUANTIFYING SAMPLE QUALITY: The two GENESIS variants achieve the best FID on both Multi-dSprites and GQN, although GENESIS-S has less interpretable components and sacrifices sensible foreground objects for intricate backgrounds on GQN.The paper also notes that FID correlates with perceptual similarity but not necessarily with downstream representation utility.

5 CONCLUSIONS

GENESIS is an unsupervised object-centric latent-variable model that decomposes rendered 3D scenes into meaningful parts and generates coherent scenes component-wise. Its autoregressive prior captures relationships between components, while sequential inference improves computational efficiency.

  • GENESIS decomposes rendered 3D scenes into semantically meaningful constituent parts while generating coherent scenes component-wise.
  • An autoregressive prior captures relationships between scene components, distinguishing GENESIS from prior object-centric models.
  • Sequential inference provides a computationally efficient complement to the learned autoregressive prior.
  • Future work targets more complex datasets, robotics and reinforcement learning applications, improved reconstruction and sample quality, lower computational cost, and higher image resolution.

A DATASETS

GENESIS is evaluated on three publicly available datasets spanning 2D compositional images and simulated 3D environments. ShapeStacks additionally provides detailed annotations for scene state and physical structure.

  • Multi-dSprites: Multi-dSprites contains 70,000 images with one to four randomly selected sprites and independently varied five-level RGB channel values for each object and the background.10,000 images are reserved for validation and 10,000 for testing.
  • GQN: GQN’s rooms-ring-camera dataset contains simulated 3D square-room scenes with varied textures and one to three objects of different shapes and sizes.
  • ShapeStacks: ShapeStacks contains simulated block towers two to six blocks high, with variation in block properties, textures, lighting, and camera viewpoints.Its annotations include tower stability, block count, individual block properties, violation locations, five wall and floor textures, five light presets, and sixteen viewpoints.

B.1 GENESIS ARCHITECTURE

GENESIS architectures encode images into sequential latent variables and decode them into spatial mixture components. GENESIS-S uses separate decoders for mixing probabilities and scene components, a design found important for decomposition.

  • Network architecture: GENESIS uses the Berg et al. architecture for encoding and decoding z^m_k, with batch normalisation inserted before GLU nonlinearities.
  • Inference network: The mask-latent pathway uses an LSTM followed by a linear layer to compute mask latents z^m_k of size 64.The LSTM state size is twice the latent size.
  • Component VAE: The component VAE uses a spatial broadcast decoder, while its multilayer perceptron conditional distribution has two hidden layers with 256 units and ELU activations.
  • GENESIS-S: GENESIS-S uses a second spatial-broadcast decoder to obtain scene components x_k in addition to the decoder computing mixing probabilities.
  • GENESIS-S: The two different decoders were important for GENESIS-S to decompose the input.
  • GENESIS-S: GENESIS-S encodes an image with an encoder and RNN, then decodes latent variables z_k into mixing probabilities π_k and scene components x_k.

B.4 OPTIMISATION

The models use a GECO-balanced reconstruction and KL objective and a fixed training setup. Segmentation covering complements ARI by penalising over-segmentation and can weight masks either by area or equally.

  • Optimisation: The Gaussian image-likelihood standard deviation is σ_x = 0.7, and GECO balances reconstruction and KL-divergence terms.The reconstruction target is deliberately weak to emphasise KL minimisation and sample quality.
  • Optimisation: All models train for 5 * 10^5 iterations with batch size 32, ADAM, and learning rate 10^-4.GENESIS training takes about two days on a single GPU, and ShapeStacks sample quality is expected to improve with further training.
  • Segmentation covering: Segmentation covering computes overlap between ground-truth and predicted masks using intersection over union.The ground-truth foreground masks form S, while predicted masks form S′.
  • Segmentation covering: The weighted segmentation covering averages mask-pair IOUs proportionally to the number of pixels in the ground-truth mask.
  • Segmentation covering: The unweighted mean segmentation covering gives equal importance to masks of different sizes.
  • Segmentation covering: Unlike ARI, both segmentation-covering variants penalise over-segmentation because it lowers the IOU for a mask pair.

D COMPONENT-WISE SCENE GENERATION - GQN

On GQN, GENESIS generates coherent novel scenes by assembling interpretable foreground objects and background components, whereas MONet and GENESIS-S produce less coherent or less interpretable samples.

  • GENESIS generates scenes with clearly distinguishable foreground objects and backgrounds, while MONet samples are mostly incoherent.
  • The DC-VAE generates decent backgrounds, but its foreground objects are blurry.
  • Although GENESIS-S achieves the best FID in Table 2, its samples emphasize high-fidelity background patterns over appropriate foreground objects.
  • GENESIS-S components are less interpretable than the components generated by GENESIS at every step.

E INFERENCE OF SCENE COMPONENTS

GENESIS and MONet can identify objects beyond regions sharing similar colour, but their decomposition quality varies across scenes and evaluation metrics.

  • Both GENESIS and MONet successfully identify two objects with the same shape and colour.The example indicates that both models use cues beyond simply grouping regions by similar colour.
  • GENESIS-S distinguishes foreground objects from the background but initially explains the foreground together and later reconstructs the background haphazardly.
  • On ShapeStacks, GENESIS achieves ARI 0.82, SC 0.68, and mSC 0.58, compared with MONet at ARI 0.39, SC 0.26, and mSC 0.35.Both methods struggle to segment foreground objects properly in this example.
  • GENESIS properly segments four foreground objects with ARI 0.83, SC 0.83, and mSC 0.83.
  • MONet can obtain higher ARI than GENESIS despite merging foreground objects with the background, because ARI does not penalise foreground over-segmentation.This example highlights a limitation of ARI for evaluating unsupervised instance segmentation.
Loading 1907.13052v4…