PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutVAE: Stochastic Scene Layout Generation From a Label Set

Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, Greg Mori

arXiv:1907.10719v3cs.CV

TL;DR

Scene layout generation from label sets lacks relational information and must represent multiple plausible arrangements. LayoutVAE uses stochastic VAE-based count and bounding-box models, with experiments showing improved performance and abnormal-layout detection.

  • Problem

    Label sets provide only category presence, so models must learn spatial and count relationships while generating diverse plausible layouts.

  • Method

    LayoutVAE decomposes layout generation into CountVAE for category counts and BBoxVAE for bounding boxes, using autoregressive conditional VAEs.

  • Results

    Experiments on MNIST-Layouts and COCO validate LayoutVAE, showing significant performance improvement and the ability to detect abnormal layouts.

  • Takeaways & Limitations

    LayoutVAE provides diverse label-conditioned layouts and can support existing layout-based scene-generation models and abnormal-layout detection.

Abstract

from arXiv · show

Recently there is an increasing interest in scene generation within the research community. However, models used for generating scene layouts from textual description largely ignore plausible visual variations within the structure dictated by the text. We propose LayoutVAE, a variational autoencoder based framework for generating stochastic scene layouts. LayoutVAE is a versatile modeling framework that allows for generating full image layouts given a label set, or per label layouts for an existing image given a new label. In addition, it is also capable of detecting unusual layouts, potentially providing a way to evaluate layout generation problem. Extensive experiments on MNIST-Layouts and challenging COCO 2017 Panoptic dataset verifies the effectiveness of our proposed framework.

1. Introduction

Scene layout generation from label sets must model plausible spatial and count relationships while representing the many layouts compatible with the same labels. LayoutVAE addresses this stochastic generation problem with separate components for count and spatial relationships, validated on MNIST-Layouts and COCO.

  • Plausible layouts require reasonable spatial relationships, such as a person standing beside or riding a horse rather than being beneath it.
  • Plausible layouts also require count relationships, such as the number of ties being no greater than the number of people.
  • Label sets omit relational details, requiring models to learn spatial and count relationships from visual data.
  • The same label set can correspond to multiple plausible layouts with different object counts and positions.For person, surfboard, and sea, the COCO training set contains 869 images sharing that label set.
  • LayoutVAE generates stochastic scene layouts from label sets and separates count-relationship modeling from spatial-relationship modeling.
  • The framework introduces MNIST-Layouts and evaluates the model on MNIST-Layouts and COCO, including analysis of unlikely scene-layout detection.

2. Related Work

Prior work generates scenes from richer sentence, scene-graph, or program descriptions, while LayoutGAN addresses layout generation without conditioning on a given label set. LayoutVAE instead uses a two-model design for counts and bounding boxes.

  • Sentence-conditioned methods use descriptions containing partial background, object, appearance, and layout information.
  • Some sentence-based pipelines predict objects, bounding boxes, segmentation masks, and images in stages, but do not model stochasticity in each bounding box.
  • Scene-graph generation can produce one scene layout from a graph, whereas PNP-Net generates abstract scenes from fully specified text-based programs.
  • PNP-Net was tested on synthetic datasets with few classes and encodes the entire image in one latent code, unlike LayoutVAE's two-stage layout generation.
  • LayoutGAN generates label sets from input noise and cannot generate a layout for a specified set of labels.
  • LayoutVAE's architecture uses CountVAE for object counts and BBoxVAE for object bounding boxes.

3. Background

The paper formulates label-set layout generation as modeling diverse contextual relationships among object counts and bounding boxes. It builds on VAEs and conditional VAEs, which introduce latent variables and conditioning descriptions for generation.

  • The task is to model contextual relationships and generate diverse, plausible scene layouts from a label set.
  • A label set L records which object categories appear, while each category has an object count and a set of bounding boxes.
  • A VAE combines a generative model pθ(x|z), a latent prior pθ(z), and an inference network qφ(z|x) for observable data and latent codes.
  • VAE training uses a KL-divergence term to keep the inferred latent distribution close to the prior and optimizes an evidence lower bound.
  • A conditional VAE adds an auxiliary description c, enabling conditional posterior inference and generation based on that description.
  • In conditional VAEs, the auxiliary input modulates the prior distribution of the latent variables.

4. LayoutVAE for Stochastic Scene Layout Generation

LayoutVAE decomposes stochastic scene-layout generation into CountVAE for label-specific object counts and BBoxVAE for bounding-box coordinates and sizes. Autoregressive conditioning and latent-variable sampling support diverse layouts and flexible generation settings.

  • LayoutVAE overview: LayoutVAE first predicts object counts with CountVAE, then predicts each object’s bounding box with BBoxVAE.The two-stage design separates count relationships from spatial relationships.
  • CountVAE: CountVAE autoregressively models each count conditioned on the label set, current label, and previously predicted counts.A predefined label order is used, and previous counts help model consistency across categories.
  • CountVAE: CountVAE uses a Poisson distribution for the current label’s occurrences, with a rate determined by the latent variable and conditioning input.Because every label count is at least one, the model learns the distribution over n_k − 1.
  • Variational learning: The latent variables model ambiguity in scene layouts, using approximate posteriors during learning and conditional priors during generation.Both distributions are modeled as multivariate Gaussians with diagonal covariance.
  • BBoxVAE: BBoxVAE autoregressively predicts bounding boxes in a predefined label order, ordering boxes left to right within each label.Each prediction conditions on all boxes from earlier labels and earlier boxes of the current label.
  • BBoxVAE: BBoxVAE models normalized x, y, width, and height with a quadrivariate Gaussian and updates its conditioning variable after each sampled box.During generation, latent variables are sampled from the conditional prior.

5. Experiments

Experiments on MNIST-Layouts and COCO evaluate LayoutVAE against adapted baselines, ablate conditioning choices, and examine layout likelihood and diversity. LayoutVAE improves baseline performance while generating diverse layouts and modeling count and spatial relationships.

  • Datasets: MNIST-Layouts contains 5,000 training images and 1,000 validation and test images, placing multiple digits on 128×128 canvases under predefined rules.The global label set is {1, 2, 3, 4}, limiting each image to at most four labels.
  • Comparison with baseline models: The likelihood comparison reports negative log-likelihood on MNIST-Layouts and COCO, with lower values indicating better performance.VAE likelihoods are estimated by drawing 1,000 samples from the conditional prior at each generation step under teacher forcing.
  • Comparison with baseline models: LayoutVAE significantly outperforms all baselines on count and bounding box generation, while sg2im struggles without scene-graph relationships.The autoregressive MLP performs better than recurrent baselines, and count prediction is similarly easy across models on MNIST-Layouts.
  • Ablation study: Both context and history improve CountVAE and BBoxVAE performance, but context matters more for CountVAE whereas history matters more for BBoxVAE.Context denotes labels or labels with counts; history denotes previous counts or bounding boxes.
  • Analysis of the label set order: Fixed label orders perform similarly whether things or stuffs come first, whereas randomizing label order across images significantly reduces performance.This analysis uses BBoxVAE NLL on COCO.
  • Quantitative and qualitative evaluation: Likelihood worsens for 92.58% of test layouts under the abnormal-layout evaluation, while some flipped layouts receive higher likelihood.The authors interpret the exception as an anomalous layout example that is equally plausible when flipped.

6. Conclusion

The conclusion presents LayoutVAE as a stochastic label-set layout generator with separate models for count and spatial relationships. It reports improved performance, learned real-world object relationships, and abnormal-layout detection.

  • Conclusion: LayoutVAE generates stochastic scene layouts from label sets using separate components for object counts and spatial relationships.The framework uses conditional autoregressive VAEs for CountVAE and BBoxVAE and does not use past latent codes during generation.
  • Conclusion: The authors report significant performance improvement over existing methods or analogues and qualitative evidence of learned intrinsic relationships in real-world scenes.The framework can also detect abnormal layouts.

B. Model Architecture

LayoutVAE uses separate CountVAE and BBoxVAE modules to model category counts and bounding boxes, conditioning each autoregressive step on labels and previously predicted elements.

  • CountVAE: CountVAE represents the label set as a multi-label vector and each current category as a one-hot vector.Category counts are encoded as M-dimensional one-hot vectors whose non-zero location contains the count value.
  • CountVAE: The CountVAE conditioning input combines representations through concatenation, fully connected layers, and a multilayer perceptron.The architecture is detailed in Figure 16.
  • BBoxVAE: BBoxVAE represents each category-count pair like CountVAE and pools the label-and-count information by summation.
  • BBoxVAE: BBoxVAE pools previously predicted bounding boxes with an LSTM over successive label-and-box vectors.Each bounding box has four coordinates, concatenated with an M-dimensional label vector; the final LSTM output is used as the pooled representation.
  • BBoxVAE: BBoxVAE predicts the mean of a four-dimensional Gaussian while assuming diagonal covariance with standard deviation 0.02.

C.1. MNIST-Layouts dataset

MNIST-Layouts is generated from label sets by sampling category counts, then sampling bounding-box locations and sizes, with a special arrangement rule for labels 4 and 2.

  • Dataset generation: The dataset-generation rules are implemented by adapting code from tf-attend-infer-repeat.
  • Dataset generation: Given a label set, category counts are sampled uniformly over possible values, followed by uniform sampling of bounding-box locations and sizes.The table omits precise location and size ranges for brevity.
  • Dataset generation: When labels 4 and 2 co-occur, one instance of 2 is randomly selected and all 4s are placed around it.

C.2. Analysis of latent code size

On COCO, CountVAE and BBoxVAE perform well across latent-code sizes from 32 to 128 and show no improvement beyond 128 dimensions.

  • Latent code size: 32–128 latent dimensions yield good NLL performance for both CountVAE and BBoxVAE on COCO.The models are reported to be insensitive to this hyperparameter within that range.
  • Latent code size: Increasing the latent space beyond 128 dimensions does not improve performance.

C.3. Detecting unlikely layouts

LayoutVAE evaluates layout plausibility through likelihood changes after vertical flipping and generates multiple stochastic layouts conditioned on label sets and scene information.

  • Detecting unlikely layouts: 92.58% of test samples have lower LayoutVAE likelihood after their original layouts are flipped upside down.The change is attributed specifically to BBoxVAE because CountVAE gives identical likelihoods when label counts remain unchanged.
  • Detecting unlikely layouts: Some flipped layouts have equal or higher BBoxVAE likelihood, indicating examples that the model considers equally or more plausible after flipping.
  • Diverse generation: LayoutVAE generates five randomly sampled layouts for each input label set.
  • Conditioning information: Conditioning information changes predicted boxes: two persons are centered in one example, whereas twelve persons are smaller and positioned toward the left in another.The examples are presented as evidence that person counts influence bounding-box prediction.
  • Stochastic generation: Additional stochastic bounding-box generation examples use the labels person, surfboard, and sea, including a scene with more objects.
Loading 1907.10719v3…