PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutTransformer: Layout Generation and Completion with Self-attention

Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry Davis, Vijay Mahadevan, Abhinav Shrivastava

arXiv:2006.14615v2cs.CVcs.LG

TL;DR

Scene layout generation requires modeling relationships among compositional primitives across diverse domains and supporting both new and partial layouts. LayoutTransformer uses self-attention with factorized primitive representations to generate layouts, and it performs competitively across multiple datasets while capturing semantic relationships automatically. The paper also identifies limitations from primitive decomposition, frequency bias, and separately trained components.

  • Problem

    Scene layout generation must capture contextual relationships among compositional primitives across images, applications, documents, and 3D objects.

    Source: §1 · pp. 1–2

  • Method

    LayoutTransformer uses self-attention and factorized primitive attributes in an autoregressive model that generates new layouts or completes partial layouts.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    LayoutTransformer performs competitively across Rico, COCO, PubLayNet, and 3D-shape datasets, while its representations capture semantic relationships automatically.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework provides a common layout-modeling approach adaptable across diverse natural and human-designed data domains.

    Source: §1 · p. 2, §5 · p. 8

    The approach requires layouts to be decomposed into compositional primitives, and generated layouts are dominated by high-frequency objects or shapes.

    Source: §5 · p. 8

Abstract

from arXiv · show

We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents, and 3D objects. Most complex scenes, natural or human-designed, can be expressed as a meaningful arrangement of simpler compositional graphical primitives. Generating a new layout or extending an existing layout requires understanding the relationships between these primitives. To do this, we propose LayoutTransformer, a novel framework that leverages self-attention to learn contextual relationships between layout elements and generate novel layouts in a given domain. Our framework allows us to generate a new layout either from an empty set or from an initial seed set of primitives, and can easily scale to support an arbitrary of primitives per layout. Furthermore, our analyses show that the model is able to automatically capture the semantic properties of the primitives. We propose simple improvements in both representation of layout primitives, as well as training methods to demonstrate competitive performance in very diverse data domains such as object bounding boxes in natural images(COCO bounding box), documents (PubLayNet), mobile applications (RICO dataset) as well as 3D shapes (Part-Net). Code and other materials will be made available at https://kampta.github.io/layout.

1. Introduction

LayoutTransformer treats layouts as arrangements of compositional primitives whose contextual relationships support generation across natural and human-designed domains. Its self-attention model generates and completes layouts while learning semantic relationships and using factorized primitive attributes.

  • Scene layouts encode contextual relationships between objects, including support, occlusion, relative likelihood, position, and size.Source: §1 · p. 1
  • Layout generation supports applications such as reducing manual UI-design effort and producing synthetic data, with layouts generated across documents, mobile apps, natural scenes, and 3D shapes.Source: §1 · p. 2
  • LayoutTransformer uses an autoregressive self-attention model to synthesize new layouts, complete partial layouts, and compute existing-layout likelihoods.Source: §1 · p. 2
  • The model factorizes layout primitives into separate attributes so attention can focus on attributes that matter, especially in symmetric document and app datasets.Source: §1 · p. 2
  • The learned feature representations capture semantic relationships between objects without explicitly using semantic embeddings.Source: §1 · p. 2
  • LayoutTransformer performs well with essentially the same architecture and hyperparameters across MNIST Layout, Rico, PubLayNet, and COCO bounding-box datasets.Source: §1 · p. 2

2. Related Work

Prior work often generates scenes from proxy inputs or imposes domain-specific constraints, while LayoutTransformer directly models layouts with fewer such restrictions. Its factorized representation and autoregressive design support high-resolution, variable-length layout generation and competitive results across domains.

  • Existing scene-generation methods commonly condition on sentences, scene graphs, layouts, images, or other proxy representations, and layout generation itself has received comparatively little evaluation.Source: §2 · p. 2
  • Some indoor-room methods assume walls, roofs, or other domain-specific structures, limiting extension to other datasets.Source: §2 · p. 3
  • LayoutTransformer factorizes primitives into structural and compositional geometric parameters, enabling high-resolution primitives and complete scenes.Source: §2 · p. 3
  • Its autoregressive design generates layouts of arbitrary lengths and can begin from partial layouts.Source: §2 · p. 3
  • Discrete position and size modeling improves performance on document and app-wireframe datasets with typically axis-aligned bounding boxes, while evaluations show competitive results across diverse domains.Source: §2 · p. 3

3. Our Approach

LayoutTransformer represents layouts as sequences of compositional primitives and uses masked self-attention to generate or complete variable-length layouts. The framework combines domain-specific primitive representations, autoregressive decoding, and training procedures for both discrete and continuous parameters.

Layout Representation

  • Layouts are modeled as fully connected graphs whose primitive information is projected into d-dimensional feature vectors, while geometry is represented separately.Primitive information may be discrete, continuous, or multidimensional; geometric information is factorized into position and scale.Source: §3.1 · pp. 3–4

Model Architecture and Training

  • The model accepts a random permutation of K visible primitives, including K = 0, and autoregressively predicts subsequent primitive parameters.This supports generation from scratch, completion from an initial set, and layouts with variable numbers of primitives.Source: §3.2 · pp. 4–5
  • Masked multi-head self-attention restricts each position to input and previously predicted latent vectors, with the final layer producing the next parameter.Each Transformer Decoder layer contains masked attention and feed-forward sublayers with residual connections and LayerNorm; training uses teacher forcing.Source: §3.2 · p. 4
  • The objective uses label-smoothed KL divergence for discrete parameters and L1 loss for continuous parameters.A softmax produces probabilities for discrete next parameters, while continuous predictions use an L1 term weighted by λ.Source: §3.2 · p. 4

3D Primitive Auto-encoding

  • For PartNet, a 3D CNN encodes voxel primitives and an implicit MLP decoder reconstructs their occupancy, while outputs are ordered by centroid raster order.Random primitive permutations are used as inputs for partial-layout completion, but generated sequences follow ascending (x, y, z) centroid order.Source: §3.2 · p. 4

4. Experiments

Experiments evaluate LayoutTransformer across 3D shapes, natural-scene bounding boxes, mobile-app wireframes, and documents using qualitative and dataset-specific quantitative comparisons. The analyses also examine learned semantic embeddings, layout distributions, downstream image generation, and failure cases.

3D Shape synthesis

  • PartNet shape completion predicts each next part’s latent encoding, position, and scale before decoding its surface geometry.The generated mesh is visualized with marching cubes from sampled surface points.Source: §4.1 · p. 5

Natural scenes

  • COCO completion generates layouts from seed layouts, while evaluation reports layout NLL and downstream image-generation IS and FID.The generated-layout figures show seed inputs above nucleus-sampled completions; Table 3 defines the evaluation metrics and reporting procedure.Source: §4.2 · p. 6, Figure 4 · p. 6, Figure 5 · p. 6

Semantic representations

  • Learned COCO category embeddings cluster super-categories and capture word2vec-style analogies without explicit semantic-embedding supervision.The embedding analysis also reports semantically related categories appearing close in the t-SNE space.Source: §4.2 · p. 6, Figure 6 · p. 7, Table 2 · p. 7

Apps and documents

  • RICO and PubLayNet samples preserve layout properties such as broad spatial coverage, low element collision, and alignment along x and y axes.The experiments include multiple completions from the same initial element, and attribute discretization plus decoupled geometric fields are especially useful for aligned boxes.Source: §4.3 · pp. 7–8, Figure 8 · p. 8, Figure 9 · p. 8

Failure Cases

  • A reported failure case produces disconnected lamp parts because the part auto-encoder and layout generator are trained separately rather than jointly.In COCO, the model also tends to generate high-frequency categories early, producing distribution differences for persons and cars.Source: §4.4 · p. 8

5. Conclusion.

LayoutTransformer uses self-attention to model relationships between layout elements, generate or complete layouts, and performs competitively across diverse domains. The authors also identify limitations involving primitive decomposition and dataset frequency bias.

  • LayoutTransformer uses self-attention to capture contextual relationships between layout elements.Source: §5 · p. 8
  • Generated document layouts contain aligned bounding boxes for various elements.Source: Figure 10 · p. 8
  • The model generates novel layouts and completes partial layouts across diverse datasets.Source: §5 · p. 8
  • The method performs competitively on Rico, COCO, PubLayNet, and 3D-shape datasets.Source: §5 · p. 8
  • The approach requires layouts or scenes to be decomposed into compositional primitives, which may not always be defined.Source: §5 · p. 8
  • Generated layouts are dominated by high-frequency objects or shapes, although improved sampling provides some diversity control.Source: §5 · p. 8

A. Architecture and training details

The experiments use a mostly fixed Transformer configuration across datasets, with a smaller configuration and learning rate for R3 experiments.

B. Ablation studies

The ablations examine element size, precision, embedding size, depth, ordering, discretization, and loss, revealing trade-offs between likelihood, layout resolution, model size, and generation speed.

Small, medium and large elements

Varying precision

Size of embedding

Model depth

Ordering of the elements

  • Raster-scan output ordering improves visual and NLL performance, while random permutation reduces dependence on input ordering and supports partial-layout completion.Source: B. Ablation studies · p. 12

Discretization strategy

  • Jointly predicting x- and y-coordinates increases vocabulary size and reduces performance but requires half as many predictions per element.Source: B. Ablation studies · pp. 12–13

Loss

C. Baselines

The baselines use different conditioning assumptions and generation mechanisms, including label sets, scene graphs, text-derived layouts, and fixed-box GAN generation. Some comparisons required reimplementation or could not reproduce reported document results.

LayoutVAE

  • LayoutVAE uses separate autoregressive VAEs for element counts and bounding-box locations and sizes.Source: C. Baselines · p. 13
  • LayoutVAE assumes a label set, so validation comparisons provide it with the dataset layouts’ label sets.Source: C. Baselines · p. 13

ObjGAN

  • ObjGAN generates bounding-box layouts from categories supplied as input instead of sentences.Source: C. Baselines · p. 13

sg2im

  • sg2im generates a scene layout from a scene graph before using it for complete-scene generation.Source: C. Baselines · p. 13

LayoutGAN

  • LayoutGAN starts from Gaussian-sampled fixed boxes and uses a GAN to assign coordinates resembling dataset layouts.Source: C. Baselines · p. 13
  • The authors could not reproduce LayoutGAN’s reported document results because bounding-box code was unavailable.Source: C. Baselines · p. 14

D. Visualizing attention

The self-attention model exposes which previously generated elements influence each new element. Attention often focuses on semantically related bounding boxes.

E. Layout Verification

Because the model assigns likelihoods to layouts, it can test whether layouts are likely under different spatial inversions. COCO layouts remain likely after horizontal flipping but become less likely after vertical flipping.

  • The model computes layout likelihoods, enabling likelihood-based verification of given layouts.Figure 12 evaluates negative log-likelihood under left-right and top-down inversions of COCO layouts.Source: E. Layout Verification · p. 14
  • COCO layouts remain likely after left-right flipping but become less likely after upside-down flipping.The latter change appears as increased negative log-likelihood.Source: E. Layout Verification · p. 14

F. More semantics in learned category embeddings

The learned category embeddings capture semantic and co-occurrence structure without task-specific analogy supervision. Additional analyses examine embedding geometry, nearest neighbors, and generated layouts across real and synthesized data.

  • The model captures word2vec-style category analogies despite being trained only to generate layouts.No additional objective function was specified for the analogical reasoning task.Source: F. More semantics in learned category… · p. 14, Table 9 · p. 14
  • Bigrams and trigrams compare frequent distinct-category pairs and triplets in real versus generated layouts.Source: F. More semantics in learned category… · p. 14, Table 10 · p. 15
  • Coordinate embeddings occupy a distinct ring-like region in a 2D t-SNE projection.The plot combines coordinate and category embeddings for COCO.Source: Figure 13 · p. 14, G. Coordinate Embedding · p. 15
  • Nearest-neighbor analysis compares generated layouts with training layouts using Chamfer distance on bounding-box coordinates.The authors use this analysis to examine whether generated layouts memorize the training dataset.Source: H. Nearest neighbors · p. 15
  • Additional examples show generated layouts and corresponding images produced with the Layout2Im tool.The text notes that existing layout-to-image methods are less effective than free-form image generation methods but may benefit downstream applications.Source: §I · p. 15

J. Dataset Statistics

This section reports category statistics across Rico wireframes and PubLayNet documents. The accompanying tables summarize category frequencies and the number of distinct layouts containing each element.

Related papers

Loading 2006.14615v2…