PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutTransformer: Layout Generation and Completion with Self-attention

Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry Davis, Vijay Mahadevan, Abhinav Shrivastava

arXiv:2006.14615v2cs.CVcs.LG

TL;DR

Scene layout generation requires modeling relationships among compositional primitives across diverse domains and supporting both new and partial layouts. LayoutTransformer uses self-attention with factorized primitive representations to generate layouts, and it performs competitively across multiple datasets while capturing semantic relationships automatically. The paper also identifies limitations from primitive decomposition, frequency bias, and separately trained components.

  • Problem

    Scene layout generation must capture contextual relationships among compositional primitives across images, applications, documents, and 3D objects.

  • Method

    LayoutTransformer uses self-attention and factorized primitive attributes in an autoregressive model that generates new layouts or completes partial layouts.

  • Results

    LayoutTransformer performs competitively across Rico, COCO, PubLayNet, and 3D-shape datasets, while its representations capture semantic relationships automatically.

  • Takeaways & Limitations

    The framework provides a common layout-modeling approach adaptable across diverse natural and human-designed data domains.

  • Takeaways & Limitations

    The approach requires layouts to be decomposed into compositional primitives, and generated layouts are dominated by high-frequency objects or shapes.

Abstract

from arXiv · show

We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents, and 3D objects. Most complex scenes, natural or human-designed, can be expressed as a meaningful arrangement of simpler compositional graphical primitives. Generating a new layout or extending an existing layout requires understanding the relationships between these primitives. To do this, we propose LayoutTransformer, a novel framework that leverages self-attention to learn contextual relationships between layout elements and generate novel layouts in a given domain. Our framework allows us to generate a new layout either from an empty set or from an initial seed set of primitives, and can easily scale to support an arbitrary of primitives per layout. Furthermore, our analyses show that the model is able to automatically capture the semantic properties of the primitives. We propose simple improvements in both representation of layout primitives, as well as training methods to demonstrate competitive performance in very diverse data domains such as object bounding boxes in natural images(COCO bounding box), documents (PubLayNet), mobile applications (RICO dataset) as well as 3D shapes (Part-Net). Code and other materials will be made available at https://kampta.github.io/layout.

1. Introduction

LayoutTransformer treats layouts as arrangements of compositional primitives whose contextual relationships support generation across natural and human-designed domains. Its self-attention model generates and completes layouts while learning semantic relationships and using factorized primitive attributes.

  • Scene layouts encode contextual relationships between objects, including support, occlusion, relative likelihood, position, and size.
  • Layout generation supports applications such as reducing manual UI-design effort and producing synthetic data, with layouts generated across documents, mobile apps, natural scenes, and 3D shapes.
  • LayoutTransformer uses an autoregressive self-attention model to synthesize new layouts, complete partial layouts, and compute existing-layout likelihoods.
  • The model factorizes layout primitives into separate attributes so attention can focus on attributes that matter, especially in symmetric document and app datasets.
  • The learned feature representations capture semantic relationships between objects without explicitly using semantic embeddings.
  • LayoutTransformer performs well with essentially the same architecture and hyperparameters across MNIST Layout, Rico, PubLayNet, and COCO bounding-box datasets.

2. Related Work

Prior work often generates scenes from proxy inputs or imposes domain-specific constraints, while LayoutTransformer directly models layouts with fewer such restrictions. Its factorized representation and autoregressive design support high-resolution, variable-length layout generation and competitive results across domains.

  • Existing scene-generation methods commonly condition on sentences, scene graphs, layouts, images, or other proxy representations, and layout generation itself has received comparatively little evaluation.
  • Some indoor-room methods assume walls, roofs, or other domain-specific structures, limiting extension to other datasets.
  • LayoutTransformer factorizes primitives into structural and compositional geometric parameters, enabling high-resolution primitives and complete scenes.
  • Its autoregressive design generates layouts of arbitrary lengths and can begin from partial layouts.
  • Discrete position and size modeling improves performance on document and app-wireframe datasets with typically axis-aligned bounding boxes, while evaluations show competitive results across diverse domains.

3. Our Approach

LayoutTransformer represents layouts as sequences of compositional primitives and uses masked self-attention to generate or complete variable-length layouts. The framework combines domain-specific primitive representations, autoregressive decoding, and training procedures for both discrete and continuous parameters.

  • Layout Representation: Layouts are modeled as fully connected graphs whose primitive information is projected into d-dimensional feature vectors, while geometry is represented separately.Primitive information may be discrete, continuous, or multidimensional; geometric information is factorized into position and scale.
  • Model Architecture and Training: The model accepts a random permutation of K visible primitives, including K = 0, and autoregressively predicts subsequent primitive parameters.This supports generation from scratch, completion from an initial set, and layouts with variable numbers of primitives.
  • Model Architecture and Training: Masked multi-head self-attention restricts each position to input and previously predicted latent vectors, with the final layer producing the next parameter.Each Transformer Decoder layer contains masked attention and feed-forward sublayers with residual connections and LayerNorm; training uses teacher forcing.
  • Model Architecture and Training: The objective uses label-smoothed KL divergence for discrete parameters and L1 loss for continuous parameters.A softmax produces probabilities for discrete next parameters, while continuous predictions use an L1 term weighted by λ.
  • 3D Primitive Auto-encoding: For PartNet, a 3D CNN encodes voxel primitives and an implicit MLP decoder reconstructs their occupancy, while outputs are ordered by centroid raster order.Random primitive permutations are used as inputs for partial-layout completion, but generated sequences follow ascending (x, y, z) centroid order.

4. Experiments

Experiments evaluate LayoutTransformer across 3D shapes, natural-scene bounding boxes, mobile-app wireframes, and documents using qualitative and dataset-specific quantitative comparisons. The analyses also examine learned semantic embeddings, layout distributions, downstream image generation, and failure cases.

  • 3D Shape synthesis: PartNet shape completion predicts each next part’s latent encoding, position, and scale before decoding its surface geometry.The generated mesh is visualized with marching cubes from sampled surface points.
  • Natural scenes: COCO completion generates layouts from seed layouts, while evaluation reports layout NLL and downstream image-generation IS and FID.The generated-layout figures show seed inputs above nucleus-sampled completions; Table 3 defines the evaluation metrics and reporting procedure.
  • Semantic representations: Learned COCO category embeddings cluster super-categories and capture word2vec-style analogies without explicit semantic-embedding supervision.The embedding analysis also reports semantically related categories appearing close in the t-SNE space.
  • Apps and documents: RICO and PubLayNet samples preserve layout properties such as broad spatial coverage, low element collision, and alignment along x and y axes.The experiments include multiple completions from the same initial element, and attribute discretization plus decoupled geometric fields are especially useful for aligned boxes.
  • Failure Cases: A reported failure case produces disconnected lamp parts because the part auto-encoder and layout generator are trained separately rather than jointly.In COCO, the model also tends to generate high-frequency categories early, producing distribution differences for persons and cars.

5. Conclusion.

LayoutTransformer uses self-attention to model relationships between layout elements, generate or complete layouts, and performs competitively across diverse domains. The authors also identify limitations involving primitive decomposition and dataset frequency bias.

  • LayoutTransformer uses self-attention to capture contextual relationships between layout elements.
  • Generated document layouts contain aligned bounding boxes for various elements.
  • The model generates novel layouts and completes partial layouts across diverse datasets.
  • The method performs competitively on Rico, COCO, PubLayNet, and 3D-shape datasets.
  • The approach requires layouts or scenes to be decomposed into compositional primitives, which may not always be defined.
  • Generated layouts are dominated by high-frequency objects or shapes, although improved sampling provides some diversity control.

A. Architecture and training details

The experiments use a mostly fixed Transformer configuration across datasets, with a smaller configuration and learning rate for R3 experiments.

  • R2 experiments use d = 512, L = 6, nhead = 8, precision = 8, and dff = 2048.
  • R2 training uses dropout 0.1, a maximum of 128 elements, Adam, and an initial learning rate of 10^-4.
  • The maximum of 128 elements covers over 99.9% of layouts in COCO, Rico, and PubLayNet.
  • R3 experiments reduce d to 128, dff to 512, and the learning rate to 10^-5.

B. Ablation studies

The ablations examine element size, precision, embedding size, depth, ordering, discretization, and loss, revealing trade-offs between likelihood, layout resolution, model size, and generation speed.

  • Small, medium and large elements: COCO NLL is 2.4, 2.5, and 1.8 for large, medium, and small boxes, respectively.
  • Varying precision: Increasing precision produces finer layouts but increases parameters and NLL.
  • Size of embedding: Increasing embedding size d improves NLL at the cost of more parameters.
  • Model depth: Increasing model depth L does not significantly improve results, so L = 6 is fixed.
  • Ordering of the elements: Raster-scan output ordering improves visual and NLL performance, while random permutation reduces dependence on input ordering and supports partial-layout completion.
  • Discretization strategy: Jointly predicting x- and y-coordinates increases vocabulary size and reduces performance but requires half as many predictions per element.
  • Loss: NLL gives better validation NLL than label smoothing, although qualitative results differ little.

C. Baselines

The baselines use different conditioning assumptions and generation mechanisms, including label sets, scene graphs, text-derived layouts, and fixed-box GAN generation. Some comparisons required reimplementation or could not reproduce reported document results.

  • LayoutVAE: LayoutVAE uses separate autoregressive VAEs for element counts and bounding-box locations and sizes.
  • LayoutVAE: LayoutVAE assumes a label set, so validation comparisons provide it with the dataset layouts’ label sets.
  • ObjGAN: ObjGAN generates bounding-box layouts from categories supplied as input instead of sentences.
  • sg2im: sg2im generates a scene layout from a scene graph before using it for complete-scene generation.
  • LayoutGAN: LayoutGAN starts from Gaussian-sampled fixed boxes and uses a GAN to assign coordinates resembling dataset layouts.
  • LayoutGAN: The authors could not reproduce LayoutGAN’s reported document results because bounding-box code was unavailable.

D. Visualizing attention

The self-attention model exposes which previously generated elements influence each new element. Attention often focuses on semantically related bounding boxes.

  • The model visualizes attended existing elements while generating each new layout element.Previously predicted bounding boxes receive the strongest attention in the illustrated examples.
  • “Snow” receives the highest attention when predicting “skis” in one example.
  • “Skis” receives the highest attention when predicting “pers” in the last-column example.

E. Layout Verification

Because the model assigns likelihoods to layouts, it can test whether layouts are likely under different spatial inversions. COCO layouts remain likely after horizontal flipping but become less likely after vertical flipping.

  • The model computes layout likelihoods, enabling likelihood-based verification of given layouts.Figure 12 evaluates negative log-likelihood under left-right and top-down inversions of COCO layouts.
  • COCO layouts remain likely after left-right flipping but become less likely after upside-down flipping.The latter change appears as increased negative log-likelihood.

F. More semantics in learned category embeddings

The learned category embeddings capture semantic and co-occurrence structure without task-specific analogy supervision. Additional analyses examine embedding geometry, nearest neighbors, and generated layouts across real and synthesized data.

  • The model captures word2vec-style category analogies despite being trained only to generate layouts.No additional objective function was specified for the analogical reasoning task.
  • Bigrams and trigrams compare frequent distinct-category pairs and triplets in real versus generated layouts.
  • Coordinate embeddings occupy a distinct ring-like region in a 2D t-SNE projection.The plot combines coordinate and category embeddings for COCO.
  • Nearest-neighbor analysis compares generated layouts with training layouts using Chamfer distance on bounding-box coordinates.The authors use this analysis to examine whether generated layouts memorize the training dataset.
  • Additional examples show generated layouts and corresponding images produced with the Layout2Im tool.The text notes that existing layout-to-image methods are less effective than free-form image generation methods but may benefit downstream applications.

J. Dataset Statistics

This section reports category statistics across Rico wireframes and PubLayNet documents. The accompanying tables summarize category frequencies and the number of distinct layouts containing each element.

  • The dataset statistics count each element’s total occurrences and the number of distinct layouts containing it.Elements are listed by occurrence frequency in descending order.
  • Tables 11 and 12 provide category statistics for Rico wireframes and PubLayNet documents, respectively.
  • The section includes generated layout and image examples alongside the dataset-statistics discussion.Figures 14 and 15 show nearest neighbors and sample layouts with corresponding images.
Loading 2006.14615v2…