PaperlayerView exact revision on arXiv

Source-linked AI summary

BLT: Bidirectional Layout Transformer for Controllable Layout Generation

Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, Irfan Essa

arXiv:2112.05112v2cs.CV

TL;DR

Layout transformers’ fixed attribute order limits controllable generation from user-specified conditions. BLT uses bidirectional masked prediction with hierarchical sampling and iterative refinement; across six benchmarks, it supports controllable generation and achieves 4–10x inference speedups over autoregressive transformer baselines.

  • Problem

    Autoregressive layout transformers struggle with user conditions that conflict with their fixed attribute dependency order, and sequential inference is inefficient for dense layouts.

    Source: §3 · p. 5

  • Method

    BLT predicts masked attributes from bidirectional context and uses hierarchical sampling in training and iterative low-confidence refinement at inference.

    Source: §1 · p. 3, §4.2 · p. 7

  • Results

    Across six benchmarks, BLT performs favorably against strong baselines for realism, alignment, and semantic relevance, with 4–10x inference speedups over autoregressive transformers.

    Source: §5 · p. 8, §6 · p. 14

  • Takeaways & Limitations

    The experiments support BLT’s effectiveness and flexibility for controllable layout generation across diverse benchmarks.

    Source: §1 · p. 3, §6 · p. 14

    On 3D-Front home-design layouts, low similarity scores indicate that housing-design layout generation remains challenging.

    Source: §5.3 · p. 11

Abstract

from arXiv · show

Creating visual layouts is a critical step in graphic design. Automatic generation of such layouts is essential for scalable and diverse visual designs. To advance conditional layout generation, we introduce BLT, a bidirectional layout transformer. BLT differs from previous work on transformers in adopting non-autoregressive transformers. In training, BLT learns to predict the masked attributes by attending to surrounding attributes in two directions. During inference, BLT first generates a draft layout from the input and then iteratively refines it into a high-quality layout by masking out low-confident attributes. The masks generated in both training and inference are controlled by a new hierarchical sampling policy. We verify the proposed model on six benchmarks of diverse design tasks. Experimental results demonstrate two benefits compared to the state-of-the-art layout transformer models. First, our model empowers layout transformers to fulfill controllable layout generation. Second, it achieves up to 10x speedup in generating a layout at inference time than the layout transformer baseline. Code is released at https://shawnkx.github.io/blt.

1 Introduction

Layout transformers struggle to generate layouts from user conditions that conflict with their fixed attribute order. BLT uses bidirectional masked prediction and hierarchical sampling to support controllable generation, with favorable results across six benchmarks and faster inference.

1 Introduction

  • Graphic layouts determine component placement and sizing, and are used across applications including interfaces, slides, magazines, and advertisements.Prior neural approaches include GANs and VAEs, while layout transformers predict attributes sequentially.Source: §1 · pp. 1–2
  • BLT predicts masked attributes using bidirectional context and applies hierarchical sampling during both training and inference.At inference, it iteratively refines low-confidence attributes.Source: §1 · p. 3
  • BLT enables controllable layout generation by addressing the fixed dependency order that limits prior layout transformers.Autoregressive decoding cannot generate an attribute when the user’s known conditions disagree with the predefined order.Source: §1 · pp. 2–3
  • Across six diverse layout datasets, BLT supports controllable generation, performs favorably against strong baselines, and reaches 4x–10x faster inference than autoregressive transformers.The experiments cover realism, alignment, and semantic relevance; BLT’s unconditional-generation quality is reported as on par with the state of the art despite not being designed for that task.Source: §1 · p. 3

2 Related Work

Prior layout generation work uses GANs, VAEs, and transformers, with transformers reported as state of the art. BLT builds on non-autoregressive bidirectional generation while addressing its quality drop with hierarchical sampling.

2 Related Work

  • GANs and VAEs are established deep generative approaches, with LayoutVAE and NDN among conditional layout-generation models.NDN models asset relations and constraints with graph convolution; BLT reports more favorable results than LayoutVAE and NDN in Table 1.Source: §2 · p. 4
  • Transformers hold state-of-the-art performance in layout generation, but their autoregressive decoding makes conditional generation difficult when user inputs conflict with the fixed attribute order.LayoutGAN++ instead takes asset labels and random code as input, whereas BLT supports unconditional and varied conditional generation.Source: §2 · p. 4
  • Non-autoregressive bidirectional transformers use context from both directions, but their decoding has shown an apparent performance drop relative to autoregressive decoding [11,12,9].Applying a non-autoregressive NLP model to layout generation also yields inferior results to the autoregressive baseline, motivating BLT’s hierarchical sampling policy.Source: §2 · p. 4

3 Problem Formulation

The paper represents each layout as a flattened sequence of object attributes. Autoregressive decoding imposes a fixed dependency order that prevents some user-conditioned generation and limits parallel inference.

3 Problem Formulation

  • Each object is represented by category, center coordinates, width, and height, with bounding-box values discretized using 8-bit uniform quantization.The example coordinate representation uses integer values from 0 to 31.Source: §3 · p. 5
  • A layout of K assets is flattened into a sequence beginning with <bos>, listing each asset’s attributes, and ending with <eos>.The start and end tokens mark the sequence boundaries, and each element is represented as an integer index or one-hot vector.Source: §3 · p. 5
  • The autoregressive model estimates layout likelihood by predicting each attribute conditioned on the preceding sequence.Training uses ground-truth attributes as input, while inference predicts sequentially from <bos> until <eos>.Source: §3 · p. 5
  • Autoregressive decoding cannot generate attributes when user conditions conflict with the training dependency order, and its sequential inference is inefficient for dense layouts.For example, the fixed order cannot generate an object’s position from its known width and height.Source: §3 · p. 5

4 Approach

BLT uses bidirectional attention to support controllable, parallel layout generation. A hierarchical masking policy guides training and iterative refinement during decoding.

4.1 Model and Training

  • The model uses the autoregressive layout-transformer architecture with bidirectional attention and learns to predict masked attributes by minimizing their negative log-likelihood.Training replaces selected attributes with a [mask] token and predicts them from the resulting sequence.Source: §4.1 · p. 6
  • Hierarchical sampling selects a semantic group, then masks a randomly sampled number of attributes from that group for training.This ensures the model predicts attributes with the same semantic meaning each time; the groups include category, position, and size.Source: §4.1 · p. 6

4.2 Parallel Decoding by Iterative Refinement

  • BLT predicts unknown attributes in parallel, then repeatedly remasks low-confidence attributes within semantic groups until all attributes are generated.The masking ratio decreases across iterations, and each iteration conditions on attributes already ascertained.Source: §4.2 · p. 7
  • The authors hypothesize that layout attributes have structural correlations that NLP non-autoregressive models may not capture, and report that their algorithm outperforms those baselines.The comparison is reported in Section 5.4.Source: §4.2 · pp. 7–8
  • For unconditional generation, BLT starts with only [mask] tokens and samples the number of objects from a prior learned from the training dataset.The sequence length must be known in advance.Source: §4.2 · p. 8

5 Experimental Results

Across six diverse benchmarks, BLT performs favorably on realism, alignment, and semantic relevance, with a reported 4x–10x speedup over autoregressive decoding. Experiments cover conditional and unconditional generation, qualitative examples, and the hierarchical sampling policy.

5 Experimental Results

  • BLT performs favorably against strong baselines across six benchmarks and achieves a 4x–10x speedup over autoregressive decoding.The evaluation examines realism, alignment, and semantic relevance.Source: §5 · p. 8

5.1 Setups

  • The six datasets span graphic interfaces, publications, magazines, commercial ads, natural scenes, and indoor layouts.The experiments include conditional and unconditional generation, with the test split used for conditional generation.Source: §5.1 · pp. 8–9

5.2 Quantitative Comparison

  • BLT supports category + size conditional generation, which the transformer baselines cannot process, and outperforms conditional VAE models across all Table 1 metrics by statistically significant margins.For unconditional generation, BLT outperforms LayoutVAE and achieves performance comparable to two autoregressive transformers.Source: §5.2 · p. 10, Table 1 · p. 10
  • In a user study on RICO and PubLayNet, participants judged BLT’s conditional layouts more similar to the reference than layouts from all baseline models.The study used 50 generated layouts per conditional setting and involved 75 unique workers.Source: §5.2 · pp. 10–11, Figure 3 · p. 11

5.3 Qualitative Result

  • Category-and-size-conditioned examples for scientific papers, mobile interfaces, and magazines yield reasonable layouts suitable for rendering.The 3D-FRONT task extends the method to predicting furniture positions in 3D bounding boxes, though its low similarity score indicates that home-design layouts remain challenging.Source: §5.3 · p. 11, Figure 4 · p. 12

5.4 Ablation Study

  • BLT’s fastest model achieves 4x speedup at around 10 objects and 10x at 20 objects; autoregressive decoding time grows with object count.The comparison generated 1,000 layouts with batch size 1 on a single GPU.Source: §5.4 · p. 12, Figure 7 · p. 13
  • The non-autoregressive NLP baseline performs worse than the autoregressive baseline, while BLT with hierarchical sampling significantly outperforms that non-autoregressive baseline.Among tested group orders, generating category first and then location or size appears better.Source: §5.4 · pp. 13–14, Table 4 · p. 14, Table 5

6 Conclusion and Future Work

BLT brings conditional and controllable generation to transformer-based layout models through a hierarchical sampling policy. Across six benchmarks, it reports 4–10 times inference speedup over autoregressive transformer baselines, while identifying content-agnostic generation as a limitation.

6 Conclusion and Future Work

  • BLT enables conditional and controllable generation with transformer-based layout models, using a hierarchical sampling policy in training and inference to produce high-quality layouts.The policy is described as essential for high-quality layouts.Source: §6 · p. 14
  • Across six benchmarks, BLT reports 4–10 times inference speedup over autoregressive transformer baselines.The paper attributes the speedup to high computation parallelism.Source: §6 · p. 14
  • BLT’s generation is content-agnostic; the authors leave visual information for future work to preserve a fair comparison with baselines that also do not use it.Source: §6 · p. 14

A Implementation Details

The implementation uses grid search for task-specific hyperparameters and averages metrics across five independent trials. Its perceptual IOU calculation divides the layout-wide overlap area by the union area of all objects.

A Implementation Details

  • Five independent trials are run for each model, with averaged metrics and standard deviations reported; grid search selects task-specific hyperparameters.The learning-rate search covers 1e−3, 3e−3, and 5e−3; dropout and attention dropout each cover 0.1 and 0.3.Source: A Implementation Details · p. 18
  • Perceptual IOU projects layouts as images and divides the overlapped area by the union area of all objects, rather than averaging IOU only across overlapping object pairs.For the toy example, the proposed calculation gives 1/13, compared with 1/3 under.Source: A Implementation Details · p. 18, Figure 8 · p. 18

B Additional Quantitative Results

Additional results compare BLT with autoregressive transformer-based models on CoCo, Magazine, and Ads. The section also presents IOU and overlap scores across decoding iterations on two datasets.

B Additional Quantitative Results

C.1 Qualitative Results for Conditional Generation

The supplementary results show conditional-generation samples across four design applications and demonstrate diverse outputs from top-k sampling. They also include comparisons with Layout-VAE and NDN.

C.1 Qualitative Results for Conditional Generation

  • Conditional-generation samples cover mobile UI, scientific paper, magazine, and natural-scene layouts, with additional comparisons to Layout-VAE and NDN.Source: §C.1 · p. 19

C.2 Examples of Diverse Conditional Generation

  • Top-k sampling with k = 10 for category-and-size conditional generation produces diverse layouts, unlike the main experiment’s greedy attribute decoding.Source: §C.2 · p. 19

C.3 More Attention Head Patterns

Masked-position attention heads can use related layout attributes, while conditional generation still produces some layouts with overlaps or weak visual quality.

C.3 More Attention Head Patterns

  • Heads attending to masked x positions also attend to objects’ width information, which supports accurate position prediction.Similar patterns appear in other heads and layers.Source: §C.3 · p. 20

C.4 Failure Cases

  • Some conditional-generation samples contain object overlaps or differ substantially from real layouts, with low visual quality.The authors identify these issues as directions for future work.Source: §C.4 · p. 20
  • In one Magazine example, the generated layout has worse alignment than its corresponding real layout.Source: §C.4 · p. 20

C.5 Iterative Refinement Process

BLT’s iterative refinement improves layout-quality metrics and makes samples more realistic over successive iterations. The appendix also presents conditional-generation examples, model comparisons, failure cases, diverse samples, attention patterns, and layouts at different refinement iterations.

C.5 Iterative Refinement Process

  • With more refinement iterations, IOU and Overlap improve and stabilize, while layouts become more realistic after low-confidence attributes are refined.The first iteration shows severe object overlaps; Fig. 15 shows layouts at different iterations on three datasets.Source: §C.5 · p. 20, Figure 15 · p. 25
  • Fig. 10 shows conditional layouts for scientific papers, user interfaces, and magazines, while Fig. 11 compares BLT and VAE-based conditional-generation results.Source: Figure 10 · p. 21, Figure 11 · p. 22
  • Fig. 12 presents failure cases, and Fig. 13 shows diverse conditional samples generated with top-k sampling.Source: Figure 12 · p. 23, Figure 13 · p. 23
  • Fig. 14 depicts attention-head patterns for masked tokens, with line darkness indicating attention strength.Some attention weights are too low to be visible.Source: Figure 14 · p. 24
Loading 2112.05112v2…