Source-linked AI summary
Variational Transformer Networks for Layout Generation
Diego Martin Arroyo, Janis Postels, Federico Tombari
TL;DR
Layout generation must handle variable-size structures whose elements mix discrete and continuous properties while capturing relationships that govern realistic designs. The paper develops Variational Transformer Networks by combining self-attention with VAEs, and reports state-of-the-art diversity and perceptual quality across public benchmarks, with application to layout detection.
Problem
Layout synthesis remains sparsely explored despite its importance for design and data-synthesis tasks, because layouts contain variable-length sets of discrete and continuous elements with interdependent relationships.
Method
VTNs instantiate VAEs with attention-based encoder-decoder networks that learn layout relationships without annotations and support variable-length layouts.
Results
VTNs achieve state-of-the-art quantitative performance across metrics and public datasets, with appealing perceptual quality and improved diversity at similar perceptual quality compared to [9].
Takeaways & Limitations
The method can synthesize diverse layouts and serve as a data synthesizer for a downstream document layout-detection task.
Takeaways & Limitations
The noted restriction originates from memory constraints and does not imply that the approach cannot learn larger layouts given sufficient memory.
Abstract
from arXiv · showhide
Generative models able to synthesize layouts of different kinds (e.g. documents, user interfaces or furniture arrangements) are a useful tool to aid design processes and as a first step in the generation of synthetic data, among other tasks. We exploit the properties of self-attention layers to capture high level relationships between elements in a layout, and use these as the building blocks of the well-known Variational Autoencoder (VAE) formulation. Our proposed Variational Transformer Network (VTN) is capable of learning margins, alignments and other global design rules without explicit supervision. Layouts sampled from our model have a high degree of resemblance to the training data, while demonstrating appealing diversity. In an extensive evaluation on publicly available benchmarks for different layout types VTNs achieve state-of-the-art diversity and perceptual quality. Additionally, we show the capabilities of this method as part of a document layout detection pipeline.
1. Introduction
Layout synthesis matters for downstream design and data-generation tasks, but remains challenging because layouts combine variable-length sets with discrete and continuous elements. The paper introduces VTNs, which use attention-based VAE components to learn layout relationships without annotations.
- Layouts support downstream tasks including neural scene synthesis, graphic design, and data synthesis, motivating flexible generation of realistic novel layouts.
- Layout generation is challenging because samples have variable lengths and combine discrete classes with continuous coordinates.
- Element positions and sizes depend on both element types and relationships with other elements, making handcrafted layout rules subjective and difficult to generalize.
- VTNs instantiate the VAE framework with attention-based architectures to investigate layout generation across multiple datasets and metrics.
- The proposed model learns high-level relationships among many layout elements without annotations and explores variational bottlenecks for variable-length sequences.
2. Related work
Prior layout-generation methods use GANs, recurrent models, graph networks, or self-attention, often relying on heuristics, conditioning, or constrained generation procedures. The paper positions attention-based VAEs as a direct way to model layout distributions while addressing relationships and variable-size layouts.
- LayoutGAN uses self-attention and differential rendering to train a GAN that synthesizes bounding-box annotations as images for discrimination.
- LayoutVAE models layouts with an autoregressive conditional VAE, LSTMs, and a second VAE for category-count distributions.
- Neural Design Networks use heuristically labeled relationships, graph convolutions, a separate layout generator, and refinement, whereas this work avoids relationship labels.
- READ also uses heuristic relationships with a VAE based on recursive neural networks to learn layout distributions.
- Self-attention layout completion uses an autoregressive decoder and beam search, but its generation depends on additional hyperparameters and heuristic regularization for diversity.
- Existing approaches span furniture-specific models, conditional graphic-design models, and methods differing in relationship supervision, layout size, and distributional guarantees.
3. Variational Transformer Networks
VTNs adapt the VAE framework to layouts using Transformer encoder-decoder networks with self-attention, modeling relationships among variable numbers of discrete-continuous elements. The design includes autoregressive and non-autoregressive decoding, learned sequence-length handling, and β-VAE optimization.
- Variational Transformer Networks: VTNs use attention layers as the main encoder and decoder building blocks of a VAE tailored to layout synthesis.
- Exploiting the Data Format of Layouts: Layouts are variable-size sets of elements containing discrete classes and continuous bounding-box features, with strong correlations among elements.
- Exploiting the Data Format of Layouts: Attention supplies an inductive bias for learning pairwise relationships from bounding-box annotations without expensive or ambiguous relationship labels.
- Architecture of VTNs: The encoder parameterizes qθ(z|x), while the decoder can be autoregressive or non-autoregressive; the autoregressive form models pφ(x|z) as a product over layout elements.
- Architecture of VTNs: For non-autoregressive decoding, the prior models both latent variables and the number of bounding boxes, allowing sequence length to be learned during training.
- Optimizing VTNs: VTNs optimize a β-weighted KL term with reconstruction loss to reduce posterior collapse, using β=1 for autoregressive and β=0.5 for non-autoregressive decoding.
- Optimizing VTNs: During inference, latent representations sampled from the prior are transformed into layouts by the self-attention-based decoder.
4. Experiments
The experiments evaluate layout generation across multiple datasets using perceptual-quality and diversity metrics, then examine decoder and prior choices, qualitative behavior, and downstream layout detection. The method produces diverse layouts with alignment metrics close to real data, outperforms LayoutVAE, and achieves comparable or better results against other baselines and explicit constraints.
- Experimental setup: The evaluation covers document, UI, natural-scene, and furniture-layout datasets using metrics for perceptual quality and diversity.The metrics include IoU, overlap, alignment, Wasserstein distances, and unique DocSim matches.
- Ablation study: The autoregressive decoder improves perceptual quality while slightly reducing diversity, and the non-learned prior provides the preferred balance with model simplicity.With an autoregressive decoder, learned and non-learned priors produce similar results; the authors therefore use the non-learned prior.
- Quantitative results: Across datasets, the method generates many distinct layouts with alignment metrics similar to real data and outperforms LayoutVAE across all reported metrics.Compared with Gupta et al., it improves diversity at similar perceptual quality.
- Quantitative results: On RICO, the method achieves superior alignment without explicit design constraints and similar performance even when all relationships are provided.This comparison uses the Neural Design Network alignment metric.
- Qualitative results: Qualitative results show strict margins, valid interpolated layouts, and relationships captured across element distances and sequence positions.The qualitative comparisons also report that LayoutVAE struggles with layouts containing many elements.
- Layout detection: Synthesized layouts alone achieve good mean average precision at 0.5 IoU when used to train a document layout detector.The experiment compares training with synthesized layouts against real layouts and real PubLayNet images.
5. Conclusion and future work
The work uses self-attention as the foundation of a VAE for layout synthesis and reports strong quantitative and perceptual results across diverse public datasets. It also identifies autoregressive decoding as important for quality and outlines extensions for richer layout properties and downstream scene synthesis.
- Self-attention layers serve as fundamental building blocks of a VAE tailored to layout synthesis.
- The approach achieves state-of-the-art quantitative performance across all evaluated metrics and produces layouts with appealing perceptual quality.
- Autoregressive decoding is an important ingredient for obtaining high-quality layouts.
- Future work includes generating additional properties, varying layout dimensions, and integrating end-to-end layout synthesis into scene synthesis pipelines.
Supplementary: Variational Transformer Networks for Layout Generation
The supplementary document is identified as arXiv:2104.02416, version 1, dated 6 April 2021.
- The supplementary document has arXiv identifier 2104.02416.
- The document is marked as version 1.
- The document date is 6 April 2021.
1. Attention analysis
The attention analysis examines how self-attention captures relationships among layout elements and how autoregressive decoding uses prior outputs. Attention develops across encoder layers and can connect elements regardless of their sequence distance.
- 1. Attention analysis: Self-attention is intended to model relationships between layout elements regardless of their distance in the input sequence.
- 1.1. Encoder: In the encoder, elements are independent in the first layer, while later layers begin incorporating other elements.
- 1.1. Encoder: Attention visualization illustrates increasing relevance among elements across subsequent encoder layers.
- 1.2. Decoder: During autoregressive decoding, the network uses the encoded document vector and previous iterations to determine the next element's location and size.
- 1.2. Decoder: The decoding process is visualized through the relevance of existing elements during autoregressive generation.
2. Latent Space Analysis
The latent-space analysis tests interpolation and consistency of latent representations. Interpolations generally yield valid layouts, while removing latent vectors appears to preserve layout structure to a substantial degree.
- 2. Latent Space Analysis: The analysis evaluates latent-space interpolation and the effect of individual latent vectors in the non-autoregressive decoder.
- 2.1. Interpolation: Intermediate latent vectors produce valid layouts, although the latent space is not perfectly smooth.The interpolated vector is z′ = z1 + λ · (z2 − z1), with λ between 0 and 1.
- 2.2. Latent-code consistency: Removing latent vectors one by one is used to test whether decoded layouts remain consistent.
- 2.2. Latent-code consistency: Qualitative latent-space experiments examine how adding elements to the latent vector changes the generated layout.
- 2.2. Latent-code consistency: The consistency experiment considers the encoder's latent distribution through its mean representation.
3. Qualitative results
The qualitative results present larger sets of synthesized layouts across document, interface, scene, furniture, and language data, including unprocessed outputs and rendered documents. The language experiment further shows that the architecture can generate realistic, grammatical reviews.
- Layout samples: Larger sample sets expose the diversity and quality of synthesized layouts without postprocessing or cherry-picking.The samples use network output coordinates directly, and comparison rows include other methods.
- Layout samples: PubLayNet examples include outputs from autoregressive and non-autoregressive decoders, including rendered document results.The autoregressive examples also show the result of feeding generated layouts to a document renderer.
- Layout samples: The appendix provides synthesized examples for RICO, COCO-Stuff, and SUN RGB-D.These figures cover interface, scene, and furniture-oriented layout data.
- Language modeling: On 800K Amazon book reviews, the autoregressive model generated reviews described as grammatically correct and highly realistic.The model used six self-attention layers in both encoder and decoder, with a 30K-word target vocabulary.
- Language modeling: The generated-review experiment suggests that the method can also be applied to other sequence-generation tasks.The experiment replaced the first fully connected layer with a learned word embedding and adapted the output shape to the vocabulary.
4. Convergence tests
The convergence test examines how training-set size affects generalization on PubLayNet by measuring unique DocSim matches across repeated trainings. For this dataset, 50K training samples are reported as sufficient to generalize well.
- Convergence tests: 50K PubLayNet training samples are enough for the autoregressive model to generalize well on this dataset.The study averages results across five identical trainings and evaluates 1,000 generated samples.
- Convergence tests: The convergence analysis evaluates the number of unique DocSim matches while varying the size of the PubLayNet training subset.Figure 9 reports the convergence results with ±1σ.
5. Distribution analysis
The distribution analysis compares bounding-box center locations in synthetic and real PubLayNet data. It uses 1,000 samples to assess how well the generated layouts capture the observed location distribution.
- Distribution analysis: Bounding-box center frequencies are compared between 1,000 synthetic samples and real PubLayNet test data.Figure 10 places the synthetic distribution above the real-data distribution.
- Distribution analysis: The comparison uses bounding-box center location as an additional metric for capturing the layout distribution.The analysis focuses on the frequency of each location as a bounding-box center.