PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutDM: Discrete Diffusion Model for Controllable Layout Generation

Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, Kota Yamaguchi

arXiv:2303.08137v1cs.CVcs.GR

TL;DR

LayoutDM addresses controllable layout generation by modeling structured layouts with discrete state-space diffusion. It uses modality-wise corruption, padding, and inference-time constraint injection, and reports stronger performance than task-agnostic baselines across most evaluated cases, with promising results against task-specific baselines.

  • Problem

    Controllable layout generation must arrange structured element attributes while accommodating optional constraints such as known categories, positions, or relationships.

  • Method

    LayoutDM uses modality-wise discrete diffusion with padding for structured variable-length layouts and injects constraints through masking or logit adjustment during inference.

  • Results

    LayoutDM outperforms task-agnostic baselines in the majority of cases and shows promising performance compared with task-specific baselines across Rico and PubLayNet tasks.

  • Takeaways & Limitations

    A single discrete-diffusion model can support diverse conditional layout-generation tasks without additional training or external models.

  • Takeaways & Limitations

    Direct comparison with reported literature results is limited because datasets lack official splits and prior methods use differing splits, preprocessing, and FID models.

Abstract

from arXiv · show

Controllable layout generation aims at synthesizing plausible arrangement of element bounding boxes with optional constraints, such as type or position of a specific element. In this work, we try to solve a broad range of layout generation tasks in a single model that is based on discrete state-space diffusion models. Our model, named LayoutDM, naturally handles the structured layout data in the discrete representation and learns to progressively infer a noiseless layout from the initial input, where we model the layout corruption process by modality-wise discrete diffusion. For conditional generation, we propose to inject layout constraints in the form of masking or logit adjustment during inference. We show in the experiments that our LayoutDM successfully generates high-quality layouts and outperforms both task-specific and task-agnostic baselines on several layout tasks.

1. Introduction

LayoutDM applies discrete state-space diffusion to controllable layout generation, progressively denoising structured layouts while injecting constraints during inference. It targets diverse conditional tasks with one model and reports strong performance across public datasets.

  • 1. Introduction: LayoutDM formulates layout generation as discrete diffusion and progressively infers complete layouts from corrupted states.The model uses modality-wise diffusion for structured layout data and a denoising backbone to infer layouts with or without conditions.
  • 1. Introduction: LayoutDM avoids autoregressive dependency-order failures and supports variable-length layouts, unlike non-autoregressive approaches that perform poorly with naive padding.Padding combined with the diffusion formulation significantly improves variable-length generation.
  • 1. Introduction: LayoutDM outperforms task-agnostic baselines in the majority of evaluated cases and shows promising performance against task-specific baselines.Evaluation uses multiple layout tasks on Rico and PubLayNet, with ablations examining quantization and positional embedding.
  • 1. Introduction: Masking and logit adjustment inject complex layout constraints during inference, enabling diverse conditional generation without additional training or external models.These mechanisms steer the denoising process for conditional tasks.
  • 1. Introduction: The paper contributes a modality-wise discrete diffusion process, padding for structured variable-length layouts, and inference-time constraint injection.These contributions are presented as the foundation for a single model across conditional layout tasks.

2. Related Work

Prior layout-generation work includes autoregressive, non-autoregressive, optimization-based, and diffusion approaches, but existing methods face limitations in partial conditioning and complex constraints. LayoutDM extends discrete diffusion with inference-time masking and logit adjustment for constrained layouts.

  • 2. Related Work: Layout generation research covers unconditional synthesis and conditional tasks including category or size control, relational constraints, completion, and refinement.These tasks appear across design applications such as magazine covers, posters, slides, interfaces, and banners.
  • 2. Related Work: Autoregressive decoders struggle with partial inputs because their fixed generation order may conflict with known elements or attributes.BLT uses fill-in-the-blank generation to address conditional input, but cannot solve decoder-based layout completion.
  • 2. Related Work: Complex user constraints such as relative box positions have been handled with handcrafted costs or constrained optimization for pretrained generative models.LayoutDM addresses these constraints through logit adjustment on task-agnostic iterative prediction.
  • 2. Related Work: Diffusion models progressively corrupt and denoise data, while discrete variants such as D3PM and VQDiffusion model categorical variables with structured corruption or mask-and-replace.These approaches motivate LayoutDM's discrete-state formulation.
  • 2. Related Work: LayoutDM extends discrete-state diffusion to conditional layout generation through hard masking or logit adjustment during reverse diffusion.The conditioning is injected into inference rather than requiring a separately trained conditional model.

3. LayoutDM

LayoutDM represents layouts as discrete, padded token sequences and applies modality-wise corruption with a Transformer denoiser. During reverse diffusion, it supports conditional generation by injecting known fields or other constraints into the sampled states.

  • 3.1. Preliminary: Discrete Diffusion Models: Discrete diffusion defines a forward categorical corruption process with transition matrices and a reverse neural denoising process.The forward process composes transition matrices across timesteps, while the reverse model predicts distributions for less-corrupted states.
  • 3.1. Preliminary: Discrete Diffusion Models: The reverse transition can be computed by combining the discrete posterior with a neural prediction of the original state.D3PM sums over possible clean states to obtain each reverse step.
  • 3.2. Unconditional Layout Generation: Layouts are quantized into category, position, width, and height tokens, then randomly shuffled by element and flattened for Transformer processing.Each layout element is represented by categorical information and four discretized bounding-box attributes.
  • 3.2. Unconditional Layout Generation: A [PAD] token enables variable-length generation by padding each layout to a maximum of M elements, yielding 5M tokens.The padded representation makes layouts fixed-dimensional for diffusion while preserving variable element counts.
  • 3.2. Unconditional Layout Generation: Modality-wise diffusion uses separate corruption matrices for category and geometric attributes, preventing tokens from switching across incompatible attribute groups.The matrices include [PAD] and [MASK] states and have dimensions determined by the corresponding category or coordinate vocabulary.
  • 3.2. Unconditional Layout Generation: Adaptive quantization balances imbalanced position and size distributions, while decoupled positional encoding separately represents element and attribute indices.Both choices are designed to preserve layout structure and improve generation performance.
  • 3.3. Conditional Generation: Conditional inference injects information into the initial and intermediate states while keeping the denoising network frozen.The injection mechanism varies with the condition type.

4. Experiment

LayoutDM is evaluated across diverse layout-generation tasks on Rico and PubLayNet, using FID, Maximum IoU, DocSim, and constraint-violation metrics. It generally performs strongly against task-agnostic and task-specific baselines while supporting conditional, refinement, and relationship-constrained generation.

  • Evaluation setup: LayoutDM is compared using FID, Maximum IoU, and, for refinement, DocSim, alongside alignment, overlap, and constraint-violation metrics.FID captures feature-space similarity, Maximum IoU measures conditional similarity, and DocSim compares predicted layouts with corresponding ground truth layouts.
  • Evaluation setup: Evaluation covers six tasks, including unconditional generation, partially conditioned field prediction, completion, refinement, and relationship-constrained generation.The experiments use Rico and PubLayNet, with layouts capped at 25 elements during preprocessing.
  • Conditional generation: LayoutDM outperforms other models except LayoutTrans in completion, with the results suggesting that padding combined with diffusion is important for generation quality.The authors also observe that FID and Maximum IoU are not highly correlated in completion.
  • Unconditional generation: Under random element order, LayoutDM achieves the best FID in unconditional generation.The authors associate poor BLT performance with train-test masking-distribution inconsistency and caution that very small alignment values can indicate trivial outputs in Rico.
  • Refinement: In refinement, imposing noisy geometric fields as a weak prior substantially improves the masking-only model and brings performance closer to RUITE.Both LayoutDM and RUITE recover complete layouts from non-trivially noisy inputs.

5. Discussion

LayoutDM’s discrete-state diffusion framework is positioned for broader controllable layout generation, with extensions to more attributes and acknowledged misuse risks.

  • LayoutDM is based on diffusion models for discrete state-space, while continuous-state diffusion is identified as an interesting future direction.
  • The proposed logit adjustment is intended to incorporate additional attributes, including area, aspect ratio, and reading order constraints.These attributes can be converted into size and location relationship constraints.
  • Automatically generating basic website or mobile-application structures could facilitate scams or misinformation.

A. Implementation Details

The implementation compares task-agnostic masking baselines and controls model size for a fair parameter-matched comparison.

  • BART and MaskGIT* are adapted as task-agnostic layout-generation baselines using random masking for discrete layout tokens.BART treats shuffled layouts as one-dimensional discrete-token sequences, while MaskGIT* uses its discrete-token generation stage.
  • Models are adjusted to approximately 12M parameters for a fair comparison.Exact parameter counts are reported in Table 5.

A.2. Relationship Guidance

Relationship guidance defines losses that penalize predicted layouts when element sizes, vertical positions, areas, aspect ratios, or reading order violate user-specified relationships.

  • The relationship loss for making element j larger than element i penalizes cases where j’s area does not exceed i’s area by the tolerance factor.The predicted coordinates are used to define differentiable geometric penalties.
  • A vertical-ordering loss compares the bottom of element j with the top of element i to encourage j to lie above i.The tolerance parameter γ is empirically set to 0.1.
  • Area, aspect-ratio, and reading-order constraints can be represented with losses over predicted geometry.Reading order is based on distance from the canvas’s left-top corner, with a margin loss encouraging i to precede j.
  • Hyperparameters for sampling and relationship guidance are selected using validation data, including distance margin and weighting-term grid searches for refinement.

A.4. Evaluation

Evaluation generates 1,000 unconditional samples from a random seed, while conditional evaluation generates one sample for each test-set input.

  • Unconditional generation produces 1,000 samples from a random seed.
  • Conditional generation uses each dataset’s test set to construct partial inputs and generates one sample per test example.

B.1. Ablation Study

The ablations examine state-space choice, model configuration, and logit-adjustment strategies for LayoutDM. Discrete state-space diffusion with the default refinement adjustment is supported by the reported comparisons.

  • Model configuration: LayoutDM’s parameter counts are summarized across the compared models.
  • State-space choice: The state-space ablation evaluates discrete and continuous choices for Rico unconditional generation.Table 6 highlights the top two results with bold and underline formatting.
  • State-space choice: Continuous-state DiffusionLM remains far behind discrete state-space models in Rico unconditional generation, despite testing different samplers and training timesteps.The comparison uses DiffusionLM with 12.6M parameters and embedding dimension d = 16.
  • Logit adjustment: The refinement ablation compares Gaussian and negation logit-adjustment priors for imposing positional constraints.Both priors use the distance between token location loc(j) and the noisy observation x̂_i, with different treatments inside and outside the margin m.
  • Logit adjustment: The distance margin m is selected from {0.2, 0.4, 0.6, 0.7, 0.8, 0.9} for the refinement comparison.
  • Logit adjustment: Default outperforms the other logit-adjustment choices by a large margin in the quantitative evaluation.
  • Efficiency: Speed-quality curves report average single-layout generation runtime using batch size 64, with shaded regions representing standard deviation across three trials.

B.3. More Results

Additional qualitative results compare LayoutDM with task-specific baselines across conditional, unconditional, and refinement tasks on PubLayNet and Rico. The examples expose recurring failure patterns and indicate that Rico is more difficult because it has more categories and less-aligned elements.

  • Qualitative comparisons: Additional figures compare task-specific baselines for conditional generation, unconditional generation, and refinement on PubLayNet and Rico.The comparisons include C→S+P, C+S→P, unconditional generation, and refinement tasks.
  • Failure cases: Typical baseline failures include frequent element overlap, unnecessarily broad blank space, and limited diversity.Frequent overlap is often observed in BLT, while broad blank space is often observed in LayoutTrans.
  • Dataset difficulty: Rico is more difficult to generate because it contains more categories and elements that are less aligned than those in PubLayNet.
  • Density and coverage: Density measures fidelity while coverage measures diversity, and iterative refinement models are evaluated as the number of prediction timesteps increases.Density reflects resemblance to real samples; coverage reflects whether generated samples span the variability of real samples.

B.5. Alignment and Overlap

Alignment and Overlap provide fidelity-focused comparisons, while density-coverage and speed-quality figures show broader trade-offs across models and tasks. LayoutDM is competitive in fixed-length generation, whereas autoregressive models are moderately better in variable-length generation.

  • Metrics: Alignment and Overlap capture fidelity, and the model closest to Real data is preferred over simply selecting the lowest score.Alignment values are scaled by 100× for visibility, and the reported definitions follow prior work.
  • Fixed-length generation: In fixed-length generation, LayoutDM performs almost comparably to VQDiffusion* and BART and better than the other models.The fixed-length tasks are C→S+P and C+S→P.
  • Variable-length generation: In variable-length generation, BART and LayoutTrans. are moderately better than LayoutDM, while diffusion-based models outperform BLT and MaskGIT*.The variable-length tasks are completion and unconditional generation.
  • Compared models: The figure materials include comparisons involving LayoutDM, VQDiffusion*, BLT, MaskGIT*, LayoutTrans., BART, LayoutVAE, NDN-none, LayoutGAN++, RUITE, and Ground Truth.
  • Conditional generation: The conditional-generation figures compare C→S+P and C+S→P on PubLayNet and Rico using three samples from each model.
  • Unconditional generation: The unconditional-generation figures compare PubLayNet and Rico using five samples from each model to demonstrate diversity.
  • Refinement: The refinement figures show three LayoutDM samples for PubLayNet and Rico.
  • Trade-offs: Density-coverage figures analyze the fidelity-diversity trade-off, while speed-quality figures compare generation efficiency and quality across models.
Loading 2303.08137v1…