Constrained Graphic Layout Generation via Latent Optimization

1 INTRODUCTION

• Layouts arrange element sizes and positions to convey information, with designers considering semantics, priority, reading order, alignment, and overlap.The introduction notes that inexperienced designers may struggle to produce high-quality presentations while maintaining these considerations.Source: §1 · p. 1
• Existing approaches may add constraint-specific losses or conditioning, whereas latent-space optimization can use existing models without fitting a new condition or loss for each constraint.The paper also introduces LayoutGAN++ as a Transformer-based layout GAN; its discriminator uses auxiliary layout reconstruction, and the model improves on LayoutGAN for unconstrained generation.Source: §1 · p. 2
• CLG-LO formulates constrained layout generation as latent-space optimization, iteratively changing an unconstrained model’s latent code to satisfy user specifications.A single pretrained GAN can incorporate various constraints without retraining the model.Source: §1 · p. 2
• Across three public datasets, LayoutGAN++ is comparable to or better than existing methods in unconstrained generation, while CLG-LO significantly improves constrained generation.The experiments use two constrained-generation settings designed to resemble real use cases.Source: §1 · p. 2
• The authors summarize contributions spanning latent-code constraint handling, a layout-GAN architecture and methodology, and experiments on public datasets.The experiments address both unconstrained and constrained layout generation.Source: §1 · p. 2

2.1 Layout Generation

• Prior layout-generation work includes VAEs, LayoutGAN, conditioned and raster generators, hierarchical autoencoders, graph-based networks, and Transformer- or attention-based models.The cited work includes LayoutVAE [12], LayoutGAN, READ, NDN, and other approaches.Source: §2.1 · p. 2
• The authors combine unconstrained and constrained layout generation by applying user specifications as constraints to a learned generator, using one model rather than NDN’s approach.Their unconstrained generator is based on LayoutGAN.Source: §2.1 · p. 2

2.2 Latent Space Exploitation

• Latent-space use was studied in image editing and restoration, while non-image examples include 3D-model shape, game-level design, and human-in-the-loop exploration.The authors position layout generation as another latent-space exploration task: finding a representation whose generated layout satisfies user-specified constraints.Source: §2.2 · p. 2

3.1 LayoutGAN++

• LayoutGAN++ uses Transformer blocks to model relationships among elements, while its discriminator scores layout realism and reconstructs input bounding boxes and labels.The generator takes element codes and labels and outputs bounding boxes.Source: generator · p. 3, §3.1 · p. 3, §3.1.2 · p. 3

3.1.2 Discriminator.

• The discriminator can underweight positional trends in aligned domains, so an auxiliary decoder reconstructs bounding boxes from its internal representation to improve positional awareness.The decoder also reconstructs labels using learnable positional embeddings.Source: §3.1.3 · p. 3

3.1.4 Training objective.

• The reconstruction loss measures similarity between bounding-box and label sets, using mean squared error for boxes and cross-entropy for labels.Bounding boxes are sorted by ground-truth position before reconstruction loss is computed.Source: §3.1.4 · p. 3

3.2 Constrained Layout Generation via Latent Optimization (CLG-LO)

• CLG-LO formulates user constraints as a constrained latent-code optimization problem, seeking layouts that satisfy constraints and appear realistic to the discriminator.It uses an augmented Lagrangian that combines the realism objective with constraint terms.Source: §3.2 · p. 4
• The augmented-Lagrangian procedure updates multipliers according to constraint violations and gradually increases the penalty parameter.The experiments use either Adam or CMA-ES for the inner optimization, stopping when violations are sufficiently small or the iteration limit is reached.Source: §3.2 · p. 4
• To reduce the risk of perceptually degraded adversarial outputs, the method clamps the discriminator objective relative to its value at the initial latent code.The procedure returns bounding boxes generated from the optimized latent codes.Source: §3.2 · p. 4

• The experiments assess both unconstrained and constrained layout generation, introducing datasets and metrics before the setup for each task.Source: §4 · p. 4

4.1 Dataset

• The evaluation uses Rico for mobile-app UI designs, PubLayNet for document images, and Magazine for magazine pages.Source: §4.1 · p. 4
• Rico and PubLayNet layouts are restricted to the 13 most frequent Rico labels, and layouts with more than 10 elements are excluded from both datasets.Source: §4.1 · p. 4
• PubLayNet uses 95% of its official training split for training, the remainder for validation, and its official validation split for testing; Rico and Magazine use 85/5/10 splits.Source: §4.1 · p. 4

4.2.1 FID.

• Table 2 compares FID scores for feature extractors trained with classification, reconstruction, or combined objectives across noise, vertical-flip, and validation-neighbor variants.Source: Table 2 · p. 5

4.2 Evaluation Metrics

• FID uses intermediate features from a network trained to distinguish real layouts from noise-added layouts, with an auxiliary decoder incorporated into training.The combined classification and reconstruction objectives improve sensitivity to different layout arrangements.Source: §4.2.1 · p. 5

4.2.2 Maximum IoU.

• Maximum IoU matches bounding boxes optimally between layouts, allowing only same-label box matches and, at the layout level, identical label sets.The assignment problems are solved using SciPy's solver.Source: §4.2.2 · p. 5
• Alignment and Overlap follow prior metrics, modified by normalization with the number of elements.Source: §4.2.3 · p. 5

4.3 Unconstrained Layout Generation

• The comparison uses LayoutGAN-W and LayoutGAN-R variants plus NDN-none; the latter generates relations, then boxes, and modifies misaligned boxes.The authors reimplemented baselines because official bounding-box baseline code was unavailable, and trained LayoutGAN++ for 200,000 iterations.Source: §4.3.1 · p. 5

4.3.1 Setup.

• LayoutGAN++ achieves results comparable to or better than NDN-none, with similar results on Rico and favorable results on PubLayNet and Magazine.LayoutGAN-W is unstable to train, while LayoutGAN-R trains more stably and outperforms LayoutGAN-W.Source: §4.3.2 · p. 5
• Quantitative results are summarized in Table 3 and qualitative results in Figure 3, with means and standard deviations across five evaluations using the same trained model.Comparisons cover Rico, PubLayNet, and Magazine.Source: §4.3.2 · p. 5

4.4 Layout Generation with Beautification Constraints

• The setting aims to post-process unconstrained layouts into aligned, non-overlapping results, using PubLayNet, whose layouts are mostly aligned and have little overlap.The initial CMA-ES sigma is set small, so optimized layouts do not change substantially from their initial layouts.Source: §4.4 · p. 5, §4.4.3 · p. 7

4.4.1 Constraints.

• Alignment is constrained by a threshold and overlap is constrained using the Overlap metric; the alignment threshold is set to 0.004.The alignment constraint is defined using the Alignment metric.Source: §4.4.1 · p. 6

4.4.2 Setup.

• The CAL baseline adds alignment and non-overlap losses to the generator objective, encouraging constraint satisfaction without explicitly enforcing it.There are no directly comparable methods for this setting, so CAL is designed as a baseline referring to similar work.Source: §4.4.2 · p. 6

4.4.3 Results.

• CLG-LO further improves Alignment and Overlap significantly, with almost no degradation in FID and Maximum IoU relative to the loss-based baseline.Table 4 reports results against the unconstrained LayoutGAN++ base model.Source: Table 4 · p. 6, §4.4.3 · p. 6
• CMA-ES appears to perform better than Adam; the authors attribute this tentatively to many local solutions in the augmented-Lagrangian objective.Mean runtime is 13.6 seconds with Adam (SD: 11.2) and 1.45 seconds with CMA-ES (SD: 1.75).Source: §4.4.3 · p. 6, §4.4.2 · p. 6
• CLG-LO with CMA-ES produces aligned, non-overlapping layouts, while the initial and optimized examples are displayed in Figure 4.The figure places initial unconditioned generations above the optimized results.Source: Figure 4 · p. 7, §4.4.3 · p. 7

4.5 Layout Generation with Relational Constraints

• Users specify size relations (smaller, larger, equal), location relations (above, bottom, left, right, overlap), or relations to the canvas, with constraints drawn from subsets of ground-truth relations.The experiment varies the percentage of relations used and reports the rate of violated constraints.Source: §4.5 · p. 7

4.5.1 Constraints.

• The size constraint sums costs for specified size relations, and the location constraint is defined analogously using bounding-box coordinates.The size-relation tolerance parameter is set to r = 0.1.Source: §4.5.1 · p. 7

4.5.2 Setup.

• CLG-LO is compared against NDN using CMA-ES and without the beautification constraints; its mean runtime is 1.96 seconds (SD: 3.48).The remaining settings follow the beautification-constraint experiment.Source: §4.5.2 · p. 7
• With 10% of relations used as constraints, CLG-LO performs comparably to or better than NDN and significantly improves the constraint-violation metric.The authors consider this proportion representative of realistic use; NDN does not guarantee inferred layouts satisfy constraints.Source: §4.5.3 · p. 7

4.5.3 Results.

• CLG-LO performs less favorably as the proportion of constrained relations rises, while NDN improves as more relations are supplied.The authors note CLG-LO performs best when users specify few relations; one proposed remedy is to let users choose among layouts stored during optimization.Source: §4.5.3 · p. 7

5 CONCLUSIONS AND DISCUSSION

• Across three public datasets, LayoutGAN++ outperforms existing approaches in unconstrained generation, while CLG-LO supports constrained generation.The authors report that experiments support the effectiveness of both methods across constrained and unconstrained tasks.Source: §5 · p. 7
• The approach has no convergence guarantee because its objective and constraints involve a complex nonlinear neural network.The authors note that many constraints can make it difficult for the optimizer to find a good solution, and suggest improved optimization or initialization as possible mitigations.Source: §5 · p. 7
• With a small part of relations specified, CLG-LO often outperforms NDN; many constraints can make the optimizer underperform existing approaches.Figure 6 compares Max. IoU, Alignment, and Violation across three datasets; Table 5 reports results with 10% of relational constraints. Figure 5 shows constraints, initial generations, and optimized outputs.Source: §5 · p. 7, Figure 6 · p. 8, Table 5 · p. 8
• CLG-LO can change its objective as well as its constraints and can use models that generate diverse plausible layouts from latent variables.With a VAE lacking an explicit layout-quality measure, the task becomes constraint satisfaction; a discriminator may be needed if output quality is problematic.Source: §5 · p. 8
• Future directions include adding human-perception constraints, exploring diverse layouts through latent codes, and testing CLG-LO beyond layout generation.Source: §5 · p. 8