Source-linked AI summary
LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators
Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, Tingfa Xu
TL;DR
Graphic-layout generation requires modeling structured elements and precise geometric relations rather than only pixels. LayoutGAN directly refines labeled elements with geometric parameters and uses differentiable wireframe rendering for visual discrimination, successfully generating layouts across multiple domains.
Problem
Graphic designs use structured primitive elements whose attributes and relations matter, while conventional image-space GANs mix layout with rendering.
Method
LayoutGAN refines randomly initialized labeled graphic elements with geometric parameters and uses a differentiable wireframe layer with CNN-based visual discrimination.
Results
LayoutGAN successfully generates layouts respecting element types and relationships across MNIST, document, clipart, tangram, and mobile-app tasks.
Takeaways & Limitations
The approach supports layout synthesis by modeling geometric relations directly and judging alignment in a visual wireframe domain.
Abstract
from arXiv · showhide
Layout is important for graphic design and scene generation. We propose a novel Generative Adversarial Network, called LayoutGAN, that synthesizes layouts by modeling geometric relations of different types of 2D elements. The generator of LayoutGAN takes as input a set of randomly-placed 2D graphic elements and uses self-attention modules to refine their labels and geometric parameters jointly to produce a realistic layout. Accurate alignment is critical for good layouts. We thus propose a novel differentiable wireframe rendering layer that maps the generated layout to a wireframe image, upon which a CNN-based discriminator is used to optimize the layouts in image space. We validate the effectiveness of LayoutGAN in various experiments including MNIST digit generation, document layout generation, clipart abstract scene generation and tangram graphic design.
1 INTRODUCTION
LayoutGAN addresses the difficulty of generating structured graphic designs by directly modeling labeled elements and their geometric relations. It combines relation-aware generation with wireframe-based visual discrimination and succeeds across several layout-generation tasks.
- Graphic designs depend on element attributes and precise spatial relations, but conventional image-space GANs conflate layout with rendering.Misalignment of only a few millimeters can ruin a design.
- LayoutGAN directly synthesizes resolution-independent sets of labeled graphic elements with class probabilities and geometric parameters.Its generator refines randomly initialized elements into a design layout.
- The generator uses self-attention-like relation modeling to refine each element using information from the other elements in the set.The relation response aggregates pairwise contributions and normalizes them by the number of elements.
- LayoutGAN uses both a structured-data discriminator and a visual-domain discriminator, covering element relations as well as rendered layout patterns.The structured discriminator is effective but less sensitive to misalignment and occlusion.
- Across MNIST, document, clipart, tangram, and mobile-app layout tasks, LayoutGAN generates layouts respecting domain-specific element types and relationships.The evaluation includes both sanity tests and graphic-design applications.
- A differentiable wireframe rendering layer maps discrete element arrangements to layouts that enable CNN-based alignment judgments.Wireframes avoid the occlusion problems caused by filled bitmap masks during back-propagation.
2 RELATED WORK
LayoutGAN extends structured-data generation from ordered sequences or regular lattices to unordered sets of labeled graphic primitives, refining their semantic and spatial relations jointly. It combines relation-based and visual-domain discriminators, using differentiable wireframe rendering to assess layout alignment, and evaluates the approach across several layout-generation tasks.
- Structured data generation: LayoutGAN represents designs as unordered sets of labeled graphic elements rather than regular image lattices or sequential data.Each element is represented by class probabilities and geometric parameters such as point coordinates, bounding-box corners, or scale and flip values.
- LayoutGAN: The generator encodes randomly initialized elements, refines each element using self-attention over all other elements, and decodes updated classes and geometry.The relation module aggregates pairwise relations through contextual residual learning before decoding the refined representation.
- Discriminators: LayoutGAN uses both a relation-based discriminator over layout parameters and a wireframe rendering discriminator that classifies rendered layouts with a CNN.The visual discriminator is intended to capture spatial properties such as alignment and occlusion that direct parameter-based discrimination may miss.
- Experiments: The wireframe discriminator produces more compact, better-aligned MNIST point layouts and a higher inception score than the relation-based discriminator.For document layouts, it also achieves lower overlap and alignment indices than the relation-based discriminator.
- Experiments: Experiments cover MNIST digits, document layouts, clipart abstract scenes, and tangram graphic design, with generated layouts respecting domain-specific element types and relationships.The clipart experiment first generates labeled element layouts and then renders the corresponding clipart elements according to predicted positions and scales.
6 APPENDIX
The appendix illustrates LayoutGAN outputs across document, clipart, tangram, and mobile-app layout tasks, alongside implementation details for generation and visual comparison.
- Layout generation results: Document, clipart, and tangram figures visualize generated layouts for distinct graphic-design problems.The figures separately present document layout generation, clipart abstract scene generation, and tangram graphic design.
- Generator architecture: The generator uses fully connected layers, self-attention-based relation modules, residual blocks, and cascaded relation blocks to refine graphic elements.Each element is represented by semantic and geometric parameters before refinement.
- Mobile app layout design: Mobile-app examples show LayoutGAN samples with corresponding real screenshots and retrieved real layouts as references.The mobile-app results include wireframe and mask layouts for visualization.