Source-linked AI summary
Neural Design Network: Graphic Layout Generation with Constraints
Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, Weilong Yang
TL;DR
Graphic layout generation must coordinate component relationships and user-specified constraints, a setting complicated by iterative design requirements and incomplete constraints. NDN completes the relation graph, predicts component bounding boxes, and refines the resulting layout. Experiments report visually appealing, constraint-following layouts, with real-design construction and layout recommendation applications.
Problem
Graphic layout generation must place components according to their attributes, inter-component relationships, and user-specified constraints.
Method
NDN represents components and constraints as a graph, completes missing relations, predicts bounding boxes iteratively, and refines the layout.
Results
Quantitative and qualitative experiments demonstrate layouts that are visually appealing and follow the specified constraints.
Takeaways & Limitations
Generated layouts support constructing real designs and a practical layout recommendation application.
Takeaways & Limitations
Performance can degrade with limited training data or component combinations that differ substantially from the training distribution.
Abstract
from arXiv · showhide
Graphic design is essential for visual communication with layouts being fundamental to composing attractive designs. Layout generation differs from pixel-level image synthesis and is unique in terms of the requirement of mutual relations among the desired components. We propose a method for design layout generation that can satisfy user-specified constraints. The proposed neural design network (NDN) consists of three modules. The first module predicts a graph with complete relations from a graph with user-specified relations. The second module generates a layout from the predicted graph. Finally, the third module fine-tunes the predicted layout. Quantitative and qualitative experiments demonstrate that the generated layouts are visually similar to real design layouts. We also construct real designs based on predicted layouts for a better understanding of the visual quality. Finally, we demonstrate a practical application on layout recommendation.
1 Introduction
Graphic layout generation must place and size interdependent components under user constraints while preserving visual quality. NDN addresses this with relation completion, iterative box prediction, and refinement, evaluated across datasets and practical design applications.
- Graphic design layout generation is challenging because component placement and sizing govern attention, information order, hierarchy, and narrative.
- NDN targets layouts that satisfy user-specified component attributes and constraints while remaining visually appealing.
- NDN represents components and user constraints as a graph, predicts complete relationships from partial inputs, generates bounding boxes iteratively, and refines alignment and visual quality.
- The method is evaluated qualitatively and quantitatively on RICO, Magazine, and a collected image banner advertisement dataset.
- Experiments assess realism, alignment, and visual quality, while generated layouts are also used to construct real designs and demonstrate layout recommendation.
2 Related Work
Graphic design layouts require stricter relational and aesthetic control than natural-scene layouts. Existing design methods use templates, exemplars, heuristics, saliency, attention, or generative models, but prior approaches do not fully handle specified components and explicit relationships.
- Natural-scene layout methods commonly serve as intermediate representations before image generation, whereas graphic layouts demand precise relationships, alignment, and aesthetics.
- Template-, exemplar-, and heuristic-based design methods require professional knowledge and are limited in capturing complex design distributions.
- LayoutGAN generates graphic layouts from noise but does not handle layouts conditioned on components with specified attributes.
- The Layout Generative Network conditions on component attributes, but conventional GAN frameworks cannot explicitly model component relationships and user-specified constraints.
- NDN models design layouts as graphs and applies graph neural networks to capture dependencies among components.
3 Graphic Layout Generation
NDN represents design components and constraints as graphs, completes missing relations, generates bounding boxes, and refines them for alignment and visual quality. Its modules support fixed-size components and iterative layout prediction from complete relation graphs.
- Problem Overview: NDN models components as graph nodes and their location and size relationships as directed edges, including unknown relations for unspecified constraints.The canvas is fixed in location and size, while other components are placed on it.
- Relation Prediction: The relation prediction module translates a partial graph into a complete graph by inferring unknown relations while preserving existing constraints.It uses conditional generation with a latent code and trains relation reconstruction with cross-entropy plus a KL loss.
- Layout Generation: The layout generation module predicts bounding boxes for all components from a complete relation graph using an iterative conditional VAE.Each prediction is conditioned on graph features and the current canvas containing boxes generated in previous iterations.
- Layout Generation: An augmented layout module reconstructs component widths and heights to support user-specified fixed-size shapes such as logos.The additional encoder encourages latent vectors to decode to bounding boxes with desired dimensions.
- Layout Refinement: The refinement module fine-tunes boxes because fixing earlier predictions can leave the overall layout suboptimal.Training simulates misalignment with uniformly sampled offsets before applying a graph convolutional network for finetuning.
4 Experiments and Analysis
Experiments evaluate NDN across three datasets and multiple constraint settings, comparing realism, alignment, prediction accuracy, and constraint consistency. Results support strong performance, flexible partial-constraint generation, practical recommendation, and identifiable failure cases.
- Experimental Setup: Evaluation uses Magazine, RICO, and image banner advertisement datasets, comparing NDN with related methods across no-constraint, all-constraint, and leave-one-out settings.The evaluation measures realism and diversity with FID, alignment among components, and leave-one-out prediction accuracy.
- Quantitative Evaluation: NDN performs favorably against other methods in both no-constraint and all-constraint settings, while also achieving statistically significant better leave-one-out accuracy.Leave-one-out accuracy is measured by L1 error between predicted and ground-truth bounding boxes, with statistical significance of at least 95%.
- Partial Constraints: Prior size and location constraints improve visual quality on the challenging RICO dataset, while the refinement module further improves FID and alignment.The partial-constraint evaluation varies the percentage of provided constraints, and the ablation identifies an additional benefit from refinement.
- Qualitative Evaluation: Generated layouts reasonably conform to user constraints and improve alignment and overlap handling relative to comparison methods in qualitative evaluations.With no constraints, NDN still produces visually good layouts; with all constraints, it better handles alignment and overlapping issues than sg2im in the reported comparison.
- Analysis: Random component ordering is used because it supports flexible user control and better matches leave-one-out prediction and layout-recommendation scenarios.The experiments find that component order affects generation results, but random order better aligns training and testing for these use cases.
- Applications and Limitations: NDN supports layout recommendation by completing graphs with unknown relations, but failures arise from undersampled latent spaces and component sets unlike the training data.The recommendation application predicts locations for specific components in partially specified designs, while the reported failure cases include unusual combinations such as three buttons and two logos.
5 Conclusion and Future Work
The paper presents neural design network as a three-module framework for generating visually appealing layouts that follow user-specified constraints. It identifies content-aware design and finer-grained layout generation beyond bounding boxes as future directions.
- NDN generates graphic layouts that follow user-specified constraints through relation prediction, layout generation, and refinement modules.The framework is designed to handle constrained design layout generation.
- Quantitative and qualitative experiments demonstrate the efficacy of the proposed model.
- The paper constructs real designs from generated layouts and demonstrates a layout recommendation application.
- Future work may incorporate content attributes such as color, font, and semantic labels, and generate layouts more finely than bounding boxes.The authors describe graphic design as a complicated process involving these content attributes.