Learning Interaction between Image and Layout Priors for Joint Image-Layout Generation in Design Templates

Shirong Yang; Bo Yang; Ying Cao

Source-linked AI summary

Learning Interaction between Image and Layout Priors for Joint Image-Layout Generation in Design Templates

Shirong Yang, Bo Yang, Ying Cao

arXiv:2609.11519v1Sep 10, 2026cs.CVcs.AIcs.GR

TL;DR

Graphic design template generation requires harmonious background images and foreground layouts, but sequential methods do not fully capture their dependency. InterIL jointly generates both modalities through communicating pretrained diffusion priors, achieving stronger image, layout, and harmony results while enabling preference control without retraining.

Problem
Graphic design templates must combine a background image and foreground layout harmoniously, while prior sequential methods inadequately capture their dependency.
Method
InterIL connects pretrained image and layout diffusion backbones through a learnable communication module for joint generation, freezing the backbones and optimizing only the interaction module.
Results
InterIL produces superior image quality, layout quality, and image-layout harmony compared with existing methods, with outputs closer to real samples.
Takeaways & Limitations
Training-free inference guidance lets users impose preferred design patterns without retraining, including reduced occlusion and improved text readability.
Takeaways & Limitations
The communication module is intentionally simple, while more sophisticated architectural designs are left for future work.

Abstract

from arXiv · show

In this paper, we address the problem of graphic design template creation, which generates a background image and a layout of foreground elements over the background to form a harmonious composition from an input text. Prior work on graphic design generation mostly adopts a sequential paradigm, where design elements are generated sequentially. We argue that such a sequential scheme falls short of faithfully capturing the dependency between the background and layout (and thus the joint image-layout distribution), which limits the quality of generated design templates. To overcome this limitation, we propose a model, InterIL, which jointly generates the two modalities, background image and layout, in a single generative process. The novel design of our joint model connects the backbones of pretrained image and layout diffusion models with a learnable communication module to explicitly model bidirectional image-layout interaction. During training, the image and layout backbones are frozen to maintain and leverage the vast pretrained single-modality prior knowledge, while only the communication module is updated, so that the model can focus on learning image-layout interaction and thereby better capture the joint image-layout distribution for improved composition harmony. Our model has no design-specific inductive bias, which allows it to better preserve the original characteristics of realistic designs. We further introduce a test-time guidance strategy to enable users to impose their specific preferences on generated results. Our experiments show that, compared with prior approaches, our model can generate significantly better results in terms of image, layout and image-layout harmonization, producing outputs closer to real samples. We also demonstrate the flexibility of our model in enforcing user preferences at inference without retraining.

1 INTRODUCTION

The paper targets automated graphic design template creation, where a background image and foreground layout must form a harmonious composition from text. InterIL jointly generates both modalities while learning their bidirectional interaction, and supports preference control at inference.

1 INTRODUCTION: Design templates provide concrete thematic and structural starting points for designers but require substantial manual effort and expertise to create.The targeted template contains a background image and a partial layout of foreground elements.
1 INTRODUCTION: Bidirectional background-layout dependency makes sequential generation inadequate for faithfully modeling the joint design distribution.Earlier approaches generate backgrounds and layouts in ordered stages, although each modality should account for the other.
1 INTRODUCTION: InterIL connects pretrained image and layout diffusion backbones with a learnable communication module while freezing the backbones during training.This preserves single-modality prior knowledge while focusing optimization on image-layout interaction.
1 INTRODUCTION: Experiments report dramatically improved visual quality and text adherence on Web-design, with more harmonious templates than alternative methods.The evaluation uses TemplateFID and TemplateCLIP, and Figure 1 presents generated examples.
1 INTRODUCTION: InterIL jointly synthesizes background images and layouts in one diffusion process, replacing prior sequential generation schemes.The model learns image-layout interaction rather than generating design elements in separate stages.
1 INTRODUCTION: Inference-time guidance accommodates user preferences without retraining, including reducing foreground occlusion and improving text readability.The guidance framework imposes preference-based constraints on generated results at inference.

2 RELATED WORK

Related work spans general graphic layout generation and text-driven graphic design generation. The paper positions its task as a specialized template-generation problem and contrasts InterIL with Desigen’s sequential two-stage approach.

2 RELATED WORK: Graphic layout generation has used GANs, VAEs, Transformers, diffusion models, and flow-based models to create spatial arrangements of design elements.Recent work also explores large language models for layout generation.
2 RELATED WORK: Text-driven graphic design methods include direct fine-tuning of pretrained text-to-image diffusion models to produce aesthetic designs.The related-work passage places these methods within broader automated graphic design generation.
2 RELATED WORK: Design template generation specializes graphic design generation by modeling a background image and element layout rather than complete designs.The paper focuses on a subset of design attributes for template creation.
2 RELATED WORK: Desigen generates backgrounds and layouts sequentially with separate models, conditioning one stage on the other’s final output.This contrasts with InterIL’s joint image-layout generation paradigm.

3 METHOD

InterIL jointly generates background images and layouts by connecting pretrained image and layout diffusion priors with a learnable module that models bidirectional interaction. The method freezes both backbones, schedules communication during denoising, and adds inference-time guidance for user preferences.

3.2 Joint Model: The model learns the conditional joint distribution of background images and layouts to produce coherent design templates from text.Each template contains a background image and foreground-element layout conditioned on a textual description.
3.1 Domain-specific Priors: The image prior uses adapted Stable Diffusion, while the layout prior is trained as a latent diffusion model over layout representations.The image backbone is fine-tuned on design backgrounds, and the layout backbone is trained from scratch on layout data.
3.2 Joint Model: InterIL jointly denoises image and layout latents using fixed pretrained backbones connected by a learnable communication module.The model combines an image prior and a layout prior in one diffusion process while preserving their pretrained knowledge.
3.3 Communication Module: The communication module is trained while the image and layout backbones remain fixed, focusing optimization on image-layout interaction.The paper identifies the simple communication design as effective but leaves more sophisticated architectures for future work.
3.3 Communication Module: Cross-attention augments intermediate image and layout representations with information from the other backbone, enabling bidirectional interaction.The augmented representations are passed into subsequent backbone blocks, while η controls whether communication is active.
3.4 Preference-based Guidance: During inference, communication is enabled for the first 30% of denoising and then disabled, while guidance adjusts layout noise predictions to impose user preferences.The guidance combines differentiable preference objectives, including minimizing foreground occlusion of salient background regions.

4 EXPERIMENTS

InterIL is evaluated against sequential design-generation baselines using domain-specific, holistic, human, efficiency, and ablation studies. Results consistently favor joint image-layout generation and bidirectional communication, while test-time guidance enables targeted preference control with trade-offs.

Evaluation Metrics: Retrieved templates were judged more similar than random templates 83.4% of the time in a 104-participant user study, validating TemplateAE embeddings across expertise groups.The study included 58 design experts and 46 non-experts and used Web-design samples for retrieval.
Comparison to Desigen: InterIL outperforms Desigen across all three image metrics, leads on 4 of 5 layout metrics, and achieves the best TemplateFID and TemplateCLIP.The results support high-fidelity backgrounds, effective layouts, stronger image-layout harmony, and text adherence.
Comparison to OpenCOLE: InterIL outperforms OpenCOLE on 9 of 10 metrics and achieves the best GPT-5 scores across image quality, layout quality, image-layout harmony, and text-design relevance.Human evaluators also strongly prefer InterIL over both baselines on all four aspects, especially image-layout harmony.
Computational Efficiency: The single-stage base model is the most compute-efficient, while Desigen’s three refinement iterations increase inference time from 7.47 s to 29.89 s.Preference-guided InterIL variants add modest cost and remain considerably faster than OpenCOLE and iterative Desigen.
Ablations: The communication module substantially improves TemplateFID and composition harmony, while bidirectional interaction outperforms unidirectional image-to-layout or layout-to-image variants.Removing communication causes independent denoising and worsens the reported harmony-related results.

5 CONCLUSION

InterIL jointly generates background images and layouts while learning their bidirectional interaction, and adds preference-based guidance for controllable generation. The model produces visually appealing images, high-quality layouts, and harmonious compositions.

5 CONCLUSION: InterIL uses a specialized communication module to jointly generate images and layouts in one latent-diffusion process, contrasting with prior sequential schemes.The module explicitly learns image-layout interaction while connecting the two modality priors.
5 CONCLUSION: Preference-based guidance lets users shift generated results toward preferred design patterns at inference time without retraining.
5 CONCLUSION: The model generates visually appealing background images, high-quality layouts, and harmonious image-layout compositions, outperforming existing methods.

A.1.1 Training Stage Attention Map Visualization

Training-stage cross-attention becomes increasingly structured in both directions: layout tokens localize to salient image regions, while salient image patches influence more layout elements.

A.1.1 Training Stage Attention Map Visualization: Layout-to-image attention sharpens during training, with layout tokens increasingly focusing on salient objects and boundaries such as text regions and buttons.
A.1.1 Training Stage Attention Map Visualization: Image-to-layout attention is strongest for salient regions such as human faces, whereas less salient background patches affect fewer layout tokens.

A.1.2 Inference Stage Attention Map Visualization

During inference, layout tokens attend to salient image regions early, then sharpen and stabilize as denoising proceeds, reflecting dynamic cross-modal grounding.

A.1.2 Inference Stage Attention Map Visualization: Layout tokens focus on salient objects and prominent anchors even at early denoising timestep t = 701, with attention sharpening and stabilizing later.

A.2.1 GPT-5 Evaluation

GPT-5 evaluates generated templates across four design-quality dimensions using scores from 1 to 10. InterIL consistently outperforms the compared baselines across these aspects.

A.2.1 GPT-5 Evaluation: GPT-5 assesses image fidelity and appeal, layout structure and usability, image-layout integration, and semantic relevance to the input text.
A.2.1 GPT-5 Evaluation: InterIL consistently outperforms Desigen and OpenCOLE across image quality, layout quality, image-layout harmony, and text-design relevance.The evaluation uses four complementary aspects, each scored from 1 to 10, with higher values indicating better performance.
A.2.1 GPT-5 Evaluation: InterIL scores 6.94 in Image Quality, 7.76 in Layout Quality, and 8.07 in Image-Layout Harmony, exceeding both named baselines on each reported aspect.

A.3 Blind Human Preference Study on Generated Templates

In a blind forced-choice study, participants preferred InterIL over Desigen and OpenCOLE across image quality, layout quality, image–layout harmony, and text–design relevance.

A.3 Blind Human Preference Study on Generated Templates: InterIL was preferred in 68% of image-quality comparisons, 78% of layout-quality comparisons, 83% of image–layout-harmony comparisons, and 71% of text–design-relevance comparisons.The study included 32 participants and 40 text prompts, with method identities hidden and display order randomized.
A.3 Blind Human Preference Study on Generated Templates: These human preferences were consistent with GPT-5 evaluations and the TemplateFID ranking, supporting the perceptual quality and coherence of InterIL templates.

A.4 Preference-Specific Guidance

Preference-specific guidance lets users steer layout placement or readability at inference time, but stronger steering can trade natural joint-distribution fidelity for the requested preference.

A.4.2 Ablation on OAG Scale: OAG progressively moves layout elements away from salient image regions during denoising, illustrating how inference-time guidance changes composition.The trajectory is shown with guidance scale s = 3, where initially overlapping elements are repositioned into less intrusive areas.
A.4.2 Ablation on OAG Scale: As OAG strength increases, occlusion falls, but excessive guidance worsens LayoutFID, alignment, and template-level realism, with TemplateFID rising from 86.63 to 114.34.TemplateCLIP also drops from 3.25 to 3.16, indicating weaker overall design quality under overly strong preference steering.
A.4.2 Ablation on OAG Scale: Moderate OAG scales, particularly s = 2 and s = 3, balance lower occlusion with competitive layout metrics and template-level realism.The paper adopts s = 3 by default; larger scales can support stricter occlusion avoidance but may reduce layout fidelity and visual realism.
A.4.3 Readability-Aware Guidance (RAG): RAG replaces OAG’s saliency objective with a clutter-based readability objective for placing text on smoother, lower-texture regions.It uses the same inference-time steering mechanism but targets a different user preference.
A.4.3 Readability-Aware Guidance (RAG): RAG can improve readability-oriented preference metrics while reducing holistic realism and fidelity to the joint data distribution, so it is an optional control rather than a replacement for the base model.
A.4 Preference-Specific Guidance: The joint framework generates templates in 5.01 seconds on average, providing a 1.5×–6× speed-up over Desigen variants and outperforming OpenCOLE at 25.8 seconds.Guided variants remain competitive at 8.69, 8.47, and 10.14 seconds, respectively.