PaperlayerView exact revision on arXiv

Source-linked AI summary

COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design

Peidong Jia, Chenxuan Li, Yuhui Yuan, Zeyu Liu, Yichao Shen, Bohan Chen, Xingru Chen, Yinglin Zheng, Dong Chen, Ji Li, Xiaodong Xie, Shanghang Zhang, Baining Guo

arXiv:2311.16974v2cs.CV

TL;DR

Graphic-design generation must handle planning, reasoning, typography, and editable layered composition, but existing systems can require expert prompts and produce uneditable or text-inaccurate results. COLE addresses this with hierarchical specialized models that transform user intentions into multi-layered editable designs and evaluates them on DESIGNINTENTION. The paper reports stronger text fidelity and message conveyance than DALL·E3 and competitive results against DALL·E3 and CanvaGPT.

  • Problem

    Graphic-design generation needs accurate typography, minimal user effort, and flexible editing beyond the capabilities and workflows of existing systems.

    Source: §1 · p. 2

  • Method

    COLE hierarchically coordinates specialized LLM, LMM, and diffusion models for design planning, layer-wise generation, typography prediction, and quality assurance.

    Source: §1 · p. 3

  • Results

    COLE achieves 69.6% and 70.5% win rates over DALL·E3 in text fidelity and message conveyance, respectively, while outperforming others in GPT-4V assessments of layout, image, and typography quality.

    Source: §1 · p. 3

  • Takeaways & Limitations

    COLE demonstrates a hierarchical route to generating multi-layered, editable graphic designs from simple user intentions.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Graphic design, which has been evolving since the 15th century, plays a crucial role in advertising. The creation of high-quality designs demands design-oriented planning, reasoning, and layer-wise generation. Unlike the recent CanvaGPT, which integrates GPT-4 with existing design templates to build a custom GPT, this paper introduces the COLE system - a hierarchical generation framework designed to comprehensively address these challenges. This COLE system can transform a vague intention prompt into a high-quality multi-layered graphic design, while also supporting flexible editing based on user input. Examples of such input might include directives like ``design a poster for Hisaishi's concert.'' The key insight is to dissect the complex task of text-to-design generation into a hierarchy of simpler sub-tasks, each addressed by specialized models working collaboratively. The results from these models are then consolidated to produce a cohesive final output. Our hierarchical task decomposition can streamline the complex process and significantly enhance generation reliability. Our COLE system comprises multiple fine-tuned Large Language Models (LLMs), Large Multimodal Models (LMMs), and Diffusion Models (DMs), each specifically tailored for design-aware layer-wise captioning, layout planning, reasoning, and the task of generating images and text. Furthermore, we construct the DESIGNINTENTION benchmark to demonstrate the superiority of our COLE system over existing methods in generating high-quality graphic designs from user intent. Last, we present a Canva-like multi-layered image editing tool to support flexible editing of the generated multi-layered graphic design images. We perceive our COLE system as an important step towards addressing more complex and multi-layered graphic design generation tasks in the future.

1. Introduction

COLE addresses graphic-design generation’s need for accurate typography, low-effort prompting, and editable outputs by hierarchically coordinating specialized models. It is evaluated against existing systems and reports stronger performance on several user-study and quality dimensions.

Motivation

  • Graphic-design generation requires creativity, visual communication, typography, composition, and imagery to structure information and convey meaning.Source: §1 · p. 2

Challenges

  • Existing systems can generate visually striking designs but may mis-render text, require complex editing operations, and demand detailed expert prompts.The cited limitations include missing or extra characters and workflows involving segmentation, erasing, and inpainting.Source: §1 · p. 2

Approach

  • COLE transforms simple user intentions into multi-layered, editable graphic designs through hierarchical planning, visual reasoning, layer generation, typography prediction, and quality assurance.A Design-LLM produces structured layer-wise captions; background, object, and typography models then handle specialized generation tasks.Source: §1 · pp. 2–3

Evaluation

  • The DESIGNERINTENTION benchmark contains approximately 500 professional prompts and about 50 creative prompts for comparisons, ablations, and design analysis.Source: §1 · p. 3
  • COLE outperforms DALL·E3 in text fidelity and message conveyance, with win rates of 69.6% and 70.5%, respectively.Non-designers slightly prefer COLE visually, while designers favor DALL·E3; designers also prefer COLE over CanvaGPT.Source: Figure 3 · p. 2, §1 · p. 3
  • The study compares COLE with DALL·E3, CanvaGPT, and earlier text-to-image models, including GPT-4V assessments of layout, image, and typography quality.Table 1 is identified as the detailed comparison of the approaches.Source: §1 · p. 3, Table 1 · p. 3

2. Related Work

Prior work addresses parts of graphic-design generation or uses LLM-based agents, whereas COLE targets autonomous generation from basic user intent through hierarchical decomposition and dedicated model fine-tuning.

Autonomous Agent based on LLMs

  • COLE differs from LLM-based agent systems by decomposing graphic-design generation hierarchically and fine-tuning dedicated models on a professional design dataset.Source: §2 · p. 3

Design Image Generation

  • Most graphic-design generation studies focus on individual aspects, while COLE aims to execute the full task from clear user-intent descriptions.The paper characterizes this autonomous formulation as more challenging than those partial approaches.Source: §2 · p. 3

3. Our Approach

COLE decomposes graphic-design generation into specialized planning, diffusion, typography, reflection, and editing stages that transform vague intentions into editable layered designs. Its pipeline combines structured JSON planning, spatially coherent visual assets, typography reasoning, quality assessment, and layer-wise rendering.

3.1. COLE Framework Overview

  • COLE organizes intention-to-design generation into hierarchical planning, image-generation, typography, editing, reflection, and quality-assessment tasks.The framework trains specialized LLMs, LMMs, and diffusion models for these coordinated stages.Source: §3.1 · p. 4, Figure 4 · p. 5

3.2. Design LLM: Intention Recaption and Layout Planning

  • The Design LLM converts vague user intentions into detailed JSON containing global, background, object, and text-layer captions and attributes.It formulates intention-to-JSON generation as masked field prediction and uses nearly 100,000 paired examples for layout planning.Source: §3.2 · p. 4

3.3. Text-to-Background Diffusion Model: Visual Planning to Generate Canvas Placeholders

  • The background diffusion model generates a canvas placeholder that reserves space for objects and text, using long captions and offset noise to support design-oriented composition.It is fine-tuned on caption–background pairs and increases the maximum text length from 77 to 512 tokens.Source: §3.3 · pp. 4–5

3.4. Text-to-Object Diffusion Model: Visual Reasoning based on the Generated Background Image

  • The text-to-object diffusion model conditions object generation on the background and predicts object pixels, alpha masks, and a composed image to improve placement and blending.The model uses approximately 55,000 quadruples and predicts about seven channels spanning the object, mask, and composed image.Source: §3.4 · p. 6

3.5. Typography LMM: Layout Planning and Attribute Reasoning for Visual Text

  • The Typography LMM predicts image-level typography attributes from text and composed images, including font, hierarchy, spacing, and other layout properties.Its training data contains nearly 100,000 triplets, and the typography JSON includes 16 attributes with text supplied by the Design LLM.Source: §3.5 · pp. 6–7, Figure 7 · p. 7

3.6. Multi-Layered SVG Editor and Renderer: Support Layer-wise Flexible User Editing

  • COLE compiles generated layers into SVG and supports repositioning, resizing, and per-text-box changes to font size, color, type, and related attributes.The editor enables separate object and text edits as well as combined editing of the composed design.Source: §3.6 · p. 7, Figure 9 · p. 7

3.7. Reflect LMM & Quality LMM

  • Reflect-LMM revises JSON attributes when initial designs misalign with user expectations, while GPT-4V evaluates design quality across multiple visual aspects but is weaker on typography and color.The framework also compares COLE with other systems using DESIGNERINTENTION benchmark materials and GPT-4V scores.Source: §3.7 · p. 7, Figure 10 · p. 8, Figure 11 · p. 8

4. Experiment

The experiments evaluate COLE on the DESIGNERINTENTION benchmark, compare it with existing generators, and test its components for layout planning, typography, diversity, and composed image prediction.

4. Experiment

  • The evaluation includes subjective assessments by approximately 20 general users and 6 professional designers, alongside comparisons with DeepFloyd/IF, SDXL, DALL·E3, and CanvaGPT.GPT4V(ision) is used for the multifaceted comparison and subjective assessment setup.Source: §4 · p. 8, §4.2 · p. 9

4.1. DESIGNERINTENTION Benchmark

  • The DESIGNERINTENTION benchmark contains approximately 500 prompts spanning five categories plus around 50 additional creative intentions.The five categories are marketing, covers and headers, posts, events, and advertising.Source: §4.1 · pp. 8–9

4.2. Main Results

  • COLE achieves competitive quality against DALL·E3 and CanvaGPT, ranking second-best across design layout, typography color, and innovation.GPT-4V(ision) provides numerical quality assessments for the comparison.Source: §4.2 · p. 9

4.3. Ablation Experiments

  • Composed image prediction markedly improves layout planning in text-to-object diffusion, while alternative methods often produce objects filling the rectangular canvas.Figure 15 compares background images with object generation without and with composed image prediction.Source: §4.3 · p. 10, Figure 15 · p. 10
  • TypographyLMM outperforms FlexDM by +4.5 and +6.2 IoU points on single- and multiple-text-box placement tasks, respectively.The evaluation uses zero-shot performance on the Crello benchmark.Source: §4.3 · p. 10, Table 3 · p. 10
  • COLE generates varied graphic designs across diffusion seeds, while qualitative results identify limitations in typography arrangement, editable visual elements, and typography color diversity.These limitations are presented as future-work directions.Source: §4.2 · p. 9, Figure 14 · p. 9, §4.3 · p. 10

5. Conclusion

The conclusion presents COLE as a hierarchical framework that coordinates specialized sub-tasks to generate multi-layered, editable graphic designs. It reports comparisons with DALL·E3 and CanvaGPT while acknowledging limitations targeted for future work.

5. Conclusion

  • COLE decomposes graphic design generation into coordinated sub-tasks that support design-oriented planning, reasoning, and layer-wise generation.The framework produces distinct outputs from manageable sub-tasks.Source: §5 · p. 10
  • The paper demonstrates COLE’s advantages through comparisons with DALL·E3 and CanvaGPT and identifies key limitations for future research.The conclusion does not specify additional quantitative results.Source: §5 · p. 10

A Graphic Design Pipeline of Human Designer

The human-designer pipeline organizes graphic design around intention interpretation, visual composition, and iterative quality assurance. COLE uses this blueprint to coordinate specialized models and produce editable, multi-layered designs.

Graphic Design Pipeline of Human Designer

  • The pipeline divides design work into creative interpretation, visual composition and enhancement, and feedback-driven quality assurance.Figure 16 presents the roles and their interactions as the conceptual foundation for COLE.Source: A Graphic Design Pipeline of Human… · p. 13, Figure 16 · p. 13
  • Creative Design and Interpretation converts user intentions into text elements, tone, style, colors, themes, and captions.The role specifies titles, subtitles, headings, subheadings, and body text.Source: A Graphic Design Pipeline of Human… · p. 13
  • Visual Composition and Enhancement generates, assesses, and adjusts spatial organization using images, icons, textures, and typographic attributes.The module refines initial concepts with suitable visual assets.Source: A Graphic Design Pipeline of Human… · p. 13
  • Feedback and Quality Assurance iteratively incorporates user feedback to meet quality, professional, and legal criteria.The module maintains continuous interaction with the first two modules during refinement.Source: A Graphic Design Pipeline of Human… · p. 13
  • The proposed DesignerAgent aims to generate graphic designs from user intention descriptions by fine-tuning LMMs, LLMs, and diffusion models on curated design data.This ambition is contrasted with systems relying heavily on professional designer expertise.Source: A Graphic Design Pipeline of Human… · p. 13

B Detailed System Prompts for GPT-3.5 and GPT-4V(ision)

The paper introduces prompts that generate design intentions and evaluate graphic designs with GPT-based systems. These prompts operationalize image interpretation, quality assurance, and pairwise comparison.

Detailed System Prompts for GPT-3.5 and GPT-4V(ision)

  • The Intention Generation Prompt uses image keywords, visible text, titles, and related information to infer a user’s design purpose.The prompt requests a JSON-formatted intention and preserves necessary information such as websites, phone numbers, and prices.Source: B Detailed System Prompts for GPT-3.5… · p. 13
  • The Quality Assurance Prompt evaluates designs across layout, content relevance, graphics and imagery, creativity, and typography and color.It uses 1–10 scoring criteria and coordinate information relative to the image’s upper-left origin.Source: B Detailed System Prompts for GPT-3.5… · pp. 13–14
  • The Pair-wise Comparison Prompt compares generated designs with other methods from six graphic-design perspectives.The paper presents it as a separate evaluation prompt alongside intention generation and quality assurance.Source: B Detailed System Prompts for GPT-3.5… · p. 13

C The Detailed Prompt List of DESIGNERINTENTION

The appendix documents DESIGNERINTENTION prompts and the GPT-4V comparison criteria used to evaluate generated graphic designs. The prompts assess alignment, composition, aesthetics, text rendering, and artistic fit.

D More Details of Bin Partition Scheme within Typography-LMM

Typography-LMM discretizes continuous typography attributes into tokenized bins, with ranges and resolutions chosen for each attribute from font size to spacing. Partition boundaries are estimated from training-set statistics.

E More Ablation Experiments

The ablations examine Design-LLM, text-to-background generation, Typography-LMM with Reflect-LMM refinement, and human or GPT-4V evaluation. Across these studies, specialized components improve structured design generation and evaluation outcomes.

Design LLM

  • Design-LLM significantly outperforms GPT-3.5 and GPT-4 on semantic similarity scores against ground-truth design attributes.The comparison uses Sentence-BERT semantic textual similarity across the test dataset.Source: E More Ablation Experiments · p. 15, Table 4 · p. 15

Text-to-Background & Text-to-Object

  • Current diffusion models often fill the image with objects, leaving insufficient space for visual text, whereas COLE’s specialized backgrounds support compositional integration.Source: E More Ablation Experiments · p. 15
  • COLE’s background images are designed to accommodate object and typography layers, enabling more flexible composition.Source: E More Ablation Experiments · p. 15

Typography LMM & Reflect LMM

  • Reflect-LMM refinement significantly improves typography localization, particularly on AP50 and AP75, using predicted text boxes refined from rendered designs.Localization is evaluated with mIoU, AP25, AP50, and AP75 against ground-truth boxes.Source: E More Ablation Experiments · pp. 15–16

Comparison by Professional Designer and GPT-4V

F Hyperparameters

The appendix provides COLE’s training hyperparameters and supplementary qualitative analyses. These materials include background-generation comparisons, designer commentary, and GPT-4V comparison results.

Hyperparameters

  • COLE comprises a hierarchical series of five generative models, with hyperparameters organized into training, model, and data settings.Source: F Hyperparameters · p. 17

Qualitative Comparisons

  • The background-generation comparison selects DeepFloyd/IF-XL and SDXL-1.0 with SDXL-Refiner-1.0 as strong baselines and marks GPT-4-enhanced prompts with †.Source: Figure 20 · p. 18

Sample Analysis: Designer’s Perspective

  • A designer analysis praises a Blue Macaw advertisement’s color balance, central focus, and triangular composition but identifies ambiguity in the “75+ years” title text.Source: F Hyperparameters · p. 19
  • A contrasting example overinterprets the prompt by adding a monitor and browser, using a bright natural background instead of the requested dark one, and producing a more crowded advertisement.Source: F Hyperparameters · p. 19

Qualitative Comparisons

Related papers

Loading 2311.16974v2…