Source-linked AI summary
OpenCOLE: Towards Reproducible Automatic Graphic Design Generation
Naoto Inoue, Kento Masui, Wataru Shimoda, Kota Yamaguchi
TL;DR
OpenCOLE addresses the reproducibility barriers of complex graphic-design systems that rely on proprietary datasets and code. It modifies COLE into an open pipeline trained with publicly available resources, and its GPT4V evaluation performs almost on par with COLE.
Problem
COLE’s proprietary datasets, code, cascaded process, and graphical-renderer dependence create significant barriers to reproduction.
Source: §1 · p. 1Method
OpenCOLE modifies COLE into an open three-stage pipeline using publicly available datasets and models, including Crello [27] and synthesized training data.
Source: §1 · p. 2, §2 · p. 2, §2.1 · p. 2Results
Takeaways & Limitations
OpenCOLE provides an open-source implementation and training results intended to support more inclusive development of automatic graphic-design generation.
Source: §1 · p. 2GPT4V assessment is a black-box dependency, and some generated images without textual content receive high rankings, motivating evaluation of both intentions and outputs.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Automatic generation of graphic designs has recently received considerable attention. However, the state-of-the-art approaches are complex and rely on proprietary datasets, which creates reproducibility barriers. In this paper, we propose an open framework for automatic graphic design called OpenCOLE, where we build a modified version of the pioneering COLE and train our model exclusively on publicly available datasets. Based on GPT4V evaluations, our model shows promising performance comparable to the original COLE. We release the pipeline and training results to encourage open development.
1. Introduction
OpenCOLE addresses COLE’s reproducibility barriers by providing an open implementation trained with publicly available data and models. It retains COLE’s staged design-generation approach while achieving comparable GPT4V-based performance.
- COLE generates editable, legible-oriented designs by separately producing background, object, and typographic layers before rendering them.Its decomposition includes typography such as font type and line spacing.Source: §1 · p. 1
- COLE’s proprietary datasets, code, cascaded process, and graphical-renderer dependence create substantial reproduction challenges.Source: §1 · p. 1
- OpenCOLE is an open-source COLE implementation using publicly available datasets and models to support reproducible graphic-design generation.The framework uses Crello [27] and synthesizes training data for each stage.Source: §1 · p. 2
- OpenCOLE matches COLE on five graphic-design aspects under GPT4V evaluation on the DESIGNERINTEN-TION benchmark [11].It performs better than DeepFloydIF but remains behind SDXL1.0 [21] and DALL-E3.Source: §1 · p. 2
- The authors’ contributions are an open automatic graphic-design pipeline and empirical results suggesting performance almost on par with COLE.Source: §1 · p. 2
2. Method
OpenCOLE converts user intentions into composed graphic designs through a modified three-stage architecture: planning, image generation, and typography generation. Its components use in-context prompting, single-stage image synthesis, and multimodal typographic prediction before rendering.
2. Method
- OpenCOLE translates a user intention into a design plan, generates an image and typography, then renders both into a final design.Source: §2 · p. 2, Figure 2 · p. 3
2.1. Design Plan Generation
- The design plan is a JSON schematic containing description, keywords, background and object captions, plus headings and subheadings.These fields guide visual generation and explicitly specify text for typography generation.Source: §2.1 · p. 2
- A pretrained LLM with five in-context user-intention/design-plan examples replaces COLE’s fine-tuned Llama [26] for JSON generation.Source: §2.1 · p. 2
2.2. Image Generation
- The image module fine-tunes a text-to-image generator on rendered text-free graphic documents, using concatenated background and object captions as input.Source: §2.2 · p. 2
- OpenCOLE uses single-stage image generation instead of COLE’s two-stage background-then-object process because reliable input-target pairs for both stages were difficult to obtain.Source: §2.2 · p. 2
2.3. Typography Generation
- The typography module fine-tunes multimodal models on the design plan and generated image to predict per-text font, size, color, spacing, and related attributes in hierarchical JSON.Source: §2.3 · pp. 2–3
3. Experiments
OpenCOLE is evaluated using publicly based implementation choices, the DESIGNER-INTENTION benchmark, and GPT4V assessment, with quantitative and qualitative comparisons to COLE and text-to-image baselines.
3.1. Implementation Details
- The implementation uses Crello templates and synthesized training data, with GPT3.5 and LLaVA models supporting design-plan and typography generation.The design-plan pipeline uses divide-and-conquer extraction for descriptions, keywords, captions, and headings, while typography merges headings and subheadings.Source: §3.1 · p. 3
- OpenCOLE fine-tunes SDXL1.0 for 10,000 iterations with a constant learning rate of 8 × 10−7 and generates 1024 × 1024 images during sampling.Training uses 1,000 warmup steps, and images with less than 0.5M pixels in area are filtered to avoid upscaling artifacts.Source: §3.1 · p. 3
3.2. Benchmark, Metrics, and Baselines
- The benchmark contains 200 intention prompts across six graphic-design categories, evaluated from 1 to 10 by GPT4V on five aspects.The aspects include design and layout, content relevance, typography and color, graphics and images, and innovation.Source: §3.2 · p. 3, Table 1 · p. 4
3.3. Quantitative Evaluation
- OpenCOLE performs competitively with COLE but lags behind SDXL1.0 and DALL-E3 across the evaluated design aspects.The baselines may produce illegible or non-editable text, unlike OpenCOLE's editable text layers.Source: §3.3 · p. 4, Table 1 · p. 4
3.4. Qualitative Evaluation
- OpenCOLE generates plausible designs, but its texts do not always accurately reflect user intentions and are often only partially related.Generated images tend to leave empty spaces for text placement, which the typography model appears to recognize when positioning text.Source: §3.4 · p. 4
- OpenCOLE's low-legibility failures involve overly long unbroken sentences, small text-placement spaces, weak color contrast, and very thin fonts.These factors are identified as contributors to the observed failure cases.Source: §3.4 · p. 4
- Compared with COLE, OpenCOLE produces more informative images with complex texts, whereas COLE produces simpler designs with more distinct text.The paper attributes these differing tendencies to dataset variations and modifications to certain modules.Source: §3.4 · p. 4
4. Discussion
The discussion highlights failure cases and comparison patterns while questioning whether GPT4V alone adequately evaluates graphic-design quality.
4. Discussion
- OpenCOLE's failure cases include low-legibility text, reinforcing the need to interpret its qualitative outputs alongside its reported comparisons.The cited failure cases are associated with long sentences, limited placement space, weak contrast, and thin fonts.Source: Figure 3 · p. 4, §4 · p. 4
- OpenCOLE and COLE both generate plausible designs but differ in complexity and text distinctness, reflecting dataset and module differences.OpenCOLE tends toward informative images with complex texts, while COLE tends toward simpler designs with distinct texts.Source: Figure 4 · p. 4
- GPT4V may highly rank generated images without textual content, especially among GPT4-augmented text-to-image baselines.The authors argue that evaluation should consider both the input intention and the generated output image.Source: §4 · p. 4