Source-linked AI summary
GlyphControl: Glyph Conditional Control for Visual Text Generation
Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang, Haisong Ding, Han Hu, Kai Chen
TL;DR
Visual text generation remains difficult because diffusion models often produce inaccurate or illegible text. GlyphControl adds rendered glyph conditioning to an off-the-shelf Stable Diffusion model, and with the LAION-Glyph benchmark it outperforms DeepFloyd IF across reported OCR, CLIP, and FID evaluations.
Problem
Diffusion-based text-to-image models often lack the ability to generate legible visual text because its structure is complex and fine-grained.
Method
GlyphControl adds glyph images through a ControlNet branch to augment an off-the-shelf text-to-image model, using LAION-Glyph for visual-text training.
Results
GlyphControl consistently outperforms recent models such as DeepFloyd IF in OCR accuracy, FID, and CLIP score.
Takeaways & Limitations
Glyph instructions support customized control over the content, locations, and sizes of generated visual text.
Takeaways & Limitations
The method cannot currently control font style or text color and uses black text in a single default font style.
Abstract
from arXiv · showhide
Recently, there has been an increasing interest in developing diffusion-based text-to-image generative models capable of generating coherent and well-formed visual text. In this paper, we propose a novel and efficient approach called GlyphControl to address this task. Unlike existing methods that rely on character-aware text encoders like ByT5 and require retraining of text-to-image models, our approach leverages additional glyph conditional information to enhance the performance of the off-the-shelf Stable-Diffusion model in generating accurate visual text. By incorporating glyph instructions, users can customize the content, location, and size of the generated text according to their specific requirements. To facilitate further research in visual text generation, we construct a training benchmark dataset called LAION-Glyph. We evaluate the effectiveness of our approach by measuring OCR-based metrics, CLIP score, and FID of the generated visual text. Our empirical evaluations demonstrate that GlyphControl outperforms the recent DeepFloyd IF approach in terms of OCR accuracy, CLIP score, and FID, highlighting the efficacy of our method.
1 Introduction
Diffusion models often struggle with legible visual text because text has complex, fine-grained structure. GlyphControl addresses this by adding explicit glyph-based spatial control to an off-the-shelf text-to-image model, with strong OCR gains over DeepFloyd IF.
- Motivation: Most diffusion-based text-to-image models produce high-quality images but struggle to render legible and readable visual text.The difficulty is attributed to visual text’s complex and fine-grained structure.
- Motivation: Textual prompts alone leave missing or merged glyphs, motivating an explicit glyph image as a spatial layout prior.The glyph image preserves the text’s content, position, and size while providing shape information.
- Method: GlyphControl augments an off-the-shelf text-to-image model with a ControlNet branch that exploits glyph shape information.The framework is presented as one of the paper’s two key innovations.
- Resources: LAION-Glyph contains approximately 10 M text-image pairs augmented with OCR detection results recording presented text information.The paper also introduces SimpleBench and CreativeBench for evaluation.
- Results: 15% and 13% OCR-accuracy gains over DeepFloyd IF are reported on SimpleBench and CreativeBench, respectively.The corresponding results are 48% versus 33% and 34% versus 21%.
- Capabilities: Glyph instructions let users control the content, locations, and sizes of generated visual text.This provides customized rendering through the model’s interface.
2 Related Work
Prior work improves text rendering through stronger text encoders or controllable generation, but glyph conditioning directly supplies spatial and shape information. GlyphControl combines this condition with a diffusion framework to support more customized visual text generation.
- Text-to-image diffusion models: Text-to-image diffusion models use progressively stronger architectures and text encoders, including latent diffusion, Stable Diffusion, and SDXL.These developments target high-quality image synthesis while retaining text-prompt conditioning.
- Controllable image generation: Controllable image-generation methods add conditions such as segmentation or depth maps to customize synthesized image content.Other approaches decompose generation into factors or support flexible and composable control.
- Visual text generation: Mainstream diffusion models often render illegible text, while prior font-focused methods do not extend to general image generation.Imagen and eDiff-I address prompt spelling through large-language-model text embeddings.
- GlyphControl framework: The framework combines a locked Stable Diffusion model with a trainable Glyph ControlNet that receives rendered glyph information.During training, glyphs are extracted and rendered into a whiteboard image used by the ControlNet branch.
- Visual text generation: GlyphControl uses glyph images as conditional maps, enabling control over text content, locations, and sizes.This differs from relying only on textual input or text embeddings.
3 Approach
GlyphControl augments Stable Diffusion with glyph-rendered condition maps through ControlNet, enabling precise visual-text generation and user control. The approach is trained and evaluated with the LAION-Glyph benchmark across multiple dataset scales.
- GlyphControl framework: The framework combines OCR detection, glyph rendering, VAE encoding and decoding, CLIP text encoding, and U-Net denoising.Users provide glyph instructions during inference, while the text prompt supplies semantic information.
- GlyphControl framework: GlyphControl feeds rendered glyph images into a ControlNet branch to add precise geometric control to Stable Diffusion.The glyph images preserve text content, position, and size while providing shape information for conditioning.
- Glyph instructions: Users can customize glyph content, line arrangement, position, font size, width-height ratio, and rotation.The framework supports single words, phrases, sentences, multiple rows, and multiple text boxes.
- LAION-Glyph benchmark: LAION-Glyph is constructed from LAION-2B-en by selecting images with abundant visual text using OCR-based filtering.The pipeline applies aesthetic filtering, OCR detection and recognition, and removes samples with unsuitable text-box configurations.
- Evaluation: GlyphControl reaches 30/19, 40/26, and 42/28 Acc(%)↑ on LAION-Glyph-100K, 1M, and 10M, respectively.The corresponding SimpleBench/CreativeBench values are reported as paired results in the table.
4 Experiment
GlyphControl is evaluated against diffusion and commercial image-generation systems using OCR, CLIP, FID, benchmark prompts, and qualitative comparisons. Results show stronger visual-text rendering, while performance depends on dataset scale, glyph size, and text density.
- Main Results: OCR accuracy is highest for GlyphControl on both benchmarks, outperforming Stable Diffusion, SDXL, and DeepFloyd IF with fewer training parameters.Stable Diffusion produces almost no legible text, while SDXL improves capitalization-insensitive accuracy but remains poor overall.
- Main Results: 48% OCR accuracy on SimpleBench, up from 37%, when GlyphControl is trained on a larger specialized visual-text dataset.The comparison indicates that larger datasets focused on visual text improve generation performance.
- Main Results: GlyphControl outperforms or matches general text-to-image models on CLIP score while maintaining prompt-image alignment.The evaluation measures whether generated images contain the specified prompt words while retaining image-text consistency.
- Main Results: GlyphControl achieves the lowest FID on LAION-Glyph, indicating realistic, high-quality visual-text images.FID is computed using 10K samples not used for training.
- Qualitative Comparison: Baseline systems produce missing, repeated, merged, misshapen, or absent glyphs, whereas glyph-conditioned control renders text more accurately.Qualitative comparisons include Stable Diffusion, SDXL, DeepFloyd IF, DALL·E 2, Midjourney, and Ideogram.
- Ablation Experiments: Small or large font sizes reduce OCR accuracy and CLIP score, while dense paragraphs preserve layout but remain difficult to render readably.Large-font degradation is associated with limited training examples, and paragraph experiments reveal failures on small text.
5 Discussion and Limitations
The discussion identifies two principal scope limitations: GlyphControl controls text position and size but not font style or color, and it struggles with abundant small text.
- Scope Limitations: GlyphControl lacks control over font style and text color, rendering black text in one default font style on whiteboard maps.The authors propose integrating artistic font rendering, style recognition, and color adapters in future work.
- Scope Limitations: GlyphControl has sub-optimal performance when generating abundant small text, partly because training filters remove samples containing more than five OCR boxes.Small dense text also challenges VAE-based pixel-level reconstruction and OCR recognition.
6 Conclusion
The paper concludes that GlyphControl combines glyph-shape conditioning with the LAION-Glyph benchmark to generate legible visual text and outperform recent models across key metrics.
- Conclusion: GlyphControl uses a glyph ControlNet to encode text-shape information from rendered glyph images.The method is paired with the large-scale LAION-Glyph benchmark.
- Conclusion: GlyphControl consistently outperforms recent models such as DeepFloyd IF on OCR accuracy, FID, and CLIP score.The conclusion identifies font style, color control, abundant small text, and caption quality as future challenges.
A More Visualized Examples
Additional examples show GlyphControl generating legible, well-formed visual text across diverse prompts and glyph conditions.
- Additional Examples: GlyphControl generates legible and well-formed visual text in diverse scenarios.These samples supplement the main visual examples.
- Additional Examples: Figure 7 presents 512 × 512 GlyphControl samples generated from different text prompts and glyph conditions.The figure provides additional examples of the method’s controllable visual-text generation.
B Comparisons across Benchmarks and Word Frequency
Performance varies across benchmarks and word-frequency buckets: SimpleBench yields higher OCR accuracy, while CreativeBench has slightly higher CLIP scores. High-frequency words are generally easier to render, whereas low-frequency words receive higher CLIP scores.
- Benchmark comparison: SimpleBench generally achieves higher OCR accuracy than CreativeBench, while CreativeBench produces slightly higher CLIP scores.The difference is attributed to CreativeBench's more diverse prompts and richer descriptions.
- Word frequency: High-frequency words are generally easier to render onto images than rarely used words.
- Word frequency: Low-frequency words consistently exhibit higher CLIP scores than high-frequency words.The paper attributes this pattern to CLIP potentially overlooking low-frequency words in prompts, causing overestimation.
- Qualitative comparison: GlyphControl shows relatively weak capability when generating abundant legible small text, despite preserving the glyph images' global arrangement.
C Failure Case Analysis
GlyphControl still exhibits several layout and legibility failures, especially with overlapping boxes, excessive yaw, and small text. TextCaps 5K ablations show higher reported metrics after U-Net decoder fine-tuning than with a frozen decoder.
- Failure cases: Rendering overlap occurs when text-box locations in glyph instructions overlap.
- Failure cases: Missing Glyphs, Wrong Glyphs, Duplicate Glyphs, Unexpected Text, and Illegible Text remain among GlyphControl's layout failures.
- Failure cases: Excessive Yaws and Small Text show poor performance, possibly because corresponding training samples are lacking.
- Decoder ablation: 48/39 Acc(%)↑, 1.21/1.73 LD ↓, and 33.8/36.3 CLIP Score↑ are reported with a frozen U-Net decoder on TextCaps 5K.
- Decoder ablation: 61/43 Acc(%)↑, 0.76/1.38 LD ↓, and 34.2/36.3 CLIP Score↑ are reported after U-Net decoder fine-tuning.
D More Ablation Studies
Additional ablations test GlyphControl's generalization on TextCaps 5K, a specialized OCR-related dataset of signs, books, and posters. Unlocking the U-Net decoder improves OCR accuracy, while dataset choice affects realism, creativity, and diversity.
- TextCaps 5K: TextCaps 5K contains images of signs, books, and posters extracted from the TextCaps v0.1 training set for OCR-related fine-tuning.
- Decoder ablation: Unlocking Stable Diffusion's frozen U-Net decoder significantly enhances OCR accuracy on visual text rendering.The improvement is attributed to better adaptation to the smaller TextCaps 5K dataset during training.
- Generated-image analysis: TextCaps 5K fine-tuning produces central visual-text regions resembling conventional signs rather than seamlessly merged backgrounds.