Source-linked AI summary
TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
TL;DR
Visual text rendering remains difficult for diffusion models, and prior explicit-guidance methods limit flexibility, layout prediction, or style diversity. TextDiffuser-2 uses language models for automated layout planning and line-level layout encoding, and experiments and user studies validate more rational layouts and greater text-style diversity. The paper also reports difficulty rendering complex languages such as Chinese because expanding the character table introduces few-shot or zeroshot scenarios.
Problem
Diffusion models remain challenged by accurate, visually appealing, and coherent visual-text rendering, while prior methods have limited flexibility, layout capability, and style diversity.
Method
TextDiffuser-2 fine-tunes one language model for prompt-based layout planning and uses another within diffusion to encode line-level text position and content.
Results
Experiments and studies with human participants and GPT-4V validate more reasonable layouts, accurate text generation, and enhanced style diversity.
Takeaways & Limitations
TextDiffuser-2 combines accurate text rendering with more diverse styles and supports flexible layout interaction, including regenerating, adding, or moving keywords.
Takeaways & Limitations
TextDiffuser-2 faces challenges rendering complex languages such as Chinese because expanding the renderable character table can lead to few-shot or zeroshot scenarios.
Abstract
from arXiv · showhide
The diffusion model has been proven a powerful generative model in recent years, yet remains a challenge in generating visual text. Several methods alleviated this issue by incorporating explicit text position and content as guidance on where and what text to render. However, these methods still suffer from several drawbacks, such as limited flexibility and automation, constrained capability of layout prediction, and restricted style diversity. In this paper, we present TextDiffuser-2, aiming to unleash the power of language models for text rendering. Firstly, we fine-tune a large language model for layout planning. The large language model is capable of automatically generating keywords for text rendering and also supports layout modification through chatting. Secondly, we utilize the language model within the diffusion model to encode the position and texts at the line level. Unlike previous methods that employed tight character-level guidance, this approach generates more diverse text images. We conduct extensive experiments and incorporate user studies involving human participants as well as GPT-4V, validating TextDiffuser-2's capacity to achieve a more rational text layout and generation with enhanced diversity. The code and model will be available at \url{https://aka.ms/textdiffuser-2}.
1. Introduction
Visual text remains difficult for diffusion models, while existing guidance-based methods trade off flexibility, layout capability, and style diversity. TextDiffuser-2 addresses these limitations with two language-model roles for automated layout planning and diverse text rendering.
- Diffusion models often generate unintended symbols or artifacts when rendering visual text, impairing image quality despite strong general image synthesis.
- Existing methods require manual glyph or keyword guidance, limiting the direct conversion of natural prompts into images.
- Single-line layout prediction and unattractive multi-line layouts constrain the applicability and visual quality of prior systems.
- Character-level guidance restricts style diversity, motivating a more flexible approach to text rendering.
- TextDiffuser-2 fine-tunes a language model for automated, chat-modifiable layout planning and uses line-level encoding to generate more diverse text images.
2. Related work
Prior work improves visual text rendering through language models or explicit position and content guidance, while TextDiffuser-2 combines language-model layout planning with line-level text encoding.
- Visual text rendering remains challenging, and improving it could increase designers’ efficiency in text-related creative tasks.
- Language-model text encoders improve spelling capabilities, whereas explicit guidance methods control the position and content of rendered text.
- GlyphControl uses multi-line glyph images and TextDiffuser uses character-level segmentation masks to guide accurate and coherent text rendering.
- TextDiffuser-2 differs from prior explicit-guidance methods by using one language model for layout planning and another for line-level text encoding.
- Layout-generation research includes document formatting, screen UI design, and image synthesis, with approaches ranging from coordinate regression to discrete language tokens.
3. Methodology
TextDiffuser-2 uses two language models in a staged architecture: one plans visual-text layouts from prompts, and the other encodes layouts for diffusion-based image generation.
- TextDiffuser-2 trains a layout-planning language model and a diffusion model in two stages.M1 uses cross-entropy loss first; M2 and the U-Net use denoising L2 loss second.
- M1 is fine-tuned on caption-OCR pairs to infer text and layouts from prompts or place user-specified keywords.Its input includes a prompt and optional keywords, while its output specifies text-line layouts.
- M1 can support interactive layout modification, including regenerating, adding, or moving keywords through conversation.
- M2 encodes the prompt and predicted layout inside the latent diffusion model, using line-level guidance rather than character-level segmentation masks.The line-level bounding box provides greater flexibility during generation.
- 3.2. Language model for layout encoding: Hybrid-granularity tokenization preserves BPE tokens for prompts while representing keywords with character and coordinate tokens.For example, words are decomposed into character tokens and positions into coordinate tokens.
4. Experiments
Experiments evaluate TextDiffuser-2’s layout planning, coordinate representation, tokenization, text rendering, diversity, and interactive capabilities. Results show strong layout quality and broad generation diversity, with character-level tokenization and two-corner box coordinates preferred for accuracy.
- Ablation studies: Fine-tuning M1 with 5k samples produces the strongest overall keyword-extraction performance across most metrics.The evaluation uses accuracy, precision, recall, F-measure, and IoU on MARIO-Eval.
- Quantitative results and user studies: TextDiffuser-2 achieves the best results under the majority of quantitative and user-study metrics.Table 3 combines benchmark results with human and GPT-4V evaluations.
- Ablation studies: 22.39% and 29.26% accuracy declines occur for center-point and LR coordinate settings versus LT-RB, motivating two-corner box representation.The selected LT-RB representation uses the top-left and bottom-right corners.
- Ablation studies: Character-level tokenization outperforms subword tokenization by 42.1% on accuracy.The authors attribute the subword deficit to insensitivity to individual token spelling.
- Experimental results: TextDiffuser-2 produces more aesthetically pleasing layouts, more diverse font styles, and greater variation in character positioning and text-line angles.Compared with TextDiffuser, it also supports more flexible line-level style control while retaining accurate text generation.
- Discussions: Interactive conversation lets M1 regenerate layouts, add or modify keywords, and manipulate box locations.The method also shows greater robustness to overlapping boxes, whereas comparison methods produce scrambled text in that setting.
5. Conclusion
TextDiffuser-2 uses language models for layout planning and layout encoding in text rendering. It generates more diverse images while maintaining text accuracy, but complex-language rendering remains challenging.
- TextDiffuser-2 uses two language models, one for layout planning and another for layout encoding.
- The method generates more diverse images while maintaining the accuracy of the rendered text.
- TextDiffuser-2 faces challenges with complex languages because expanding the character table can require few-shot or zeroshot rendering.
A. Choice of the maximum length L
The composed sequence length limits coverage and inference efficiency. Setting L=128 covers 94.0% of samples, while L=256 covers 99.2% but increases inference time.
- 94.0% of samples have composed sequences shorter than 128, and all evaluation samples fall below this threshold.
- Increasing the maximum length from 128 to 256 raises training coverage to 99.2% but increases single-sample inference time from 6 to 7 seconds.
- The length limit balances model capability against computational efficiency.
B. More visualizations of layout prediction
The layout planner organizes specified keywords into related lines and aesthetically pleasing layouts. It can also make logical substitutions and correct misspellings in predicted layouts.
- The layout planner places related specified keywords on the same line and produces aesthetically pleasing layouts.
- The planner corrects misspelled prompt terms, replacing “RRAINBOW” with “RAINBOW”.
- Single-point conditions produce diverse text sizes and angles but a significant portion of rendered text is inaccurate.
D. Generation with additional angle condition
The model supports angle-conditioned text generation by adding angle tokens to the layout representation. Generated samples align well with the specified angle instructions.
- Generated text aligns well with angle instructions after adding 181 angle tokens spanning -90° to 90°.
- The angle token is placed after the four coordinate tokens in the language-format layout.
- The angle-guidance visualizations use the prompt “A text image of hello” to compare generation under different angle conditions.
E. Details of compared methods and evaluation
The evaluation compares TextDiffuser-2 with diffusion, API-based, and explicit-guidance baselines under text-rendering settings. It uses established metrics and Microsoft Read API OCR evaluation.
- Compared methods: The compared methods span SD-XL, PixArt-α, Ideogram, DALLE-3, GlyphControl, and TextDiffuser, covering general text-to-image and explicit text-guidance baselines.The baselines vary in architecture, parameter count, output resolution, and conditioning format.
- Compared methods: GlyphControl conditions SD 2.1 with multi-line glyph images and cannot generate images directly from prompts.For comparison, it uses layouts produced by TextDiffuser-2 to create the glyph image.
- Compared methods: TextDiffuser is a two-stage, SD 1.5-based framework that relies on user-specified rendering keywords and produces 512×512 images.It contains 884M parameters in total.
- Evaluation: Evaluation uses the metrics from TextDiffuser and Microsoft Read API to assess OCR performance.
- Compared methods: Character-Aware Model and GlyphDraw are compared using samples from their papers because their source code, weights, and demos are unavailable.
F. Comparisons with samples in other papers
Comparisons with published samples show that TextDiffuser-2 improves rendering accuracy over Character-Aware Model and supports multiple text lines beyond GlyphDraw. Additional analyses examine overlapping-layout robustness and text-inpainting implementation.
- Comparisons with samples in other papers: TextDiffuser-2 renders more accurately than Character-Aware Model and supports multiple text lines, outperforming GlyphDraw in the reported comparison.The Character-Aware samples contain typos, including a missing “r” in “from” and incorrect spelling of “Chimpanzees”.
- User studies: User studies evaluate layout aesthetics, style diversity, text quality, text-image matching, and inpainting ability with human participants and GPT-4V.Questions use single-selection or multiple-selection formats depending on the evaluated aspect.
- Text inpainting: TextDiffuser-2 can be trained for text inpainting by adding a five-dimension feature to the U-Net input.The addition comprises a one-dimension mask and four non-inpainted-area features.
- Text inpainting: The inpainting modification adds 14,400 parameters to a 922M-parameter architecture and uses classifier-free guidance 7.5 with 50 sampling steps.Generation takes 6 seconds on one A100 GPU card.
- Overlapping layouts: The overlapping-layout analysis compares GlyphControl, TextDiffuser, and TextDiffuser-2 using glyph images, character-level masks, and text-conditioned bounding boxes.The corresponding visualizations assess robustness when layout boxes overlap.
J. Conformity of the positional guidance
TextDiffuser-2 is evaluated for adherence to positional guidance, robustness to overlapping layouts, backbone effects, and quadrilateral-box conditioning. The reported detection metrics indicate strong positional conformity.
- Conformity of positional guidance: Precision 0.9524, recall 0.9635, and F-measure 0.9579 show that generated text usually complies with provided positional constraints.The assessment uses a widely used text-detection tool.
- Overlapping layouts: TextDiffuser-2 demonstrates enhanced robustness under overlapping layouts compared with GlyphControl and TextDiffuser.The comparison uses bounding boxes with text versus glyph-image and character-level segmentation guidance.
- Backbone comparison: The SD 2.1 version renders more detail and small characters more accurately than the SD 1.5 version.The improvement is attributed to the stronger pretrained model and higher latent-space resolution.
- Quadrilateral guidance: Quadrilateral bounding boxes guide slanted text by changing the layout planner to output four corner coordinates per line and adapting the layout encoder.The sequence format is “textline x0, y0, x1, y1, x2, y2, x3, y3”.
- User studies: The user-study protocol covers layout aesthetics, style diversity, text quality, text-image matching, and inpainting ability, with GPT-4V used alongside human participants.