PaperlayerView exact revision on arXiv

Source-linked AI summary

TextDiffuser: Diffusion Models as Text Painters

Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei

arXiv:2305.10855v5cs.CV

TL;DR

Diffusion models still struggle to render accurate, coherent text, despite progress in image generation and the importance of text images. TextDiffuser predicts keyword layouts with a Transformer and uses them to condition latent diffusion, supported by the 10-million-pair MARIO-10M dataset and MARIO-Eval benchmark. Experiments and user studies report flexible, controllable text-image generation, including template conditioning and text inpainting.

  • Problem

    Existing diffusion models struggle with accurate, coherent text rendering, and no specialized large-scale dataset supports this task.

  • Method

    TextDiffuser uses a two-stage framework that predicts keyword layouts and character-level masks, then conditions latent diffusion on those masks and text prompts.

  • Results

    Experiments and user studies demonstrate TextDiffuser's superiority over existing methods for text rendering and its controllable text-image generation capabilities.

  • Takeaways & Limitations

    TextDiffuser supports text generation from prompts or template images and reconstructs incomplete text images through text inpainting.

  • Takeaways & Limitations

    The latent-space VAE limits reconstruction of small characters, and long-text generation has failure cases; part-image generation lacks comparable methods for evaluation.

Abstract

from arXiv · show

Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, we contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs with text recognition, detection, and character-level segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, we show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. The code, model, and dataset will be available at \url{https://aka.ms/textdiffuser}.

1 Introduction

TextDiffuser addresses the difficulty of generating accurate, coherent text in images, where existing diffusion models and conventional tools remain limited. It introduces a controllable two-stage framework and the MARIO-10M dataset to support text-image generation.

  • Existing diffusion models struggle to render visually pleasing, accurate, and coherent text on images.High-quality text images are important across posters, book covers, and memes, but often require professional design skills and substantial effort.
  • Traditional tools such as Photoshop can create unnatural artifacts when text interacts with complex backgrounds or lighting.
  • TextDiffuser uses a Layout Transformer to locate prompt keywords and produce character-level segmentation masks before diffusion-based image generation.The second stage conditions latent diffusion on the generated masks and text prompts, with an additional character-aware loss.
  • MARIO-10M provides 10 million high-quality image-text pairs with recognition, detection, and character-level segmentation annotations.

2 Related Work

Prior work improved text rendering through stronger text encoders and introduced small text-rendering benchmarks, while image inpainting addressed broader visual completion tasks. This paper expands evaluation and applies diffusion-based inpainting specifically to text rendering.

  • Text Rendering: Recent diffusion systems improve text rendering by using large T5 encoders or character-aware text encoders instead of relying only on CLIP.
  • Text Rendering: Existing text-rendering benchmarks contain relatively few prompts, limiting the breadth of evaluation.Examples include DrawBench with 21 text-rendering prompts, DrawTextCreative with 175, and ChineseDrawText with 218.
  • Text Rendering: MARIO-Eval combines prior prompts with additional data to establish a larger benchmark for comprehensive text-rendering comparisons.
  • Image Inpainting: TextDiffuser treats text inpainting as text-conditional image inpainting guided by character-level segmentation masks.Unlike prior inpainting work focused on natural backgrounds or objects, it reconstructs image regions containing rendered text.
  • Optical Character Recognition: OCR research supports text image recognition, detection, and related applications, providing context for the annotations used by this work.

3 Methodology

TextDiffuser separates keyword layout prediction from latent diffusion image synthesis. The layout stage predicts keyword boxes and character masks, while the image stage uses those masks and feature conditions to support whole-image generation, part-image generation, and controllable inference.

  • 3. Methodology: The methodology has Layout Generation and Image Generation stages.The first predicts keyword layouts; the second generates images conditioned on the resulting segmentation masks.
  • 3.1 Stage1: Layout Generation: A Transformer encoder-decoder predicts bounding boxes for K keywords from prompt, keyword, width, and positional embeddings.The decoder autoregressively produces B ∈ R^K×4, and Pillow renders the boxes into character-level segmentation masks.
  • 3.2 Stage2: Image Generation: The image stage encodes images into latent features, adds diffusion noise, and conditions denoising on segmentation and masking features.Whole-image generation masks all feature regions, whereas part-image generation masks the user-selected region while preserving the remaining feature.
  • 3.2 Stage2: Image Generation: A character-aware loss guides the diffusion model to focus on text regions alongside the denoising objective.A fixed pretrained U-Net supplies character-level segmentation guidance through cross-entropy loss.
  • Inference: TextDiffuser supports prompt-based generation, editable layouts or text, template-image conditioning, and text inpainting.

4 MARIO Dataset and Benchmark

MARIO-10M provides about 10 million diverse image-text pairs with comprehensive OCR annotations, while MARIO-Eval offers a broader benchmark for text-rendering evaluation.

  • MARIO-10M Dataset: MARIO-10M contains 10,061,720 image-text pairs, with 10,000,000 training samples and 61,720 testing samples.The dataset combines natural images, posters, book covers, and other text-image sources.
  • MARIO-10M Dataset: Each MARIO-10M image includes text detection, recognition, and character-level segmentation annotations produced with automated tools and filtering rules.Detection uses DB, recognition uses PARSeq, and segmentation uses a manually trained U-Net.
  • MARIO-10M Dataset: MARIO-LAION, MARIO-TMDB, and MARIO-OpenLibrary contribute filtered text images from LAION, movie posters, and book covers, respectively.TMDB and OpenLibrary captions are constructed from movie or book titles when off-the-shelf captions are unavailable.
  • MARIO-Eval Benchmark: MARIO-Eval comprises 5,414 prompts from prior text-rendering benchmarks, MARIO-10M test subsets, and multiple image sources.Its MARIO-10M-derived portion includes LAIONEval4000, TMDBEval500, and OpenLibraryEval500.
  • MARIO-Eval Benchmark: MARIO-Eval assesses text rendering using FID, CLIPScore, and OCR-based accuracy, precision, recall, and F-measure metrics.These criteria cover image-distribution similarity, image-text alignment, and generated-text detection and recognition.

5 Experiments

Experiments evaluate TextDiffuser’s layout and character-aware components, text-to-image quality, controllability, efficiency, and text inpainting. The method achieves strong OCR-related performance and user-perceived rendering quality while supporting multiple input and editing modes.

  • Ablation Studies: Adding width embedding improves Layout Transformer IoU by 2.1%, 2.9%, and 0.3% with 1, 2, and 4 layers, respectively.The optimal IoU uses two Transformer layers with width embedding.
  • Ablation Studies: Without character-level segmentation masks, generated text becomes inaccurate and incoherent with the background.The ablation highlights the role of explicit character guidance during generation.
  • Ablation Studies: A character-aware loss weight of λchar = 0.01 increases Accuracy by 9.8% over the λchar = 0 baseline.The study evaluates detected words against prompt keywords using Microsoft Read API on DrawBenchText.
  • Qualitative Results: TextDiffuser produces more readable text coherent with generated backgrounds and supports prompt-based, template-based, and text-inpainting generation.User studies report significantly better generation performance than existing methods and satisfaction with inpainting in most cases.
  • Controllability and Efficiency: Text color can be controlled through language descriptions, while image-generation time remains independent of the number of queries at 7.12±0.77s.Layout-generation time increases with keyword count, reaching 1.95±0.28s for 32 keywords.

6 Discussion and Conclusion

TextDiffuser generalizes beyond text rendering and supports text editing and removal, but its latent representation limits reconstruction of small characters and long-text generation can fail.

  • Discussion: TextDiffuser maintains general image-generation capability without text rendering and produces more diverse images than a text editing model.The discussion also presents potential for text removal and applications in posters, book covers, and secondary creation.
  • Limitations and Broader Impact: The VAE latent space limits reconstruction of images containing small characters, and long text remains a failure case.The authors also identify possible misuse of text inpainting for forging documents and recommend text-tampering detection techniques.

A Architecture of U-Net and Design of Character-Aware Loss

The U-Net processes latent features through four downsampling and four upsampling operations, while the dataset and caption construction support layout learning from diverse text images.

  • Architecture of U-Net and Design of Character-Aware Loss: The U-Net downsamples inputs to at most 1/16 and upsamples them through four corresponding operations.The character-aware loss uses 64 × 64 latent features and predicts 96 channels: 95 alphabet characters plus a null non-character class.
  • MARIO-10M Dataset: About 90% of MARIO-10M text regions are horizontally oriented with rotation angles below 5 degrees and no perspective changes.The layout model therefore predicts horizontal boxes using left-top and bottom-right coordinates.
  • Caption Templates: Caption templates express poster, book, movie, and TV-show contexts by inserting titles into fixed text prompts.Examples include “A poster with a title text of {XXX}” and “A book cover with logo {XXX} on it.”
  • Data Filtering: Strict filtering retains images larger than 256 pixels, excludes NSFW-triggering samples, and limits detected text boxes to 1–8.The rules target legibility, ethical concerns, and sufficient text area for recognition.

F Analysis of OCR Performance on MARIO-10M

MARIO-10M OCR annotations are evaluated against manual annotations across recognition, detection, and character-level segmentation, with examples illustrating the dataset’s varied text.

  • OCR Evaluation: OCR performance on MARIO-10M is assessed using manually annotated samples for recognition, detection, and character-level segmentation.The authors note that existing OCR methods perform below their results on dedicated detection and spotting benchmarks.
  • OCR Evaluation: The evaluation reports binary-pixel IoU for character segmentation, while recognition is included within the spotting task.Table 6 defines IoU as a two-class character-versus-non-character evaluation.
  • Dataset Examples: MARIO-10M examples include varied printed and scene text such as “GREENE HIGH,” “LONORES POLISH,” and “RELEASE.”The accompanying visualizations show OCR annotations for the dataset samples.

G Samples in MARIO-Eval

MARIO-Eval combines existing prompts with a large subset of MARIO-10M to evaluate text rendering across image quality, text-image alignment, OCR, and human judgments.

  • The benchmark includes prompts for posters, book covers, storefronts, signs, logos, graffiti, and text embedded in everyday scenes.
  • Evaluation uses FID, CLIPScore, OCR metrics, and human evaluation to assess generated-image quality and text rendering.
  • The Layout Transformer produces keyword layouts that are visualized alongside generated images, with the model producing reasonable layouts.

J Experiment without Explicit Guidance of Segmentation Masks

Experiments remove positional and content guidance from TextDiffuser to test explicit segmentation-mask conditioning, alongside comparisons, user studies, and general-image evaluation.

  • Experiment without Explicit Guidance of Segmentation Masks: With position guidance but character pixels removed, the model generates words resembling keywords but introduces grammatical errors such as a missing letter.
  • Experiment without Explicit Guidance of Segmentation Masks: Removing segmentation masks worsens text rendering quality, demonstrating the significance of explicit position and content guidance.Without masks, the experiment is equivalent to fine-tuning a pretrained latent diffusion model directly on MARIO-10M.
  • User Studies: User studies evaluate whole-image generation using text-rendering quality and correspondence between generated images and text descriptions.The whole-image questionnaire contains 15 cases and two multiple-choice questions per case.
  • User Studies: Part-image generation is rated on text-rendering quality and harmony between the generated region and the unmasked region.Ratings range from 1 to 4, with 4 indicating the best result.
  • Generating Images without Text: TextDiffuser retains good general-image generation ability after fine-tuning on MARIO-10M, according to qualitative and MSCOCO FID evaluations.The comparison uses Stable Diffusion 1.5 because both models share the same backbone.

P Experimental Results of Text Removal

TextDiffuser can transform its text-inpainting setup into text removal by masking text regions and marking them as non-character areas.

  • Text removal is obtained by providing a mask and setting all regions to non-character in the segmentation mask.
  • Experimental results show that the method achieves results similar to the ground truth on the demonstrated text-removal cases.

Q Limitations and Failure Cases

TextDiffuser has failure cases with small characters and long text, reflecting limitations in latent-space detail preservation and crowded keyword layouts.

  • Generating images with small characters: Small characters can become vague or disjointed because VAE compression loses detail, reducing text legibility.Using Stable Diffusion 2.1 at 768×768 increases latent resolution and mitigates the issue, but raises inference latency from 8.5s to 12.0s.
  • Generating images from long text: Long text with many keywords can produce disordered and overlapping words in the generated layouts.
  • Generating images from long text: Dense, small-text training examples may introduce noise that increases detection and recognition errors for long-text generation.The paper suggests improving OCR tools as a possible future response.
Loading 2305.10855v5…