PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyText: Multilingual Visual Text Generation And Editing

Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng, Xuansong Xie

arXiv:2311.03054v5cs.CV

TL;DR

Existing diffusion models generate high-fidelity images but often render visual text inaccurately, motivating a multilingual generation and editing framework. AnyText combines text-specific latent and OCR-based embedding modules with dedicated diffusion and perceptual losses, and its evaluations report substantially stronger performance than competing approaches.

  • Problem

    Diffusion image generators still struggle to produce well-formed, legible, and readable visual text, partly because training data, text encoders, and losses lack text-specific support.

  • Method

    AnyText uses an auxiliary latent module for glyph, position, and masked-image inputs plus an OCR-based text embedding module, text-control diffusion loss, and text perceptual loss.

  • Results

    AnyText outperforms competing methods by a large margin on Chinese and English text generation across OCR accuracy and realism metrics.

  • Takeaways & Limitations

    The framework provides multilingual text generation and editing with multi-line, irregular-region, and plug-and-play functionality.

Abstract

from arXiv · show

Diffusion model based Text-to-Image has achieved impressive achievements recently. Although current technology for synthesizing images is highly advanced and capable of generating images with high fidelity, it is still possible to give the show away when focusing on the text area in the generated image. To address this issue, we introduce AnyText, a diffusion-based multilingual visual text generation and editing model, that focuses on rendering accurate and coherent text in the image. AnyText comprises a diffusion pipeline with two primary elements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph, position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We employed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy. AnyText can write characters in multiple languages, to the best of our knowledge, this is the first work to address multilingual visual text generation. It is worth mentioning that AnyText can be plugged into existing diffusion models from the community for rendering or editing text accurately. After conducting extensive evaluation experiments, our method has outperformed all other approaches by a significant margin. Additionally, we contribute the first large-scale multilingual text images dataset, AnyWord-3M, containing 3 million image-text pairs with OCR annotations in multiple languages. Based on AnyWord-3M dataset, we propose AnyText-benchmark for the evaluation of visual text generation accuracy and quality. Our project will be open-sourced on https://github.com/tyxsspa/AnyText to improve and promote the development of text generation technology.

1 INTRODUCTION

AnyText addresses the difficulty of rendering legible visual text by combining text-specific controls, OCR-based embeddings, and dedicated supervision. It supports multilingual generation, editing, and varied text layouts.

  • Current diffusion models often struggle to produce well-formed, legible, and readable visual text despite strong overall image quality.
  • This difficulty reflects limited annotated image-text data, character-insensitive text encoders, and losses without dedicated text-region supervision.
  • Figure 1 illustrates specified-position text generation and text editing that preserves consistency with surrounding text style.
  • AnyText combines an auxiliary latent module for glyphs, positions, and masked images with an OCR-based text embedding module and text perceptual loss.
  • AnyText supports multi-line, irregular-region, multilingual, and text-editing operations while integrating with existing diffusion models.

2 RELATED WORKS

Prior work improves controllable image synthesis and text rendering through glyph conditions, text encoders, and perceptual supervision, but important limitations remain for character accuracy and non-Latin text.

  • Recent diffusion research has expanded from image generation to interactive editing and multi-condition controllable synthesis.
  • Visual text integration remains challenging even as general image synthesis has substantially improved.
  • Glyph-based methods control text through explicit glyph images, locations, font-related information, segmentation masks, or masked images.
  • Large language-model text encoders can improve accuracy, but character-blind encoders and non-Latin rendering remain problematic.
  • Perceptual-supervision methods constrain intermediate OCR features or character-level latent representations, with differing model requirements.

3 METHODOLOGY

AnyText combines a text-control diffusion pipeline with auxiliary latent and text embedding modules, using glyph, position, masked-image, caption, and OCR-derived information to generate or edit visual text. Text-control diffusion and text perceptual losses provide supervision for text accuracy, including in irregular regions and multiple languages.

  • Framework: AnyText’s framework combines a text-control diffusion pipeline, auxiliary latent module, text embedding module, and text perceptual loss.The overall objective combines text-control diffusion loss with text perceptual loss, weighted by λ.
  • Text-control diffusion pipeline: TextControlNet adds auxiliary features to noisy latents while a frozen base UNet preserves the underlying image-generation capability.The trainable encoding layers focus on text generation, and modular binding allows other base diffusion models to generate text.
  • Auxiliary latent module: The auxiliary latent module fuses glyph, position, and masked-image conditions into a latent feature map for text generation and editing.Glyph and position inputs are transformed into feature maps, while the masked image is downsampled with a VAE encoder before convolutional fusion.
  • Text embedding module: The text embedding module combines caption semantics with OCR features extracted from rendered glyph lines to represent text for multilingual generation.Generated text lines use a placeholder in the caption, while a centered glyph rendering is passed to an OCR recognition model for feature extraction.
  • Text perceptual loss: The text perceptual loss compares OCR feature maps from predicted and original text regions after reconstructing an image from the denoiser’s latent prediction.Position conditions locate text regions, and the loss focuses on writing correctness rather than background, position, color, or font differences; its weighting uses φ(t) = ᾱ_t.

4 DATASET AND BENCHMARK

The paper introduces AnyWord-3M as a large multilingual text-image dataset and constructs AnyText-benchmark to evaluate visual text generation accuracy and quality. The benchmark uses defined OCR-based metrics and compares AnyText with competing methods under reported training and evaluation conditions.

  • AnyWord-3M: AnyWord-3M contains images from publicly available image and OCR datasets, covering diverse text-containing scenes and multiple languages.OCR annotations are used directly for OCR datasets; other images are processed with PP-OCRv3, and captions are regenerated with BLIP-2.
  • AnyWord-3M: 3,034,486 images contain over 9 million text lines and more than 20 million characters or Latin words.The dataset includes approximately 1.6 million Chinese images, 1.39 million English images, and 10,000 images in other languages.
  • AnyText-benchmark: AnyText-benchmark uses 1,000 randomly selected images from each of the Wukong and LAION subsets to evaluate Chinese and English generation, respectively.The remaining images form the AnyWord-3M training set.
  • Evaluation metrics: Sentence Accuracy counts a generated line as correct only when OCR output completely matches the ground truth, while Normalized Edit Distance provides a less stringent measure.The benchmark uses these metrics to assess text-generation accuracy and quality.
  • Comparative evaluation: Table 2 compares AnyText with competing methods, including models trained on LAION-Glyph-10M or fine-tuned on TextCaps-5k, using officially released models.The cited evaluation uses the benchmark and metrics described for the comparison.

5 EXPERIMENTS

Experiments compare AnyText with competing methods, qualitative systems, and ablations. AnyText achieves stronger multilingual text generation and integration, while ablations identify OCR-based embeddings, positional conditioning, and perceptual loss as effective components.

  • 5.2 Quantitative Results: AnyText outperforms competing methods by a large margin on Chinese and English text generation across OCR accuracy and realism metrics.The comparison uses Sen.ACC, NED, and FID under matched sampling settings.
  • 5.2 Qualitative Results: AnyText integrates generated English and Chinese text with backgrounds more naturally than compared models in qualitative examples.Examples include carved stone text, signboard reflections, chalkboard writing, and clothing-fold distortions.
  • 5.3 Ablation Study: Enabling the editing branch causes a slight decrease because joint text generation and editing make model convergence more difficult.Subsequent experiments disable editing to focus on text generation.
  • 5.3 Ablation Study: 25.7% improvement in Sen. Acc over Exp.2 results from using the pretrained PP-OCRv3 model for text embedding.Alternative CLIP vision and convolutional feature extractors produced unsatisfactory or limited results.
  • 5.3 Ablation Study: More accurate positional conditioning improves performance and enables text generation in irregular regions.The comparison is between Exp.5 and Exp.6, where position is added beyond implicitly available glyph information.
  • 5.3 Ablation Study: A perceptual-loss weight of λ = 0.01 yields a 4.0% Sen. Acc improvement over Exp.5.The loss improves results despite slowing training because it requires transitioning from latent space to image space.

6 CONCLUSION AND LIMITATIONS

The paper concludes that AnyText is a multilingual diffusion framework for visual text generation and editing, supported by dedicated conditioning, OCR-based embeddings, and text-focused losses. It also introduces AnyWord-3M and evaluates the approach on AnyText-benchmark.

  • Conclusion: AnyText combines auxiliary latent conditioning, OCR-derived text embeddings, and text-control and perceptual losses for multilingual visual text generation and editing.The auxiliary module uses glyph, position, and masked-image information, while OCR features are merged with image-caption semantics.
  • Conclusion: AnyWord-3M contains 3 million multilingual text-image pairs with OCR annotations for training and evaluation.The paper also introduces AnyText-benchmark to evaluate visual text generation accuracy and quality.
  • Conclusion: Comprehensive experiments on AnyText-benchmark show superiority over existing methods.

A.1 MORE EXAMPLES ON THE FLEXIBILITY OF ANYTEXT

AnyText preserves ordinary text-to-image generation while allowing community base models to gain text-generation capability through modular integration.

  • Flexibility: AnyText preserves the base model’s conventional text-to-image generation capability.The paper illustrates this behavior with examples of images generated without text.
  • Flexibility: Modularly binding AnyText lets community base models with diverse styles generate text.Examples include Oil Painting, Guoha Diffusion, Product Design, and Moon Film models.

A.2 PARAMETER SIZE AND COMPUTATIONAL OVERHEAD OF ANYTEXT

AnyText adds little computational overhead relative to ControlNet while retaining flexible text-generation integration with base models.

  • Computational Overhead: AnyText takes 3512 ms/image versus 3476 ms/image for ControlNet at batch size 4 on one Tesla V100.The reported inference-time difference is 36 ms/image.
  • Parameter Size: AnyText’s parameter-size comparison with ControlNet is summarized in Table 4.The supplied material identifies the comparison but does not provide the table values.
  • Dataset Statistics: The supplied dataset statistics are summarized in Table 5 for AnyWord-3M subsets and line counts.The passage identifies the table’s scope without reporting its numeric entries.

A.3 MORE DETAILS ABOUT DATASET PREPARATION

AnyWord-3M preparation applies image-, text-region-, and OCR-level filters, then regenerates captions and replaces caption placeholders with glyph embeddings. The process is designed to retain high-quality images and accurately recognized text lines.

  • Dataset filtering: The authors apply stricter analogous filtering rules to LAION-400M because it contains more English data than the Chinese source dataset.
  • Image-level filtering: Images must have dimensions of at least 256 pixels, an aspect ratio between 0.67 and 1.5, text covering at least 10% of the image, and no more than 8 text lines.
  • Text-line filtering: Each retained text line must be at least 30 pixels high, receive an OCR recognition score of at least 0.7, and contain non-whitespace content.
  • Caption construction: Captions are regenerated with BLIP-2, stripped of specific placeholders, and augmented with randomly selected text-related statements.
  • Caption construction: The final textual description appends one placeholder per text line, whose embedding is later replaced by an embedding of the corresponding text glyph information.

A.4 STASTIC AND EXAMPLES OF ANYWORD-3M

The appendix reports AnyWord-3M composition statistics, dataset growth, OCR annotation quality, and qualitative AnyText examples. It also describes invalid-text masking and training or model implementation details associated with the reported examples and improvements.

  • Dataset statistics: Table 6 summarizes language-specific line counts, character or word counts, unique characters or words, mean characters per line, and the number of lines with at most 20 characters.
  • Dataset statistics: Figure 9 shows example images from the AnyWord-3M dataset, while Table 8 reports changes in watermark and pseudo-text tested on a private model and dataset.
  • Qualitative examples: Figures 10 and 11 show English and Chinese benchmark examples generated with the same fixed random seed and positive and negative prompts, while Figures 12 and 13 provide additional generation and editing examples.
  • OCR annotations: English OCR annotation error was 2.4%, compared with 11.8% for Chinese in a random sample; Chinese annotations were regenerated with PP-OCRv4.
  • Training implementation: Invalid text lines are masked during training when recognition is too low, text is too small, or the line is not among five randomly chosen lines; the corresponding loss is set to zero.
  • Training implementation: The v1.0 model was fine-tuned for 5 epochs on AnyWord-3M v1.1, with the final 2 epochs using data with wm score < 0.5, and later replaced SD1.5 with Realistic Vision.
Loading 2311.03054v5…