PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Text Image Generation with Diffusion Models

Yuanzhi Zhu, Zhaohai Li, Tianwei Wang, Mengchao He, Cong Yao

arXiv:2306.10804v1cs.CV

TL;DR

Collecting and labeling enough diverse real text images is difficult, so text recognition has relied heavily on synthesis and augmentation. CTIG-DM uses diffusion with image, text, and style conditions to generate text images; experiments report improvements in image quality and recognizer performance, alongside potential for OOV generation and domain adaptation.

  • Problem

    Collecting and annotating enough real text images to capture real-world complexity and diversity is difficult, leaving recognition systems reliant on synthesis and augmentation.

    Source: §1 · p. 1

  • Method

    CTIG-DM uses a conditional diffusion model with image, text, and style conditions, configurable into synthesis, augmentation, recovery, and imitation modes.

    Source: §1 · p. 2

  • Results

    Experiments on handwritten and scene text report improved image quality and performance of existing text recognizers, with potential for OOV generation and domain adaptation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude CTIG-DM can boost existing text recognizers and has potential for generating OOV images and supporting domain adaptation.

    Source: §1 · p. 2, §5 · p. 8

    The cited prior augmentation methods cannot create OOV samples, and most prior synthesis methods were tested only on handwritten Latin text.

    Source: §2.2 · pp. 2–3

Abstract

from arXiv · show

Current text recognition systems, including those for handwritten scripts and scene text, have relied heavily on image synthesis and augmentation, since it is difficult to realize real-world complexity and diversity through collecting and annotating enough real text images. In this paper, we explore the problem of text image generation, by taking advantage of the powerful abilities of Diffusion Models in generating photo-realistic and diverse image samples with given conditions, and propose a method called Conditional Text Image Generation with Diffusion Models (CTIG-DM for short). To conform to the characteristics of text images, we devise three conditions: image condition, text condition, and style condition, which can be used to control the attributes, contents, and styles of the samples in the image generation process. Specifically, four text image generation modes, namely: (1) synthesis mode, (2) augmentation mode, (3) recovery mode, and (4) imitation mode, can be derived by combining and configuring these three conditions. Extensive experiments on both handwritten and scene text demonstrate that the proposed CTIG-DM is able to produce image samples that simulate real-world complexity and diversity, and thus can boost the performance of existing text recognizers. Besides, CTIG-DM shows its appealing potential in domain adaptation and generating images containing Out-Of-Vocabulary (OOV) words.

1. Introduction

Text recognition depends heavily on synthesized and augmented data because collecting and labeling enough diverse real text images is difficult. CTIG-DM uses diffusion with three configurable conditions to generate text images, and experiments report improvements in image quality and text recognition.

1. Introduction

  • Collecting and labeling sufficient real text images is difficult, so existing recognition methods rely heavily on synthesis and augmentation.The paper frames data volume and diversity as important for recognition performance.Source: §1 · p. 1
  • Text image generation must preserve the specified text content while also producing faithful and diverse images.The introduction contrasts text-image requirements with natural-image generation, where content validity is not the same constraint.Source: §1 · p. 2
  • CTIG-DM combines image, text, and writer-style conditions into four modes: synthesis, augmentation, recovery, and imitation.The conditions guide image attributes, textual content, and style.Source: §1 · p. 2
  • Experiments on handwritten and scene text report improved image quality and performance of existing text recognizers.The paper also reports potential for OOV image generation and domain adaptation.Source: §1 · p. 2

2. Related Work

Related work covers text recognition, text-image synthesis and augmentation, and diffusion models. The paper positions its approach against prior methods' limits in OOV generation, generalization, and controllable text-image content.

2.1. Text Recognition

  • Scene-text recognition progressed from CNN–RNN–CTC sequence models to attention methods for irregular text and Transformer-based language-model approaches; handwriting work addresses style diversity and cursive joins.The cited handwriting literature also includes domain adaptation, writer-adaptive learning, and synthesis or augmentation.Source: §2.1 · p. 2

2.2. Text Image Augmentation and Synthesis

  • Prior augmentation methods perturb images or features but cannot create OOV samples, leaving diversity bounded by training data; much synthesis work also lacks broader text-type evaluation.GAN-based handwritten synthesis covers variable-length words and style/content modeling, but can be difficult to train and prone to mode collapse without careful hyperparameter design.Source: §2.2 · pp. 2–3

2.3. Diffusion Models

  • Diffusion models have shown strong image-generation diversity and support conditional generation, but text images require character-level context and texture, motivating text-recognizer-derived conditions.Prior diffusion work includes unconditional generation, online handwriting, classifier guidance, and text-conditional image synthesis.Source: §2.3 · p. 3

3. Methodology

The method encodes image appearance, text content, and writer style as conditions for a conditional diffusion model. It trains the model to predict noise and uses different condition combinations for four text-image generation modes.

3.1. Conditional Encoder

  • The image condition uses a pretrained text-recognizer encoder to capture character-focused visual features, with attention pooling aggregating patch features into an image-level representation.The design targets textural and color characteristics rather than noisy background features.Source: §3.1 · p. 3
  • The text condition adapts classifier weights to encode character content from text labels, allowing the model to represent out-of-vocabulary words.One-hot text labels and character positions are projected into the text condition's dimensionality.Source: §3.1 · p. 3
  • The style condition represents a specific writer's characteristics, including slants, cursive joins, and stroke widths, and is used only for handwritten text generation.The writer ID supplies the style condition.Source: §3.1 · pp. 3–4

3.2. Conditional Diffusion Model

  • The conditional UNet predicts added Gaussian noise from noisy images and the proposed conditions, using squared-error supervision computed directly from the original image.The model follows a diffusion process that gradually adds Gaussian noise to an initial image.Source: §3.2 · p. 4
  • At generation time, the UNet iteratively subtracts predicted noise from random Gaussian input to produce an image.The current denoising image and configured conditions are used at each generation step.Source: §3.2 · p. 4
  • Condition combinations define synthesis, augmentation, recovery, and imitation modes; text condition governs diversity, while image and style conditions influence fidelity.The text condition is also described as helpful for generating OOV images.Source: §3.2 · p. 4, Table 1 · p. 4

4. Experiments

Experiments cover handwritten and scene text datasets, with implementation choices and evaluation criteria specified for the recognition and generation settings.

4.1. Datasets

  • Handwritten experiments use IAM, RIMES, and CVL, while CASIA-HWDB 1.0-1.1 supplies offline Chinese characters for OOV generation.IAM contains over 115,000 English words from 657 writers; RIMES has over 60,000 French words from over 1,000 authors; CVL includes writing from 311 writers.Source: §4.1 · p. 4
  • Scene-text training uses MJSynth, Synth-Text, and Real-L, with regular and irregular datasets used for testing.The listed test sets include IIIT, SVT, IC03, IC13, IC15, SVTP, and CUTE.Source: §4.1 · p. 4

4.2. Implementation Details

  • Handwritten images are resized to 64-pixel height with aspect-ratio-preserving width capped at 256 pixels, and recognition is evaluated with WER and CER.Generation quality is measured with FID, GS, SSIM, RMSE, and LPIPS.Source: §4.2 · p. 4
  • Scene-text comparisons keep prior methods' experimental settings unchanged, except that CTIG-DM-generated data is used.Source: §4.2 · p. 4
  • The recognizer uses CRNN and the diffusion model follows DDPM; training uses AdamW, cosine-annealed learning rate 0.0001, and batch sizes of 64 and 256, respectively.The experiments use NVIDIA Tesla V100 GPUs.Source: §4.2 · pp. 4–5

4.3. Ablation Study

The ablation study examines how the conditional encoder’s conditions affect diffusion generation. Adding image, text, and style conditions progressively improves generation quality, while the modes show different diversity and fidelity trade-offs.

4.3. Ablation Study

  • FID falls from 33.42 without conditions to 9.34 with all conditions, with image and text conditions producing the largest improvements.Adding image and text conditions together brings FID to 9.76; adding style further reduces it to 9.34.Source: §4.3 · p. 5, Table 2 · p. 5
  • Text-only synthesis produces diverse samples, while image-only augmentation can lose character details and generate wrong characters.Adding text condition to image condition rarely produces wrong characters while retaining diversity; style condition further improves handwritten-image fidelity.Source: §4.3 · p. 5
  • The generated MIX data uses equal proportions of synthesis and imitation modes, or recovery mode for scene text, to combine diversity and fidelity.Synthesis has the best diversity, while imitation or scene-text recovery has the best fidelity.Source: §4.3 · p. 5

4.4. Recognition Performance

The experiments evaluate generated data for handwritten and scene text recognition. CTIG-DM improves recognition performance across these settings, with gains varying by dataset and training-data scale.

4.4.1 Handwritten Text Recognition

  • On IAM, adding generated data reduces WER by 4.78% and CER by 1.87%; on RIMES, WER falls by 5.20% and CER by 1.52% versus the baseline.On IAM, 900K MIX yields CER of 5.60% and WER of 14.03%; about 200K generated samples achieve performance comparable to the real-data baseline.Source: §4.4.1 · p. 6, Figure 6 · p. 6, Table 3 · p. 6
  • CTIG-DM outperforms previous data-augmentation and data-synthesis methods on handwritten text, and adding random geometric augmentation further improves its performance.The comparison excludes methods using additional real data or language models; the authors describe CTIG-DM as complementary to prior work.Source: §4.4.1 · p. 6, Table 3 · p. 6

4.4.2 Scene Text Recognition

  • With generated data, CRNN improves by an average of 1.7% and ABINet by 0.5% across six scene-text test datasets.On Real-L, the gains are 4.9% for CRNN and 1.5% for ABINet.Source: §4.4.2 · pp. 6–7, Table 4 · p. 7
  • Scene-text gains are less pronounced than handwritten-text gains, which the authors attribute partly to STR training sets of about 14M synthesized images limiting CTIG-DM’s diversity and fidelity.Despite this boundary, generated data improves advanced recognizers without network changes.Source: §4.4.2 · p. 7

4.5. Quality of Generated Images

The section compares CTIG-DM with prior methods on generated-image quality for handwritten text. It reports that CTIG-DM performs better across all evaluated metrics on both datasets.

4.5. Quality of Generated Images

  • The image-quality comparison calculates FID using 25k real and 25k generated images, and GS using 5k of each.The evaluation follows the settings of prior work [45].Source: §4.5 · p. 7
  • The reported quality evaluation is scoped to handwritten-text recognition datasets.Table 5 identifies HTR datasets as the comparison scope.Source: Table 5 · p. 7
  • CTIG-DM outperforms state-of-the-art methods by a notable margin on all metrics across both handwritten-text datasets.Table 5 compares image-generation quality; lower values are preferable.Source: §4.5 · p. 7, Table 5 · p. 7

4.6. Generating Out-of-Vocabulary Images

The section evaluates CTIG-DM’s ability to generate out-of-vocabulary text images. It uses text and style conditions and reports higher OOV image-generation quality than previous methods.

4.6. Generating Out-of-Vocabulary Images

  • Synthesis mode depends only on text condition, so changing the input text allows generation of OOV images.For a fair comparison with prior methods, the study uses both text and style conditions.Source: §4.6 · pp. 7–8
  • Using text and style conditions, CTIG-DM significantly improves OOV image-generation quality over previous methods.The comparison is on the IAM dataset.Source: §4.6 · p. 8, Table 6 · p. 8
  • The visualized examples indicate control over generated text and style, and the displayed OOV words do not appear in the training set.Figure 7 shows images generated using specified styles and OOV text.Source: Figure 7 · p. 8, §4.6 · p. 8

4.7. Domain Adaptation

The domain-adaptation experiment trains CTIG-DM on IAM, then uses generated samples with the CVL training lexicon to train and evaluate a recognizer on CVL. Against the baseline, WER and CER decrease, and the paper reports gains exceeding those of previous methods.

4.7. Domain Adaptation

  • The comparison evaluates domain adaptation on the CVL dataset, with SYN denoting generated data using the CVL training lexicon.Source: Table 7 · p. 8
  • WER and CER decrease by 16.40% and 7.60%, respectively, compared with the baseline on CVL.The paper reports that the gains far exceed those of previous methods.Source: §4.7 · p. 8, Table 7 · p. 8
  • CTIG-DM is trained on IAM, then generates 100K samples using the CVL training lexicon; the IAM training set and generated samples train the recognizer evaluated on CVL.Results average five repeated training runs.Source: §4.7 · p. 8

4.8. Applications

CTIG-DM is explored for generating out-of-vocabulary Chinese characters and ancient rare characters. The examples include characters absent from training data and, for Chinese, even nonexistent characters.

4.8. Applications

  • Chinese characters are decomposed into radical sequences used as text conditions to generate characters absent from training, including nonexistent characters.The examples include typos that may be relevant to typo-recognition scenarios.Source: §4.8 · p. 8
  • CTIG-DM can generate rare ancient characters, addressing a setting where training samples are lacking because of a long-tailed character distribution.Source: §4.8 · p. 8
  • Combining CTIG-DM with radicals for character generation is presented as promising when rare ancient characters have extremely few samples.Source: §4.8 · p. 8

5. Conclusion and Future Work

The conclusion presents CTIG-DM as a diffusion-based text-image generator built around three conditions and four generation modes. Experiments demonstrate its effectiveness, while future work will explore more modes and text-image types.

5. Conclusion and Future Work

  • CTIG-DM adapts diffusion models to text-image generation using three conditions and four generation modes.Source: §5 · p. 8
  • Experiments demonstrate CTIG-DM's effectiveness and advantage, including its use to boost existing text recognizers.Source: §5 · p. 8
  • Future work will explore more generation modes on more types of text images and generate images in styles of unseen writers.Source: §5 · p. 8

Related papers

Loading 2306.10804v1…