PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Generation of Handwritten Text Images Conditioned on Sequences

Eloi Alonso, Bastien Moysset, Ronaldo Messina

arXiv:1903.00277v1cs.CV

TL;DR

Handwriting recognition systems need large annotated datasets, whose creation is costly. The paper generates handwritten-word images conditioned on the target text using an adversarial model with an auxiliary recognizer. It reports realistic French and Arabic images and a slight recognition improvement when synthetic images are added to training data.

  • Problem

    Handwritten-text recognition requires large amounts of annotated image-transcript data, and annotating text images is costly and time-consuming.

    Source: §I · p. 1

  • Method

    The model conditions adversarial handwritten-word generation on a character-sequence embedding and adds an auxiliary recognizer to encourage the generated text to be recognizable.

    Source: A. Auxiliary Recognizer Generative… · p. 2, §IV · p. 6

  • Results

    The model produces realistic French and Arabic word images, and synthetic images slightly reduce error rate for the French model trained on combined data.

    Source: §IV · p. 6

  • Takeaways & Limitations

    Controlled generation can provide training material that covers rare character sequences beyond the textual contents of distortion- and noise-based augmentation.

    Source: §I · p. 2

Abstract

from arXiv · show

State-of-the-art offline handwriting text recognition systems tend to use neural networks and therefore require a large amount of annotated data to be trained. In order to partially satisfy this requirement, we propose a system based on Generative Adversarial Networks (GAN) to produce synthetic images of handwritten words. We use bidirectional LSTM recurrent layers to get an embedding of the word to be rendered, and we feed it to the generator network. We also modify the standard GAN by adding an auxiliary network for text recognition. The system is then trained with a balanced combination of an adversarial loss and a CTC loss. Together, these extensions to GAN enable to control the textual content of the generated word images. We obtain realistic images on both French and Arabic datasets, and we show that integrating these synthetic images into the existing training data of a text recognition system can slightly enhance its performance.

I. INTRODUCTION

The introduction motivates generating handwritten-word images from text to address the cost of annotated training data. It proposes a sequence-conditioned adversarial model and reports realistic French and Arabic images, with a slight recognition improvement on RIMES.

I. INTRODUCTION

  • Generating an image from a given word is proposed as a way to address the large, costly annotated datasets required to train handwritten-text recognition networks.Source: §I · p. 1
  • Template concatenation struggles to reproduce real-world image distributions and cursive writing, while online trajectory synthesis does not capture offline features such as background texture or line-thickness variation.Source: §I · p. 1
  • Prior GAN work conditions on classes or textual descriptions, whereas the cited text-image synthesis approach uses style transfer to render isolated handwritten Chinese characters from printed fonts.Source: §I · pp. 1–2
  • Controlled generation can create training material covering rare character sequences, which data augmentation using distortion and additive noise cannot add to the textual contents.Source: §I · p. 2
  • The proposed adversarial architecture generates realistic handwritten-word images, using bidirectional LSTMs to encode the target sequence and an auxiliary recognizer to control generated text.Source: §I · p. 2
  • A French recognition model trained on combined data slightly improves performance with synthetic images, while realistic word images are obtained on French and Arabic datasets.The reported recognition improvement is on RIMES.Source: §I · p. 2

II. PROPOSED ADVERSARIAL MODEL

The section introduces the proposed adversarial model and maps the presentation of its objectives, network architectures, and optimization settings.

II. PROPOSED ADVERSARIAL MODEL

  • The section introduces the adversarial model for handwritten-word generation and its training objectives.Source: §II · p. 2
  • Network architectures are detailed in Section II-B.Source: §II · p. 2
  • Optimization settings are described in Section II-C.Source: §II · p. 2

A. Auxiliary Recognizer Generative Adversarial Networks

The model modifies a standard GAN to generate images conditioned on a character sequence and to encourage legible text through an auxiliary recognizer. Its objectives combine adversarial and CTC losses.

A. Auxiliary Recognizer Generative Adversarial Networks

B. Networks architecture

The architecture encodes each target character sequence with a bidirectional LSTM and uses that embedding to condition the generator. Separate generator and discriminator ResBlocks up-sample and down-sample, while an auxiliary recognition network checks generated text.

  • The sequence encoder embeds characters in R^128 and outputs the final layer of a four-layer bidirectional LSTM with hidden size 128.Source: B. Networks architecture · p. 3
  • The generator splits 128-dimensional noise into eight chunks, combines seven with the sequence embedding in up-sampling ResBlocks, and outputs a 1 × 128 × 512 image.A self-attention layer sits between the fourth and fifth ResBlocks.Source: B. Networks architecture · p. 3
  • The discriminator uses seven down-sampling ResBlocks, a self-attention layer, and a final normal ResBlock before projecting its pooled output to a scalar.Source: B. Networks architecture · p. 3
  • The auxiliary recognizer is a Gated Convolutional Network with a five-layer convolutional encoder and a decoder of two stacked bidirectional LSTM layers.It uses Tanh activations and convolutional gates, then max-pools vertically before decoding.Source: B. Networks architecture · p. 3
  • Generator ResBlocks concatenate noise chunks with ϕ(s) in 512-unit CBN layers and use nearest-neighbor up-sampling; discriminator blocks use Batch Normalization and average-pooling down-sampling.The generator uses ReLU and the discriminator LeakyReLU; a 1×1 convolution is used only when channel counts differ.Source: Figure 3 · p. 3

C. Optimization settings

The generator and sequence encoder learn from discriminator and recognition-network gradients, which can differ greatly in magnitude. The training procedure balances these signals while training the recognizer only on real images.

  • Training uses spectral normalization in the generator and discriminator, Adam with learning rate 2×10−4 and β1 = 0, β2 = 0.999, and gradient clipping in the discriminator and recognizer.Mini-batches contain 64 images, either real or generated.Source: C. Optimization settings · p. 3
  • The recognizer trains on real data only, while the generator and sequence encoder send generated images through both the discriminator and recognizer for feedback.Source: C. Optimization settings · p. 3
  • Recognition gradients can be 10^2 to 10^3 times larger than discriminator gradients, so the authors apply an affine transformation to balance the signals before updating the generator and sequence encoder.The relative recognition weight α is set to 1.Source: C. Optimization settings · pp. 3–4

III. RESULTS

The experiments use French RIMES and Arabic OpenHaRT word images, resized and padded to a common shape. Generation quality and text-recognition performance are assessed with separate image and transcription metrics.

A. Experimental setup

  • Images are resized to 128 pixels high, widths above 512 pixels are removed, and remaining images are padded to 512 pixels, on the right for French and left for Arabic.The datasets are RIMES and OpenHaRT.Source: A. Experimental setup · p. 4
  • Generated words are sampled from French Wikipedia or OpenHaRT word lists, and RIMES recognition experiments use a separate validation set of 6060 images.Source: A. Experimental setup · p. 4
  • Generation is evaluated with FID and Geometry Score, for which lower is better; text-recognition performance is measured by character-level Edit Distance and Word Error Rate.FID uses 25k real and 25k generated images; Geometry Score uses 5k of each with 100 repetitions, and textual content is checked visually.Source: A. Experimental setup · p. 4

B. Ablation study

The RIMES ablation tests how gradient balancing trades off readable text against image realism. The best reported compromise uses α = 1, while weaker or stronger recognition gradients lose realism or textual control.

B. Ablation study

1) Gradient balancing:

  • Without gradient balancing, ||∇R||2 was typically 102 to 103 times greater than ||∇D||2, biasing learning toward R; the word is readable, but FID is high (141.35) and the image is unrealistic.The background is noisy, and the letters are too far apart.Source: 1) Gradient balancing · p. 4
  • At α = 0.1, FID is 72.93 and the image looks more realistic, but the generated text loses control because the recognition task receives little weight.Source: 1) Gradient balancing · p. 4, Table II · p. 4
  • At α = 10, conditioning on the text succeeds, but FID rises to 222.47 and the images show strong visual artifacts and noise.Source: 1) Gradient balancing · p. 4, Table II · p. 4
  • FID 23.94 and GS 8.58 × 10−4 are the best reported scores at α = 1, where the generated word is both readable and realistic.The authors use α = 1 for the remaining experiments.Source: 1) Gradient balancing · p. 4, Table II · p. 4

2) Adversarial loss:

The Geometric GAN performs best on FID and GS, while vanilla GAN and LSGAN show signs of style collapse and uncontrolled textual content.

2) Adversarial loss:

  • Geometric GAN leads on both FID and GS among the three adversarial training procedures.LSGAN fails to produce text-like outputs in three of five trials.Source: 2) Adversarial loss · pp. 4–5, Table III · p. 5
  • Vanilla GAN produces realistic images by FID, but its high GS indicates style collapse and uncontrolled textual content.The same style-collapse and content-control issues are reported for LSGAN; visual inspection confirmed the metric trends.Source: 2) Adversarial loss · p. 5

3) Self-attention:

Self-attention improves FID and GS, while feeding the generator through conditional batch-normalization layers improves realism and reduces mode collapse.

3) Self-attention:

  • Self-attention improves performance in both FID and GS compared with training without the module.Without self-attention, samples remain realistic and their textual content remains correct.Source: 3) Self-attention · p. 5, Table IV · p. 5

4) Conditional Batch Normalization:

  • Feeding generator inputs through CBN layers improves realism and reduces mode collapse compared with concatenating the whole noise z and ϕ(s) into G’s first linear layer with standard batch normalization, according to FID and GS.Visual inspection confirmed these trends and showed that the first-linear-layer alternative prevents correct conditioning on textual content.Source: 4) Conditional Batch Normalization · p. 5

C. Generation of handwritten text images

The model generates cursive words in French and Arabic, including French words absent from RIMES; adding generated images to RIMES slightly improves recognition, but synthetic-only training is not competitive.

C. Generation of handwritten text images

  • The model generates cursive, variable-length words in French and Arabic, though some French samples are barely readable and writing style shows slight collapse.The examples are randomly generated rather than cherry-picked.Source: C. Generation of handwritten text images · p. 5
  • French generation includes words absent from RIMES, because the generator was trained on words from all Wikipedia.Examples absent from RIMES include olibrius, inventif, ionique, gorille, and ski.Source: C. Generation of handwritten text images · p. 5

D. Data augmentation for handwritten text recognition

IV. CONCLUSION

The paper presents a sequence-conditioned adversarial model for handwritten-word images, with realistic French and Arabic outputs and a slight error-rate reduction for the French model trained on combined data.

IV. CONCLUSION

  • The model combines recurrent word embeddings, an auxiliary recognition network, and balanced generator gradients from the discriminator and recognizer.The recognition network is intended to help generate images with legible text.Source: §IV · p. 6
  • Realistic word images are obtained in French and Arabic, with a slight error-rate reduction for the French model trained on combined data.Source: §IV · p. 6
  • The experiments do not yet cover more challenging datasets such as those with textured backgrounds; reducing style collapse and generating variable-size line images remain future directions.Source: §IV · p. 6

Related papers

Loading 1903.00277v1…