Source-linked AI summary
ScrabbleGAN: Semi-Supervised Varying Length Handwritten Text Generation
Sharon Fogel, Hadar Averbuch-Elor, Sarel Cohen, Shai Mazor, Roee Litman
TL;DR
Handwritten text recognition needs more varied labeled data, while annotation is costly and handwriting styles vary widely. ScrabbleGAN generates arbitrary-length handwritten text with controllable style and uses unlabeled data for adaptation. The paper reports improved HTR performance from synthetic data, with both style and lexicon contributing to gains.
Problem
Handwritten text recognition is limited by scarce, costly-to-label data and insufficiently versatile annotated handwriting.
Method
ScrabbleGAN uses a semi-supervised generative model with a fully convolutional character-wise generator for arbitrary-length handwritten text and controllable style.
Results
Synthetic data using both target-domain style and lexicon improves HTR, producing an additional 5% WER boost and NED better than the oracle in the reported CVL experiment.
Takeaways & Limitations
The approach can enrich HTR training data across handwriting styles and lexicons while adapting to unlabeled target-domain images.
Takeaways & Limitations
The generator gives characters the same receptive-field width, unlike scripts where letters such as ‘i’ are narrower than ‘w’.
Abstract
from arXiv · showhide
Optical character recognition (OCR) systems performance have improved significantly in the deep learning era. This is especially true for handwritten text recognition (HTR), where each author has a unique style, unlike printed text, where the variation is smaller by design. That said, deep learning based HTR is limited, as in every other task, by the number of training examples. Gathering data is a challenging and costly task, and even more so, the labeling task that follows, of which we focus here. One possible approach to reduce the burden of data annotation is semi-supervised learning. Semi supervised methods use, in addition to labeled data, some unlabeled samples to improve performance, compared to fully supervised ones. Consequently, such methods may adapt to unseen images during test time. We present ScrabbleGAN, a semi-supervised approach to synthesize handwritten text images that are versatile both in style and lexicon. ScrabbleGAN relies on a novel generative model which can generate images of words with an arbitrary length. We show how to operate our approach in a semi-supervised manner, enjoying the aforementioned benefits such as performance boost over state of the art supervised HTR. Furthermore, our generator can manipulate the resulting text style. This allows us to change, for instance, whether the text is cursive, or how thin is the pen stroke.
1. Introduction
ScrabbleGAN addresses the shortage of versatile annotated handwriting data by synthesizing realistic text with controllable style and arbitrary length, including unlabeled-data adaptation.
- ScrabbleGAN manipulates handwriting style while generating long text, including cursive examples such as a 34-letter word.
- Handwritten text recognition remains behind printed-text OCR because versatile annotated handwriting is difficult and costly to obtain.
- ScrabbleGAN introduces a fully convolutional architecture that generates arbitrarily long words or complete sentences without character-level annotation.
- The method trains its generator semi-supervisedly, enabling adaptation to unlabeled data, including test-time images.
- The authors report that training lexicon richness matters no less than style richness for HTR training.
2. Previous Work
Previous work spans online and offline handwriting generation, recognition, and synthetic augmentation, but ScrabbleGAN targets variable-length offline text and broader lexicon control.
- Handwriting recognition methods process either online pen-stroke data or offline rasterized images, with offline methods lacking stroke trajectories.
- Modern HTR commonly uses convolutional-recurrent architectures and related visual representations for recognition or word spotting.
- Earlier handwriting-generation methods synthesized online or author-specific data, often requiring recurrent models or time-consuming character-level annotation.
- Generative augmentation adds synthesized examples to training data, including methods that warp existing handwriting with parametric functions.
- Unlike ScrabbleGAN, the cited warping approach cannot generate words outside a given lexicon.
- An online generator with rendering cannot adapt to the versatility or typical noise of unseen offline datasets.
3. Method
ScrabbleGAN combines a fully convolutional, character-wise generator with adversarial and recognition objectives to produce readable handwriting whose style and length can vary.
- The discriminator promotes realistic handwriting style, while the recognizer promotes readable output matching the input text.
- 3.1. Fully convolutional generator: The generator assembles character-conditioned patches with overlapping receptive fields, allowing neighboring letters to interact and producing smooth transitions, including cursive writing.
- 3.1. Fully convolutional generator: Character filters are concatenated and multiplied by a noise vector, enabling generation of arbitrary-length words from repeated or distinct character filters.
- A fixed noise vector is reused across characters so the generated word or sentence retains a consistent style.
- The discriminator is convolutional with overlapping receptive fields, allowing training on unlabeled images without character-level class supervision.
- The generator balances adversarial and recognizer losses, with α controlling their relative importance and standard-deviation balancing avoiding a gradient-sign change.
4. Results
ScrabbleGAN generates variable-length handwritten text with controllable styles and improves HTR through synthetic-data augmentation and domain adaptation. Experiments also show that balancing recognition and adversarial objectives is important for readable, stylistically varied outputs.
- Comparison to Alonso et al. [2]: Compared with Alonso et al., ScrabbleGAN produces clearer images with fewer artifacts, especially for shorter words, but the reported FID and geometric-score comparison is not direct because the baseline data were unavailable.The qualitative comparison identifies fewer artifacts such as redundant or missing letters; baseline metrics were copied from the earlier paper.
- Generating different styles: Changing noise vector z generates cursive or non-cursive handwriting with bold or thin strokes while preserving the synthesized words across styles.Each row in the style examples uses a shared noise vector, while columns vary the word being synthesized.
- Generating different styles: Linear noise interpolation changes letter size, stroke width, and inter-letter connections gradually, while also reproducing IAM-like background variation.The interpolations connect two randomly selected handwriting styles in each column.
- Boosting HTR performance: ScrabbleGAN-generated samples significantly improve HTR performance over affine augmentation alone on RIMES and IAM.The augmentation experiment adds 100k synthetic images and compares original data, affine transformations, GAN augmentation, and refinement.
- Gradient balancing ablation study: With α = 1, gradient balancing produces the best RIMES HTR results among the tested recognition–adversarial loss configurations.The ablation varies no balancing, the prior scheme, and the proposed balancing approach across α values.
5. Conclusion and Future Work
ScrabbleGAN generates variable-length handwritten text with diverse styles and uses both labeled and unlabeled data to improve HTR training. The authors identify future work in author adaptation, finer style control, and character-specific widths.
- Conclusion: ScrabbleGAN generates offline handwritten text by assembling character images, supporting variable word lengths, diverse stroke widths, cursive and non-cursive styles, and style adaptation from unlabeled data.The approach assumes character writing is local and uses overlapping receptive fields to model interactions between neighboring characters.
- Conclusion: Generated words from a new lexicon are beneficial when combined with a new handwriting style.
- Future Work: Future work includes generative representation learning for few-shot author adaptation and disentanglement for finer control of cursiveness and pen width.
- Future Work: The generator currently gives every character the same receptive-field width, unlike scripts in which characters such as “i” and “w” differ in width.Proposed remedies include character-dependent filter widths or spatial transformer layers.
Supplementary Materials
The supplementary materials identify the paper’s authors and affiliations.
- Authors: The paper lists Sharon Fogel, Hadar Averbuch-Elor, Sarel Cohen, Shai Mazor, and Roee Litman as authors.
- Supplementary Materials: The author list includes affiliations marked by dagger and section symbols.
- Affiliations: The affiliations are Amazon Rekognition, Israel, and Cornell Tech, Cornell University.
A. Visual Results
The visual results show handwritten quotations and demonstrate ScrabbleGAN’s ability to generate complete sentences by varying output length.
- Visual Results: The visual examples use a quotation attributed to A.A. Milne’s Winnie-the-Pooh.
- Visual Results: ScrabbleGAN generates complete sentences, illustrating its varying-length generation capability.The quotations were originally single images and were split into several lines to fit one column.
- Visual Results: Figure 7 presents quotations about handwriting, with each example originally contained in one image.Some examples were split into multiple lines for column layout.
B. Ablation Study
The ablation study examines recognizer architecture and GAN noise conditioning choices for synthetic data generation. Simpler recognition networks performed best, while removing conditional batch normalization increased WER by about 1%.
- Ablation Study: Table 5 compares HTR performance across synthetic datasets generated with different generator and recognizer architectures.
- Recognizer architecture selection: A simpler recognizer without a recurrent subnetwork produced the best synthetic images for HTR training among the tested configurations.The authors report that the weakest of three alternatives performed best, and a seven-convolution feed-forward recognizer further improved results.
- GAN noise input selection: Removing conditional batch-normalization layers increased WER by about 1%, so the generator retained them.This ablation tested replacing conditional batch normalization with standard batch normalization.
C. Architecture Details
ScrabbleGAN uses BigGAN-based generator and discriminator modules with architecture changes for character-sized output patches, plus an RCNN-based recognizer without Bi-LSTM. The generator also conditions different image regions on different character classes.
- Generator and discriminator: The generator architecture is specified through a modified G arch helper, with channel settings described in Table 6.The table caption states that listed input and output channels are scaled by the default ch=64 parameter.
- Generator and discriminator: The discriminator architecture is specified through a modified D arch helper, with channel settings described in Table 7.The table caption notes that the first block receives the one-channel image, while later channel counts scale with the default ch=64 parameter.
- Generator and discriminator: The generator and discriminator are based on BigGAN, with architecture helpers modified to produce 16 × 32-pixel output patches per character.The Generator class was also adapted to support different width and height up-sampling parameters.
- Generator and discriminator: Unlike BigGAN’s image-level class conditioning, ScrabbleGAN conditions different image regions on different character classes.The spatial conditioning is imposed in the first layer, where characters do not overlap.
- Recognizer: The recognizer is based on an RCNN implementation, with the Bi-LSTM network removed following an ablation study.