Source-linked AI summary
Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition
Lei Kang, Pau Riba, Marçal Rusiñol, Alicia Fornés, Mauricio Villegas
TL;DR
Recurrent HTR models fit sequential text and language dependencies but hinder training parallelization and face memory limits on long sequences. This paper introduces a transformer architecture with multi-head self-attention at visual and textual stages, enabling non-recurrent, character-level recognition without a fixed vocabulary. The method achieves state-of-the-art performance and supports few-shot training scenarios.
Problem
Recurrent HTR architectures impose sequential processing that limits training parallelization and creates memory constraints for longer sequences.
Method
The paper uses multi-head self-attention at visual and textual stages in a non-recurrent, character-level transformer architecture.
Results
The method achieves new state-of-the-art performance on the public IAM dataset and competitive results with limited real annotated training data.
Takeaways & Limitations
Character-level decoding removes closed-vocabulary constraints, while the method's few-shot capability extends its applicability to real industrial use cases.
Abstract
from arXiv · showhide
The advent of recurrent neural networks for handwriting recognition marked an important milestone reaching impressive recognition accuracies despite the great variability that we observe across different writing styles. Sequential architectures are a perfect fit to model text lines, not only because of the inherent temporal aspect of text, but also to learn probability distributions over sequences of characters and words. However, using such recurrent paradigms comes at a cost at training stage, since their sequential pipelines prevent parallelization. In this work, we introduce a non-recurrent approach to recognize handwritten text by the use of transformer models. We propose a novel method that bypasses any recurrence. By using multi-head self-attention layers both at the visual and textual stages, we are able to tackle character recognition as well as to learn language-related dependencies of the character sequences to be decoded. Our model is unconstrained to any predefined vocabulary, being able to recognize out-of-vocabulary words, i.e. words that do not appear in the training vocabulary. We significantly advance over prior art and demonstrate that satisfactory recognition accuracies are yielded even in few-shot learning scenarios.
1. Introduction
Handwritten text recognition remains difficult because writing varies widely and conventional recurrent architectures limit training parallelization. The paper proposes a transformer-based, non-recurrent, character-level architecture that jointly recognizes characters and models language dependencies without a fixed vocabulary.
- Handwritten text recognition remains challenging because stroke variability and ambiguity differ substantially across writing styles.
- Sequential HTR models process text temporally and use recurrent states to handle variable-length signals, but recurrent attention pipelines restrict training parallelization and impose memory limits on longer sequences.
- The proposed architecture removes recurrence and applies multi-head self-attention at visual and textual stages for character recognition and language-dependency modeling.
- Character-level decoding avoids a predefined vocabulary, allowing recognition of out-of-vocabulary words unseen during training.
- Synthetic-data pre-training supports competitive recognition with limited real annotated data, while ablation and comparative experiments evaluate the approach.
2. Related Work
Prior HTR systems largely rely on sequential recurrent processing, whereas transformers offer attention-based parallelization and language modeling without recurrent steps. This paper positions transformer-based HTR as a previously unexplored direction for handwriting recognition.
- HMMs, BLSTMs, MDLSTMs, and encoder-decoder networks share a sequential paradigm that processes text lines through temporal model states.
- Transformers avoid recurrent steps through attention mechanisms, facilitating parallel training and supporting language modeling in vision and language tasks.
- Before this work, transformer architecture had not been used to tackle handwriting recognition, although related adaptations addressed text in natural-scene images.
- The paper proposes a unified transformer architecture that recognizes long character sequences and models language at character level without predefined lexicons.
3. Proposed Method
The proposed recognizer combines visual and textual processing with transformer-style self-attention, avoiding recurrent networks while decoding handwritten text at character level. A mutual-attention stage aligns image and language features for final character predictions.
- Overall architecture: The architecture combines a visual feature encoder with a character-level text transcriber in an end-to-end model.The visual encoder extracts image features, while the transcriber models language and decodes characters.
- Visual feature encoder: CNN features are reshaped into width-ordered visual vectors and augmented with sine-cosine positional encodings to preserve horizontal order without recurrence.The encoding makes otherwise shift-invariant visual representations sensitive to image position.
- Visual self-attention: Four eight-head self-attention blocks refine visual features by computing correlations among the position-indexed image representations.The visual attention uses the same feature representation as query, key, and value.
- Text transcriber: The text transcriber operates at character level, learning n-gram-like dependencies from transcription embeddings rather than relying on a predefined word vocabulary.Special start, end, and padding symbols extend transcriptions to a fixed maximum prediction length.
- Text transcriber: Masked transformer decoding exposes only preceding character positions during training, enabling concurrent decoding steps that replace recurrent iteration.At inference, greedy decoding feeds each predicted character back until the end symbol or maximum length is reached.
- Mutual attention: A final mutual-attention module uses textual representations as queries and visual representations as keys and values to align both information sources before prediction.The combined representation is passed through a linear layer and softmax to produce the final character sequence.
4. Experimental Evaluation
The evaluation uses the RWTH partition of the IAM handwritten English dataset and measures recognition with character- and word-level edit-error rates.
- Dataset: The RWTH IAM partition contains 6482 training, 976 validation, and 2914 test text lines.Images are resized to 64 pixels high while preserving aspect ratio, with batching padded to width 2227 pixels.
- Dataset: The dataset uses an alphabet of 83 symbols and a maximum output sequence length of 89 characters.The alphabet includes special symbols in addition to ordinary characters.
- Performance measures: Character Error Rate measures character substitutions, insertions, and deletions relative to the ground-truth character count.CER is based on Levenshtein distance.
- Performance measures: Word Error Rate measures word substitutions, insertions, and deletions relative to the ground-truth word count.WER applies the same edit-distance principle at the word level.
4.2. Implementation Details
The implementation uses repeated multi-head attention blocks with regularization and Adam optimization, including label smoothing to reduce over-confident predictions.
- Network configuration: The network uses feature size f = 1024 and four visual and language self-attention blocks.Each self-attention module has eight heads.
- Regularization and sequence setup: A dropout setting of 0.1 is applied to every dropout layer.Transcriptions include start and end symbols and are padded to length 89.
- Optimization: Label smoothing is used as regularization to prevent over-confident predictions.The ground-truth labels are initially one-hot vectors, and the smoothing value is set to 0.4.
- Optimization: Adam optimization starts with a learning rate of 2 · 10^-4, which is halved every 20 epochs.The implementation is based on PyTorch and runs on an NVIDIA Cluster.
4.3. Pre-training with Synthetic Data
The study uses line-level synthetic handwriting data for pre-training because public synthetic datasets are typically word-level and real annotated data are scarce. Ablations show that temporal encoding and language self-attention materially improve recognition, while qualitative attention maps align decoded characters with image regions.
- Pre-training with Synthetic Data: Line-level synthetic handwriting data are created from an English text corpus to compensate for scarce real annotated training data.Existing popular synthetic handwriting datasets are described as word-level, motivating line-level pre-training data.
- Function of Temporal Encoding: Temporal encoding at the text level reduces error from 7.72% to 4.86% without image-level encoding and from 6.33% to 5.52% with it.The best performance uses temporal encoding for both image and text representations.
- Role of Self-Attention Modules: Language self-attention improves performance from 7.71% to 4.86% without visual self-attention and from 7.78% to 4.89% with it.The visual self-attention module barely improves performance in this ablation.
- Role of Self-Attention Modules: The language self-attention module learns character-level language-modeling information from whole-line context while retaining out-of-vocabulary prediction.This contextual information is identified as the source of the module’s recognition benefit.
- Qualitative Analysis: Attention-map visualizations show successful alignment between decoded characters and corresponding image regions.The visualizations average mini attention maps across layers and heads.
4.5. Detailed Comparison with Seq2Seq Model
The paper compares its transformer recognizer with a reimplemented state-of-the-art recurrent sequence-to-sequence pipeline under identical training and evaluation conditions. The comparison reports training-time and recognition-error differences, with synthetic pre-training benefiting both models, especially the transformer.
- Comparison Setup: A state-of-the-art recurrent sequence-to-sequence recognizer is reimplemented and evaluated under the exact same circumstances as the proposed transformer.The recurrent baseline uses an encoder, decoder, attention mechanism, BGRUs, and a GRU decoder.
- Training Comparison: The transformer takes less time per training epoch than the sequence-to-sequence model despite having more parameters.The comparison reports model size and time taken per epoch.
- Synthetic Pre-training: Both models benefit from synthetic pre-training, with a noticeably larger reduction in final error rates for the transformer.The sequence-to-sequence model also improves, but its boost is described as less drastic.
4.6. Few-shot Training
Few-shot experiments vary the available real IAM training data to assess recognition under annotation scarcity. Performance generally improves with more real data, while synthetic pre-training makes the transformer particularly effective in few-shot settings.
- Few-shot Training: The few-shot study trains models with 20%, 40%, 60%, and 80% of the IAM training set to simulate limited real annotated data.The transformer is compared with a sequence-to-sequence approach.
- Few-shot Training: Both approaches improve as more real training data becomes available.This trend is reported for the line-level test set.
- Few-shot Training: The transformer generally outperforms the sequence-to-sequence model except when only 20% of real annotated training data is available.The larger transformer struggles under this extreme data-scarcity condition.
- Qualitative Analysis: Qualitative results include text-line transcriptions and attention-map visualizations.The supplied figure description identifies both recognition outputs and attention maps.
- Synthetic Pre-training: With synthetic pre-training, the transformer excels in few-shot settings.Figure 4 provides qualitative examples from models trained with reduced training sets, all pre-trained on synthetic data.
4.7. Language Modelling Abilities
The paper tests whether its recognizer already captures language-specific knowledge by adding a separate recurrent language model as post-processing. The post-processing does not improve performance, which the authors interpret as evidence that language context is already incorporated through language self-attention.
- Language Modelling Abilities: A shallow-fusion language model with 2,400 LSTM units, trained on 130,000 English text-lines, is evaluated as post-processing.Its additive fusion weight is set to 0.2.
- Language Modelling Abilities: The language-model post-processing is useless, indicating that the proposed recognizer already incorporates language-specific contextual information.The authors attribute this information to the language self-attention module.
4.8. Comparison with the State-Of-The-Art
The paper compares handwritten text-recognition approaches across established architectural families and reports that its method achieves the strongest result among approaches without a closed lexicon.
- The comparison groups prior methods into HMM or early-neural, recurrent-CTC, and encoder-decoder sequence-to-sequence families.
- Table 7 reports results on the IAM line-level dataset across state-of-the-art approaches.
- The proposed method obtains the best result among methods without using a closed lexicon while competing with most closed-vocabulary approaches.
5. Conclusion
The paper concludes that its transformer-based recognizer provides a non-recurrent, open-vocabulary approach to handwritten text-line recognition. It reports state-of-the-art performance, few-shot capability, and character-level integration of visual and language knowledge.
- The proposed method is non-recurrent and open-vocabulary, and is presented as the first transformer approach for handwritten text recognition.
- The evaluations demonstrate state-of-the-art performance and capability in few-shot training scenarios.
- Character-level design avoids closed-vocabulary constraints while combining visual and language-specific learned knowledge.