Source-linked AI summary
Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention
Théodore Bluche, Jérôme Louradour, Ronaldo Messina
TL;DR
Existing handwriting systems require line-segmented inputs, limiting end-to-end paragraph recognition. This paper uses an MDLSTM attention encoder-decoder to transcribe multi-line text without explicit segmentation, with encouraging feasibility results but substantial computational cost.
Problem
Current handwriting-recognition systems require segmented text lines, although real-world documents rarely provide them readily.
Method
The model encodes a two-dimensional text image with MDLSTM features and uses an attention-based sequential decoder whose attention is implemented by an MDLSTM network.
Results
The results show that full-paragraph recognition without explicit line segmentation is feasible, including preliminary right-to-left and bidirectional reading-order behavior for Arabic and mixed Arabic/Latin inputs.
Takeaways & Limitations
The work supports end-to-end handwriting recognition without explicit segmentation and suggests that attention could combine document layout analysis with text recognition.
Takeaways & Limitations
The system is very slow, with attention computed for each character in turn, and its time and memory consumption are prohibitive for most industrial applications.
Abstract
from arXiv · showhide
We present an attention-based model for end-to-end handwriting recognition. Our system does not require any segmentation of the input paragraph. The model is inspired by the differentiable attention models presented recently for speech recognition, image captioning or translation. The main difference is the covert and overt attention, implemented as a multi-dimensional LSTM network. Our principal contribution towards handwriting recognition lies in the automatic transcription without a prior segmentation into lines, which was crucial in previous approaches. To the best of our knowledge this is the first successful attempt of end-to-end multi-line handwriting recognition. We carried out experiments on the well-known IAM Database. The results are encouraging and bring hope to perform full paragraph transcription in the near future.
1 Introduction
Offline handwriting recognition maps variable-sized two-dimensional images to character sequences, but cursive writing and multi-line layouts make segmentation difficult. The paper proposes attention-based recognition that avoids explicit line segmentation and handles complex reading orders.
- Offline handwriting recognition converts variable-sized two-dimensional images into character sequences, with cursive writing making character-level segmentation difficult.
- MDLSTM-RNNs combined with CTC became state-of-the-art for handwriting recognition because they handle two-dimensional inputs and sequential predictions.
- Existing systems require segmented text lines, so practical document transcription needs automatic line-segmentation algorithms.
- Multi-line recognition requires attention to select image content in order while encapsulating left-to-right and top-to-bottom reading orders.
- The proposed system decouples sequence generation from information extraction, allowing variable prediction counts and iterative selection of input regions without assuming reading order.
2 Handwriting Recognition with MDLSTM and CTC
MDLSTM-RNNs process two-dimensional handwriting through directional recurrent layers and subsampling convolutions, then collapse spatial features for character prediction. The paper replaces this collapse/CTC sequence mechanism with an attention-based decoder to extend recognition beyond single lines.
- MDLSTM-RNNs generalize LSTM architectures to multidimensional inputs using four layers that scan the image in different directions.
- At each position, the MDLSTM recurrence uses the input feature vector together with horizontal and vertical predecessor states, depending on scan direction.
- Subsampling convolutions follow each LSTM layer, increasing feature count as spatial maps shrink.
- A collapsing layer sums features vertically to produce prediction vectors, which softmax normalizes into character predictions.
- CTC uses a blank label and considers possible sequence labelings to map T predictions into a final sequence of N ≤ T labels.
- The proposed model replaces collapse and CTC with an attention-based decoder that directly predicts character sequences spanning potentially several lines.
3 An Attention-Based Model for End-to-End Handwriting Recognition
The proposed model encodes a two-dimensional text image and uses MDLSTM attention to sequentially select feature information for character prediction. This attention-based decoder replaces the collapse and CTC framework, enabling direct prediction of character sequences spanning multiple lines.
- The model combines a 2D image encoder with a sequential decoder that predicts characters from attention-weighted feature summaries.At each timestep, the attention network weights encoded feature vectors into a single summary used to update the state and predict the next character.
- The architecture is fully differentiable and is trained by backpropagation using the negative log-likelihood of the correct transcription.The target is a character sequence, and the network outputs its conditional probabilities given the input image.
- Replacing collapse and CTC with an attention-based decoder allows the model to predict character sequences potentially spanning several lines.This change moves beyond the single-line recognition paradigm while retaining the MDLSTM encoder for high-level image features.
- MDLSTM attention computes weights from encoded features, the previous attention map, and the preceding state vector.The MDLSTM can track both content and position, allowing attention to depend on context across the image.
- The attention summary updates the state LSTM, while the decoder uses the updated state and image summary to output character probabilities.An EOS token marks the end of the predicted sequence, and the decoder is a one-hidden-layer multilayer perceptron with softmax output.
4 Related Work
The model builds on attention-based encoder–decoder systems for translation, image captioning, and speech recognition while adapting recurrence to two-dimensional handwriting images. Its distinguishing aim is multi-line cursive-text recognition without explicit line segmentation.
- The approach couples an input encoder, a language-token decoder, and attention that selects relevant encoded signal parts for each prediction.
- Unlike speech-recognition systems using bidirectional LSTMs, this model uses MDLSTM units to handle images and long character sequences.
- The model weights and sums feature vectors instead of interpolating or explicitly predicting attention coordinates.
- Its attention combines whole-image content with location information represented implicitly through MDLSTM recurrences.
- The paper presents what the authors believe is the first attempt to recognize multiple lines of cursive text without explicit line segmentation.
5 Experiments
Experiments on IAM evaluated attention-based recognition from multi-word inputs through multiple lines and paragraphs. Paragraph training used truncated backpropagation, curriculum learning, and augmented line sequences to address memory and alignment challenges.
- Experimental setup: The experiments used the IAM database, with 747 training documents, 116 validation documents, and 336 test documents.The corresponding splits contain 6,482, 976, and 2,915 lines, respectively.
- Experimental setup: The attention model retained the baseline encoder while removing its collapse and softmax layers and adding attention and state LSTM components.The attention network used 16 or 32 hidden units in each direction, while the state LSTM used 128 or 256 units.
- Word and line recognition: For multi-word and single-line recognition, longer inputs generally performed better, while complete-line results remained close to baseline performance.These experiments are reported as CER percentages in Table 1.
- Learning line breaks: For two-line recognition, the model learned to shift attention from the first line toward the second in the required reading order.The attention could split between the end of the first line and the beginning of the second line.
- Paragraph recognition: Paragraph training addressed memory and alignment by truncating decoder backpropagation to 30 steps, adding lines and successive-line concatenations, and applying curriculum learning.The curriculum progressively increased target lengths and recognized additional lines over training epochs.
- Paragraph recognition: After twelve epochs on 150 dpi images, paragraph transcription was promising, although attention sometimes returned to earlier lines and produced insertion rates that could push CER above 100%.CERs were evaluated for sub-paragraphs of one to twelve lines and complete paragraphs.
6 Discussion
The model demonstrates feasible full-paragraph transcription without explicit line segmentation, while preliminary tests extend its behavior to Arabic and scene digits. The approach remains constrained by decoder design and prohibitive computational costs.
- Discussion: Full-paragraph recognition without explicit line segmentation is feasible, supporting future end-to-end recognition of complex documents.The authors suggest attention could combine document layout analysis and text recognition.
- Discussion: The model reads Arabic text from right to left and handles bidirectional Arabic/Latin reading order in several images.These were preliminary experiments without cropping, rescaling, or preprocessing.
- Discussion: For SVHN, the model finds digits in scene images.This experiment also used images without cropping, rescaling, or preprocessing.
- Discussion: The decoder is not conditioned on the previous character, making language-model integration more complicated than in classical CTC-trained models.The authors identify this as an issue for future work.
- Discussion: Memory consumption and runtime are prohibitive for most industrial applications because attention is computed separately for each character.The authors describe the system as very slow and computationally expensive.
7 Conclusion
The paper presents an MDLSTM attention model that transcribes complete paragraphs without explicit line segmentation and learns reading orders across multiple lines. The results support explicit segmentation as unnecessary, while the architecture retains an important decoder limitation.
- Conclusion: The method transcribes complete paragraphs without explicit line segmentation using MDLSTM-RNNs and an attention-based model.The proposed system recognizes multiple lines and learns encapsulated reading orders.
- Conclusion: The decoder is not conditioned on the previous token, unlike similar models.Future work proposes adding this architectural modification to enable beam-search decoding.
- Conclusion: The results show that explicit segmentation is not necessary for end-to-end handwriting recognition.The authors describe this as an important contribution toward end-to-end recognition.