Source-linked AI summary
Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition
Théodore Bluche
TL;DR
Offline handwriting recognition depends on costly and error-prone line segmentation. The paper replaces the standard MDLSTM-RNN collapse layer with recurrent attention that transcribes one line at a time, achieving competitive results on Rimes and IAM while leaving automatic step-count prediction unresolved.
Problem
Handwriting recognition systems require cropped, line-level images and segmentation, whose annotation is costly and whose automatic errors can compromise recognition.
Method
The model uses an attention-based recurrent replacement for the MDLSTM-RNN collapse layer, iteratively focusing on and transcribing individual paragraph text lines.
Results
Results on Rimes and IAM paragraphs are competitive with state-of-the-art systems using ground-truth line segmentation.
Takeaways & Limitations
Line segmentation can be learned inside the transcription model, advancing end-to-end handwriting recognition toward full-page processing.
Takeaways & Limitations
The decoder predicts a fixed experimenter-defined number of line sequences because it has no stop token and cannot automatically determine when to stop.
Abstract
from arXiv · showhide
Offline handwriting recognition systems require cropped text line images for both training and recognition. On the one hand, the annotation of position and transcript at line level is costly to obtain. On the other hand, automatic line segmentation algorithms are prone to errors, compromising the subsequent recognition. In this paper, we propose a modification of the popular and efficient multi-dimensional long short-term memory recurrent neural networks (MDLSTM-RNNs) to enable end-to-end processing of handwritten paragraphs. More particularly, we replace the collapse layer transforming the two-dimensional representation into a sequence of predictions by a recurrent version which can recognize one line at a time. In the proposed model, a neural network performs a kind of implicit line segmentation by computing attention weights on the image representation. The experiments on paragraphs of Rimes and IAM database yield results that are competitive with those of networks trained at line level, and constitute a significant step towards end-to-end transcription of full documents.
1 Introduction
Handwriting recognition systems still require segmented text lines, while this paper proposes an attention-based MDLSTM-RNN that transcribes paragraphs line by line without explicit line segmentation. Experiments on Rimes and IAM produce results competitive with systems using ground-truth line segmentation.
- Current handwriting recognition models require segmented text lines, making automatic line segmentation a crucial part of full-document processing.
- The proposed model replaces the MDLSTM-RNN collapse layer with an attention network that iteratively focuses on and transcribes individual text lines.Attention weights modulate the importance of different image positions during successive decoding steps.
- Iterative application of the attention-based collapse enables purely segmentation-free recognition of full handwritten paragraphs.
- Experiments on Rimes and IAM paragraphs report results competitive with state-of-the-art systems using ground-truth line segmentation.
2 Related Work
The paper builds on attention-based methods that implicitly localize relevant image regions, while addressing the inefficiency of predicting one character at a time for long handwritten paragraphs. Its decoder instead represents and processes complete text lines.
- The model integrates line segmentation into the neural network, sharing intermediate features between segmentation and transcription models trained jointly on transcription error.
- Attention-based models iteratively select relevant parts of encoded signals for successive predictions, implicitly performing localization inside the neural network.This paradigm has been applied to machine translation, image captioning, speech recognition, and scene text recognition.
- Prior paragraph-recognition attention models predicted each character in turn, but full paragraphs typically contain about 500 characters, making this costly in memory and time.
- The proposed system encodes the image as sequences of vectors representing full text lines rather than summarizing it as one vector at each timestep.
- This line-level iterative design provides a huge speedup while returning to the MDLSTM-RNN architecture with an augmented collapse layer.
3 Handwriting Recognition with MDLSTM and CTC
MDLSTM-RNNs transform two-dimensional handwriting images into prediction sequences through multidirectional recurrent and convolutional layers followed by vertical collapse. CTC then maps these predictions to text while accommodating alternative labelings.
- MDLSTM layers generalize LSTMs to two-dimensional inputs by scanning in four directions, with each layer followed by convolution.
- Convolutional strides reduce representation resolution while increasing the number of extracted features, producing one top-level feature map for each character.
- A non-character label and CTC mapping convert T predictions into N labels while considering all possible sequence labelings during training.
- The collapse layer converts two-dimensional feature maps into height-one vector sequences by summing across the vertical dimension.
- Vertical collapse reduces all information at a given horizontal position to one vector regardless of vertical location, preventing recognition of multiple lines in this framework.
4 An Iterative Weighted Collapse for End-to-End Handwriting Recognition
The model replaces MDLSTM-RNNs’ simple vertical collapse with a recurrent, attention-based weighted collapse that recognizes one text line per timestep. It supports paragraph-level CTC training and implicitly segments lines while transcribing, but requires a fixed number of steps and remains constrained by whole-width outputs.
- 4.2 Attention: The weighted collapse replaces the standard vertical sum with neural-network-predicted weights that focus on specific image regions.The weights are computed recurrently at each timestep, enabling recognition of a text line at each timestep.
- 4.2 Attention: The attention computation depends on both the encoded image and previous attention features, with softmax normalization applied to each column.This produces an attention map whose weights define the weighted feature aggregation at each timestep.
- 4.2 Attention: The attention module repeatedly converts encoded image features into feature-vector sequences intended to represent individual text lines.Because it shares encoder features with transcription and is trained for transcription accuracy, it functions as a soft line-segmentation network.
- 4.3 Decoder: The decoder predicts character sequences from collapsed feature vectors and may use a BLSTM to model dependencies across the paragraph’s text lines.The decoder can process the concatenation of all collapsing steps, while alternative decoding applies it to sub-sequences and concatenates the outputs.
- 4.4 Training: CTC can train either independently on line segments with line transcripts or on the concatenation of all line predictions using paragraph-level transcripts.The experiments mainly use paragraph-level CTC with a BLSTM decoder over all collapsing steps.
- 4.5 Limitations: The iterative decoder is 20-30 times faster than the earlier character-step model but cannot infer the required number of steps automatically.The experiment fixes T to the dataset’s maximum number of lines; the collapsing paradigm also forces outputs across the full image width, complicating complex layouts.
5 Experiments
Experiments on IAM and Rimes evaluate the model’s architecture, implicit line segmentation, and comparison with published systems. The attention-based paragraph model improves over the baseline and remains competitive with systems using explicit segmentation.
- Experimental setup: The experiments use IAM and Rimes paragraph databases, with the network architecture combining an MDLSTM encoder, attention network, and BLSTM decoder.The encoder uses different output sizes for IAM and Rimes, while the attention network computes softmax weights over columns.
- Impact of the decoder: 20% relative improvement on IAM and 50% on Rimes: the proposed model outperforms the baseline in validation CER.The reported gains may be attributed to both the BLSTM decoder and the attention mechanism.
- Impact of line segmentation: Automatic line segmentation increases baseline error rates by an absolute 1% in the best case, whereas the proposed system performs paragraph transcription without explicit segmentation.The comparison includes ground-truth positions, three automatic segmentation algorithms, and the end-to-end attention model at multiple resolutions.
- Impact of line segmentation: The model performs better than explicit automatic-segmentation methods and achieves comparable or better results than ground-truth segmentation, even at half resolution.Figure 4 visualizes attention weights as implicit line segmentation and colors transcription chunks by attention timestep.
- Comparison to published results: On Rimes, 150 dpi results outperform the state of the art in CER% while remaining competitive in WER%, and 300 dpi is comparable to the best single system in WER%.The 300 dpi system has significantly better CER% than that best single system.
- Comparison to published results: On IAM, the language model is important, while punctuation is often missed by the attention mechanism.At 300 dpi, WER% improves little but CER% is lower; published comparisons use explicit ground-truth line segmentation and language models.
6 Discussion
The model handles paragraph transcription without explicit line segmentation but remains constrained by reading order, stopping, document layout, and training alignment.
- 6 Discussion: The model cannot handle arbitrary reading orders because it implements implicit line segmentation rather than general document reading.Its iterative collapse uses a fixed number of timesteps, although it can handle variable numbers of text lines.
- 6 Discussion: Full-document transcription requires prior document layout analysis to detect paragraph regions.The current system is inherently limited to paragraph images because weighted collapse spans the encoded image width.
- 6 Discussion: Variable-sized line crops remain unavailable because line size is determined by the text-block size.Suggested directions include spatial transformers, hierarchical attention, or reinforcement-learning-based crop prediction.
- 6 Discussion: Training complex documents is harder because reading order may be difficult to infer and prediction-to-ground-truth matching must be addressed.The issue arises in addition to the complexity of the task itself.
- 6 Discussion: CTC convergence is much faster with a pretrained encoder.A recommended procedure is to train an MDLSTM-RNN with standard collapse on text lines, then fine-tune with attention-based collapse on paragraphs.
7 Conclusion
The paper learns implicit line segmentation within an MDLSTM transcription model, enabling paragraph transcription without explicit segmentation. It reports comparable error rates to the state of the art on two public databases and identifies full-page recognition as the next step.
- 7 Conclusion: The model transcribes full handwritten paragraphs without explicit line segmentation.It replaces the standard collapse layer with an iterative mechanism that focuses on individual text lines.
- 7 Conclusion: Implicit line segmentation is learned jointly with the network through backpropagation by minimizing paragraph-level CTC error.The segmentation mechanism is integrated into the transcription model rather than handled as a separate preprocessing stage.
- 7 Conclusion: Comparable error rates to the state of the art were reported on two public databases.The conclusion presents line segmentation learning as a step toward end-to-end handwriting recognition at full-page level.