Source-linked AI summary
Fast Multi-language LSTM-based Online Handwriting Recognition
Victor Carbune, Pedro Gonnet, Thomas Deselaers, Henry A. Rowley, Alexander Daryin, Marcos Calvo, Li-Lun Wang, Daniel Keysers, Sandro Feuz, Philippe Gervais
TL;DR
The paper addresses online handwriting recognition across many languages and scripts. It introduces an end-to-end neural recognizer with Bézier-curve inputs and script-level models combined with language-specific decoding. The system improves relative error rates by 20%–40% depending on language, uses smaller and faster models, and improves over the previous best published IAM-OnDB result.
Problem
Online handwriting recognition must support users whose languages and scripts are difficult to represent with intuitive keyboard layouts, including languages with large alphabets or grapheme clusters.
Method
The system uses an end-to-end neural network with Bézier-curve input representations, script-specific recognition models, and language-specific knowledge during CTC decoding.
Results
20%–40% relative error-rate improvement is reported depending on language, with smaller and faster models and improved performance over the previous best published IAM-OnDB result.
Takeaways & Limitations
A single script-level recognition model can serve multiple languages when combined with language-specific decoding knowledge.
Takeaways & Limitations
The reported Vietnamese competition comparison is limited because the system was not trained on the competition training dataset, so comparable performance is inferred rather than directly measured.
Abstract
from arXiv · showhide
We describe an online handwriting system that is able to support 102 languages using a deep neural network architecture. This new system has completely replaced our previous Segment-and-Decode-based system and reduced the error rate by 20%-40% relative for most languages. Further, we report new state-of-the-art results on IAM-OnDB for both the open and closed dataset setting. The system combines methods from sequence recognition with a new input encoding using Bézier curves. This leads to up to 10x faster recognition times compared to our previous system. Through a series of experiments we determine the optimal configuration of our models and report the results of our setup on a number of additional public datasets.
1 Introduction
The paper presents an end-to-end deep-learning online handwriting recognizer designed for many languages and scripts, replacing a heuristic-heavy segment-and-decode system. It reports broader support, improved accuracy, faster and smaller models, Bézier-curve inputs, and extensive evaluations.
- System overview: The system supports 102 languages across multiple scripts using a deep-learning recognition architecture.It trains separate models per script and combines them with language-specific language models and feature functions during decoding.
- System overview: The new recognizer replaces a segment-and-decode pipeline that relied on preprocessing, segmentation, feature extraction, and heuristic character-hypothesis construction.The new architecture uses bidirectional LSTMs, a logits layer, and CTC loss instead of those manually engineered stages.
- Evaluation: The authors provide detailed architecture tuning, comparisons with the previous stack, ablations, and results across supported languages and public datasets.The experiments include comparisons of recognition accuracy for eight languages and analyses of individual system components.
- Input representation: Bézier-curve interpolation produces shorter input sequences, enabling faster recognition while maintaining accuracy relative to raw touch inputs.The resulting curve sequences are roughly 4× shorter than corresponding raw representations.
- Evaluation: The paper reports new state-of-the-art results on IAM-OnDB for both open and closed training-set settings.It also compares performance on publicly available datasets and introduces an IBM-UB-1 evaluation protocol.
2 End-to-end Model Architecture
The system replaces explicit segmentation and handcrafted feature pipelines with end-to-end bidirectional LSTM models trained with CTC. It supports raw touch-point and Bézier-curve inputs, with compact curve sequences improving recognition latency and one script model serving multiple languages.
- Model architecture: The model processes input sequences with bidirectional LSTM layers and produces per-time-step character probabilities through a final softmax layer.The architecture is trained end-to-end using CTC and decoded with language-specific knowledge sources.
- Model architecture: The end-to-end structure removes explicit feature extraction, segmentation, stroke reordering, and character-hypothesis construction from the previous system.These heuristics are instead learned implicitly from training data.
- Raw Touch Points: Raw inputs are represented as 5-dimensional touch-point sequences containing coordinates, timestamps, pen state, and stroke-start indicators.The representation can use coordinate and time differences, with minimal normalization and resampling for variation across writing surfaces and devices.
- Bézier Curves: Bézier curves encode stroke trajectories as sequences of 10-dimensional curve representations derived from cubic polynomials in the curve parameter s.Curve coefficients are fitted by minimizing squared error to observed points, with alternating coefficient and correspondence updates until convergence.
- Bézier Curves: 4× shorter sequences from Bézier representations lead to faster recognition and better latency, while usually having little impact on accuracy.The curves compactly represent multiple points and are roughly four times shorter than the corresponding raw representation.
- Decoding: One recognition model per script can serve multiple languages by combining softmax outputs with language-specific language models, character information, and learned decoder weights.This combination guides beam-search CTC decoding and supports languages that share a script.
3 Training
Training proceeds in two stages: end-to-end CTC neural-network learning on a large dataset, followed by decoder-weight tuning on a separate smaller dataset. Decoder weights are optimized with Bayesian optimization, with transfer across languages within each script.
- Training procedure: Two-stage training combines end-to-end CTC model learning with separate decoder-weight tuning.The neural network is trained on a large dataset, while decoder weights are tuned on a smaller, distinct dataset.
- Training procedure: Separate tuning data reduces bias toward the neural network’s implicitly learned language model.Using the same data for decoder-weight learning could produce weights biased toward the neural network model because it is overconfident on its training data.
- Neural-network training: CTC trains without frame-aligned labels by treating input-label alignment as hidden and introducing a blank label.The blank label supports joint learning of alignments and character hypotheses.
- Decoder optimization: Decoder weights are optimized with batched Gaussian-process bandits using expected improvement in Google Vizier.Each recognizer uses seven studies of 500 trials, and additional trials did not improve results.
- Multilingual training: A separate neural network model is trained for each script, while language-specific models and feature functions support multiple languages per script.Decoder weights are trained on selected languages with sufficient data and transferred to other languages within the script.
4 Experimental Evaluation
The evaluation compares the new system with prior methods across public and internal datasets, training settings, architectures, inputs, and decoding configurations. It reports state-of-the-art IAM-OnDB results, substantial IBM-UB-1 gains, competitive cross-dataset performance, and diminishing returns from larger networks.
- Evaluation scope: The evaluation covers closed and open public-dataset settings, internal data, prior systems, and state-of-the-art comparisons.The experiments include IAM-OnDB, IBM-UB-1, Chinese, Vietnamese, and internal datasets.
- Architecture and inputs: Three or five LSTM layers outperform shallower networks, while additional layers provide little improvement; 64 nodes per layer is sufficient for good accuracy.The study compares raw and curve inputs with and without tuned language-model and character-class feature weights.
- IAM-OnDB: IAM-OnDB results establish a new state of the art for both closed-data training and production-style in-house training.The in-house model was not tuned for IAM-OnDB and used none of its data for training.
- IBM-UB-1: 38% lower CER and 32% lower WER are reported on IBM-UB-1 relative to the previous segment-and-decode approach.The comparison concerns the IBM-UB-1 test set and contrasts the current system with the previous system.
- Cross-dataset evaluation: On CASIA data, the system remains competitive despite supporting almost four times as many characters and lacking task-specific tuning.Only two of 224,590 test samples produced more than one character and were counted as errors.
- Internal and multilingual evaluation: 54% relative CER improvement is reported on internal Vietnamese data, decreasing CER from 7.2% to 3.3% versus the old system.The new system’s CER is reported as 3.3%, compared with 7.2% for the previous Segment-and-Decode system.
- Internal-data tuning: Increasing network depth and width yields diminishing returns quickly on internal data, with less pronounced overfitting than on smaller public datasets.The reduced overfitting is attributed in the passage to substantially larger internal datasets.
5 System Performance and Discussion
The study evaluates model configurations, ablations, and dataset transfer across handwriting benchmarks. The new architecture delivers 20%–40% relative error-rate improvement over almost all evaluated languages, with dataset characteristics strongly affecting cross-dataset performance.
- Model configuration: Four–five bidirectional LSTM layers with Bézier spline input processing formed the best-performing setup, selected separately for each script.The final softmax layer followed the recurrent stack.
- Ablation results: The largest relative improvement came from the overall network architecture, followed by the character language model and other feature functions.
- Accuracy results: 20%–40% relative error-rate improvement was achieved over almost all languages with evaluation datasets containing more than 2 000 items.
- Dataset comparison: Cross-dataset evaluation trained separate recognizers on IAM-OnDB, IBM-UB-1, and the internal Latin dataset, using Bézier-processed inputs and a fixed five-layer bidirectional-LSTM architecture.
- Dataset comparison: Dataset mismatch affected transfer: IBM-UB-1 models could not predict spaces, while IAM-OnDB’s printed writing differed from IBM-UB-1’s predominantly cursive writing.
- Dataset comparison: Models trained only on IBM-UB-1 or IAM-OnDB performed significantly worse on the internal dataset, whose distribution includes varied handwriting and input conditions.The internal data includes sloppy handwriting, overlapping characters, nonuniform sampling rates, and partially rotated inputs.
6 Conclusion
The paper presents a deep-learning online handwriting recognizer deployed for 102 languages in 26 scripts, replacing the previous Segment-and-Decode system. It improves accuracy by 20% to 40% relative depending on language, while using smaller and faster models, and reports improved results on public benchmarks.
- The deployed system supports 102 languages in 26 scripts and replaces the previous Segment-and-Decode system.
- 20% to 40% relative accuracy improvement was obtained depending on the language, with smaller and faster models.
- The system improved over the previous best published result on IAM-OnDB while also being compared on IBM-UB-1 and CASIA.