Source-linked AI summary
Dropout improves Recurrent Neural Networks for Handwriting Recognition
Vu Pham, Théodore Bluche, Christopher Kermorvant, Jérôme Louradour
TL;DR
Unconstrained handwriting recognition needs models that capture long sequence context from images, but dropout had not previously been applied to RNNs. The paper applies dropout only to feed-forward connections, preserving recurrent connections, and evaluates the approach across multiple handwriting datasets. Dropout improves recognition error rates across the reported settings, with larger relative gains when applied at multiple LSTM layers.
Problem
Handwriting recognition requires modeling long text sequences from images, while dropout had not previously been applied to RNNs.
Method
The paper applies dropout at selected feed-forward connections in a deep LSTM-based RNN while leaving recurrent connections untouched.
Results
Dropout improves error rates across the reported handwriting experiments, with relative CER and WER reductions of almost 30-40% when applied at multiple LSTM layers.
Takeaways & Limitations
Dropout is effective in deep handwriting-recognition architectures containing recurrent and shared connections.
Abstract
from arXiv · showhide
Recurrent neural networks (RNNs) with Long Short-Term memory cells currently hold the best known results in unconstrained handwriting recognition. We show that their performance can be greatly improved using dropout - a recently proposed regularization method for deep architectures. While previous works showed that dropout gave superior performance in the context of convolutional networks, it had never been applied to RNNs. In our approach, dropout is carefully used in the network so that it does not affect the recurrent connections, hence the power of RNNs in modeling sequence is preserved. Extensive experiments on a broad range of handwritten databases confirm the effectiveness of dropout on deep architectures even when the network mainly consists of recurrent and shared connections.
I. INTRODUCTION
Unconstrained handwriting recognition requires modeling long text sequences from images, motivating recurrent architectures that capture character context. LSTM-enhanced RNNs achieve strong recognition performance, and this work adapts dropout to regularize them without disrupting recurrent sequence modeling.
- Recognition problem: Unconstrained offline handwriting recognition maps image-only lines of text to recognized sequences in a given language.Preprocessing typically extracts snippets containing one word or line before recognition.
- Recognition problem: HMMs struggle with long-term dependencies because their hidden states follow a first-order Markov chain.An HMM with n hidden states can typically carry only log (n) bits about its dynamics.
- RNN motivation: RNN recurrent connections can store past input events as activations, enabling modeling of long sequences with complex structures.Their depth in time and multiple layers make parameter optimization difficult.
- RNN motivation: LSTM-enhanced RNNs achieved superior sequence-modeling performance and currently held the best known handwriting-recognition results.LSTM cells use carefully designed recurrent neurons for sequence modeling.
- Paper contribution: Dropout had been effective in deep convolutional and feed-forward networks but had not been applied to RNNs in prior work.The paper applies it at selected layers while leaving recurrent connections unaffected.
II. RECURRENT NEURAL NETWORKS FOR HANDWRITING RECOGNITION
The paper uses a deep handwriting-recognition architecture built from multidirectional LSTM and convolutional layers, with CTC producing trainable sequence recognition without explicit alignment. This architecture had previously served as the optical model in a winning OpenHaRT 2013 system.
- Architecture: The architecture feeds 2 × 2 image blocks into four LSTM layers scanning in different directions.Each LSTM output is separately processed by convolutional layers, whose activations are summed element-wise.
- Architecture: The network repeats convolutional processing with different filter sizes and feature counts, then uses a fully-connected top layer and softmax output.The final activations are summed vertically before the softmax layer.
- Key components: Multidirectional LSTM layers provide access to context from multiple scanning directions while storing information over long periods.Four LSTM layers operate in parallel, each with a particular scanning direction.
- Key components: CTC computes sequence negative log-likelihood so the full architecture can be trained without explicitly aligning input images with target sequences.This removes the need for manually specified input-target alignment.
- Prior system: The architecture was used as the optical model in the authors’ winning OpenHaRT 2013 Arabic handwriting-recognition system.The paper further improves that optical model using dropout.
III. DROPOUT FOR RECURRENT NEURAL NETWORKS
Dropout is adapted to deep RNNs by removing hidden activations during training while preserving recurrent connections. The approach also applies to convolutional stages and uses the conventional p = 0.5 rate effectively in the evaluated architecture.
- Dropout: Dropout randomly removes hidden units during training and retains all units during testing with activations weighted by p.The common dropout probability is p = 0.5.
- RNN adaptation: The proposed RNN implementation applies dropout only to feed-forward connections, leaving recurrent connections untouched.This design is intended to conserve the RNN’s ability to model sequences.
- RNN adaptation: Dropout can be inserted as a separate layer at any stage in the deep architecture, increasing design flexibility.The layer outputs its input except at locations selected for dropping.
- Comparison: Compared with DropConnect in convolutional layers, dropping convolutional inputs samples from a larger pool of models because inputs typically outnumber shared weights.DropConnect drops connections, whereas this approach drops hidden-unit inputs.
- Empirical design: The typical dropout rate p = 0.5 yielded superior performance in this deep architecture despite prior concerns that it could slow convergence or increase error.The authors attribute the improvement to keeping recurrent connections untouched.
A. Experimental setup
The experiments evaluate the system on three handwritten-language datasets containing isolated words and lines. Performance is measured with CER and WER under a fixed training and evaluation protocol using dropout as the only additional regularizer.
- Datasets: The study evaluates Rimes, IAM, and OpenHaRT, containing French, English, and Arabic handwriting respectively.The datasets are split into disjoint training, validation, and evaluation subsets.
- Datasets: The evaluated images contain isolated words or isolated lines scanned at or scaled to 300 dpi.The network architecture is designed for this resolution.
- Metrics: CER normalizes character edit distance, while WER is classification error for isolated words and normalized word-sequence edit distance for lines.Both metrics assess recognition errors at different sequence levels.
- Training: Models are trained with online stochastic gradient descent at learning rate 10^-3 using CTC negative log-likelihood.Weights are initialized from a zero-mean Gaussian with standard deviation 10^-2, and early stopping is used.
- Training: Dropout is the only additional regularization method, and enabled experiments consistently use probability p = 0.5.The model with highest validation performance is selected for test evaluation.
1) Dropout at the topmost LSTM layer:
Applying dropout at the topmost LSTM layer regularizes the handwriting recognizer and improves validation and test performance across model sizes and datasets, with a dataset- and capacity-dependent exception.
- Topmost-layer setup and results: Dropout at the topmost LSTM layer significantly improves IAM and Rimes performance by 10–20% across all tested hidden-unit counts.The topmost layer contains 50 features, and its subsampled inputs make dropout relatively inexpensive during training.
- Topmost-layer setup and results: On OpenHaRT, dropout helps models with 50, 100, or 200 topmost units but hurts performance with 30 units, likely because that model is underfitted.The experiments vary the topmost layer among 30, 50, 100, and 200 units while keeping other parameters fixed.
- Evaluation procedure: For each dataset and setting, the highest-validation-performance model is selected before evaluation on the corresponding test set.This selection procedure is used for the reported comparisons.
- Convergence and regularization: Without dropout, validation NLL increases after a point, indicating overfitting; with dropout, networks are better regularized and achieve higher final validation performance.For OpenHaRT, 200 units overfit without dropout but perform very well with dropout, whereas 30 units are inadequate and converge slowly.
2) Dropout at multiple layers:
Applying dropout across multiple LSTM layers increases regularization and generally improves recognition, but it also increases training time and can expose overfitting in small datasets.
- 2) Dropout at multiple layers:: Dropout is tested at the topmost LSTM layer, the top two layers, and all three LSTM layers.When dropout is applied, LSTM units at that layer are doubled to preserve the average number of active hidden units with p = 0.5.
- 2) Dropout at multiple layers:: With dropout, CER and WER decrease by almost 30–40% relatively, and applying it at all three LSTM layers is generally helpful.The expanded dropout models use 4, 20, and 100 units instead of the baseline 2, 10, and 50 units.
- 2) Dropout at multiple layers:: Applying dropout at three LSTM layers significantly lengthens training in both epochs to convergence and CPU time per epoch.The added modeling power from doubling the LSTM-layer sizes is accompanied by this training-cost increase.
C. Line Recognition with Lexical Constraints and Language Modeling
Line recognition combines RNN outputs with HMM, lexicon, and language-model constraints in an FST decoding graph. Across Rimes, IAM, and OpenHaRT, dropout improves recognition errors, but comparisons with some published systems are not direct because vocabulary and lexical constraints differ.
- Scope of comparisons: The reported Table III results cannot be directly compared with published state-of-the-art results because the RNNs output unconstrained character sequences.Complete large-vocabulary recognition systems additionally use a lexicon and language model to constrain and rescore hypotheses.
- Recognition and decoding: The system builds a hybrid HMM/RNN model whose character, whitespace, and blank emissions derive from RNN posterior probabilities.The HMM includes one state per label, with priors estimated from the training set.
- Recognition and decoding: Lexical constraints are incorporated during decoding by composing HMM, lexicon, and language-model finite-state transducers into the graph searched for the best word sequence.Optical and language-model scaling factors, word insertion penalty, and prior scaling are tuned independently on validation data.
- Results: Dropout provides 7 to 27% relative improvement for unconstrained RNNs, while lexical constraints alone improve CER by more than 40% relative to systems without them.These results are reported across the Rimes, IAM, and OpenHaRT evaluations.
- Results: With dropout and lexical constraints enabled, CER improves by 5.7% on Rimes, 19.0% on IAM, and 4.1% on OpenHaRT.The corresponding WER improvements are 2.4%, 14.5%, and 3.2%, respectively.
D. Effects of dropout on the Recurrent Neural Networks
Dropout regularizes recurrent handwriting-recognition networks by reducing overfitting and altering weight and activation distributions without saturating LSTM activations.
- Dropout makes classification weights smaller, producing a regularization effect similar to L1 or L2 weight decay.Its probability hyper-parameter p is reported as easier to tune than weight-decay hyper-parameters.
- Dropout increases LSTM weight magnitudes and widens the spread of LSTM activations.The authors hypothesize that randomly dropping units encourages the remaining units to emit stronger, more independently useful activations.
- Dropout keeps LSTM activations unsaturated, preserving error-gradient propagation needed to learn long-term dependencies.
- Dropout greatly reduces overfitting, with especially significant gains when the network is relatively large compared with the dataset.The learning curves in Fig. 3 show the regularization effect.
V. CONCLUSION
The conclusion reports that dropout improves deep handwriting-recognition networks across recurrent and convolutional layers, with larger gains from applying it at multiple LSTM layers. Experiments also indicate easier tuning than weight decay and applicability beyond handwriting recognition.
- Dropout at the topmost layer reduces CER and WER by 10-20%, while applying it at multiple LSTM layers further improves performance by 30-40%.
- Dropout improves complete-line recognition error rates whether RNNs operate alone or are constrained by a lexicon and language model.
- The experiments report the best known results on the Rimes and OpenHaRT databases.
- Dropout behaves similarly to weight decay, while its hyper-parameter is easier to tune.
- Although evaluated on handwritten datasets, the technique is described as applicable to other RNN applications.