Source-linked AI summary
Accurate, Data-Efficient, Unconstrained Text Recognition with Convolutional Neural Networks
Mohamed Yousef, Khaled F. Hussain, Usama S. Mohammed
TL;DR
Unconstrained text recognition must handle varied visual forms, while prior approaches often rely on sequential recurrence or task-specific models. This paper proposes an end-to-end, recurrence-free convolutional model with generic augmentation, reporting state-of-the-art performance across seven public datasets and an ICFHR2018 competition win.
Problem
Unconstrained text recognition spans varied visual forms, and prior systems often rely on sequential RNNs or task-specific models rather than one demonstrated across a broad task range.
Source: §1 · p. 1Method
The paper proposes an end-to-end fully convolutional CNN without recurrent connections, trained with CTC on full-line or word labels and accompanied by generic text-recognition augmentations.
Source: §1 · pp. 1–2Results
Across seven public datasets it reports state-of-the-art performance and won ICFHR2018 with more than 25% relative decrease in Character Error Rate (CER) versus the second-place entry.
Source: §1 · p. 2Takeaways & Limitations
The same architecture achieved state-of-the-art performance across varied recognition tasks with minimal hyperparameter changes, and ablations highlighted the importance of its submodules.
Source: §5 · pp. 10–11The comparison with RCN is not like-for-like: RCN evaluations are case-insensitive, while this paper uses case-sensitive evaluation.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Unconstrained text recognition is an important computer vision task, featuring a wide variety of different sub-tasks, each with its own set of challenges. One of the biggest promises of deep neural networks has been the convergence and automation of feature extractors from input raw signals, allowing for the highest possible performance with minimum required domain knowledge. To this end, we propose a data-efficient, end-to-end neural network model for generic, unconstrained text recognition. In our proposed architecture we strive for simplicity and efficiency without sacrificing recognition accuracy. Our proposed architecture is a fully convolutional network without any recurrent connections trained with the CTC loss function. Thus it operates on arbitrary input sizes and produces strings of arbitrary length in a very efficient and parallelizable manner. We show the generality and superiority of our proposed text recognition architecture by achieving state of the art results on seven public benchmark datasets, covering a wide spectrum of text recognition tasks, namely: Handwriting Recognition, CAPTCHA recognition, OCR, License Plate Recognition, and Scene Text Recognition. Our proposed architecture has won the ICFHR2018 Competition on Automated Text Recognition on a READ Dataset.
1 INTRODUCTION
The introduction frames unconstrained text recognition as difficult because text varies widely, and proposes a generic convolutional approach supported by results across diverse tasks.
1 INTRODUCTION
- Text-recognition inputs vary across handwriting, fonts, layouts, visual distortions, and scripts, contributing to the complexity of unconstrained recognition.The introduction describes text in documents and natural scenes, with possible degradation, occlusion, spacing, slant, and curvature.Source: §1 · p. 1
- Prior systems often required sequential RNN processing and separate task-specific models, creating latency and model-selection burdens for practitioners.The introduction argues that pure-visual recognition depends on local neighborhoods rather than long-range dependencies.Source: §1 · p. 1
- The proposed architecture is a feed-forward, fully convolutional CNN using depthwise separable convolutions, residual connections, gating, and CTC training on full-line or word labels.The authors emphasize that it has no recurrent connections and uses efficient convolutional primitives.Source: §1 · pp. 1–2
- The system reports state-of-the-art results on several benchmarks, including a >20% absolute increase in CAPTCHA recognition rate over RCN and a win in ICFHR2018.It also reports state-of-the-art performance on SVHN, IAM, KHATT, UW3, and all AOLP divisions, and a more than 25% relative Character Error Rate (CER) decrease over the second-place ICFHR2018 entry.Source: §1 · p. 2
- The authors propose generic data-augmentation procedures intended to boost performance across text-recognition tasks and neural-network architectures.They also study how the procedures affect system performance.Source: §1 · p. 2
- Experiments on seven benchmark datasets assess the architecture's generality, while ablations examine the importance of its submodules.The paper also reports extensive experiments to support its claims.Source: §1 · p. 2
2 RELATED WORK
The related-work review centers on deep-learning sequence-transduction approaches, contrasting CTC with encoder-decoder models and placing the paper's recurrence-free CNN+CTC approach among prior recognition systems.
2 RELATED WORK
2.1 CTC Models
- CTC models turn framewise symbol likelihoods into output-sequence probabilities, with greedy decoding or beam search used to predict sequences [12].The paper distinguishes its CNN+CTC work by its different CNN architecture and state-of-the-art results across a wide range of tasks [41], [42], [43].Source: §2.1 · p. 2
- Prior work used BLSTM+CTC and MDLSTM+CTC for handwriting, while CNN+BLSTM+CTC systems delivered state-of-the-art results across several text-recognition tasks,,, [8].The cited literature also reports MDLSTM performance equaled by faster CNN+BLSTM+CTC systems,.Source: §2.1 · p. 2
2.2 Encoder-Decoder Models
- Encoder-decoder models read the source sequence and use attention to select relevant regions for token-by-token decoding [22],, [24]; cited work includes multiview signs and unsegmented paragraphs,.One handwriting system achieved competitive results but had prohibitive inference time because it required a very wide beam search [48].Source: §2.2 · pp. 2, 4
3 METHODOLOGY
The method combines a fully convolutional recognition network with GateBlocks and generic data augmentation, then trains with CTC on unsegmented image–transcription pairs. Evaluation uses greedy CTC decoding and character error metrics, including an upper-bound measure for possible reranking gains.
3.1 Overall Architecture
- The fully convolutional architecture centers on depthwise separable convolutions, normalization and GateBlocks; global height pooling lets it accept inputs of any height.Source: §3.1 · p. 4
3.2 GateBlocks
- GateBlocks replace a single gated transform with two learned, input-conditioned transforms; channel-only subsampling lets gates operate on lower-dimensional representations.Source: §3.2 · p. 5
3.3 Data Augmentation
- The proposed label-preserving augmentation techniques are generic to text recognition and are applied independently to training samples.Source: §3.3 · p. 5
3.3.1 Projective Transforms
- Projective augmentation randomly moves image corners, changing either the horizontal or vertical coordinates while keeping connected-corner distances between half and twice their original length.Source: §3.3.1 · p. 5
3.3.2 Elastic Distortions
- Coarse-grid elastic warping displaces control points to preserve localities without Gaussian smoothing; unlike, it skips baseline alignment.Source: §3.3.2 · p. 5
3.3.3 Sign Flipping
- Sign flipping exploits text’s color or texture invariance and improves performance even when training and testing images have matching text and background colors, as previously noted in [72].Source: §3.3.3 · pp. 5–6
3.4 Model Training and Evaluation
- CTC training uses unsegmented line-image and transcription pairs without character/frame alignment, and applies the same random augmentation values across each batch.Source: §3.4 · p. 6
- Greedy CTC decoding uses no language model; CER measures normalized character edit distance, while CER@TopN estimates the upper bound on gains from secondary reranking.Source: §3.4 · p. 6
4 EXPERIMENTS
Experiments compare the proposed architecture with prior methods on public text-recognition benchmarks. On reCAPTCHA, the augmented, deeper model outperforms RCN and exceeds the estimated human performance reported in that work.
4.1 Implementation Details
- Experiments use a single Nvidia TITAN Xp GPU and TensorFlow; model variants are parameterized by GateBlock count and first- and third-block channel sizes.Source: §4.1 · p. 6
4.2 Vicarious reCAPTCHA Dataset
- With augmentation and a deeper model, the system is 20% better than RCN and exceeds estimated human performance, despite case-sensitive evaluation versus RCN’s case-insensitive evaluation.Source: Table 1 · p. 6, §4.2 · p. 6
- The reCAPTCHA benchmark has 5,500 images (500 training, 5,000 test) for unseen-CAPTCHA generalization; the authors split training into 475 training and 25 validation images.Source: §4.2 · p. 6
4.3 SVHN
On SVHN, the model improves full-sequence error with compact parameter counts. The experiments use resized grayscale images and data augmentation.
4.3 SVHN
- The dataset contains around 249k house-number images, with 235k for training and validation and 13k for testing.Each image contains a sequence of 1–5 digits, varying in texture, scale, and spatial arrangement.Source: §4.3 · pp. 6–7
- 3.3% full-sequence error with 0.9M parameters advances the prior 3.6% result; a 3.4M-parameter model reaches 3.1%.The 0.9M-parameter model uses 25X fewer parameters than the second-best system while achieving better classification accuracy.Source: Table 2 · p. 7, §4.3 · p. 7
- Inputs were converted to grayscale and resized to 32×32 pixels, with the study's data augmentations applied.The authors report that 32×32 pixels was sufficient for their model and led to faster processing.Source: §4.3 · p. 7
4.4 University of Washington Database III
On UW3, a small model achieves state-of-the-art character error without text-line normalization or script-specific knowledge. A deeper model improves CER further, while the models converge in far fewer epochs and use smaller input dimensions.
4.4 University of Washington Database III
- 0.10% CER is state of the art for a small model without text-line normalization or script-specific knowledge; a deeper model reaches 0.08% CER.The models also take two orders of magnitude fewer epochs to converge and use smaller input spatial dimensions.Source: Table 3 · p. 7, §4.4 · p. 7
- UW3 contains 1,600 document-image pages; the experiments use 100k text lines split into 90k training and 10k testing lines.The benchmark includes small original input resolutions and a requirement for very low CER.Source: §4.4 · p. 7
- The input text-line images are resized to 32 pixels high while maintaining their aspect ratio.Geometric line normalization was designed for Latin scripts and is described as a fairly complex, script-specific operation.Source: §4.4 · p. 7
4.5 Application-Oriented License Plate (AOLP) Dataset
The model achieves state-of-the-art license-plate recognition on all three AOLP subsets, with its largest margin on the challenging RP subset. The evaluation trains on two subsets and tests on the third.
4.5 Application-Oriented License Plate (AOLP) Dataset
- The model achieves state-of-the-art results with small margins on AC and LE and a big margin on challenging RP.Table 4 reports whole-sequence and character recognition accuracy across the three AOLP divisions.Source: Table 4 · p. 7, §4.5 · p. 7
- AOLP contains 2,049 Taiwanese license-plate images across AC, LE, and RP, evaluated by training on two subsets and testing on the third.For example, RP testing uses AC and LE for training.Source: §4.5 · p. 7
- The comparison reports state-of-the-art results using the authors' described augmentations and grayscale inputs, whereas uses extensive shape and color augmentation.The authors note that grayscale conversion removes most color information.Source: §4.5 · p. 7
4.6 KHATT database
On KHATT, the method substantially reduces CER relative to prior state of the art in comparisons restricted to pure optical results. The experiments use the database's unique-paragraph split and remove excess white space from some line images.
4.6 KHATT database
- KHATT is an offline handwritten Arabic database comprising 4,000 paragraphs written by 1,000 writers across 18 countries.Its scanned pages were collected at 200, 300, and 600 dpi.Source: §4.6 · p. 7
- The method significantly reduces CER against prior state of the art on KHATT when comparisons are limited to pure optical results.The authors attribute part of the reduction to Arabic-script complexity and the limited prior work on KHATT.Source: Table 5 · p. 8, §4.6 · p. 8
- Experiments use the 2,000 unique randomly selected paragraphs, split into 4,825 training, 966 test, and 937 validation text lines.The dataset also includes 2,000 fixed paragraphs with the same text content covering Arabic character shapes.Source: §4.6 · p. 8
- The preprocessing removes large extra white spaces from some line images to avoid using images whose height could greatly exceed the line height.Source: §4.6 · p. 8
4.7 IAM Handwriting Database
The IAM experiments compare visual-only recognition against prior systems and test whether multiple candidate outputs improve recognition. The section also describes the dataset and excludes methods using additional linguistic data.
4.7 IAM Handwriting Database
- IAM contains 657 writers and 1,539 scanned pages, with writer-independent training, validation, and test partitions of 6,161, 966, and 2,915 lines.The dataset has 79 character classes.Source: §4.7 · p. 8
- The comparison excludes methods that use additional linguistic data, such as language models, because the study focuses on visual data and CER.Source: §4.7 · p. 8
- An absolute 1% decrease in CER (relative 16% improvement) beats previous state-of-the-art CNN+BLSTM methods,.The comparison uses down-sampled input relative to; uses synthetic-data pre-training and test-time augmentation.Source: §4.7 · p. 8
- A second candidate reduces CER by 15% relatively for both models; the larger model matches IAM state of the art with five output sequences.The comparison is based on full-line output sequences.Source: §4.7 · p. 8
4.8 ICFHR2018 Competition on Automated Text Recognition
The competition evaluates recognition on an unknown corpus with limited document-specific training pages. The proposed system leads the reported comparisons and achieves useful transcription quality on most tested documents with 16 additional pages.
4.8 ICFHR2018 Competition on Automated Text Recognition
- The competition targets adaptation to an unknown corpus using only a few pages, a setting the authors describe as more realistic than assuming thousands of annotated lines.The dataset contains 22 heterogeneous documents written in different periods and languages.Source: §4.8 · p. 8
- More than 25% relative CER decrease over second place; across 0-to-16-page settings, the system leads every reported metric and cuts CER nearly 80%.Table 8 reports CER by number of additional pages and average CER by test document.Source: §4.8 · pp. 8, 10, Table 8 · p. 9
- With 16 additional pages, four of five tested documents have CER below 10%; the Italian Ricordi document is the exception at 11.4%.The paper cites below 10% CER as a helpful initial transcription that a human operator can correct quickly.Source: Table 9 · p. 9, §4.8 · p. 10
4.9 Ablation Study
The ablation study uses a small, fixed-budget model on IAM to examine gating and normalization choices. Its results distinguish the effects of gating variants from the substantial role of normalization.
4.9.1 Gating Function
- Baseline gating yields a 20% relative CER reduction over the prior-work gate [42],, [93], [94]; a one-transformation variant is slightly better.The authors retain the baseline because it offers roughly the same performance with two transformation functions.Source: Table 10 · p. 10, §4.9.1 · p. 10
4.9.2 Normalization
- Removing Layer Normalization has the largest effect; dropping only its first and last layers raises CER by almost 30%, while Batch Normalization also matters.The study also tests removing Softmax normalization before GateBlocks or replacing it with Tanh.Source: Table 11 · p. 10, §4.9.2 · p. 10
4.9 Ablation Study
- The ablations use IAM, a small baseline model, roughly 71k parameters, and capped, down-sampled inputs to limit computation.Input images are resized to 32 pixels high, width is limited to 200 pixels, and feature maps are down-sampled to at most 4 x 50 pixels.Source: §4.9 · p. 10
5 CONCLUSION
The paper concludes that one architecture achieves state-of-the-art performance across seven public benchmarks with minimal hyperparameter changes. It also identifies automated speech recognition as a direction for further investigation.
5 CONCLUSION
- The same architecture achieves state-of-the-art performance on seven public benchmark datasets with minimal hyperparameter changes.The experiments are presented as evidence of the architecture's generality and ease of adoption for new text-recognition tasks.Source: §5 · pp. 10–11
- The end-to-end architecture handles variable-sized images and variable-sized line-level transcriptions.The paper characterizes the model as data- and computation-efficient.Source: §5 · p. 10
- The ablation study highlights the importance of each model submodule.Source: §5 · p. 11
- Applying the architecture to automated speech recognition is proposed as a direction for further research.The conclusion points to its sample efficiency and noise robustness in handwriting and scene-text recognition.Source: §5 · p. 11