Source-linked AI summary
What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis
Jeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, Hwalsuk Lee
TL;DR
STR model comparisons are confounded by inconsistent training and evaluation datasets, making reported improvements difficult to assess fairly. The paper standardizes the datasets, organizes models into a four-stage framework, and evaluates module contributions across accuracy, speed, and memory, while documenting remaining failure cases.
Problem
Inconsistent training and evaluation datasets prevent fair assessment of whether proposed STR modules improve over existing models.
Method
The paper analyzes dataset inconsistencies, introduces a four-stage STR framework, and evaluates module combinations under unified datasets across accuracy, speed, and memory.
Results
The study provides fair comparisons among key STR methods and identifies module combinations with competitive performance while characterizing accuracy, speed, and size gains.
Takeaways & Limitations
Consistent datasets and a common stage-based framework make module-wise STR performance comparisons more rigorous.
Takeaways & Limitations
Among 8,539 benchmark examples, 644 images, or 7.5%, are misrecognized by all 24 evaluated models, revealing unresolved STR failure cases.
Abstract
from arXiv · showhide
Many new proposals for scene text recognition (STR) models have been introduced in recent years. While each claim to have pushed the boundary of the technology, a holistic and fair comparison has been largely missing in the field due to the inconsistent choices of training and evaluation datasets. This paper addresses this difficulty with three major contributions. First, we examine the inconsistencies of training and evaluation datasets, and the performance gap results from inconsistencies. Second, we introduce a unified four-stage STR framework that most existing STR models fit into. Using this framework allows for the extensive evaluation of previously proposed STR modules and the discovery of previously unexplored module combinations. Third, we analyze the module-wise contributions to performance in terms of accuracy, speed, and memory demand, under one consistent set of training and evaluation datasets. Such analyses clean up the hindrance on the current comparisons to understand the performance gain of the existing modules.
1. Introduction
STR models are difficult to compare fairly because prior studies use inconsistent training and evaluation datasets. The paper addresses this with dataset analysis, a unified four-stage framework, and module-wise evaluation under consistent settings.
- Prior work: Prior STR models use multi-stage pipelines whose neural-network stages address distinct challenges, including variable-length recognition and curved-text normalization.Examples include recurrent sequence modeling, CTC, and transformation modules.
- Motivation: Inconsistent training and evaluation datasets make reported STR results difficult to compare directly.Different IC13 subsets alone may produce performance disparities exceeding 15%.
- Contributions: The paper analyzes commonly used STR training and evaluation datasets and identifies missing examples and incomparable results caused by dataset inconsistency.It reports 7 missing examples in IC03 and 158 in IC13.
- Contributions: The proposed framework divides STR models into transformation, feature extraction, sequence modeling, and prediction stages.This common decomposition supports analysis of existing methods and possible module variants.
- Contributions: Under unified experiments, the paper evaluates modules by accuracy, speed, and memory demand, finding overlooked combinations that improve over state-of-the-art methods.The study also examines benchmark failure cases and remaining STR challenges.
2. Dataset Matters in STR
Training and evaluation dataset choices materially affect STR comparisons. The paper catalogs synthetic training data and real-world benchmarks, documents inconsistent subsets, and measures the resulting comparison bias.
- Synthetic datasets for training: MJSynth provides 8.9 M synthetic word-box images, while SynthText provides 5.5 M cropped and filtered training examples for STR.SynthText was originally designed for scene text detection but is also used for STR after word-box cropping.
- Dataset inconsistency: Prior STR studies combine different training sources, making it unclear whether improvements come from model modules or better or larger training data.The paper recommends clearly reporting training datasets and comparing models with the same training set.
- Real-world datasets for evaluation: Seven real-world STR benchmarks are categorized as regular or irregular according to text difficulty and geometric layout.Regular datasets contain horizontally arranged, evenly spaced characters; irregular datasets include curved, rotated, or distorted text.
- Evaluation discrepancies: Different benchmark subsets produce inconsistent evaluations, including IC03 versions with 860 versus 867 images and IC13 versions with 857 versus 1,015 images.The paper also describes differing IC15 evaluation versions of 1,811 and 2,077 images.
- Measured impact: 7 omitted IC03 examples can create a 0.8% performance gap, while IC13 and IC15 have even larger differences in evaluated example counts.These discrepancies directly undermine comparisons across prior reported results.
3. STR Framework Analysis
The paper represents STR as a four-stage pipeline and studies alternative modules within each stage. The framework connects image normalization, visual feature extraction, sequence modeling, and character prediction for systematic comparison.
- Four-stage framework: The STR framework consists of transformation, feature extraction, sequence modeling, and prediction stages derived from commonalities across existing models.Each stage performs a distinct operation in the recognition pipeline.
- Transformation: Transformation normalizes text geometry, reducing the burden on downstream feature extraction for curved or tilted text.Thin-plate spline transformation is presented as a flexible spatial-transformer variant.
- Feature extraction: Feature extraction maps the input image to a visual feature map whose columns correspond to distinguishable horizontal receptive fields.The study evaluates VGG, RCNN, and ResNet feature extractors.
- Sequence modeling: Sequence modeling adds contextual information across feature frames, with BiLSTM selectable or removable to trade computational complexity and memory consumption.The sequence is formed by reshaping feature-map columns into frames.
- Prediction: Prediction converts sequence features into characters using either CTC or attention-based sequence prediction.CTC supports non-fixed-length output from a fixed number of input features.
4. Experiment and Analysis
Using a unified dataset and four-stage framework, the paper evaluates 24 STR module combinations across accuracy, speed, and memory, identifying trade-offs, efficient upgrade orders, and persistent failure cases.
- Experimental setup: 24 STR module combinations were evaluated under a common training and evaluation dataset from the four-stage framework.The experiments used a unified setting to compare all possible combinations.
- Training datasets: 81.3% accuracy came from combining 20% of MJSynth and 20% of SynthText, outperforming either dataset used individually.The combined subset contained 2.9M examples, suggesting training-data diversity matters beyond dataset size.
- Accuracy-time trade-offs: 69.5%→82.9% accuracy accompanied a 1.3ms→10.9ms slowdown as ResNet, BiLSTM, and TPS were added along the accuracy-time frontier.Adding Attn raised accuracy by only 1.1% while increasing inference time to 27.6 ms.
- Accuracy-memory trade-offs: 75.4%→82.3% accuracy required only 1.9M→7.2M parameters before ResNet increased memory to 49.6M for a further 1.7% accuracy.Transformation, sequential, and prediction modules contributed relatively little memory compared with heavy feature extractors.
- Module-wise effects: Prediction modules most strongly affected speed, whereas feature extraction modules most strongly affected memory.Attn formed a slower accuracy-speed cluster, while the feature extractor determined the main accuracy-memory clustering.
- Failure analysis: 644 of 8,539 benchmark images, or 7.5%, were misrecognized by all 24 models, with remaining challenges including calligraphic, vertical, occluded, and low-resolution text.The analysis also identified special-character handling and label noise as failure-related issues.
5. Conclusion
The paper establishes a fair comparison of key STR methods using a common framework and consistent datasets, while analyzing module contributions and remaining failures.
- The study uses seven benchmark evaluation datasets and two training datasets, MJ and ST, for consistent comparison.
- The common framework enables analysis of which modules provide the greatest accuracy, speed, and size gains.
- The authors also analyze module-wise contributions to typical STR challenges and examine remaining failure cases.
Appendix B : Dataset Matters in STR - examples
The appendix documents dataset examples, implementation checks, architectural details, and the experimental results used to support the STR framework.
- The appendix illustrates problematic datasets, verifies four reproduced STR models, and documents every framework module.
- The reproduced models are CRNN, RARE, GRCNN, and FAN without the Focus Net.
- Architectural details are provided for all modules in the framework.
Appendix E : STR Framework - full experimental results
The appendix reports comprehensive experiments, including fine-tuning, training-set-size variation, and testing on COCO.
- The appendix provides comprehensive experimental results and discusses them in detail.
- The experiments include fine-tuning, varying the training dataset size, and testing on the COCO dataset.
B. Dataset Matters in STR - examples
This material covers dataset inconsistencies and the STR framework’s transformation, feature, sequence, and prediction components, including their implementation and training roles.
- Dataset Matters in STR - examples: IC03 evaluation has versions with 860 and 867 images; the smaller version omits 7 word boxes.
- Dataset Matters in STR - examples: IC15 contains 2,077 evaluation examples, while prior works filtered it to 1,811 without unambiguous discard specifications.
- Dataset Matters in STR - examples: Examples of duplicated scene images and filtered or missing evaluation examples illustrate dataset-construction discrepancies.
- Dataset Matters in STR - examples: Overlapping data include 215 duplicate word boxes, requiring caution when assessing models on IC13 evaluation data.
- STR Framework - architectural details: The implementation reproduces CRNN, RARE, GRCNN, and FAN without Focus Net to verify the experimental platform.
- Transformation stage: The transformation stage maps an input image X to normalized image X̃, while TPS uses fiducial points and grid sampling.
- Feature extraction stage: Feature extraction applies CNNs to produce a visual feature map, with VGG, RCNN, and ResNet offered as architectures.
- Sequence and prediction stages: Sequence modeling can use BiLSTM contextual states, or no sequence module, before prediction produces character sequences through CTC or attention.
E. STR Framework - full experimental results
The experiments compare 24 STR module combinations using accuracy–time and accuracy–memory trade-off plots, with colors separating feature extraction, sequence modeling, and prediction choices.
- Figure 11–14 visualize 24 STR combinations through accuracy-versus-time and accuracy-versus-memory trade-off plots.The combinations are color-coded by module to show module effectiveness.
- Feature extraction stage: VGG has the lowest average accuracy and inference time, while RCNN achieves higher accuracy than VGG with the longest inference time.
- Sequence modeling stage: BiLSTM provides a larger average accuracy boost than TPS with similar inference-time or parameter-size concessions.The figure describes their effects as similar overall.
- Prediction stage: The CTC-versus-Attn choice produces the clearest inference-time increase for each percentage of accuracy gained.The same clear pattern is not observed for parameter-count increases.
F.1. Fine-tuning on real datasets
The paper fine-tunes models on held-out real data from in-distribution evaluation sets, while excluding unsuitable out-distribution training sources. Fine-tuning improves accuracy near the test distribution but can reduce accuracy on out-distribution data.
- Fine-tuning uses the union of IIIT, SVT, IC13, and IC15 held-out subsets as in-distribution real training data.IC03, SP, and CT lack suitable held-out training subsets, and IC03 has overlap concerns with IC13 evaluation data.
- Real-data fine-tuning improves in-distribution accuracy by 2.2 pp and all-benchmark accuracy by 1.5 pp, but decreases out-distribution accuracy by 1.3 pp.Fine-tuning used 10 epochs.
- The authors conclude that real-data fine-tuning is effective when the real data are close to the test-time distribution.They report that fine-tuning may otherwise harm performance.
F.2. Accuracy with varying training dataset size
Across 24 STR combinations, more training data generally improves accuracy, but saturation depends on the module configuration. Higher-capacity modules retain improvement potential at larger dataset sizes than lower-capacity alternatives.
- The average accuracy of all 24 models tends to increase with more training data.
- Transformation stage: Models without TPS remain unsaturated at 100% training-data size, whereas TPS models saturate at 80%.The authors conjecture that TPS normalizes the additional data rather than improving accuracy.
- Each varying-dataset-size result represents Total accuracy (%) from one trial per STR combination and may differ slightly from Table 8.
- Feature extraction stage: ResNet curves remain unsaturated at 100% training-data size, while VGG and RCNN averages saturate at 60% and 80%, respectively.The authors conjecture that VGG and RCNN have lower capacity and reach their limits earlier.
- Sequence modeling stage: BiLSTM curves remain unsaturated at 100% training-data size, while curves without BiLSTM saturate at 80%.The authors conjecture that BiLSTM’s higher capacity leaves room for further accuracy improvement.
- Prediction stage: Attn curves remain unsaturated at 100% training-data size, whereas CTC curves saturate at 80%.The authors conjecture that Attn’s higher capacity explains its remaining improvement potential.
F.3. Evaluation on COCO-Text dataset
COCO-Text evaluates the 24 STR methods on complex, low-resolution images with special characters, noise, and occlusions. Accuracy is lower overall, but the relative ordering of methods is largely preserved.
- COCO-Text is considered more challenging than the seven previously used benchmarks because it contains complex, low-resolution images, special characters, heavy noise, and occlusions.
- The relative ordering among the 24 methods is largely preserved on COCO-Text despite lower overall accuracy.Figure 19 presents accuracy–time and accuracy–space trade-off plots.