PaperlayerView exact revision on arXiv

Source-linked AI summary

What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labels

Jeonghun Baek, Yusuke Matsui, Kiyoharu Aizawa

arXiv:2103.04400v2cs.CV

TL;DR

STR research commonly relies on large synthetic datasets, although suitable synthetic examples can be difficult to create or unavailable for some domains and languages. This paper consolidates real labeled data and applies augmentation and semi- and self-supervised methods. It reports competitive STR performance using real data, while noting that the dataset's diversity prevents concluding that 276K labels alone are sufficient.

  • Problem

    STR with fewer real labels is under-studied because real data has been assumed insufficient, even though synthetic data can be difficult to generate or unavailable for some domains and languages.

    Source: §1 · p. 1

  • Method

    The authors consolidate public real labeled data and improve real-data STR with simple augmentation and semi- and self-supervised methods using unlabeled data.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    TRBA trained only on real data with Pseudo-Label and RotNet reaches 89.3% and is competitive with state-of-the-art methods.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    The accumulated public real data supports satisfactory STR training without synthetic data, with augmentation and unlabeled-data methods further improving performance.

    Source: §1 · p. 2, §6 · p. 8

    Because the training set combines 11 diverse datasets, the authors caution that the results do not establish 276K as a sufficient amount of real data in general.

    Source: §5.3 · p. 6

Abstract

from arXiv · show

Scene text recognition (STR) task has a common practice: All state-of-the-art STR models are trained on large synthetic data. In contrast to this practice, training STR models only on fewer real labels (STR with fewer labels) is important when we have to train STR models without synthetic data: for handwritten or artistic texts that are difficult to generate synthetically and for languages other than English for which we do not always have synthetic data. However, there has been implicit common knowledge that training STR models on real data is nearly impossible because real data is insufficient. We consider that this common knowledge has obstructed the study of STR with fewer labels. In this work, we would like to reactivate STR with fewer labels by disproving the common knowledge. We consolidate recently accumulated public real data and show that we can train STR models satisfactorily only with real labeled data. Subsequently, we find simple data augmentation to fully exploit real data. Furthermore, we improve the models by collecting unlabeled data and introducing semi- and self-supervised methods. As a result, we obtain a competitive model to state-of-the-art methods. To the best of our knowledge, this is the first study that 1) shows sufficient performance by only using real labels and 2) introduces semi- and self-supervised methods into STR with fewer labels. Our code and data are available: https://github.com/ku21fan/STR-Fewer-Labels

1. Introduction

STR with fewer real labels has received little study because real data was assumed insufficient, despite its importance where synthetic data is difficult to create or unavailable. The paper consolidates accumulated real labels, finds performance close to synthetic-data training, and explores augmentation and unlabeled-data methods.

1. Introduction

  • The paper addresses the under-studied real-label setting, where the common assumption was that too little real data yields low accuracy.The authors say this assumption may have hindered research on STR with fewer labels.Source: §1 · p. 1
  • Training with fewer real labels matters when synthetic examples are difficult to generate for handwritten or artistic text, or unavailable for other languages.The paper notes that producing appropriate examples can be difficult for people who do not know the target language.Source: §1 · p. 1
  • With 276K real labels, CRNN and TRBA accuracy is close to training with 16M synthetic data.The authors argue this is enough to train STR models satisfactorily using real data alone.Source: §1 · p. 2, Figure 1 · p. 2
  • Simple data augmentation, plus semi- and self-supervised methods using unlabeled data, improves real-data STR and yields a competitive model.The paper also investigates whether its methods help when both synthetic and real data are available.Source: §1 · p. 2

2. Common Practice in STR Dataset

STR models are generally trained on large synthetic datasets because obtaining enough real training data is difficult. The section describes the major synthetic training datasets and real datasets used for evaluation.

2. Common Practice in STR Dataset

  • Real data has generally been reserved for evaluation because its labeling cost makes obtaining enough training examples difficult.The benchmark study cited by the authors describes this constraint.Source: §2 · p. 2

2.1. Synthetic Datasets for Training

  • MJSynth contains 9M word boxes generated from a 90K English lexicon and over 1,400 Google Fonts.It was generated specifically for STR.Source: §2.1 · p. 2
  • SynthText contains 7M word boxes rendered onto scene images, which are cropped for STR training.The dataset was originally generated for scene text detection.Source: §2.1 · p. 2

2.2. Real Benchmark Datasets for Evaluation

  • Six real datasets serve as STR evaluation benchmarks, grouped into regular and irregular text sets.Regular sets mainly contain horizontal text; irregular sets mainly contain perspective or curved text.Source: §2.2 · p. 2
  • The evaluation benchmarks span street imagery, web-crawled images, competition data, perspective text, and curved text.For example, SVT has 647 evaluation images, while CUTE80 has 288 cropped evaluation images.Source: §2.2 · p. 2

3. Consolidating Public Real Datasets

The authors consolidate public real labeled datasets accumulated from 2011 to 2019, alongside unlabeled scene-image collections. They preprocess these resources for STR training, including filtering content and removing duplicates.

3. Consolidating Public Real Datasets

  • Public real labeled datasets accumulated from 2011 to 2019 provide the basis for a consolidated STR training set.The section points to an increasing variety of irregular text in later datasets.Source: §3 · p. 2, §3.1 · p. 3

3.1. Real Labeled Datasets Have Increased

  • Real labeled datasets grew in number over time and include varied text, such as Chinese, multilingual, perspective, curved, and unusually arranged text.ICDAR competitions released seven of the eleven datasets listed in Table 1.Source: §3.1 · p. 3

3.2. Real Unlabeled Datasets

  • Three unlabeled datasets supply scene images without word-region annotations, so a pretrained text detector is used to crop words.The collections include Book32, TextVQA, and ST-VQA; overlapping evaluation datasets are excluded from ST-VQA.Source: §3.2 · p. 3

3.3. Preprocessing Real Datasets

  • Preprocessing removes duplicate data, non-alphanumeric or symbol-only language content outside the selected set, unreadable or do-not-care labels, vertical text, and overly long words.The authors also split training data to create validation sets; they focus mainly on horizontal text.Source: §3.3 · p. 4

4. STR With Fewer Labels

The paper combines STR models with semi- and self-supervised learning to make better use of limited real labels and unlabeled data. It describes the model components and the training strategies explored.

4. STR With Fewer Labels

  • Because real labeled data remains far smaller than synthetic data, the paper introduces semi- and self-supervised methods to improve STR with fewer labels.The real labeled data is described as about 1.7% of synthetic data.Source: §4 · p. 4

4.1. STR Model Framework

  • CRNN and TRBA combine transformation, feature extraction, sequence modeling, and prediction, with different architectures and accuracy-speed trade-offs.CRNN uses VGG, BiLSTM, and CTC and is faster and lighter; TRBA uses TPS, ResNet, BiLSTM, and attention and is larger and slower but more accurate.Source: §4.1 · p. 4

4.2. Semi-Supervised Learning

  • The paper introduces Pseudo-Label and Mean Teacher as semi-supervised approaches for using unlabeled data in STR.Pseudo-Label retrains on predicted labels; Mean Teacher trains a student against an EMA-updated teacher using consistency loss.Source: §4.2 · pp. 4–5, Figure 5 · p. 5

4.3. Self-Supervised Learning

  • RotNet and MoCo are investigated as self-supervised methods that pretrain on a surrogate task before training for STR.RotNet predicts image rotations, while MoCo uses instance discrimination and a contrastive InfoNCE loss.Source: §4.3 · p. 5

5. Experiment and Analysis

Across experiments, real-only training with 276K labels approaches synthetic-data baselines, and augmentation plus pseudo-labeling and RotNet produces a competitive result. Performance depends on data diversity and training setup; semi- and self-supervision can help with real-only training but may hurt fine-tuning.

5.2. Comparison to State-of-the-Art Methods

  • 89.3%: TRBA-PR, trained only on real data, achieves competitive performance with state-of-the-art methods.PR combines Pseudo-Label and RotNet; Table 2 compares CRNN and TRBA across six benchmark datasets and includes reproduced synthetic baselines and real-only models.Source: Table 2 · p. 6, §5.2 · p. 6

5.3. Training Only on Real Labeled Data

  • With 276K real labels, CRNN and TRBA approach synthetic-data accuracy using only 1.7% as many labeled examples, but dataset diversity prevents concluding that 276K is generally sufficient.The study indicates a need for 146K–276K real examples; its 11 datasets have high diversity, which may matter more than dataset count alone.Source: §5.3 · p. 6
  • 277K synthetic images yield lower accuracy than 276K real images: 65.1% vs. 74.8% for CRNN and 75.9% vs. 86.6% for TRBA.The synthetic subset is 1.74% of the full synthetic data and approximately matches the real-data amount.Source: §5.3 · p. 6
  • For real-only training, Crop+Rot improves CRNN accuracy by 5.2%, while Blur+Crop improves TRBA accuracy by 0.9% over each model’s real-data baseline.The paper uses cropping, rotation, and blur to address text truncation, rotated or curved text, and blur; applying augmentation to synthetic data reduced accuracy.Source: §5.3 · p. 7

5.4. Semi- and Self-Supervised Learning

  • Pseudo-Label improves CRNN by 2.8% and TRBA by 1.7%, whereas Mean Teacher decreases them by 0.2% and 0.4%; RotNet and MoCo have model-dependent results.RotNet improves CRNN by 1.3% and TRBA by 0.09%, while MoCo improves CRNN by 0.8% and decreases TRBA by 0.8%; pretraining both TPS and ResNet sharply decreases TRBA accuracy.Source: §5.4 · p. 7
  • Combining Pseudo-Label and RotNet outperforms either alone and improves accuracy over augmented real-data baselines by 3.4% for CRNN and 1.8% for TRBA.The combined PR setting uses RotNet-pretrained models to predict pseudo-labels and adds augmentation.Source: §5.4 · p. 7

5.5. Varying Amount of Real Labeled Data

  • With labeled-data ratios reduced from 40% to 20%, baseline accuracy drops 13.2% for CRNN and 27.0% for TRBA; augmentation improves accuracy at 20% by 13.0% and 20.0%, respectively.PR substitutes for over 221K labeled examples for CRNN and 110K for TRBA; at 20% labels CRNN-PR exceeds the full-label baseline by 2.0%, and at 60% labels TRBA-PR exceeds it by 0.4%.Source: §5.5 · p. 8, Figure 6 · p. 8

5.6. Training on Both Synthetic and Real Data

  • Fine-tuning on real labels improves accuracy, but adding PR during fine-tuning performs worse than fine-tuning on real labels alone; with synthetic and real data, PR can outperform the real-only baseline.Real-label fine-tuning improves CRNN by 6.3% and TRBA by 4.3%; PR fine-tuning scores 76.6% vs. 82.1% for CRNN and 87.5% vs. 90.0% for TRBA, while PR with synthetic data is reported higher than Baseline-real with synthetic data.Source: §5.6 · p. 8, Table 5 · p. 8

6. Conclusion

The conclusion argues that accumulated public real data is sufficient for STR training despite being much smaller than synthetic data, and that augmentation and unlabeled-data methods improve performance.

6. Conclusion

  • Despite being only 1.7% of synthetic data, accumulated public real data can train STR models sufficiently.The paper presents this as a step toward wider study of STR with fewer labels.Source: §6 · p. 8
  • Simple augmentations and semi- and self-supervised methods further improve performance using millions of real unlabeled examples.The conclusion identifies these as ways to improve models trained with fewer real labels.Source: §6 · p. 8
  • The authors hope this work will facilitate future research on STR with fewer labels.Source: §6 · p. 8

Supplementary Material

The supplementary material expands the paper’s discussion of real-label needs, dataset preprocessing, methods, and experimental analyses.

Supplement B: Needs for STR with Fewer Real Labels

Supplement C: STR Datasets - Details and More Examples

  • Supplement C details dataset preprocessing and adds examples of public real data.Source: A. Contents · p. 12

Supplement D: STR With Fewer Labels - Details

  • Supplement D gives further details on STR models and semi- and self-supervised methods.Source: A. Contents · p. 12

Supplement E: Experiment and Analysis - Details

B. Needs for STR with Fewer Real Labels

The section motivates STR with fewer real labels through domains that are hard to synthesize and reports stronger results from real data on additional evaluation sets. It also introduces the following dataset-preprocessing details.

B. Needs for STR with Fewer Real Labels

  • The section frames fewer-label STR as efficiently exploiting limited real annotations and describes two practical needs for it.Source: B. Needs for STR with Fewer Real Labels · p. 12
  • Real-trained TRBA predicts handwritten and artistic text better than its synthetic-trained counterpart in the illustrated examples.The authors note that current synthetic engines have difficulty generating artistic text and may not cover the diversity of handwriting.Source: B. Needs for STR with Fewer Real Labels · p. 12, Figure 7 · p. 12, Figure 8 · p. 12
  • For languages other than English, few real labels are useful when appropriate synthetic data is difficult to create.Preparation may require target-language vocabulary, characters, and fonts; Arabic also has right-to-left writing and context-dependent character shapes.Source: B. Needs for STR with Fewer Real Labels · p. 12
  • 58.0% vs. 52.3% for CRNN and 69.2% vs. 60.4% for TRBA: real-data baselines outperform synthetic-data baselines across seven additional datasets.The best setting PR improves the real-data baselines by an average of 5.8% for CRNN and 3.1% for TRBA; the authors presume target-domain overlap may explain the advantage.Source: B. Needs for STR with Fewer Real Labels · pp. 12–13, Table 6 · p. 13
  • Prior fewer-label studies are difficult to compare with recent STR benchmarks because evaluation datasets and lexicon use differ.The surveyed older methods did not evaluate on IC15, SP, and CT, while lexicon-assisted accuracy is generally higher; a spotting result also does not directly report recognition performance.Source: B. Needs for STR with Fewer Real Labels · p. 13

C. STR Datasets - Details and More Examples

C.1. Preprocessing Real Datasets

The supplementary sections document real-dataset preparation, model choices, learning objectives, and experimental settings. They also illustrate accumulated public datasets and explain the implementation details used in the experiments.

C.1. Preprocessing Real Datasets

  • The authors collect public datasets, define train-validation-evaluation splits where needed, and use a real-data-trained detector to crop words from unlabeled scenes.Some evaluation sets lack labels and are treated as unavailable; unlabeled scene images have no word-region annotations.Source: §C.1 · pp. 14–15
  • Preprocessing filters selected languages, don’t-care labels, vertical text, and duplicated examples to prepare the real-data sets.The authors identify 27 duplicated ArT-training/CT-evaluation scene images and 122 word boxes, and report similar labeled-image overlap between ArT and LSVT training sets.Source: §C.1 · pp. 14–15

C.2. More Examples of Public Real Data

  • The supplementary figures show benchmark examples and public real datasets accumulated from 2011 through 2019, plus cropped unlabeled examples.The benchmark examples distinguish regular sets with many horizontal texts from irregular sets with many perspective or curved texts.Source: §C.2 · p. 15, Figure 11 · p. 16, Figure 12 · p. 16

D. STR With Fewer Labels - Details

D.1. Description of STR Models

  • CRNN and TRBA represent lightweight and higher-performing baseline architectures, respectively.CRNN uses VGG-like CNNs, BiLSTM, and CTC; TRBA combines TPS, ResNet, BiLSTM, and attention.Source: §D.1 · p. 18

D.2. Objective Function

  • The objective-function details cover supervised recognition, pseudo-labeling, teacher-student consistency, rotation prediction, and contrastive learning.The text defines labeled and unlabeled inputs, describes mean-squared-error consistency between augmented student and teacher predictions, and presents rotation prediction as a four-class task.Source: §D.2 · p. 18

E. Experiment and Analysis - Details

E.1. Implementation Detail

  • Training uses 32 × 100 inputs, 94 prediction characters, balanced dataset sampling, and lowercased alphanumeric word-level accuracy.The authors also document departures from the benchmark base code, including optimizer, schedule, batch size, character set, and evaluation aggregation.Source: §E.1 · p. 19

E.2. Comparison to State-of-the-Art Methods

The comparison covers STR methods trained on synthetic data or mixtures of synthetic and real data, while noting that unequal real-data amounts complicate comparisons. The authors also discuss character-level supervision and report a competitive mixed-data result for TRBA-PR.

E.2. Comparison to State-of-the-Art Methods

  • Table 9 extends the comparison to methods using SynthAdd or mixed synthetic and real data across six benchmarks.The table reports original-paper results and identifies methods using character-level labels with an asterisk.Source: §E.2 · p. 19
  • Methods using both synthetic and real data report better performance than their synthetic-only counterparts, but their real-data quantities differ: 50K, 16K, and 7K.The methods are [25], [63], and [72], respectively, so their results are difficult to compare fairly.Source: §E.2 · p. 19
  • Methods using character-level information tend to achieve higher accuracy on irregular texts, and some use character-level labels from ST [26] [63].Source: §E.2 · p. 19
  • 90.0% is the reported competitive performance of TRBA-PR trained on MJ+ST and real data; adding SynthAdd raises it to 90.3%.SynthAdd is generated with the same synthetic engine as SynthText to compensate for missing special characters.Source: §E.2 · p. 19

E.3. Training Only on Real Labeled Data

The appendix examines how accumulated real data and augmentation choices affect STR accuracy, and gives implementation details for the semi- and self-supervised framework. Accuracy approaches synthetic-data performance as real datasets accumulate, while augmentation effects vary by model and operation.

E.3. Training Only on Real Labeled Data

  • As accumulated real labeled data increases, accuracy approaches the accuracy obtained using synthetic data.Figure 16 plots accuracy against dataset increments; Table 10 reports accuracy by increment.Source: §E.3 · p. 19, Figure 16 · p. 20, Table 10 · p. 21
  • Weak augmentations help, but 1° Rot improves CRNN accuracy by 0.3% and decreases TRBA accuracy by 0.8%.At intensity 1, Blur and Crop also improve accuracy for both models; TRBA performs best at 99% Crop.Source: §E.3 · p. 21
  • CRNN’s best settings—5r Blur, 90% Crop, and 15° Rot—improve accuracy over no augmentation by 0.9%, 4.0%, and 4.7%; TRBA’s 5r Blur and 99% Crop improve it by 0.2% and 0.5%.The study adjusts Gaussian-blur maximum radius, minimum crop percentage, and maximum rotation degree; §5.3 uses 15° Rot for TRBA to investigate combined augmentations.Source: §E.3 · p. 21
  • Rotation improves RBA accuracy by 3.1% at 30°, whereas it decreases TRBA accuracy; the authors suggest rotation may hinder TRBA's TPS normalization.RBA is TRBA without TPS, and the authors note that rotation may not be compatible with TPS.Source: §E.3 · p. 21

E.4. Semi- and self-supervised framework

  • For semi- and self-supervised learning, the appendix specifies Mean Teacher loss weight α=1.0 and 200K iterations for RotNet and MoCo.RotNet uses batches of 512 with four rotations; MoCo uses batches of 256, with both using SGD and the stated augmentation policy for MoCo.Source: §E.4 · p. 21

E.5. Varying Amount of Real Labeled Data

The appendix varies the amount of real labeled data while keeping dataset diversity fixed at 11 datasets. It identifies tables for the word-box counts and corresponding accuracy results, but the supplied passages do not state their values.

E.5. Varying Amount of Real Labeled Data

  • The experiment varies each dataset's real-labeled-data ratio while retaining all 11 datasets.Source: §E.5 · p. 21
  • Table 12 reports the number of word boxes used at each real-data amount.Source: §E.5 · p. 21, Table 12 · p. 22
  • Table 13 reports accuracy for varying amounts of real labeled data, with values plotted in Figure 6.Source: Table 13 · p. 22
Loading 2103.04400v2…