PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Scene Text Recognition for Character-Level Long-Tailed Distribution

Sunghyun Park, Sunghyo Chung, Jungsoo Lee, Jaegul Choo

arXiv:2304.08592v1cs.CVcs.CL

TL;DR

STR models are less effective on languages with many characters because rare characters follow a long-tailed distribution and contextual and visual learning can conflict. The paper analyzes balanced and long-tailed synthetic data, then proposes CAFE-Net, which combines context-aware and context-free experts through confidence-based ensembling. Experiments show improved STR performance on such languages and applicability across model architectures.

  • Problem

    STR research has focused mainly on English, while languages with many characters suffer degradation on rarely occurring characters under character-level long-tailed distributions.

  • Method

    CAFE-Net trains separate context-aware and context-free experts on long-tailed common-word data and balanced character data, then aggregates them using confidence scores.

  • Results

    CAFE-Net improves STR performance on languages with numerous characters, including tail-character and whole-word prediction, and applies across various STR models.

  • Takeaways & Limitations

    Separating contextual and visual representation learning enables a single confidence-ensemble approach to address character-level imbalance across STR settings.

Abstract

from arXiv · show

Despite the recent remarkable improvements in scene text recognition (STR), the majority of the studies focused mainly on the English language, which only includes few number of characters. However, STR models show a large performance degradation on languages with a numerous number of characters (e.g., Chinese and Korean), especially on characters that rarely appear due to the long-tailed distribution of characters in such languages. To address such an issue, we conducted an empirical analysis using synthetic datasets with different character-level distributions (e.g., balanced and long-tailed distributions). While increasing a substantial number of tail classes without considering the context helps the model to correctly recognize characters individually, training with such a synthetic dataset interferes the model with learning the contextual information (i.e., relation among characters), which is also important for predicting the whole word. Based on this motivation, we propose a novel Context-Aware and Free Experts Network (CAFE-Net) using two experts: 1) context-aware expert learns the contextual representation trained with a long-tailed dataset composed of common words used in everyday life and 2) context-free expert focuses on correctly predicting individual characters by utilizing a dataset with a balanced number of characters. By training two experts to focus on learning contextual and visual representations, respectively, we propose a novel confidence ensemble method to compensate the limitation of each expert. Through the experiments, we demonstrate that CAFE-Net improves the STR performance on languages containing numerous number of characters. Moreover, we show that CAFE-Net is easily applicable to various STR models.

1. Introduction

This paper addresses character-level long-tailed distributions in scene text recognition, where models struggle especially with rare characters in languages such as Chinese and Korean. It analyzes contextual and visual representations and proposes CAFE-Net, which combines specialized experts to improve recognition across character frequencies.

  • Languages with thousands of characters, such as Chinese and Korean, pose greater STR challenges than English, which has 26 letters.
  • Character-level long-tailed distributions make models favor frequent head characters, causing substantial performance degradation on rare tail characters.
  • Balancing tail-character samples improves individual character recognition but can remove everyday word contexts, while common-word data preserves context but underrepresents rare characters.
  • CAFE-Net trains context-aware and context-free experts separately, then aggregates their predictions using confidence scores.
  • Experiments report improved performance on many, medium, and few characters, with applicability across various STR models.

2. Related Work

Prior STR research largely focused on English or multilingual methods without modeling each language’s character distribution. This paper relates its approach to long-tailed recognition and specialized multi-expert models while introducing character-level distribution analysis for STR.

  • Scene Text Recognition: Most STR studies focus on English, while existing multilingual approaches generally do not account for language-specific character distributions.
  • Scene Text Recognition: The paper addresses a character-level long-tailed setting that previous STR work had not directly considered.
  • Long-tailed Recognition: Long-tailed recognition research commonly uses loss design, tail-class augmentation, logit adjustment, and specialized experts for different label distributions.
  • Figure 2 compares WikiSynth, RandomSynth, and CombinedSynth distributions; RandomSynth and CombinedSynth provide sufficient samples across all character classes unlike WikiSynth.

3. Motivation

The analysis examines how character-level distributions affect Korean and Chinese scene text recognition, revealing a tradeoff between contextual word recognition and individual-character recognition. Balanced random-character data improves character recognition, while common-word data better preserves context.

  • 3.1. Synthetic Data: RS is character-balanced but context-free, while CS mixes WS and RS to compensate for their respective limitations.RS randomly combines characters and contains the same number of images as WS; CS uses equal numbers of WS and RS images.
  • 3.2. Character-Level F1 Score: Character-level F1 evaluates aligned per-character performance and is better suited than word accuracy for imbalanced character distributions.The metric aligns ground-truth and predicted sequences, computes per-character F1, and averages the scores.
  • 3.2. Character-Level F1 Score: Characters are grouped as many, medium, or few according to training frequency, with few characters receiving substantially lower char F1 under WS.The thresholds are ni ≥1500 for many, 100 ≤ ni <1500 for medium, and ni <100 for few.
  • 3.3. Tradeoff between Context-Free and Context-Aware Learning: Training with WS improves RealEasy word accuracy, whereas RS and CS improve character-level recognition compared with WS.WS preserves contextual information, while RS and CS expose models to randomly combined characters.
  • 3.3. Tradeoff between Context-Free and Context-Aware Learning: The experiments show that visual representations support individual-character recognition, whereas contextual representations support predicting complete words.The paper concludes that both representations are needed to recognize tail characters and whole words effectively.

4. Method

CAFE-Net trains separate context-aware and context-free experts on datasets with different character distributions, then selects between their predictions using confidence scores. Shared feature extraction reduces inference complexity while the expert roles target contextual and visual representations.

  • 4. Method: CAFE-Net trains a context-aware expert on WS and a context-free expert on RS to learn contextual and visual representations separately.The two experts use different training distributions: WS contains common-word context, while RS balances characters through random sequences.
  • 4. Method: The experts share a feature extractor, which reduces inference complexity and permits different extractor architectures such as ResNet or ViT.The shared extractor produces separate context-aware and context-free feature representations.
  • 4. Method: The context-free expert is trained with balanced characters and therefore prioritizes visual representation over contextual representation.Random character sequences lack semantic meaning, helping the expert predict few characters more effectively than the context-aware expert.
  • 4. Method: The context-aware expert uses an external language model to capture semantic information for accurate whole-word prediction.Its output feature is produced after incorporating contextual information from the language model.
  • 4. Method: Both experts are trained with the same negative-log-likelihood objective for conditional word prediction.The objective is applied to the experts’ word-label predictions.
  • 4. Method: During inference, CAFE-Net selects the word prediction from the expert with the higher length-normalized confidence score.Confidence is based on maximum softmax probabilities for predicted characters, normalized by predicted word length.

5. Experiments

CAFE-Net improves STR across Korean and Chinese evaluation datasets by combining experts specialized for contextual and visual representations. Confidence-based selection lets each expert compensate for the other's weaknesses, with negligible added computational cost.

  • Effectiveness of CAFE-Net: CAFE-Net consistently improves accuracy across Korean and Chinese Real, RealEasy, RealHard, and Synthtest evaluations.The comparison uses multiple CNN- and ViT-based STR architectures, demonstrating applicability across model architectures.
  • Comparison with Baselines: CAFE-Net outperforms long-tailed recognition baselines in accuracy while maintaining comparable char F1 performance.τ-norm generally achieves the best char F1 score but has degraded accuracy, whereas CAFE-Net achieves the best accuracy.
  • Analysis on Confidence Score: Confidence ensemble filters low-confidence predictions from one expert and selects the other expert's higher-confidence prediction.This mechanism addresses visually confusing characters for the context-free expert and few characters for the context-aware expert.
  • Analysis on Confidence Score: Context-aware experts have higher averaged probabilities for many characters, whereas context-free experts have higher probabilities for few characters.The context-free expert tends to handle few or medium characters, while the context-aware expert more often handles many-character instances.
  • Computational Cost: CAFE-Net improves performance across model architectures while requiring only a negligible amount of additional computational cost.The computational-cost analysis uses accuracy, FLOPs, and parameter counts on the Real dataset.

6. Conclusions

The paper identifies character-level long-tailed distributions as an overlooked challenge in STR and proposes CAFE-Net to jointly learn contextual and visual representations. Experiments show improved performance on languages with numerous characters and applicability across STR models.

  • 6. Conclusions: The paper proposes CAFE-Net, which trains separate experts for contextual information and visual representation before aggregating them with confidence ensemble.The method targets languages whose characters follow long-tailed distributions.
  • 6. Conclusions: CAFE-Net improves STR performance across many, medium, and few characters and achieves state-of-the-art performance on languages with character-level long-tailed distributions.The authors also report applicability to various STR models.
  • 6. Conclusions: The work highlights character-level long-tailed STR as an under-explored direction compared with English-focused STR research.The conclusion states that the authors hope this work inspires future research on languages with numerous characters.

Supplementary Material

The supplementary material provides additional experimental details, explains the character-level F1 score, and documents dataset preprocessing.

  • Supplementary Material: The supplementary material covers additional Section 3 experiment details, character-level F1 computation, and dataset preprocessing.These topics are organized into Sections A.1, A.2, and A.3.

A.1. Experimental Setup in Section 3

The Section 3 experiments use the TRBA architecture and a standardized four-stage STR pipeline. They train on Korean synthetic datasets under different character distributions and evaluate character-level performance with grouped F1 scores.

  • A.1. Experimental Setup in Section 3: The experiments use TRBA for all Section 3 comparisons to maintain a common model architecture.TRBA contains transformation, feature extraction, sequence modeling, and prediction stages.
  • A.1. Experimental Setup in Section 3: The STR pipeline applies TPS transformation, ResNet feature extraction, BiLSTM-style sequence modeling, and an attention-based decoder.The decoder is selected because it is reported to outperform a CTC decoder.
  • A.1. Experimental Setup in Section 3: Models are individually trained on Korean WikiSynth, RandomSynth, and CombinedSynth datasets using the previous work's objective function.Training uses Adadelta with learning rate 1.0 for 200,000 iterations, batch size 192, and NVIDIA A100 GPUs.
  • A.1. Experimental Setup in Section 3: Character-level F1 is computed after Hirschberg sequence alignment and averaged across characters instead of using per-character accuracy.Characters are grouped as many when n_i ≥1500, medium when 100 ≤ n_i <1500, and few when n_i <100.

A.3. Pre-processing Dataset

The experiments use Korean and Chinese real-world test sets alongside synthetic datasets with different character-level distributions. Real images are filtered and cropped to form suitable text-recognition inputs.

  • AI Hub contains 151,105 Korean cropped images, while ReCTS contains 77,709 Chinese cropped images from real-world signs.
  • WikiSynth and RandomSynth are synthetic datasets designed to represent different character-level distributions.
  • Pre-processing crops word boxes, removes characters outside the predefined set, and excludes vertically oriented images meeting the stated size condition.

B. Implementation Details of CAFE-Net

CAFE-Net provides CNN- and ViT-based implementations with separate context-free and context-aware experts. The experts use different training distributions and objectives to emphasize visual character recognition or contextual word prediction.

  • CAFE-Net has CNN-based and ViT-based versions, supporting two widely used STR backbone families.
  • The context-free expert trains on RandomSynth to improve few-character recognition through balanced character exposure and visual representations.
  • The context-aware expert trains on WikiSynth and uses an external language model to learn contextual information needed for accurate whole-word prediction.
  • In the ViT implementation, transformer features are reshaped into sequences before separate prediction modules produce context-free and context-aware outputs.
  • The implementation includes linear and attention decoders, with linear decoding predicting words in parallel and attention decoding producing sequential predictions.

D. Experiments on English

English experiments evaluate CAFE-Net under case-insensitive and more challenging case-sensitive settings. The method shows limited gains when the character set is small but improves performance in the case-sensitive setting.

  • The English experiments construct WikiSynth, RandomSynth, and CS synthetic datasets from the newsgroup corpus and compare them with established STR training data.
  • The case-sensitive setting contains 94 characters, including uppercase letters, lowercase letters, numbers, and symbols.
  • CAFE-Net enhances English STR performance in the case-sensitive setting, while case-insensitive results remain comparable to existing baselines.
  • Case-insensitive English evaluation uses only 36 letters and numbers, limiting the performance improvement available from CAFE-Net.

E. Additional Results

Additional Chinese and Korean analyses examine how the confidence ensemble selects experts across character distributions. The ensemble generally exceeds either individual expert by filtering low-confidence predictions.

  • The confidence ensemble outperforms both individual experts except on Korean Synthtest and Chinese RealHard.
  • Qualitative results show the ensemble improves recognition by filtering one expert’s low-confidence prediction and selecting the other expert’s high-confidence prediction.
Loading 2304.08592v1…