Source-linked AI summary
Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition
Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, Yongdong Zhang
TL;DR
Scene text recognition needs effective linguistic modeling because visual evidence is often degraded and existing language models can be implicit, unidirectional, or noise-sensitive. ABINet addresses these issues with autonomous vision-language units, bidirectional cloze reasoning, iterative correction, and ensemble self-training. It achieves state-of-the-art results on mainstream benchmarks, with particular strength on low-quality images and promising improvement toward human-level recognition.
Problem
Effectively modeling linguistic behavior in end-to-end scene text recognition remains an open problem, especially when blur, noise, or occlusion weakens visual evidence.
Method
ABINet decouples vision and language learning, uses a bidirectional cloze network, iteratively corrects predictions, and applies ensemble self-training to unlabeled data.
Results
ABINet achieves state-of-the-art results on mainstream benchmarks, especially on low-quality images, while self-training improves performance across benchmark datasets.
Takeaways & Limitations
Explicit, bidirectional, and iterative linguistic modeling, together with unlabeled-data training, is promising for improving scene text recognition toward human-level recognition.
Takeaways & Limitations
Current methods remain vulnerable to noise input, and parallel-prediction methods can suffer from unaligned text and visual-prediction lengths.
Abstract
from arXiv · showhide
Linguistic knowledge is of great benefit to scene text recognition. However, how to effectively model linguistic rules in end-to-end deep networks remains a research challenge. In this paper, we argue that the limited capacity of language models comes from: 1) implicitly language modeling; 2) unidirectional feature representation; and 3) language model with noise input. Correspondingly, we propose an autonomous, bidirectional and iterative ABINet for scene text recognition. Firstly, the autonomous suggests to block gradient flow between vision and language models to enforce explicitly language modeling. Secondly, a novel bidirectional cloze network (BCN) as the language model is proposed based on bidirectional feature representation. Thirdly, we propose an execution manner of iterative correction for language model which can effectively alleviate the impact of noise input. Additionally, based on the ensemble of iterative predictions, we propose a self-training method which can learn from unlabeled images effectively. Extensive experiments indicate that ABINet has superiority on low-quality images and achieves state-of-the-art results on several mainstream benchmarks. Besides, the ABINet trained with ensemble self-training shows promising improvement in realizing human-level recognition. Code is available at https://github.com/FangShancheng/ABINet.
1. Introduction
ABINet addresses the open problem of modeling linguistic behavior in scene text recognition through autonomous, bidirectional, and iterative language modeling. It combines explicit language learning, bidirectional cloze-style context, iterative correction, and ensemble self-training.
- Motivation: Scene text recognition benefits from linguistic context when blur, noise, or occlusion makes visual character discrimination difficult.Characters can be reasoned from surrounding context when visual evidence is degraded.
- Motivation: Human-reading assumptions motivate autonomous, bidirectional, and iterative language modeling for scene text recognition.Autonomous learning separates vision and language, bidirectional reasoning uses both sides of an illegible character, and iterative reasoning progressively corrects predictions.
- Limitations of prior methods: Existing language models implicitly learn linguistic rules, often use effectively unidirectional features, and remain vulnerable to noisy visual inputs.Auto-regression accumulates recognition errors, while parallel prediction still passes visual-model errors to the language model.
- ABINet: ABINet blocks gradient flow between vision and language models, introduces a bidirectional cloze network, and iteratively corrects predictions.The language model can be trained independently, BCN jointly conditions on left and right context, and repeated execution progressively refines predictions.
- Results: ABINet achieves state-of-the-art performance on mainstream benchmarks, especially on low-quality images, while ensemble self-training improves prospects for human-level recognition.The self-training method exploits unlabeled data and reports promising improvement toward human-level recognition.
3.1. Vision Model
The vision model extracts visual features and uses position attention to transcribe them into character probabilities in parallel. Its query-based design uses positional encodings for character order.
- Architecture: The vision model combines a backbone network with a position attention module.ResNet and Transformer units serve as feature extraction and sequence modeling components.
- Position attention: Position attention transcribes visual features into character probabilities in parallel using a query-based paradigm.The module follows prior query-based approaches for parallel character prediction.
- Position attention: Q consists of positional encodings for character orders, with T representing the character-sequence length.The positional encoding matrix has dimensions T × C.
- Architecture: The feature-processing network contains five residual blocks, with down-sampling after the first and third blocks.This implementation detail describes the residual backbone configuration.
3.2. Language Model
The language model is designed as an autonomous, bidirectional, and iterative unit that learns linguistic knowledge explicitly, represents context in both directions, and corrects noisy visual predictions.
- Autonomous: Blocking gradient flow makes the language model an independent functional unit that can be trained separately from images and text.This design explicitly encourages learning linguistic rules rather than implicitly coupling them with visual recognition.
- Bidirectional Representation: BCN uses cloze-style attention masks to prevent each character from seeing itself while producing bidirectional representations.Character probability vectors provide keys and values, while positional encodings provide queries.
- Bidirectional Representation: BCN computes bidirectional features in parallel and requires only half the computations and parameters of ensemble unidirectional models.The comparison is made against ensemble models used to combine two unidirectional representations.
- Iterative Correction: Iterative language modeling repeatedly feeds predictions from the vision or fusion model back into the language model to correct recognition errors.The first iteration uses vision predictions; later iterations use the preceding fusion prediction.
- Iterative Correction: Iterative fusion can alleviate Transformer-based recognition failures when predicted character length is unaligned with the ground truth.Repeated visual-linguistic fusion gives the model several opportunities to address this problem.
3.3. Fusion
The fusion stage aligns visual and linguistic features through a gated mechanism for the final recognition decision.
- Fusion: A gated mechanism combines visual and linguistic features to produce the final decision.The visual and language models operate on different modalities, so gating is used to align their features.
- Fusion: The fusion parameters include Wf ∈R2C×C and G ∈RT ×C.
3.4. Supervised Training
ABINet is trained end-to-end with multitask objectives that supervise the vision, language, and fusion outputs across iterative predictions.
- Supervised Training: The multitask training objective combines cross-entropy losses from the vision, language, and fusion outputs.These losses are denoted Lv, Ll, and Lf, respectively.
- Supervised Training: The objective includes losses from each iterative prediction and balances vision and language terms with λv and λl.
3.5. Semi-supervised Ensemble Self-training
Ensemble self-training extends ABINet with unlabeled images by generating, filtering, and repeatedly updating soft pseudo-labels using iterative predictions.
- Procedure: The training algorithm periodically updates pseudo-labels with the current model and refilters them before continuing optimization.The update occurs at step Nupl and training proceeds until Nmax.
- Procedure: The method generates soft pseudo-labels for unlabeled images and retrains ABINet using labeled and filtered unlabeled batches.Training begins with labeled data and then incorporates selected pseudo-labeled examples.
- Pseudo-label Filtering: Text certainty is defined by the minimum character confidence within each text instance.This criterion filters pseudo-labels whose least-confident character falls below threshold Q.
- Pseudo-label Filtering: Iterative predictions for each character are ensembled to smooth the influence of noisy pseudo-labels.
4. Experiment
Experiments evaluate ABINet’s vision, autonomous, bidirectional, iterative, and semi-supervised components across standard scene-text benchmarks. The results show gains from stronger vision models, independent language modeling, BCN, iterative correction, and self-training, especially on challenging images.
- Datasets and implementation: Experiments use synthetic MJSynth and SynthText for training and six standard benchmarks for testing.The benchmarks are IC13, IC15, IIIT, SVT, SVTP, and CUTE.
- Vision model: Upgrading the vision model improves accuracy substantially, but increases parameter count and reduces speed.The study recommends position attention for feature extraction and a deeper Transformer for sequence modeling.
- Autonomous strategy: Allowing gradient flow between vision and language models decreases performance by 0.9% on average.The authors attribute this to language-model overfitting enabled by the vision model during training.
- Bidirectional representation: BCN improves accuracy and speed over unidirectional representations, while outperforming bidirectional SRN ensembles especially on challenging datasets.ABINet with BCN is reported to be about 20%-25% faster than the bidirectional SRN ensemble.
- Iterative correction: Iterative correction increases accuracy by up to 1.3% and 1.0% on SVT for SV and LV respectively.Accuracy saturates after more than three iterations, while inference time increases linearly with iteration count.
- State-of-the-art comparison and semi-supervised training: ABINet-SV outperforms reproduced SRN-SV by 0.5%, 2.3%, 0.4%, 1.4%, 0.6%, and 1.4% on IC13, SVT, IIIT, IC15, SVTP, and CUTE respectively.ABINet shows particular strength on low-quality images, while ensemble self-training further improves performance using unlabeled Uber-Text images.
5. Conclusion
ABINet uses autonomous, bidirectional, and iterative language modeling to exploit linguistic knowledge in scene text recognition, with ensemble self-training extending the approach to unlabeled data.
- ABINet combines autonomous, bidirectional, and iterative principles to explicitly learn linguistic rules, use two-sided character context, and progressively correct predictions.
- An ensemble self-training method built on ABINet uses unlabeled data for semi-supervised scene text recognition.
- Experiments show ABINet performs especially well on low-quality images, while unlabeled-data exploitation is presented as promising for human-level recognition.