Source-linked AI summary
Focusing Attention: Towards Accurate Text Recognition in Natural Images
Zhanzhan Cheng, Fan Bai, Yunlu Xu, Gang Zheng, Shiliang Pu, Shuigeng Zhou
TL;DR
Attention-based scene text recognizers can lose accurate feature-to-target alignments on complicated or low-quality images, a failure mode termed attention drift. FAN combines an attention network with a focusing network and a ResNet-based encoder, and extensive benchmark experiments show substantial performance gains over existing methods.
Problem
Existing attention-based methods perform poorly on complicated or low-quality images because they cannot reliably align feature areas with target characters.
Method
FAN combines an attention network for character recognition with a focusing network that evaluates and corrects attention, using a ResNet-based encoder for deeper representations.
Results
Extensive experiments on several benchmarks show that FAN substantially outperforms existing methods.
Takeaways & Limitations
Focusing attention can rectify drifted attention in complicated and low-quality scene-text images while supporting accurate text recognition.
Takeaways & Limitations
The AN model is easily affected by complicated or low-quality data and is hard to train on huge datasets.
Abstract
from arXiv · showhide
Scene text recognition has been a hot research topic in computer vision due to its various applications. The state of the art is the attention-based encoder-decoder framework that learns the mapping between input images and output sequences in a purely data-driven way. However, we observe that existing attention-based methods perform poorly on complicated and/or low-quality images. One major reason is that existing methods cannot get accurate alignments between feature areas and targets for such images. We call this phenomenon "attention drift". To tackle this problem, in this paper we propose the FAN (the abbreviation of Focusing Attention Network) method that employs a focusing attention mechanism to automatically draw back the drifted attention. FAN consists of two major components: an attention network (AN) that is responsible for recognizing character targets as in the existing methods, and a focusing network (FN) that is responsible for adjusting attention by evaluating whether AN pays attention properly on the target areas in the images. Furthermore, different from the existing methods, we adopt a ResNet-based network to enrich deep representations of scene text images. Extensive experiments on various benchmarks, including the IIIT5k, SVT and ICDAR datasets, show that the FAN method substantially outperforms the existing methods.
1. Introduction
Existing attention-based scene text recognizers struggle on complicated or low-quality images because their feature-to-target alignments can drift. FAN addresses this problem with a focusing network alongside an attention network and a ResNet-based encoder, outperforming existing methods in benchmark experiments.
- Existing framework: Attention-based recognition encodes images as feature-vector sequences, then uses alignment factors, glimpse vectors, and an RNN decoder to generate target characters.Each feature vector corresponds to an input-image region, enabling character-level alignment.
- Motivation: Attention-based recognizers often perform poorly on complicated or low-quality images because their learned alignments may deviate from target character regions.The paper names this mismatch between attention regions and target areas “attention drift.”
- Attention drift: A real example shows AN changing the input text “83KM” into “831K” when attention regions for the final characters are substantially displaced.The example is drawn from experiments rather than a toy case.
- Proposed method: FAN combines AN for character recognition with FN, which evaluates attention regions and automatically redirects deviated attention toward target areas.With FN, the final attention centers are rectified and FAN outputs “83KM” correctly.
- Proposed method: FAN uses a ResNet-based CNN to enrich deep scene-text representations.This encoder is adopted in addition to the focusing mechanism.
- Results: Extensive experiments on several benchmarks show that FAN substantially outperforms existing methods.The paper presents this as a performance-superiority finding across benchmark evaluations.
2. Related work
Scene text recognition has progressed from character-level bottom-up approaches to direct or sequence-based top-down recognition. This work adds deeper ResNet representations and a focusing network that evaluates and corrects attention alignment.
- Traditional approaches: Traditional scene text systems mainly use bottom-up pipelines that detect individual characters and integrate them into the output text.Examples include sliding windows, connected components, and Hough voting.
- Top-down approaches: Top-down approaches predict entire text strings directly from images, including CNN structured-output and large-vocabulary word-classification systems.Later methods formulate recognition as sequence recognition with separately encoded image patches and character sequences.
- This work: The paper extracts deeper image representations with a ResNet-based CNN before feeding feature sequences to an attention network.The authors describe this as possibly the first use of a ResNet-based CNN for scene text recognition.
- This work: FAN adds a focusing network that evaluates glimpse vectors and feeds back information to help the attention network generate more reasonable alignments.This mechanism is designed to keep attention on target-character regions.
- Relation to prior work: Although attention drift had been observed in speech recognition, this paper formally introduces the concept for scene text recognition and designs a focus mechanism for it.The authors also report that combining CTC and AN did not work well for their scene-text setting.
3. The FAN Method
FAN addresses attention drift by combining an attention network that recognizes characters with a focusing network that redirects attention to target regions. It computes attention centers, crops feature patches, performs dense focusing, and trains recognition and focusing objectives jointly.
- Focusing Network (FN): FN performs dense prediction over cropped attention regions using the corresponding glimpse vector and computes a probability distribution over the selected region.The focusing network uses trainable parameters for energy computation, with K denoting the number of label classes.
- Attention Network (AN): The attention network generates alignment factors and glimpse vectors from CNN-LSTM features while decoding the target character sequence recurrently.The decoder uses attention weights over sequential feature vectors and includes an EOS token for variable-length outputs.
- Attention Network (AN): Bad alignments between attention regions and target characters can produce poor recognition, especially when scene images are complicated or low quality.The AN lacks an alignment constraint on glimpse-vector integration, allowing attention regions to mismatch ground-truth regions.
- Focusing Network (FN): The focusing mechanism estimates each target’s attention center, crops a feature patch around it, and applies focusing over the selected attention region.Attention centers are obtained from receptive-field centers and combined into a weighted position before cropping.
- FAN Training: FAN combines a ResNet-based feature extractor, AN, and FN, and trains target generation and attention focusing simultaneously with a tunable loss trade-off.The parameter λ controls the relative impact of AN and FN, and training uses standard back-propagation.
- Decoding: During decoding, lexicon-free recognition selects the most probable character, whereas constrained recognition selects the highest-probability word from the available lexicon.The constrained settings use lexicons of sizes “50”, “1k” and “full”.
4. Performance Evaluation
FAN is evaluated against existing methods and controlled baselines across scene-text benchmarks, with additional analyses of NED, attention focusing, λ, and pixel labeling. Results consistently support FAN’s effectiveness, while showing that both its focusing mechanism and ResNet-based encoder contribute to performance.
- Performance on General Recognition Datasets: FAN substantially outperforms 18 existing methods on almost all benchmarks, except IC03 with the “Full” lexicon and IC15.It also performs better than the AN-plus-ResNet baseline in all cases.
- Performance on General Recognition Datasets: FAN significantly improves total normalized edit distance over the baseline in both constrained and unconstrained cases.NED is defined as edit distance(pred, gt)/|gt|.
- Performance on General Recognition Datasets: FAN rectifies attention drift and correctly recognizes more characters than AN in real-image examples.Figure 5 compares AN and FAN outputs and marks their attention centers and character correctness.
- Performance on General Recognition Datasets: FAN still significantly outperforms AN in accuracy and total NED when both use Shi et al.’s image encoder.This isolates the attention focusing mechanism without the ResNet-based encoder.
- The Effect of Parameter λ: FAN performs stably and achieves relatively higher performance with λ = 0.01.λ trades off the impact of AN and FN; λ = 0 removes FN, whereas λ = 1 removes AN.
- The Effect of Pixel Labeling: Even with only a small fraction of pixel-labeled training samples, FAN can guide the model to achieve satisfactory performance.The experiments vary the pixel-labeled fraction from 0 to 30% on unconstrained benchmarks.
5. Conclusion
The paper introduces attention drift to explain poor recognition on complicated or low-quality images and proposes FAN to rectify it with a focusing network. Experiments across several benchmarks show that FAN significantly outperforms existing methods.
- 5. Conclusion: The paper defines attention drift as a problem underlying poor scene-text recognition on complicated or low-quality images.FAN addresses this problem by using a focusing network to rectify the AN model’s drifted attention.
- 5. Conclusion: Extensive experiments across several benchmarks show that FAN significantly outperforms existing methods.The conclusion also identifies text detection and related tasks as future extension targets.