Source-linked AI summary
Edit Probability for Scene Text Recognition
Fan Bai, Zhanzhan Cheng, Yi Niu, Shiliang Pu, Shuigeng Zhou
TL;DR
Attention-based scene text recognition is trained with frame-wise probabilities, but missing or superfluous characters misalign ground truth and outputs, misleading and increasing the cost of training. The paper introduces edit probability (EP), which estimates string probability while accounting for these edits, and reports significantly improved recognition performance on several benchmarks.
Problem
Frame-wise training can be misled when missing or superfluous characters misalign ground-truth strings with attention output probability sequences.
Method
EP estimates the probability of generating a string from an output probability sequence while considering possible missing and superfluous characters.
Results
EP significantly improves scene text recognition performance across experiments on several benchmarks.
Takeaways & Limitations
EP provides an effective way to handle misalignment in attention-based scene text recognition.
Takeaways & Limitations
EP assumes that edit operations over the ground-truth and output sequences are conditionally independent.
Abstract
from arXiv · showhide
We consider the scene text recognition problem under the attention-based encoder-decoder framework, which is the state of the art. The existing methods usually employ a frame-wise maximal likelihood loss to optimize the models. When we train the model, the misalignment between the ground truth strings and the attention's output sequences of probability distribution, which is caused by missing or superfluous characters, will confuse and mislead the training process, and consequently make the training costly and degrade the recognition accuracy. To handle this problem, we propose a novel method called edit probability (EP) for scene text recognition. EP tries to effectively estimate the probability of generating a string from the output sequence of probability distribution conditioned on the input image, while considering the possible occurrences of missing/superfluous characters. The advantage lies in that the training process can focus on the missing, superfluous and unrecognized characters, and thus the impact of the misalignment problem can be alleviated or even overcome. We conduct extensive experiments on standard benchmarks, including the IIIT-5K, Street View Text and ICDAR datasets. Experimental results show that the EP can substantially boost scene text recognition performance.
1. Introduction
Attention-based encoder-decoder models use frame-wise probabilities, but missing or superfluous characters can misalign ground-truth text with output distributions and mislead training. The paper proposes edit probability (EP), which accounts for these edits so training can focus on problematic characters and improve recognition.
- Existing framework: Attention-based encoder-decoder models are state of the art for scene text recognition and optimize ground-truth likelihood using frame-wise output probabilities.The encoder-decoder framework produces probability distributions expected to align with text characters.
- Misalignment problem: Missing or superfluous characters during training create misalignment between ground-truth strings and output probability sequences.For example, the model may output “DVE#” or “DOOVE#” instead of the ground truth “DOVE#”.
- Misalignment problem: Frame-wise methods can assign low probabilities to correctly labeled characters under misalignment, causing misleading error propagation, costly training, and degraded recognition accuracy.In the example, ‘O’, ‘V’ and ‘E’ receive low probabilities in their corresponding distributions.
- Proposed approach: Edit probability (EP) estimates the probability of generating a string while allowing missing and superfluous characters in the alignment.The method is proposed under the attention-based encoder-decoder framework and is inspired by sequence alignment and edit distance.
- Proposed approach: EP enables training to focus on missing, superfluous, and misclassified characters, and experiments show substantially improved recognition performance.The paper evaluates EP on several scene text recognition benchmarks.
2. Related Work
Scene text recognition has progressed from handcrafted and naïve deep-learning pipelines to sequence-learning methods, especially attention-based models. The paper positions EP as an alternative response to attention-training misalignment and reports empirical gains over existing methods.
- Earlier approaches: Earlier scene text recognition methods used handcrafted features and heuristic rules to detect, classify, and assemble individual characters.These methods included character classifiers based on features such as aspect ratio, hole area ratio, and HOG.
- Earlier approaches: Naïve deep neural-network methods replaced handcrafted features with learned visual representations but still relied on character segmentation or other challenging preprocessing.Complicated backgrounds and small distances between consecutive characters make such preprocessing difficult.
- Sequence learning: Sequence-learning methods encode text images with deep neural networks and directly generate character sequences using techniques such as CTC or attention.Attention-based methods are identified as the state of the art in text recognition.
- Sequence learning: Frame-wise optimization in attention-based methods can be misled by misalignment between ground-truth and output sequences, resulting in poor performance.The paper identifies this issue as the motivation for improving attention-based recognition training.
- Edit probability: EP addresses attention-training misalignment by modeling missing and superfluous characters and empirically outperforms existing methods.The paper distinguishes EP from a joint CTC-attention approach that reportedly does not work well for scene text recognition.
3. The EP Method
EP estimates string probability by summing edit paths that accommodate missing and superfluous characters in attention decoder outputs. Dynamic programming makes this calculation efficient, while EP-Trie and sequence-generation mechanisms support prediction.
- Edit probability formulation: EP evaluates a string’s probability by summing conditional probabilities of edit paths transforming an empty string and output sequence into the target string.Edit paths use consumption, insertion, and deletion operations, whose probabilities are combined under a conditional-independence assumption.
- Edit operations: Consumption aligns a target character with an output distribution, insertion models a missing character, and deletion treats an output distribution as superfluous.After the EOS token, deletion is the only allowed operation; insertion can append a missing target character without consuming output.
- Dynamic programming: The edit-state dynamic program computes EP in O(|T| · |y|) instead of enumerating all possible edit paths.The recurrence is applied over states pairing target prefixes with prefixes of the output probability-distribution sequence.
- Edit-path example: For the illustrated DOVE# example, the maximal-probability path inserts ‘O’ and deletes superfluous output distributions while consuming the remaining target characters.The path consumes y1, y2, y3, and y4 for ‘D’, ‘V’, ‘E’, and ‘#’, then deletes y5 and y6.
- Training and prediction: EP training minimizes negative log-likelihood and can be optimized with standard backpropagation, while prediction selects the string maximizing EP.Searching the complete answer set is costly, motivating efficient lexicon-free and lexicon-based generation mechanisms.
- Efficient lexicon-based prediction: EP-Trie stores each lexicon prefix with a vector of ep(S, y1:j) values and computes child vectors from parent vectors in O(|y|) time.Shared prefixes reduce repeated computation as the lexicon grows.
4. Performance Evaluation
EP is evaluated by replacing frame-wise training and prediction in established attention-based text-recognition structures, using standard benchmarks and real training data. It improves recognition performance, handles misalignment more appropriately than FP, and adds little training time while supporting efficient lexicon-based prediction.
- Comparison with Existing Methods: EP-based methods significantly outperform Shi’s and Cheng’s frame-wise baselines across all evaluated benchmarks.Cheng’s + EP outperforms existing methods in unconstrained recognition and is best on IIIT5K in constrained recognition.
- Comparison with Existing Methods: Cheng’s + EP outperforms all existing methods without lexicons, leads on constrained IIIT5K, and is comparable to FAN on SVT and IC03.FAN requires word-level and character-level bounding-box annotations, whereas EP uses only word-level ground truth.
- Performance of EP Training: EP avoids FP’s continued probability decline after missing or superfluous characters, allowing correctly recognized later outputs to remain less affected by misalignment.FP propagates error to correctly recognized probability distributions following the misalignment; EP instead models insertion or deletion operations.
- Performance of EP Training: 6.8ms/7.0ms more per iteration is the EP training overhead for Shi’s/Cheng’s structures, compared with 170.5ms/536.0ms baseline costs.The measurement uses batch size 75.
- Performance of EP Prediction: Accuracy rises as λ increases from 0.5 to 0.98 but drops rapidly near 1 because of over-correction; λ=0.5 matches lexicon-free prediction.A ground-truth-unrelated 50k-word lexicon helps when λ is in the proper 0.9–0.98 range.
- Performance of EP Prediction: 0.11 second per image is the EP-Trie prediction cost, versus 2.566 seconds for enumeration, with identical recognition results.The comparison uses a 50k-word lexicon.
5. Conclusion
The paper introduces edit probability to handle misalignment between training text and output probability sequences caused by missing or superfluous characters. Experiments across several benchmarks show that EP significantly improves scene text recognition performance.
- Edit probability is proposed for accurate scene text recognition.
- EP handles misalignment between training text and output probability sequences caused by missing or superfluous characters.
- Experimental results across several benchmarks show that EP significantly improves recognition performance.