Source-linked AI summary
From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network
Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, Yongdong Zhang
TL;DR
Scene text recognition needs linguistic information when visual cues are confused, but separate language models add computation and complicate fusion. VisionLAN gives a vision model language capability by training it to recognize character-wise occluded features, then uses that unified model for prediction. It reports a 39% speed improvement, state-of-the-art results on several benchmarks, and introduces the OST dataset for missing character-wise cues.
Problem
Confused visual cues make scene text recognition difficult, while separate language models add computation and create fusion challenges.
Method
VisionLAN trains a vision model with character-wise occluded feature maps so it learns linguistic information directly in visual context without an extra language model.
Results
VisionLAN improves speed by 39% and achieves state-of-the-art performance on seven benchmarks and the OST dataset.
Takeaways & Limitations
The unified structure adaptively considers visual and linguistic information for recognition when character-wise visual cues are missing or confused.
Abstract
from arXiv · showhide
In this paper, we abandon the dominant complex language model and rethink the linguistic learning process in the scene text recognition. Different from previous methods considering the visual and linguistic information in two separate structures, we propose a Visual Language Modeling Network (VisionLAN), which views the visual and linguistic information as a union by directly enduing the vision model with language capability. Specially, we introduce the text recognition of character-wise occluded feature maps in the training stage. Such operation guides the vision model to use not only the visual texture of characters, but also the linguistic information in visual context for recognition when the visual cues are confused (e.g. occlusion, noise, etc.). As the linguistic information is acquired along with visual features without the need of extra language model, VisionLAN significantly improves the speed by 39% and adaptively considers the linguistic information to enhance the visual features for accurate recognition. Furthermore, an Occlusion Scene Text (OST) dataset is proposed to evaluate the performance on the case of missing character-wise visual cues. The state of-the-art results on several benchmarks prove our effectiveness. Code and dataset are available at https://github.com/wangyuxin87/VisionLAN.
1. Introduction
Scene text recognition struggles when visual cues are confused, while separate language modules add computation and complicate information fusion. VisionLAN gives the vision model language capability during training, enabling unified recognition and evaluation on occluded text.
- VisionLAN uses only the vision model for testing, adaptively considering linguistic information in visual space when visual cues are confused.
- Scene text recognition reads textual content from natural images, but occlusion and noise can make visual cues insufficient for recognition.
- Previous two-step systems use a vision model for character texture and a separate language model for relationships among characters.
- Language-model computation grows with word length, while bidirectional reasoning further increases the burden and limits real-application efficiency.
- VisionLAN trains the vision model to reason about occluded characters and learn linguistic information directly from visual context.
- The paper introduces the OST dataset for evaluating recognition when character-wise visual cues are missing and reports state-of-the-art performance on seven benchmarks and OST.
2. Related Work
Prior scene text recognition methods divide into language-free visual approaches and language-aware approaches that add linguistic modeling. VisionLAN learns linguistic information by masking visual features at character positions rather than adding a separate language-modeling structure.
- Language-free methods treat scene text recognition as visual classification or matching and rely mainly on visual information for prediction.
- Language-aware methods use linguistic rules or learned character relationships to assist recognition through architectures including RNNs, CNNs, and transformers.
- RNN-based reasoning is serial and time-dependent, while transformer-based semantic reasoning introduces a separate language-modeling process.
- VisionLAN applies a feature-level masking strategy inspired by cloze learning, masking visual features in 2D spatial space rather than token-level information.
- The masking approach supports scene text recognition's weak-label setting by learning linguistic information with word-level annotations instead of character-wise pixel annotations.
3. Proposed Method
VisionLAN is an end-to-end framework that masks character-wise visual features during training and uses a unified visual reasoning structure to infer missing characters from context. Its modules jointly guide linguistic learning without requiring an additional language model.
- VisionLAN is an end-to-end framework containing a backbone network, Masked Language-aware Module, and Visual Reasoning Module.
- Masked Language-aware Module: The mask occludes character-wise visual messages, producing an occluded feature map for word-level prediction by the Visual Reasoning Module.
- Masked Language-aware Module: The Masked Language-aware Module takes visual features and a randomly selected character index to generate a position-aware character mask.
- Masked Language-aware Module: Weakly-supervised Complementary Learning uses parallel masked and complementary branches to cover the selected character without overlapping other characters.
- Visual Reasoning Module: The Visual Reasoning Module models visual and linguistic information together, using visual context to reason about word-level predictions from occluded features.
- Visual Reasoning Module: During testing, the VSR layer adaptively enhances visual features when current character semantics are confused, supplementing the occluded “r” in “better” and correcting predictions.
- Training Objective: The training objective combines recognition, masked-character, and other-character losses, with λ1 = λ2 = 0.5.
4. Experiment
Experiments evaluate VisionLAN’s components, training design, efficiency, benchmark performance, and behavior on visually confused characters. MLM, WCL, and VRM improve recognition, while VisionLAN achieves strong accuracy with lower computational cost.
- Experimental setup: The experiments train on SynthText and SynthText90K and evaluate six benchmarks spanning regular and irregular scene-text conditions.The benchmarks are IIIT5K, IC13, IC15, SVT, SVTP, and CUTE80.
- Experimental setup: The OST dataset contains 4,832 images manually occluded at weak or heavy degree on one character per image.It is collected from six existing benchmarks and targets missing character-wise visual cues.
- Ablation study: A 1:1 occlusion ratio substantially improves MLM performance, while increasing the ratio to 2:1 slightly reduces performance.The authors attribute the drop to disrupting the balance between rich and weak visual cues during training.
- Ablation study: MLM improves irregular-dataset accuracy by at least 2% and regular-dataset accuracy by 0.9%, 0.8%, and 1.7% on IIIT5K, IC13, and SVT.These results support using linguistic learning to address blur, occlusion, noise, and other confused visual cues.
- Ablation study: Weakly-supervised Complementary Learning outperforms training with only the occluded-character branch or only the remaining-string branch.The comparison tests whether both complementary branches are needed during MLM training.
- State-of-the-art comparisons: VisionLAN achieves state-of-the-art performance across six public datasets, with gains of 1%, 0.2%, and 0.2% on IIIT5K, IC13, and SVT.On irregular datasets, gains are 1%, 0.9%, and 0.7% on IC15, SVTP, and CUTE80; speed improves by at least 39% without extra parameters.
- Qualitative analysis: VisionLAN improves recognition of confusing, background-disturbed, and blurred characters when linguistic information is used.The qualitative examples include correcting “f” to “e” in “before” and suppressing interference from occlusion, illumination, and textures.
5. Conclusion
The conclusion presents VisionLAN as a unified scene-text recognizer that equips the vision model with language capability. It reports efficient, strong performance and introduces OST for evaluating missing character-wise visual cues.
- Conclusion: VisionLAN transforms two-step scene-text recognition into one-step recognition by unifying visual and linguistic information without an extra language model.The architecture is described as concise and effective.
- Conclusion: Figure 7 shows input images above generated Maskc maps with corresponding character indices P.The figure illustrates character-wise masking generated by the model.
- Conclusion: Figure 8 compares VisionLAN predictions without MLM against predictions with VisionLAN for confused, background-disturbed, and blurred characters.The top string is the prediction without MLM; the bottom string is the VisionLAN prediction.
- Conclusion: Experiments on seven benchmarks and the OST dataset demonstrate VisionLAN’s effectiveness and efficiency.The conclusion also states that VisionLAN has stronger language capability while maintaining high efficiency.