Source-linked AI summary
SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition
Mingxin Huang, Yuliang Liu, Zhenghao Peng, Chongyu Liu, Dahua Lin, Shenggao Zhu, Nicholas Yuan, Kai Ding, Lianwen Jin
TL;DR
Existing end-to-end scene text spotters often share a backbone without directly exploiting feature interaction between detection and recognition. SwinTextSpotter uses Transformer-based set prediction with Recognition Conversion to connect the tasks and guide detection through recognition supervision. It reports superior performance across multi-oriented, arbitrarily-shaped, and multilingual benchmarks, while remaining concise and avoiding character-level annotations and a dedicated rectification module.
Problem
Existing end-to-end systems often share a backbone, but do not directly exploit feature interaction between detection and recognition; cascaded systems also suffer error accumulation and inefficient separate optimization.
Method
SwinTextSpotter combines a Transformer-based query detector with Recognition Conversion, which back-propagates recognition information to detection features for joint optimization.
Results
SwinTextSpotter achieves superior performance across six benchmarks, including 88.0% detection F-measure on SCUT-CTW1500 and Total-Text and a 9.8% 1-NED gain over ABCNet v2 on ReCTS.
Takeaways & Limitations
The concise framework supports arbitrarily-shaped and multilingual text spotting without character-level annotation or a specifically designed rectification module.
Takeaways & Limitations
Long arbitrarily-shaped text remains challenging because limited training data leaves the recognition decoder insufficiently trained, especially for SCUT-CTW1500.
Abstract
from arXiv · showhide
End-to-end scene text spotting has attracted great attention in recent years due to the success of excavating the intrinsic synergy of the scene text detection and recognition. However, recent state-of-the-art methods usually incorporate detection and recognition simply by sharing the backbone, which does not directly take advantage of the feature interaction between the two tasks. In this paper, we propose a new end-to-end scene text spotting framework termed SwinTextSpotter. Using a transformer encoder with dynamic head as the detector, we unify the two tasks with a novel Recognition Conversion mechanism to explicitly guide text localization through recognition loss. The straightforward design results in a concise framework that requires neither additional rectification module nor character-level annotation for the arbitrarily-shaped text. Qualitative and quantitative experiments on multi-oriented datasets RoIC13 and ICDAR 2015, arbitrarily-shaped datasets Total-Text and CTW1500, and multi-lingual datasets ReCTS (Chinese) and VinText (Vietnamese) demonstrate SwinTextSpotter significantly outperforms existing methods. Code is available at https://github.com/mxin262/SwinTextSpotter.
1. Introduction
End-to-end scene text spotting addresses limitations of cascaded detection-recognition pipelines by jointly optimizing both tasks. SwinTextSpotter strengthens their interaction through Recognition Conversion and achieves strong results across diverse benchmarks without character-level annotations or a dedicated rectification module.
- Cascaded spotting accumulates detection errors, separately optimizes detection and recognition, and consumes substantial memory with low inference efficiency.
- Existing end-to-end systems remain limited because detectors can be distracted by background noise and detection-recognition interaction is insufficient when only backbones are shared.
- SwinTextSpotter uses Transformer-based set prediction and Recognition Conversion to exploit detection-recognition synergy and guide localization through recognition supervision.
- Recognition Conversion back-propagates recognition information to the detector, suppresses background noise, and supports joint optimization without character-level annotation or a rectification module.
- Experiments span six multi-oriented, arbitrarily-shaped, and multilingual benchmarks, with state-of-the-art performance reported across public scene text benchmarks.
2. Related Work
Related work progresses from separate detection and recognition pipelines toward end-to-end systems that exchange information between the tasks. Methods for arbitrarily-shaped text use segmentation, feature transformation, rectification, or alignment to connect recognition with irregular text.
- Earlier scene text spotting methods treated detection and recognition as separate tasks without exchanging information between them.
- End-to-end methods unify detection and recognition, using rotated-box sampling or attention-based recognition to connect the two stages.
- Arbitrarily-shaped text methods avoid explicit rectification through character segmentation, segment-based feature transformation, thin-plate-spline transformation, or BezierAlign.
3. Methodology
SwinTextSpotter combines a Swin-Transformer backbone, query-based iterative detection, and Recognition Conversion to couple localization with recognition. Recognition features and losses are fed back into detection representations, while the framework uses multi-scale feature processing and compact mask representations.
- The architecture combines a Swin-Transformer backbone, query-based text detector, Recognition Conversion module, and attention-based recognizer.
- Query-Based Text Detector: Detection begins with learnable proposal boxes and features, which are enriched with global image information and refined recurrently across K stages.Each stage fuses RoI features with the previous proposal features before predicting updated boxes and masks.
- Query-Based Text Detector: The query-based detector treats text detection as set prediction using learnable proposal boxes and semantic proposal features instead of massive region proposals.A Transformer encoder with dynamic head processes the proposals, while bipartite matching associates predictions with ground truths.
- Query-Based Text Detector: The detector uses multi-stage self-attention and conditioned convolutions to embed prior detection information into later proposal features.RoI features extracted from previous-stage boxes are processed with dynamically generated convolutional parameters, producing the next-stage proposal features.
- Recognition Conversion: Recognition Conversion injects detection features into recognition through a Transformer encoder, upsampling, text-region masks, and multi-scale feature integration.Its recognition features are sent to the recognizer at the highest resolution.
- Recognition Conversion: Recognition loss back-propagates through Recognition Conversion to detection features and proposal features, coupling recognition supervision with localization.The resulting masks can suppress background noise more tightly than detection-head masks.
4. Experiments
Experiments evaluate SwinTextSpotter across multi-oriented, multilingual, and arbitrarily-shaped scene-text benchmarks, alongside component ablations and qualitative analyses. The method achieves strong detection and spotting results, while long arbitrarily-shaped text remains a limitation.
- Experimental Setup: Experiments cover multi-oriented, multilingual, and arbitrarily-shaped scene-text benchmarks, with ablations conducted on Total-Text.The evaluation includes RoIC13, ICDAR 2015, ReCTS, VinText, Total-Text, and SCUT-CTW1500.
- Multi-oriented and Multilingual datasets: SwinTextSpotter achieves the best strong-lexicon results on ICDAR 2015 without character-level annotations.The comparison also evaluates rotation robustness on RoIC13.
- Irregular text: 88% H-mean is achieved for detection on both Total-Text and SCUT-CTW1500, exceeding prior methods by 1.0% and 3.3%, respectively.These results concern arbitrarily-shaped text detection.
- Irregular text: 74.3% F-measure on Total-Text exceeds ABCNet v2 by 3.9% and MANGO by 1.4% for end-to-end text spotting.With a ResNet-50 backbone, the result is 72.4%, compared with MANGO’s 72.9%.
- Ablation Studies: Recognition Conversion improves detection by 3.0% and end-to-end spotting by 6.9% on Total-Text.The authors attribute the gains to more discriminative text-region features that improve recognition and benefit detection.
- Ablation Studies: The SwinTransformer improves end-to-end results by 2.9% over ResNet-50, while dilated convolution adds 0.7% for detection and 0.5% for end-to-end results.The SwinTransformer alone does not improve detection in the reported ablation.
- Ablation Studies: Two-level self-attention improves text detection by 0.8% and end-to-end text spotting by 0.9%.The mechanism is reported to compensate for the dilated Swin-Transformer’s weaker fine-grained feature capture.
- Limitation and Discussion: Long arbitrarily-shaped text remains difficult because large attention maps can mismatch the recognizer and the recognition decoder requires more training data.The limited amount of long arbitrarily-shaped data leaves the decoder insufficiently trained.
5. Conclusion
SwinTextSpotter couples detection and recognition through Recognition Conversion in a Transformer-based, set-prediction framework. Experiments report superior end-to-end spotting performance on arbitrarily-shaped and multilingual text without rectification or character-level annotation.
- Conclusion: Recognition Conversion makes detection results differentiable with respect to recognition loss, suppressing background noise in recognition features.This tightly couples detection and recognition rather than only sharing backbone information.
- Conclusion: The design removes the rectification module and enables joint optimization without character-level annotation.The framework is presented as a concise text spotter for end-to-end scene text spotting.
- Conclusion: SwinTextSpotter achieves superior end-to-end scene-text spotting performance on arbitrarily-shaped and multilingual text.The conclusion bases this claim on extensive experiments on public benchmarks.
A. Qualitative Comparisons
Qualitative comparisons show SwinTextSpotter handling difficult text instances that previous methods miss. The reported explanation is its use of detection–recognition synergy.
- Qualitative Comparisons: Previous methods fail on difficult text instances such as “Party,” whereas SwinTextSpotter handles the case.The qualitative analysis attributes this difference to exploiting the synergy between text detection and recognition.
- Qualitative Comparisons: The qualitative comparison reports that SwinTextSpotter produces more accurate detection results.This statement accompanies the comparison with previous methods.
B. Ablation study of Recognition Conversion
The Recognition Conversion ablation shows that removing RC reduces both detection and end-to-end spotting performance.
- Recognition Conversion: Without Recognition Conversion, detection performance drops from 83.2% to 82.8%.The comparison uses otherwise included components without RC.
- Recognition Conversion: Without Recognition Conversion, end-to-end scene-text spotting performance drops from 66.9% to 63.4%.The reported decline is larger for end-to-end spotting than for detection.
C. Comparison different backbone in different frameworks
Replacing ResNet50 with Swin-Transformer improves ABCNet’s end-to-end text-spotting result but does not improve detection, mirroring the SwinTextSpotter case.
- 1.8% improvement is achieved in ABCNet’s end-to-end text-spotting result when ResNet50 is replaced with Swin-Transformer.The comparison uses ABC-R50 for ABCNet with ResNet50 and ABC-Swin for ABCNet with SwinTransformer.
- No improvement is observed for ABCNet’s detection result after replacing ResNet50 with Swin-Transformer.The passage states that this pattern is similar to the case in SwinTextSpotter.
- Table 9 compares different backbones across architectures using separate detection and end-to-end text-spotting results.Det. denotes detection result, while E2E denotes end-to-end text spotting result.