PaperlayerView exact revision on arXiv

Source-linked AI summary

PAN++: Towards Efficient and Accurate End-to-End Spotting of Arbitrarily-Shaped Text

Wenhai Wang, Enze Xie, Xiang Li, Xuebo Liu, Ding Liang, Zhibo Yang, Tong Lu, Chunhua Shen

arXiv:2105.00405v4cs.CV

TL;DR

Efficient and accurate end-to-end spotting of arbitrarily-shaped text remains challenging, particularly for curved text and real-time deployment. PAN++ uses a kernel-based representation with lightweight feature, detection, and recognition modules, achieving 64.9% end-to-end F-measure at 29.2 FPS on Total-Text.

  • Problem

    Existing end-to-end spotters struggle with arbitrarily-shaped text and real-world efficiency because some assume straight layouts while accurate methods can use heavy models or complicated pipelines.

  • Method

    PAN++ combines a kernel representation with stacked FPEMs, Pixel Aggregation detection, Masked RoI, and a lightweight attention-based recognition head.

  • Results

    64.9% end-to-end F-measure at 29.2 FPS on Total-Text, 11 FPS faster than the previous best method with competitive accuracy.

  • Takeaways & Limitations

    PAN++ provides an efficient end-to-end framework that maintains competitive accuracy while spotting arbitrarily-shaped text.

  • Takeaways & Limitations

    PAN++ still fails on large character spacing, text-like areas, and fancy font styles; the authors attribute some cases to insufficient training samples.

Abstract

from arXiv · show

Scene text detection and recognition have been well explored in the past few years. Despite the progress, efficient and accurate end-to-end spotting of arbitrarily-shaped text remains challenging. In this work, we propose an end-to-end text spotting framework, termed PAN++, which can efficiently detect and recognize text of arbitrary shapes in natural scenes. PAN++ is based on the kernel representation that reformulates a text line as a text kernel (central region) surrounded by peripheral pixels. By systematically comparing with existing scene text representations, we show that our kernel representation can not only describe arbitrarily-shaped text but also well distinguish adjacent text. Moreover, as a pixel-based representation, the kernel representation can be predicted by a single fully convolutional network, which is very friendly to real-time applications. Taking the advantages of the kernel representation, we design a series of components as follows: 1) a computationally efficient feature enhancement network composed of stacked Feature Pyramid Enhancement Modules (FPEMs); 2) a lightweight detection head cooperating with Pixel Aggregation (PA); and 3) an efficient attention-based recognition head with Masked RoI. Benefiting from the kernel representation and the tailored components, our method achieves high inference speed while maintaining competitive accuracy. Extensive experiments show the superiority of our method. For example, the proposed PAN++ achieves an end-to-end text spotting F-measure of 64.9 at 29.2 FPS on the Total-Text dataset, which significantly outperforms the previous best method. Code will be available at: https://git.io/PAN.

1 INTRODUCTION

PAN++ addresses the challenge of efficient, accurate end-to-end spotting for arbitrarily-shaped text by combining a kernel-based representation with lightweight detection, feature-enhancement, and recognition components. It targets the limitations of separate detection and recognition, straight-text assumptions, and slow existing pipelines.

  • Motivation: Existing methods often treat text detection and recognition separately, limiting their ability to exploit complementarity between the two tasks.
  • Motivation: Straight-text spotters can fail on curved text because both detection and recognition components assume regular layouts.
  • Motivation: Efficient and accurate end-to-end spotting of arbitrarily-shaped text remains unresolved because accurate methods may rely on heavy models or complicated pipelines.
  • Kernel representation: The kernel representation models each text line as a central text kernel surrounded by peripheral pixels, supporting curved and adjacent text while enabling one-stage prediction.
  • Framework: PAN++ detects text lines through segmentation and Pixel Aggregation, then recognizes content with Masked RoI and an attention-based decoder.
  • Results: 64.9% end-to-end F-measure at 29.2 FPS on Total-Text demonstrates PAN++'s balance of accuracy and speed, 11 FPS faster than the previous best method.

2 RELATED WORK

Related work spans unified text spotting, arbitrarily-shaped text representations, scene text detection, and recognition. PAN++ builds on pixel-based representations and integrates detection with recognition in a unified framework while targeting inference efficiency.

  • End-to-End Text Spotting: End-to-end text spotters unify text-line detection and recognition, with prior methods broadly divided into regular and arbitrarily-shaped text spotters.The cited review organizes representative deep-learning text spotters into these two categories.
  • Arbitrarily-Shaped Text Spotting: Prior arbitrarily-shaped spotters include Mask TextSpotter, CharNet, TextDragon, and attention-based extensions of Mask R-CNN, but some require character-level annotations.Mask TextSpotter and CharNet are described as requiring character-level supervision for training.
  • Arbitrarily-Shaped Text Spotting: Pixel-based representations are described as more flexible and reliable than Bezier curves, which represent only curve series under specific constraints.The comparison concerns representation flexibility and reliability for irregular text instances.
  • Arbitrarily-Shaped Text Spotting: Most prior arbitrarily-shaped spotters emphasize accuracy, while inference speed is rarely addressed; this motivates an improved pixel-based representation and efficient framework.The paper identifies efficiency as an under-addressed aspect of prior work.
  • Scene Text Detection: Scene text detection methods progressed from box regression for horizontal or oriented text to segmentation-based approaches capable of curved text, typically using separate recognition models.The related work contrasts detection-only methods with unified spotting systems.
  • Scene Text Detection: PAN++ integrates improved detection with a recognition module, using a unified model rather than treating recognition as a separate model.The framework combines advances from PSENet and PAN with a recognition module.
  • Scene Text Recognition: Attention-based recognizers can handle irregular shapes by fusing visual and temporal features, motivating PAN++’s lightweight attention-based recognition head.The proposed head uses multi-head attention with CNN and RNN features.

3 PROPOSED METHOD

PAN++ uses a kernel-based, pixel-level representation and lightweight components to detect arbitrarily shaped text, assemble text lines, and recognize their contents efficiently.

  • Kernel representation: The kernel representation uses central text kernels and surrounding pixels to flexibly describe arbitrary shapes and distinguish adjacent instances.Its pixel-based formulation can be predicted by a fully convolutional network.
  • Feature enhancement: PAN++ enhances lightweight-backbone features with stacked Feature Pyramid Enhancement Modules that fuse multiscale information with limited computation.FPEMs use separable convolutions and can enlarge receptive fields as their stack number increases.
  • Detection: The detection head predicts text regions, text kernels, and instance vectors, which Pixel Aggregation combines into complete text lines.Text regions preserve shape, kernels separate adjacent instances, and instance vectors guide pixels toward the corresponding kernel.
  • Detection: Pixel Aggregation uses aggregation and discrimination objectives to pull pixels toward their text kernels while separating kernels from one another and the background.The losses optimize instance vectors used for clustering and assembly.
  • Recognition: Masked RoI extracts masked text-line features for recognition, removing background or other-text noise while avoiding spatial rectification.The recognition head uses a lightweight multi-head attention design with two LSTM layers and one attention layer.

4 EXPERIMENT

PAN++ is evaluated on curved-text detection and end-to-end spotting benchmarks using controlled training and testing protocols. It achieves strong accuracy while maintaining substantially higher inference speed than prior methods.

  • Experimental Setup: The evaluation covers text detection and end-to-end spotting across Total-Text, CTW1500, ICDAR 2015, MSRA-TD500, and RCTW-17.Experiments use established datasets and compare detection-only and end-to-end settings.
  • Curved Text Detection: 84.0% F-measure and 36 FPS are achieved on CTW1500 after SynthText pre-training, exceeding all compared methods while remaining fastest.The result uses PAN++ with the shorter side set to 640 pixels.
  • Curved Text Detection: 85.3% F-measure is obtained on Total-Text at nearly 40 FPS, surpassing prior methods by over 1.7 points and running about three times faster than TextSnake.At a shorter-side scale of 320 pixels, PAN++ also reaches 84.9 FPS with 80.0% F-measure.
  • Curved Text Spotting: 66.4% end-to-end F-measure is achieved at 1.3 times the speed of ABCNet, while the 512-pixel configuration reaches 64.9% at 29.2 FPS.The best reported Total-Text spotting result is 68.6% F-measure at 21.1 FPS, compared with 4.8 FPS for the second-best method.
  • Curved Text Spotting: The experiments report state-of-the-art text spotting performance on Total-Text, with especially large gains in inference speed.Qualitative results also demonstrate curved-text spotting in the reported benchmark setting.

4.3 Experiments on Straight Text Datasets

PAN++ remains effective on straight-text datasets for both detection and end-to-end spotting. The results show a consistent accuracy–speed balance across IC15 and MSRA-TD500.

  • Straight Text Detection: 81.2% F-measure at 28.2 FPS is achieved on IC15 without external pre-training, surpassing EAST by 3.0 points while running twice as fast.With SynthText fine-tuning, PAN++ reaches 83.1% and runs 25 times faster than TextSnake.
  • Straight Text Detection: 87.5% F-measure is obtained on IC15 with high-resolution input, while inference remains fastest at 12.6 FPS.This configuration exceeds all compared counterparts under the stated high-resolution setting.
  • Straight Text Detection: 88.4% F-measure is achieved on MSRA-TD500, at least 5.4 points above previous methods.The method also reports 80.9% and 84.7% under the two external-data training conditions.
  • Straight Text Spotting: 69.2% generic-lexicon F-measure is achieved for IC15 at 13.8 FPS, exceeding the previous best by 1.2 points while remaining fastest.At a shorter-side scale of 736 pixels, PAN++ reaches 66.2% at 24.5 FPS.
  • Straight Text Spotting: The results support an accuracy–speed balance for both curved and straight text detection and recognition.Qualitative straight-text spotting results are also reported.

4.4 Experiments on Chinese Text Dataset

PAN++ is tested on Chinese text using RCTW-17, where recognition is evaluated with average edit distance. It improves detection, recognition, and speed over the baseline.

  • Experiment Setting: RCTW-17 evaluates Chinese text with text-detection F-measure and average edit distance, using a validation split and no external text datasets.The dataset presents a large character-category challenge, with up to 3,000 categories.
  • Chinese Text Results: The qualitative evaluation includes text detection and recognition results on RCTW-17.The dataset is used to assess performance on Chinese text distinct from English text.
  • Chinese Text Results: 68.0% detection F-measure and 22.2 average edit distance are achieved with ResNet50 and larger input scale.These results improve over the baseline by 14.1 and 2.9 points while running at 10.7 FPS versus 5.9 FPS.

4.5 Comparison with Conference Versions

PAN++ extends earlier conference-version detectors into an end-to-end arbitrarily-shaped text spotter. Architectural changes improve detection performance while preserving comparable or faster inference speed.

  • Architecture Evolution: PAN++ adds Masked RoI and a lightweight recognition head to evolve the earlier detector into an end-to-end text spotter.The rebuilt architecture integrates feature extraction and recognition with the existing detection pipeline.
  • Architectural Improvements: The method improves FPEM and PA while systematically comparing kernel representation with mainstream alternatives for real-time suitability.FPEM is simplified by combining earlier modules, and PA is enhanced with background awareness.
  • Architecture Evolution: PAN++ enables simultaneous detection and recognition of arbitrarily-shaped text, which conference-version models did not support.The extension also reports improved text-detection performance over its predecessors.
  • Comparison Results: 84.5% F-measure on Total-Text is reported for PAN++, improving over PAN by 1.0 point and PSENet by 6.4 points at similar or faster speed.The comparison is presented as evidence that the incorporated improvements are effective for a fast end-to-end spotter.

4.6 Ablation Study

The ablation studies examine the roles of text kernels, shrinking rate, FPEMs, PA, Masked RoI, recognition design, backbone choice, and end-to-end integration in PAN++. These components affect detection quality, recognition robustness, efficiency, or the balance between accuracy and speed.

  • Text kernels: Text kernels cannot serve alone as detection results because they do not cover complete text shapes and cannot directly support recognition.Using only kernels sharply lowers detection F-measure, while CRNN fails to recognize text cropped to the kernel.
  • Shrinking rate: When the shrinking rate is too small or too large, detection F-measure drops; at r ≥0.9, kernels may fail to separate nearby text lines.At r = 1, the detector degenerates into binary segmentation without PA and loses adjacent-line separation.
  • FPEM: 66.4% Total-Text text spotting F-measure with two FPEMs is 1.9 points above the no-FPEM model, while performance levels off beyond two modules.The two-stacked FPEM design also reaches 66.4% versus 65.9% for FPN and 66.9% for NAS-FPN, with lower time and parameter costs.
  • Pixel Aggregation: PA improves text-kernel quality and raises F-measure by over 1.0 points compared with removing it.It reduces cases where one text line is split or adjacent lines are connected by combining auxiliary loss and post-processing aggregation.
  • Recognition head: Masked RoI improves text spotting F-measure by at least 1.1% over naive RoI extraction, which can fail in noisy backgrounds.The proposed recognition head also uses half SAR’s recognition time, 2.3 ms versus 5.0 ms.
  • End-to-end integration: 86.0% detection F-measure and 66.4% end-to-end spotting F-measure are 1.1 and 3.0 points above the separate model, with higher inference speed.Heavier backbones improve both datasets by over 1 point but reduce FPS.

4.7 Network Output Visualization

PAN++ visualizations show that its intermediate representations preserve text-line shapes, separate instances, and support recognition of irregular text through focused attention masks.

  • Intermediate representations: Text regions preserve complete text-line shapes, while text kernels clearly distinguish different text lines.Instance-vector projections further visualize the separation of pixels and kernels belonging to different text lines.
  • Recognition visualization: Decoder attention masks focus on correct positions even when recognizing irregular text lines.The starter attention mask identifies the beginning of the string, and decoder masks track positions at each recognition step.

4.8 Robustness Analysis

The robustness analysis evaluates cross-dataset detection without external datasets or a recognition head, covering word-level and text-line-level benchmarks. PAN++ retains competitive cross-dataset detection performance under these settings.

  • Evaluation setting: Cross-dataset evaluation tests PAN++ across word-level and text-line-level datasets without external datasets during training or a recognition head.The tested datasets include Total-Text, CTW1500, IC15, and MSRA-TD500, with image scales set separately by dataset.
  • Cross-dataset results: Models trained on SynthText achieve fairish IC15 and Total-Text performance, while models trained on manually annotated data exceed 62 text detection F-measure cross-dataset.The reported results indicate detection remains effective across datasets under the stated training setup.

4.9 Speed Analysis

Recognition accounts for half of PAN++’s total time cost, making parallel recognition a direct route to reducing practical inference time.

  • Component timing: Recognition consumes half of PAN++’s total time cost on Total-Text.A producer-consumer implementation that runs recognition in parallel can reduce total time to the original one-half.

4.10 Failure Cases and Discussion

PAN++ performs well on most arbitrarily-shaped text cases but still fails on difficult images involving large character spacing, text-like areas, and fancy fonts.

  • Failure Cases: PAN++ still fails on images with large character spacing, text-like areas, and fancy font styles.The authors attribute large character spacing to a problem shared by other state-of-the-art methods, while linking the latter two failures to insufficient training samples.
  • Discussion: Large character spacing remains difficult because solving it requires linguistic features.
  • Discussion: Text-like areas and fancy font styles are attributed mainly to a lack of corresponding training samples.The authors expect these problems to lessen with sufficient training data.

5 CONCLUSION

PAN++ extends kernel-based text spotting into an efficient end-to-end framework by speeding up detection and recognition with lightweight modules. Experiments on text detection and end-to-end spotting report advantages in both accuracy and inference speed over several prior state-of-the-art methods.

  • Conclusion: PAN++ extends kernel representation into an efficient end-to-end arbitrarily-shaped text spotting framework.The framework uses stacked FPEMs, a PA-based detection head, and a recognition head with Masked RoI.
  • Conclusion: The framework speeds up all parts of text spotting through lightweight feature enhancement, detection, and recognition modules.
  • Conclusion: PAN++ shows advantages in both accuracy and inference speed on text detection and end-to-end text spotting tasks compared with several prior state-of-the-art methods.
Loading 2105.00405v4…