Source-linked AI summary
Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network
Wenhai Wang, Enze Xie, Xiaoge Song, Yuhang Zang, Wenjia Wang, Tong Lu, Gang Yu, Chunhua Shen
TL;DR
Efficient arbitrary-shaped text detection remains challenging because prior approaches trade speed against accuracy or use costly processing pipelines. PAN addresses this with a lightweight FPEM/FFM segmentation head and Pixel Aggregation, achieving strong accuracy and real-time speed on curved-text benchmarks.
Problem
Prior arbitrary-shaped text detectors often have low inference speed because of heavy models or complicated post-processing, while efficient detectors are mostly designed for quadrangular text.
Method
PAN combines a lightweight segmentation head built from FPEM and FFM with Pixel Aggregation that learns similarity vectors to reconstruct complete text instances.
Results
PAN achieves state-of-the-art performance on two curved-text benchmarks while maintaining 58 FPS inference speed.
Takeaways & Limitations
PAN provides an efficient framework for detecting arbitrary-shaped text in real time, with experiments demonstrating advantages in speed and accuracy over previous state-of-the-art detectors.
Abstract
from arXiv · showhide
Scene text detection, an important step of scene text reading systems, has witnessed rapid development with convolutional neural networks. Nonetheless, two main challenges still exist and hamper its deployment to real-world applications. The first problem is the trade-off between speed and accuracy. The second one is to model the arbitrary-shaped text instance. Recently, some methods have been proposed to tackle arbitrary-shaped text detection, but they rarely take the speed of the entire pipeline into consideration, which may fall short in practical applications.In this paper, we propose an efficient and accurate arbitrary-shaped text detector, termed Pixel Aggregation Network (PAN), which is equipped with a low computational-cost segmentation head and a learnable post-processing. More specifically, the segmentation head is made up of Feature Pyramid Enhancement Module (FPEM) and Feature Fusion Module (FFM). FPEM is a cascadable U-shaped module, which can introduce multi-level information to guide the better segmentation. FFM can gather the features given by the FPEMs of different depths into a final feature for segmentation. The learnable post-processing is implemented by Pixel Aggregation (PA), which can precisely aggregate text pixels by predicted similarity vectors. Experiments on several standard benchmarks validate the superiority of the proposed PAN. It is worth noting that our method can achieve a competitive F-measure of 79.9% at 84.2 FPS on CTW1500.
1. Introduction
PAN addresses the unresolved challenge of detecting arbitrary-shaped text while balancing accuracy and inference speed. It combines a lightweight segmentation pipeline with feature-enhancing modules, learnable pixel aggregation, and strong benchmark results.
- 1. Introduction: Arbitrary-shaped text detection remains difficult because accurate methods often use heavy models or complicated post-processing, while efficient detectors mainly target quadrangular text.These limitations constrain deployment in real-world environments.
- 1. Introduction: PAN uses a two-step pipeline that predicts text regions, kernels, and similarity vectors before rebuilding complete text instances from predicted kernels.The design targets both segmentation and post-processing efficiency.
- 1. Introduction: PAN’s low-cost segmentation head combines Feature Pyramid Enhancement Module (FPEM) and Feature Fusion Module (FFM) to improve representations from a lightweight backbone.FPEM is U-shaped and uses separable convolutions, while FFM fuses features from FPEMs of different depths.
- 1. Introduction: 83.7% F-measure is achieved by PAN-640 on CTW1500, while PAN-320 reaches 84.2 FPS and is four times faster than EAST.The experiments cover CTW1500, Total-Text, ICDAR 2015, and MSRA-TD500.
- 1. Introduction: PAN’s Pixel Aggregation learns similarity vectors that selectively aggregate pixels near text kernels to reconstruct text instances.This learnable post-processing is designed to handle arbitrary-shaped instances precisely.
- 1. Introduction: PAN achieves state-of-the-art performance on two curved-text benchmarks while maintaining 58 FPS inference speed.The paper describes it as the first algorithm to detect curved text precisely in real time.
2. Related Work
Prior scene text detectors use anchor-based or anchor-free formulations, with different trade-offs between accuracy, efficiency, and shape flexibility. PAN follows segmentation-based approaches while targeting efficient detection of arbitrary-shaped text.
- 2. Related Work: Deep-learning text detectors are broadly divided into anchor-based and anchor-free methods.Some prioritize accuracy with heavy frameworks or complicated pipelines, while others use simpler structures to balance speed and accuracy.
- 2. Related Work: Anchor-based methods adapt object-detection frameworks to text through specialized anchors, quadrangle regression, or rotation-sensitive features.Examples include TextBoxes, TextBoxes++, and RRD.
- 2. Related Work: Anchor-free methods formulate text detection as segmentation and combine predicted text structures with post-processing to separate instances.PixelLink, EAST, DeepReg, TextSnake, and PSENet use different pixel-, geometry-, or instance-level representations.
- 2. Related Work: PAN’s architecture enhances lightweight-backbone features with FPEM and FFM, then predicts text regions, kernels, and similarity vectors.This places PAN within segmentation-based detection while explicitly modeling instance reconstruction.
- 2. Related Work: Real-time detectors use concise prediction and post-processing pipelines or parallelizable graph-based clustering, but the cited discussion identifies shape limitations for prior efficient methods.EAST uses score maps and coordinates followed by NMS, while MCN avoids NMS through graph-based clustering.
3. Proposed Method
PAN uses an efficient segmentation architecture and learnable Pixel Aggregation post-processing to detect arbitrary-shaped text. Its modules enhance multiscale features, fuse representations, and reconstruct complete text instances from kernels.
- Overall Architecture: PAN uses a lightweight backbone and segmentation-based pipeline to predict text regions, kernels, and similarity vectors before rebuilding text instances.The pipeline first performs segmentation and then reconstructs complete instances from predicted kernels.
- Feature Pyramid Enhancement Module: FPEM is a cascadable U-shaped module that enhances multiscale features through up-scale and down-scale enhancement phases.It fuses low-level and high-level information across feature maps with strides from 4 to 32 pixels.
- Feature Pyramid Enhancement Module: FPEM uses separable convolution to enlarge receptive fields and deepen the network with low computation overhead.Its FLOPS are about 1/5 of FPN.
- Feature Fusion Module: FFM combines enhanced feature pyramids by adding corresponding-scale maps and then upsampling and concatenating them into a compact feature representation.This avoids the large channel count produced by directly concatenating all feature pyramids.
- Pixel Aggregation: Pixel Aggregation reconstructs complete text instances by treating kernels as cluster centers and text pixels as samples guided by learned similarity vectors.During testing, neighboring text pixels are merged when their similarity-vector distance is below a threshold.
- Loss Function: The loss combines text-region, kernel, aggregation, and discrimination terms to supervise segmentation and maintain separable kernel representations.The balancing weights are α=0.5 and β=0.25 in all experiments, while δagg=0.5 and δdis=3.
4. Experiment
Experiments across curved, multi-oriented, and long-text benchmarks show that PAN balances detection accuracy with inference speed. Ablations attribute this balance to lightweight feature enhancement and fusion, learnable pixel aggregation, and a carefully chosen FPEM depth.
- Ablation study: Two cascaded FPEMs provide the default balance because F-measure rises and then levels off for nc ≥2, while each additional FPEM reduces speed by about 2-5 FPS.The default setting is nc = 2.
- Ablation study: One FPEM improves F-measure by about 1.5% over no FPEM with tiny extra computation.The comparison uses nc = 0 versus nc = 1.
- Ablation study: ResNet18 + 2 FPEMs + FFM reaches almost the same performance as ResNet50 + PSPNet while running over 5 times faster and using a 12.25M-parameter model.The comparison is made under the same setting.
- Ablation study: Removing FFM drops F-measure by 0.6%-0.8%, while the proposed FFM matches direct concatenation with greater efficiency.The result indicates that shallow features contribute to semantic segmentation.
- Ablation study: Removing PA causes an F-measure drop of over 1%, supporting its role in selectively aggregating text pixels around kernels.The no-PA variant merges all neighboring text pixels during post-processing.
- State-of-the-art comparisons: On CTW1500, PAN-320 achieves 77.1% F-measure at 84.2 FPS without external pre-training, while SynthText fine-tuning raises it to 79.9%.PAN-512 reaches at least 1.2% higher F-measure than other methods at 58 FPS.
- State-of-the-art comparisons: On Total-Text, PAN-640 reaches 85.0% F-measure with SynthText pre-training and nearly 40 FPS, exceeding the second-best SPCNet by 2.1%.Without external pre-training, PAN-320 runs at 82.4 FPS with 77.1% performance.
5. Conclusion
PAN is an efficient real-time framework for arbitrary-shaped text detection, combining a lightweight segmentation head with Pixel Aggregation to improve speed and accuracy.
- PAN uses a lightweight segmentation head with Feature Pyramid Enhancement Module and Feature Fusion Module for efficient feature extraction.The modules add only minor extra computation while benefiting feature extraction.
- Pixel Aggregation predicts similarity vectors between text kernels and surrounding pixels to aggregate text pixels.
- PAN demonstrates superior speed and accuracy on Total-Text and CTW1500 compared with previous state-of-the-art text detectors.
6. Appendix
The appendix compares PAN with other segmentation methods, reports cross-dataset performance, and illustrates both its broad detection abilities and remaining failure cases.
- 6.1. Cross-Dataset Validation: PAN is evaluated across word-level and text-line-level datasets in cross-dataset experiments.SynthText, ICDAR 2015, and Total-Text are word-level datasets; CTW1500 and MSRA-TD500 are text-line-level datasets.
- 6.2. Comparisons with Other Semantic Segmentation Methods: PAN addresses the challenge of preserving high efficiency with high-resolution feature maps through ResNet18, two FPEMs, and FFM.High-efficiency methods often predict on 1/8 feature maps, while direct 1/4-resolution prediction can sharply reduce speed.
- 6.3. Failure Samples: PAN still fails on large character spacing, symbols, and false positives in some difficult cases.Large character spacing remains unresolved, while symbol and false-positive errors may be alleviated with more training data.
- 6.4. More Detected Results on CTW1500, Total Text, ICDAR 2015 and MSRA-TD500: PAN detects adjacent, arbitrary-shaped, multi-oriented, long, multilingual, and variably illuminated text instances across four datasets.The examples cover CTW1500, Total-Text, ICDAR 2015, and MSRA-TD500.