Source-linked AI summary
ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting
Yuliang Liu, Chunhua Shen, Lianwen Jin, Tong He, Peng Chen, Chongyu Liu, Hao Chen
TL;DR
ABCNet v2 targets the challenge of end-to-end spotting for arbitrarily shaped text, where existing approaches may require character annotations or complex processing. It uses adaptive Bezier curves, BezierAlign, coordinate convolution, and adaptive end-to-end training in a unified framework. Experiments on bilingual benchmarks report state-of-the-art performance with high efficiency, while a limitation remains for instances whose reading order conflicts with the curve orientation.
Problem
End-to-end text spotting remains challenging for arbitrarily shaped text because existing character-based and segmentation-based approaches may require character-level annotations or complex processing.
Method
ABCNet v2 uses parameterized Bezier curves for text detection, BezierAlign for curved-region feature extraction, coordinate convolution, and adaptive end-to-end training.
Results
ABCNet v2 achieves state-of-the-art text-spotting performance while maintaining high efficiency across bilingual benchmark datasets.
Takeaways & Limitations
The framework provides a concise, efficient pipeline for detecting and recognizing arbitrarily shaped text with Bezier-based structured representations and lightweight recognition features.
Takeaways & Limitations
Instances whose reading order conflicts with the longer-side Bezier orientation can produce rotated features and recognition errors.
Abstract
from arXiv · showhide
End-to-end text-spotting, which aims to integrate detection and recognition in a unified framework, has attracted increasing attention due to its simplicity of the two complimentary tasks. It remains an open problem especially when processing arbitrarily-shaped text instances. Previous methods can be roughly categorized into two groups: character-based and segmentation-based, which often require character-level annotations and/or complex post-processing due to the unstructured output. Here, we tackle end-to-end text spotting by presenting Adaptive Bezier Curve Network v2 (ABCNet v2). Our main contributions are four-fold: 1) For the first time, we adaptively fit arbitrarily-shaped text by a parameterized Bezier curve, which, compared with segmentation-based methods, can not only provide structured output but also controllable representation. 2) We design a novel BezierAlign layer for extracting accurate convolution features of a text instance of arbitrary shapes, significantly improving the precision of recognition over previous methods. 3) Different from previous methods, which often suffer from complex post-processing and sensitive hyper-parameters, our ABCNet v2 maintains a simple pipeline with the only post-processing non-maximum suppression (NMS). 4) As the performance of text recognition closely depends on feature alignment, ABCNet v2 further adopts a simple yet effective coordinate convolution to encode the position of the convolutional filters, which leads to a considerable improvement with negligible computation overhead. Comprehensive experiments conducted on various bilingual (English and Chinese) benchmark datasets demonstrate that ABCNet v2 can achieve state-of-the-art performance while maintaining very high efficiency.
1 INTRODUCTION
ABCNet v2 addresses the difficulty of real-time end-to-end spotting for arbitrarily shaped text by using a unified, efficient framework. It replaces unstructured or multi-stage processing with Bezier-based representation, aligned recognition features, and streamlined inference.
- End-to-end spotting is motivated by the limitations of separately optimized detection and recognition modules, which cannot share features effectively.
- Existing character-based and segmentation-based methods often require character-level annotations, grouping hyper-parameters, or unstructured contour outputs.
- ABCNet v2 introduces a concise parametric representation of curved scene text using Bezier curves with negligible computation overhead over bounding boxes.
- BezierAlign provides feature alignment for curved text, while shared backbone features support a lightweight recognition branch and efficient inference.
- The detection model supports multi-scale text through bidirectional multi-scale pyramid global textual features.
- ABCNet v2 jointly detects and recognizes horizontal, multi-oriented, and arbitrarily shaped text in a single shot while maintaining real-time inference speed.
- Quantization further increases inference speed with only marginal accuracy reduction.
- Experiments on various benchmarks report state-of-the-art text-spotting performance in both accuracy and speed.
2 RELATED WORK
Scene text spotting has progressed from separate detection and recognition modules toward unified end-to-end systems, while detection and recognition research increasingly addresses arbitrarily shaped text.
- End-to-end Scene Text Spotting: End-to-end spotting integrates detection and recognition in one network, replacing earlier pipelines that separately optimized two models.Early end-to-end systems used RoI Pooling or anchor-free mechanisms, initially focusing mainly on horizontal or focused text.
- Scene Text Detection: Text detection has evolved from horizontal rectangles to rotated boxes and then instance masks or polygons for arbitrarily shaped text.This progression reflects increasing flexibility in representing text geometry.
- Scene Text Recognition: Scene text recognition moved from character-level segmentation and grouping toward CNN-RNN systems that directly transcribe text sequences.Character-based recognition requires costly character-level annotations and may generalize poorly without large training datasets.
- Scene Text Recognition: Recognition methods for irregular text are commonly divided into rectification-based approaches and rectification-free approaches.STN and TPS are representative rectification methods, while AON and 2D-attention exemplify rectification-free designs.
- End-to-end Scene Text Spotting: Arbitrarily shaped spotting methods use masks, polygons, character supervision, text grouping, or fused segment features, often adding annotation or computation requirements.Mask TextSpotter uses character-level supervision, RoI Masking requires extra computation for polygon fitting, and CharNet uses character-level data and TextField grouping.
3 OUR METHOD
ABCNet v2 is an end-to-end, anchor-free framework for real-time arbitrarily shaped text spotting. It represents text with cubic Bezier curves, aligns curved features for recognition, adds coordinate encoding, and uses shared features for efficient inference.
- Framework: ABCNet v2 uses a single-shot, anchor-free convolutional detector for end-to-end arbitrarily shaped scene text spotting.Detection is densely predicted on output feature maps from the detection head.
- Bezier Curve Detection: Cubic Bezier curves provide a compact parametric representation for arbitrarily shaped text with negligible overhead compared with rectangular bounding boxes.The authors empirically find cubic curves sufficient for varied curved text, while higher orders may help text-line datasets with multiple waves.
- Bezier Curve Detection: Bezier curve detection reformulates arbitrary-shaped text detection as regression over eight control points, with straight text handled as a special case.Additional control points are interpolated along the long sides of four-vertex straight text instances.
- Bezier Ground-truth Generation: Bezier ground truth is generated by fitting a parameterized Bezier curve to polygon annotations using standard least squares.The procedure uses annotated boundary points and converts polygonal annotations into a parameterized curve.
- BezierAlign: BezierAlign samples intermediate convolution features along curved text regions and improves recognition by accurately aligning features to the detected shape.The sampling locations are obtained from upper and lower Bezier boundaries and evaluated with bilinear interpolation.
4 EXPERIMENTS
Experiments evaluate ABCNet v2 across multi-oriented, bilingual, and arbitrarily shaped scene-text benchmarks, with ablations, qualitative analysis, error analysis, and quantization studies. The results report strong accuracy and speed, while identifying limitations in unusual reading orders, fonts, and extreme curvature.
- Experimental setup: Experiments cover ICDAR’15, MSRA-TD500, ReCTS, Total-Text, and SCUT-CTW1500, with ablations on Total-Text and SCUT-CTW1500.The setup uses common ResNet-50 and FPN components, multiscale RoIAlign and BezierAlign, augmentation, and a 260K-iteration training schedule.
- Ablation study: Attention-based recognition improves results by 2.7% on Total-Text and 7.9% on SCUT-CTW1500.
- Ablation study: Coordinate convolution improves results by 2.8% and 2.9% on the two datasets without noticeable computation overhead.
- Ablation study: BezierAlign dramatically improves end-to-end results, while Bezier curve detection adds negligible computation overhead compared with standard bounding-box detection.The comparison uses previous sampling methods and a small training and testing scale.
- Benchmark results: ABCNet v2 achieves state-of-the-art performance across four detection datasets and best end-to-end performance on SCUT-CTW1500 and ICDAR 2015.On ReCTS, it is worse than Mask TextSpotter in 1-NED but has clear inference-speed advantages without the provided character-level bounding boxes.
- Limitations: Error analysis identifies failures from unusual whole-instance reading orders, unfamiliar calligraphy fonts, and extremely curved CTW1500 text with more than three crests.The latter can exceed the representation capacity of a lower-order Bezier curve, while unusual fonts generally require more training images.
5 CONCLUSION
ABCNet v2 uses parameterized Bezier curves and BezierAlign to support efficient end-to-end spotting of arbitrarily shaped text. Experiments demonstrate the effectiveness of its components and its potential for real-time deployment.
- ABCNet v2 detects arbitrarily shaped scene text with parameterized Bezier curves at negligible computation cost compared with standard bounding boxes.
- BezierAlign connects a lightweight recognition branch to Bezier curve detections for accurate feature extraction, especially on curved text.
- Comprehensive experiments demonstrate the effectiveness of the attention recognition module, biFPN structure, coordinate convolution, and adaptive end-to-end training strategy.
- Quantization techniques support deploying ABCNet v2 for real-time tasks and indicate potential across a wide range of applications.