Source-linked AI summary
Character Region Awareness for Text Detection
Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, Hwalsuk Lee
TL;DR
Rigid word-level representations limit detection of arbitrarily shaped text, and character-level annotations are scarce. CRAFT instead predicts character regions and affinities, using weak supervision for real word-level datasets. It outperforms state-of-the-art detectors and shows flexibility on long, curved, and arbitrarily shaped texts.
Problem
Word-level bounding boxes inadequately represent arbitrary text shapes, while most datasets lack affordable character-level annotations.
Method
CRAFT localizes individual characters with region scores, links them using affinity scores, and estimates character-level ground truths through weak supervision.
Results
CRAFT outperforms state-of-the-art text detectors and demonstrates flexibility on long, curved, and arbitrarily shaped texts.
Takeaways & Limitations
Character-level region and affinity modeling supports bottom-up detection across varied text shapes without requiring character-level annotations for real datasets.
Takeaways & Limitations
CRAFT distinguishes Bangla and Arabic characters less effectively than Latin, Korean, Chinese, and Japanese because these cursive scripts are difficult to segment individually.
Abstract
from arXiv · showhide
Scene text detection methods based on neural networks have emerged recently and have shown promising results. Previous methods trained with rigid word-level bounding boxes exhibit limitations in representing the text region in an arbitrary shape. In this paper, we propose a new scene text detection method to effectively detect text area by exploring each character and affinity between characters. To overcome the lack of individual character level annotations, our proposed framework exploits both the given character-level annotations for synthetic images and the estimated character-level ground-truths for real images acquired by the learned interim model. In order to estimate affinity between characters, the network is trained with the newly proposed representation for affinity. Extensive experiments on six benchmarks, including the TotalText and CTW-1500 datasets which contain highly curved texts in natural images, demonstrate that our character-level text detection significantly outperforms the state-of-the-art detectors. According to the results, our proposed method guarantees high flexibility in detecting complicated scene text images, such as arbitrarily-oriented, curved, or deformed texts.
1. Introduction
Conventional word-level text detection struggles with curved, deformed, and extremely long text, while character-level awareness offers a bottom-up alternative. CRAFT detects individual characters, links them into text instances, and uses weak supervision to address missing character annotations.
- Word-level bounding boxes are difficult to apply to curved, deformed, or extremely long text.
- Character-level awareness handles challenging text by linking successive characters in a bottom-up manner.
- Most existing datasets lack character-level annotations, making character-level ground-truth creation costly.
- CRAFT predicts character region and affinity scores to localize individual characters and group them into text instances.
- CRAFT uses weakly supervised learning to estimate character-level ground truths for existing real word-level datasets.
- CRAFT outperforms state-of-the-art detectors and remains flexible on long, curved, and arbitrarily shaped texts.
2. Related Work
Prior scene text detectors mainly use regression, segmentation, or end-to-end recognition frameworks, but word-level representations remain structurally limited for irregular shapes. Character-level approaches address this limitation with finer units, while differing in annotation and representation requirements.
- Deep learning scene text detectors commonly adopt regression, segmentation, or end-to-end recognition architectures.
- Regression-based text detectors: Regression-based detectors modify kernels, anchors, or rotation-sensitive filters, but remain structurally limited in capturing all possible text shapes.
- Segmentation-based text detectors: Segmentation-based detectors estimate text regions at the pixel level, including word bounding areas and partial text segments.
- End-to-end text detectors: End-to-end detectors jointly train detection and recognition modules to leverage recognition results for detection accuracy.
- Word-level annotation boundaries are ambiguous because words can be separated by meaning, spaces, or color, weakening regression and segmentation ground truths.
- Character-level text detectors: Earlier character-level methods rely on MSER, character annotations, or rectangular anchors, limiting robustness, supervision flexibility, or deformation handling.
3. Methodology
CRAFT detects text by predicting character regions and inter-character affinities, then grouping characters into text instances. Its weakly supervised training estimates character annotations for real images and supports flexible outputs for arbitrarily shaped text.
- Network and representations: CRAFT uses a fully convolutional network to produce character-region and character-affinity score maps.The region score localizes character centers, while the affinity score represents spaces between adjacent characters.
- Network and representations: Gaussian heatmaps encode character-center probabilities instead of rigid binary segmentation regions.The same heatmap representation is used for both region and affinity scores.
- Network and representations: Affinity boxes connect adjacent character boxes by using centers of their upper and lower triangles.This representation provides supervision for linking neighboring characters into text instances.
- Network and representations: The ground-truth design lets convolutional filters focus on intra-character and inter-character structure, enabling detection of long text with small receptive fields.Unlike box regression, the model need not cover an entire long text instance with one receptive field.
- Weakly-supervised learning: For real word-level data, an interim model predicts character regions, watershed splitting generates pseudo-ground-truth boxes, and coordinates are transformed back to the original image.The pipeline crops word images, predicts region scores, splits characters, and unwarps the resulting boxes.
- Weakly-supervised learning: Training combines real and synthetic images, using true ground truth for synthetic data and progressively improving confidence for real-image predictions.The model learns unfamiliar text appearances, including irregular fonts and synthesized text distributions differing from SynthText.
- Weakly-supervised learning: Pseudo-ground-truth quality is weighted using confidence derived from predicted character counts relative to transcription length.When confidence falls below 0.5, estimated boxes are replaced by equal-width character divisions and the confidence is set to 0.5.
- Inference: At inference, connected components from region and affinity maps yield word boxes, while polygon generation provides outputs for curved or arbitrarily shaped text.Thresholded score maps are labeled and enclosed by minimum-area rotated rectangles; polygons are generated around character regions for curved text.
4. Experiment
Experiments evaluate CRAFT across quadrilateral- and polygon-type datasets, including challenging long, curved, and arbitrarily shaped text. The method achieves strong benchmark performance and retains flexibility without broad fine-tuning or multi-scale testing.
- Training protocol: CRAFT is trained only on ICDAR datasets because other benchmarks lack the quadrilateral annotations or transcriptions required for weakly supervised training.
- Quadrilateral-type datasets: CRAFT achieves state-of-the-art performance on ICDAR and MSRA-TD500 datasets.
- Quadrilateral-type datasets: CRAFT outperforms competing methods on MSRA-TD500 despite no fine-tuning on that dataset.
- Polygon-type datasets: CRAFT’s character localization enables robust detection of arbitrarily shaped text, including TotalText’s curved and deformed examples.
- Polygon-type datasets: On CTW-1500, LinkRefiner refines affinity scores for long texts before polygon generation, achieving state-of-the-art performance while CRAFT remains frozen.
- Discussions: Single-scale experiments support robustness to scale variance because CRAFT localizes individual characters rather than entire text instances.
- Discussions: The model distinguishes Bangla and Arabic characters less effectively than Latin, Korean, Chinese, and Japanese characters because cursive writing complicates individual character segmentation.
5. Conclusion
The conclusion presents CRAFT as a character-level detector that uses region and affinity scores to represent varied text shapes despite scarce character annotations. It reports strong benchmark performance and generalization without fine-tuning, while identifying end-to-end recognition integration as future work.
- CRAFT detects individual characters without requiring character-level annotations.
- Character region and affinity scores together represent varied text shapes in a bottom-up manner.
- Weakly supervised learning generates pseudo-ground truths from an interim model because real character-level datasets are rare.
- CRAFT achieves state-of-the-art performance on most public datasets and demonstrates generalization without fine-tuning.
- Future work will integrate CRAFT with a recognition model end to end for scene text spotting.
A. LinkRefiner for CTW-1500 dataset
LinkRefiner adapts CRAFT to CTW-1500’s line-level arbitrary polygons by refining links between distant characters and words. The refined links support long-polygon generation and curved-text rectification.
- CTW-1500 combines line-level annotations with arbitrary polygons and lacks spaces as separation cues, conflicting with CRAFT’s affinity assumption.
- LinkRefiner takes region scores, affinity scores, and CRAFT’s intermediate feature map as input to produce refined affinity scores.
- Atrous Spatial Pyramid Pooling provides a large receptive field for combining distant characters and words on the same text line.
- The LinkRefiner ground truth connects paired polygon control points with lines whose widths follow their distances.
- The refined link score replaces the original affinity score, after which polygon generation proceeds as for TotalText.
- The method detects challenging long and curved texts and rectifies curved polygons into straight text images.