Source-linked AI summary
Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection
Shi-Xue Zhang, Xiaobin Zhu, Jie-Bo Hou, Chang Liu, Chun Yang, Hongfa Wang, Xu-Cheng Yin
TL;DR
Arbitrary-shape text detection is difficult because existing representations and CNN-based linkage methods inadequately model relations among separated components. The paper introduces an end-to-end graph network that builds local graphs from predicted component geometry and performs GCN-based relational reasoning. It reports state-of-the-art performance on polygon and quadrilateral datasets.
Problem
Arbitrary-shape text detection is challenging because conventional text representations and CNN-based linkage methods inadequately capture relations between separated components.
Method
The method estimates text-component geometry with a proposal network, constructs local graphs, and applies graph-based relational reasoning to infer linkages before grouping components.
Results
The proposed method achieves state-of-the-art performance on both polygon and quadrilateral text datasets.
Takeaways & Limitations
A local graph can bridge CNN-based component proposals and GCN-based relational reasoning in an end-to-end arbitrary-shape text detector.
Abstract
from arXiv · showhide
Arbitrary shape text detection is a challenging task due to the high variety and complexity of scenes texts. In this paper, we propose a novel unified relational reasoning graph network for arbitrary shape text detection. In our method, an innovative local graph bridges a text proposal model via Convolutional Neural Network (CNN) and a deep relational reasoning network via Graph Convolutional Network (GCN), making our network end-to-end trainable. To be concrete, every text instance will be divided into a series of small rectangular components, and the geometry attributes (e.g., height, width, and orientation) of the small components will be estimated by our text proposal model. Given the geometry attributes, the local graph construction model can roughly establish linkages between different text components. For further reasoning and deducing the likelihood of linkages between the component and its neighbors, we adopt a graph-based network to perform deep relational reasoning on local graphs. Experiments on public available datasets demonstrate the state-of-the-art performance of our method.
1. Introduction
Arbitrary-shape text detection remains difficult because conventional representations and CNN-based linkage methods inadequately capture relations among separated components. The paper addresses this with an end-to-end graph network that reasons over local component graphs and achieves state-of-the-art performance.
- Motivation: Regular-shape text detectors often fail on arbitrary-shaped text because their representations are limited.Earlier methods perform well in controlled environments but struggle with irregular text shapes.
- Motivation: CC-based methods use linked components to form text instances but have not fully exploited relations that support component aggregation.Learned link or embedding relationships are more robust than predefined rules, especially for long and curved text.
- Motivation: CNN-based linkage prediction is inadequate for non-adjacent components because separate text components form non-Euclidean data and local convolutions cannot directly capture their relations.The paper represents components as graph nodes and uses local graph edges to provide contextual information for linkage estimation.
- Approach: The proposed network estimates component geometry with a text proposal network, constructs local graphs, and uses graph reasoning to infer linkage relationships before grouping components.A local graph bridges the CNN proposal model and the relational reasoning network, enabling end-to-end training.
- Contributions: The framework is unified and end-to-end trainable, bridging a CNN-based text proposal network with a GCN-based relational reasoning network.This bridge is identified as a principal contribution of the method.
- Contributions: The method is presented as an early attempt to perform deep relational reasoning through graph convolution for arbitrary-shape text detection and reports state-of-the-art performance on polygon and quadrilateral datasets.The paper explicitly claims both methodological novelty and state-of-the-art results across the two dataset types.
2. Related Work
Prior work includes regression-, segmentation-, and connected-component-based detectors, while relational reasoning methods focus on learning links or embeddings for grouping text parts. The paper positions graph convolution as a way to address CNN limitations in modeling distant component relations.
- Regression-Based Methods: Regression-based methods use object-detection frameworks with word- or line-level priors to accommodate varied text aspect ratios and shapes.Examples modify anchor ratios, convolutional kernels, or anchor boxes, while EAST predicts pixel-level quadrangles.
- Segmentation-Based Methods: Segmentation-based methods detect text through word bounding areas, pixel linkages, progressive kernels, or embedding features for instance grouping.PixelLink predicts neighboring-pixel relationships, PSENet expands predefined kernels, and embedding methods provide instance information.
- CC-Based Methods: Connected-component methods first detect individual text parts or characters and then link or group them into final text instances.This approach has been used extensively in both traditional and deep-learning scene text detection.
- Relational Reasoning: CC-based detectors are robust for long or non-quadrilateral text, but their performance depends strongly on robust grouping or linkage results.Pixel links, embeddings, region maps, and affinity maps are used to form instances from detected components.
- Proposed Architecture: The proposed architecture combines shared convolutions, text component prediction, local graphs, relational reasoning, and Link Merging.These components form the overall pipeline described by the paper.
- Relational Reasoning: CNN-based methods cannot directly capture relations between distant component regions because local convolutional operators have limited receptive relationships.The paper addresses this limitation by transforming separate text components into graphs and applying graph convolutional reasoning.
3. Proposed Method
The method predicts geometric text components with shared CNN features, then constructs local graphs and applies relational reasoning to infer component linkages for arbitrary-shape text detection.
- Shared Architecture: Shared VGG-16 with FPN convolutions support both text component proposal and deep relational reasoning networks.The proposal network estimates component geometry, while the graph network reasons over linkages.
- Text Component Prediction: Each text instance is represented as ordered rectangular components with position, height, width, and orientation attributes.The component geometry is D = (x, y, h, w, cos θ, sin θ).
- Text Component Prediction: Text components are generated from text regions, with center regions extracted by shrinking the center line and expanding it by 0.3h.The center-region construction also shrinks the two center-line ends by 0.5w end pixels.
- Text Component Prediction: The prediction head uses four classification channels for TR/TCR and four regression channels for h1, h2, cos θ, and sin θ.Final detections use thresholding and locality-aware NMS on positive samples.
- Local Graph Generation: Local graphs use nearby component nodes, with 1-hop and 2-hop neighbors, to efficiently estimate linkage likelihoods from local context.The method uses eight nearest 1-hop neighbors and four nearest 2-hop neighbors, while selecting neighbors by Euclidean similarity.
- Deep Relational Reasoning: The relational reasoning network combines RROI and geometry features, embeds geometry with sine and cosine functions, and performs reasoning over graph structure.The resulting geometry embedding has 6 · Cε dimensions and is concatenated with RROI features.
4. Experiments
Experiments evaluate the method on polygon and quadrilateral datasets, with ablations and comparisons across curved, oriented, multilingual, and arbitrary-scale text. The method reports strong Hmean results, while CTW-1500 annotations introduce evaluation ambiguities.
- Datasets: The evaluation covers Total-Text, CTW-1500, MSRA-TD500, ICDAR2015, and ICDAR2017, spanning polygonal, quadrilateral, curved, oriented, multilingual, and small-scale text.Total-Text and CTW-1500 use polygon annotations; MSRA-TD500, ICDAR2015, and ICDAR2017 use quadrilateral annotations.
- Implementation Details: The network is pre-trained on SynthText or ICDAR2017-MLT, then fine-tuned on benchmark datasets using VGG16, multi-scale training, and standard augmentation.Training uses two pre-training epochs and 600 fine-tuning epochs, with Adam during pre-training and SGD during fine-tuning.
- Comparison with the state-of-the-arts: 85.73% Hmean is achieved on Total-Text, significantly outperforming other methods on curved and multi-oriented text.The method also separates close text instances of arbitrary shapes in the reported examples.
- Comparison with the state-of-the-arts: 84.45% Hmean and 83.02% recall are achieved on CTW-1500, while Hmean improves over TextSnake by 8.85% on CTW-1500 and 6.6% on Total-Text.The paper attributes less remarkable CTW-1500 performance partly to confusing or incomplete annotations.
5. Conclusion
The paper concludes with a CC-based arbitrary-shape text detector that uses spectral graph convolution to learn component linkages. Across five benchmarks, it reports good performance on arbitrary-shape, oriented, and multilingual text.
- Conclusion: The proposed CC-based method uses a spectral graph convolution network to learn text-component linkage relationships and guide post-processing.The learned relationships connect components into text instances.
- Conclusion: Experiments on five benchmarks report good performance for arbitrary-shape text detection and for oriented and multilingual text.The paper identifies an end-to-end arbitrary-shape text reading system with graph networks as future work.