PaperlayerView exact revision on arXiv

Source-linked AI summary

Arbitrary-Oriented Scene Text Detection via Rotation Proposals

Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, Xiangyang Xue

arXiv:1703.01086v3cs.CV

TL;DR

Natural scene text detection must handle arbitrary orientations and challenging image conditions. The paper introduces RRPN and RRoI within a rotation-based region-proposal framework, reporting effective and efficient detection across three scene-text datasets.

  • Problem

    Axis-aligned proposals are not robust for scene text because text commonly appears in unnatural shapes and different orientations.

  • Method

    The framework generates inclined proposals with orientation angles, refines rotated bounding boxes, and uses RRoI pooling to project arbitrary-oriented proposals for classification.

  • Results

    Experiments on MSRA-TD500, ICDAR2013, and ICDAR2015 showed the proposed RRPN and RRoI were effective and efficient for text detection, including 0.3 s per testing image on MSRA-TD500.

  • Takeaways & Limitations

    Rotation-aware proposals and pooling provide a region-proposal-based approach for detecting text with multiple orientations while retaining computational efficiency.

Abstract

from arXiv · show

This paper introduces a novel rotation-based framework for arbitrary-oriented text detection in natural scene images. We present the Rotation Region Proposal Networks (RRPN), which are designed to generate inclined proposals with text orientation angle information. The angle information is then adapted for bounding box regression to make the proposals more accurately fit into the text region in terms of the orientation. The Rotation Region-of-Interest (RRoI) pooling layer is proposed to project arbitrary-oriented proposals to a feature map for a text region classifier. The whole framework is built upon a region-proposal-based architecture, which ensures the computational efficiency of the arbitrary-oriented text detection compared with previous text detection systems. We conduct experiments using the rotation-based framework on three real-world scene text detection datasets and demonstrate its superiority in terms of effectiveness and efficiency over previous approaches.

I. INTRODUCTION

Arbitrary-oriented scene text challenges horizontal proposal methods and can make segmentation-based systems inefficient. The paper addresses this with an end-to-end rotation-based region-proposal framework using orientation-aware proposals and rotated feature pooling.

  • Motivation: Natural scene text varies in lighting, blur, perspective, and orientation, making detection difficult.
  • Problem: Most existing methods use horizontal annotations and return horizontal regions, which poorly fit non-horizontal text in real applications.
  • Related limitations: Segmentation-based arbitrary-orientation systems often require time-consuming segmentation and post-processing to produce oriented proposals.
  • Approach: RRPN generates inclined proposals with orientation angles, adapts those angles for bounding-box regression, and uses RRoI pooling for arbitrary-oriented regions.
  • Contributions: The region-proposal architecture predicts text-line orientation, fits proposals more closely, and supports computationally efficient detection compared with segmentation-based systems.
  • Evaluation: Experiments on MSRA-TD500, ICDAR2013, and ICDAR2015 report improved accuracy and efficiency compared with previous approaches.

III. HORIZONTAL REGION PROPOSAL

The conventional RPN generates horizontal proposals with scale and aspect-ratio anchors, but these are not robust for arbitrarily oriented scene text. The framework therefore represents text boxes with orientation-aware tuples and rotation-based processing.

  • Horizontal region proposal: RPN slides over convolutional feature maps and uses classification and box-regression layers to generate horizontal proposals.
  • Horizontal region proposal: Scale controls anchor size and aspect ratio controls anchor shape, but axis-aligned anchors are not robust for differently oriented scene text.
  • Rotated bounding box representation: Ground-truth text regions use rotated boxes represented by (x, y, h, w, θ), encoding center, short-side height, long-side width, and orientation.
  • Rotated bounding box representation: The orientation is normalized over half the angular space because a text direction and its opposite identify the same detected region.
  • Rotated bounding box representation: The tuple representation simplifies angle differences and angle regression compared with an eight-point representation.
  • Rotated bounding box representation: Image rotation preserves the rotated box width and height while updating its orientation, supporting rotation-based training augmentation.

B. Rotation Anchors

The paper replaces traditional anchors with rotation anchors that encode text orientation while balancing orientation coverage against computational efficiency.

  • Rotation anchors add an orientation parameter to proposals, addressing the insufficiency of scale-and-aspect-ratio-only anchors for in-the-wild text.
  • Six orientations are used: −π/6, 0, π/6, π/3, π/2, and 2π/3.These orientations trade off orientation coverage and computational efficiency.
  • Rotation anchors use aspect ratios 1:2, 1:5, and 1:8 while retaining scales 8, 16, and 32 to cover diverse text-line shapes.
  • Each proposal generated from an R-anchor is represented by five variables: (x, y, h, w, θ).

C. Learning of Rotated Proposal

Rotated proposals are learned using orientation-aware sampling, multitask classification and regression, and visualized training responses that increasingly align with text orientation.

  • Positive R-anchors have the highest or greater-than-0.7 IoU with ground truth and an intersection angle below π/12.Unselected anchors are excluded from training.
  • The proposal objective combines classification and regression losses as L(p, l, v∗, v) = Lcls(p, l) + λlLreg(v∗, v).The balancing parameter λ controls the trade-off between the two terms.
  • Regression uses predicted and ground-truth tuples for position, size, and orientation, with angular differences normalized into [−π/4, 3π/4).Background RoIs are ignored for bounding-box regression, which uses smooth-L1 loss for text RoIs.
  • The six orientation fit domains divide [−π/4, 3π/4) so every ground-truth orientation can match an R-anchor within an intersection angle below π/12.
  • As training iterations increase, feature-map responses focus more strongly on text regions and predicted orientations move closer to the text-instance orientation.
  • Skew IoU computation constructs an intersection polygon from rectangle intersections and contained vertices, then triangulates it to calculate area.

D. Accurate Proposal Refinement

The framework refines arbitrary-oriented proposals with skew-aware overlap computation, skew NMS, and rotation-aware pooling operations.

  • Axis-aligned IoU can be inaccurate for skew proposals, so skew IoU computes intersection polygons from crossing points and contained rectangle vertices.The ordered polygon is triangulated before the final IoU is computed.
  • The RRoI pooling procedure divides an inclined proposal into a grid and samples feature-map values through rotated coordinates before max pooling.The displayed algorithm uses proposal dimensions, orientation θ, and spatial scale SS.
  • Skew-NMS first keeps the maximum-IoU proposal above 0.7, then uses minimum angle difference when all IoUs lie between 0.3 and 0.7.The angle difference must remain below π/12 in the second phase.

E. RRoI Pooling Layer

RRoI pooling adapts region-of-interest feature extraction to arbitrary-oriented proposals by preserving each proposal’s orientation during subdivision and pooling.

  • Traditional RoI pooling handles only axis-aligned proposals, motivating the rotation RoI pooling layer for RRPN-generated proposals.
  • RRoI pooling divides a proposal into Hr×Wr equal subregions, each with the proposal’s orientation.
  • The subregion vertices are computed using a similarity transformation, and max pooling is performed within every rotated subregion.

V. EXPERIMENTS

The experiments evaluate the rotation-based framework on three text-detection benchmarks and examine implementation choices affecting efficiency. The framework uses data augmentation and rotated anchors while filtering proposals to maintain runtime comparable to prior approaches.

  • The framework is evaluated on MSRA-TD500, ICDAR2015, and ICDAR2013 using their benchmark protocols.
  • Random-angle image rotation is used for data augmentation because it improves efficiency and measurements.
  • The rotation-anchor strategy produces nearly 6 times more proposals per image than Faster-RCNN.
  • Border-padding runtime is measured against the original Faster-RCNN using a single NVIDIA Titan X GPU.
  • Filtering rotation anchors that cross image borders keeps training and testing speed similar to previous works.

A. Ablation Study

The ablation study improves the baseline by incorporating rotation proposals, contextual enlargement, more training data, border padding, and scale jittering. These strategies raise detection quality while addressing orientation, context, proposal coverage, and small-text robustness.

  • Baseline and rotation proposals: 57.4% precision, 54.5% recall, and 55.9% F-measure establish the baseline, exceeding the original Faster-RCNN results.The original Faster-RCNN scores are 38.7% precision, 30.4% recall, and 34.0% F-measure.
  • Baseline and rotation proposals: Rotation proposals detect text more accurately with less background area than horizontal region proposals.
  • Baseline limitations: Blur, uneven lighting, extremely small text, and extremely long text lines remain difficult for the baseline system.Text lines with a height-width ratio larger than 1:10 may be split into shorter proposals and scored as false detections.
  • Ablation strategies: Enlarging the text-region context increases F-measure across the experiments, possibly because more context helps orientation prediction.The text bounding box is enlarged by a factor of 1.X during preprocessing and divided back during testing.
  • Ablation strategies: Training on 700 images after adding HUST-TR400 raises F-measure to 60.8%.The authors report improvement in all measurements and greater robustness to noisy inputs.
  • Ablation strategies: Border padding combined with context enlargement and training-set enlargement raises F-measure to 63.3%.Border padding reserves positive proposals when enlarged rotated regions exceed image borders.
  • Ablation strategies: Inputs with a 1300-pixel long side achieve 71.1% precision, 65.3% recall, and 68.1% F-measure.Scale jittering with random long-side sizes below 1300 pixels performs better than the corresponding experiment without jittering.

B. Performance on Benchmarks

The framework is tested across horizontal and arbitrarily oriented benchmarks, with post-processing used only for MSRA-TD500 text-line annotations. Results show strong performance while revealing sensitivity to long, small, and unreadable text instances.

  • MSRA-TD500: MSRA-TD500 contains very long text lines that fixed R-anchor ratios may cover with several short bounding boxes.A text-linking post-processing step is incorporated for this benchmark.
  • Qualitative and sensitivity analyses: Figures compare detection across input scales, benchmarks, ICDAR2015 examples, and unreadable-text proportions.The ICDAR2015 visualization marks IoU > 0.5 detections in yellow and missed text regions in red.
  • MSRA-TD500: 42.7% precision, 37.6% recall, and 40.0% F-measure are obtained for the Faster-RCNN comparison on MSRA-TD500.The comparison is reported as verifying the need for a rotation-based framework for robust text detection.
  • ICDAR2015: ICDAR2015 baseline performance is 45.42% precision, 72.56% recall, and 55.87% F-measure.The authors attribute lower precision partly to small incidental text, unreadable instances, and a smaller training set.
  • ICDAR2015: Small text regions are addressed by jittering image patches with random long sides below 1,700 pixels.
  • ICDAR2015: Removing unreadable training instances leaves recall near 72%-73% while significantly increasing precision.The authors randomly remove 80% of unreadable instances while retaining the complete testing set.
  • ICDAR2013: ICDAR2013 performance reaches 90.22% precision, 71.89% recall, and 80.02% F-measure, reported as a 7% improvement over Faster-RCNN.The rotation proposals are fit into horizontal-aligned rectangles for this benchmark.

C. More Results

Across three datasets, RRPN achieves competitive or improved detection performance while requiring only 0.3 seconds per testing image on MSRA-TD500. Its gains over published ICDAR results support rotation proposals and rotation RoI pooling for efficient text detection.

  • Figure 15 compares precision-recall curves for RRPN and RRPN* across three datasets.
  • 0.3 s per testing image: RRPN is more efficient than other approaches on MSRA-TD500.
  • RRPN reaches the same magnitude of performance as state-of-the-art approaches on MSRA-TD500.
  • Substantial performance gains over published works on ICDAR benchmarks support using rotation region proposals and rotation RoI pooling.
  • On ICDAR2013, RRPN performs better than DeepText across different evaluation measurements.

VI. CONCLUSIONS

The paper concludes that rotation-based proposals and RRoI pooling enable arbitrary-oriented text detection by incorporating text-region orientation information. Comparisons on MSRA-TD500, ICDAR2013, and ICDAR2015 show the proposed RRPN and RRoI are effective and efficient.

  • Inclined rectangle proposals use text-region orientation information from higher convolutional layers to detect text with multiple orientations.
  • The novel RRoI pooling layer is adapted to rotated RoIs within the rotation-based detection framework.
Loading 1703.01086v3…