PaperlayerView exact revision on arXiv

Source-linked AI summary

EAST: An Efficient and Accurate Scene Text Detector

Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, Jiajun Liang

arXiv:1704.03155v2cs.CV

TL;DR

Existing scene text detectors often rely on multiple stages that can be sub-optimal and time-consuming. EAST uses a two-stage FCN-based pipeline to directly predict word- or text-line-level geometries, and experiments report state-of-the-art accuracy with low computation cost. The paper also identifies limits for very long and vertical text instances.

  • Problem

    Existing scene text detection methods often combine multiple stages and components, leaving accuracy and efficiency unsatisfactory.

  • Method

    EAST uses a fully convolutional network to directly predict word- or text-line-level geometries, followed by NMS.

  • Results

    EAST substantially outperforms previous methods in accuracy and efficiency across standard benchmarks.

  • Takeaways & Limitations

    The simple pipeline achieves state-of-the-art performance while keeping computation cost low.

  • Takeaways & Limitations

    The receptive-field size limits prediction of very long text lines, and sparse vertical-text training examples may cause missed or imprecise predictions.

Abstract

from arXiv · show

Previous approaches for scene text detection have already achieved promising performances across various benchmarks. However, they usually fall short when dealing with challenging scenarios, even when equipped with deep neural network models, because the overall performance is determined by the interplay of multiple stages and components in the pipelines. In this work, we propose a simple yet powerful pipeline that yields fast and accurate text detection in natural scenes. The pipeline directly predicts words or text lines of arbitrary orientations and quadrilateral shapes in full images, eliminating unnecessary intermediate steps (e.g., candidate aggregation and word partitioning), with a single neural network. The simplicity of our pipeline allows concentrating efforts on designing loss functions and neural network architecture. Experiments on standard datasets including ICDAR 2015, COCO-Text and MSRA-TD500 demonstrate that the proposed algorithm significantly outperforms state-of-the-art methods in terms of both accuracy and efficiency. On the ICDAR 2015 dataset, the proposed algorithm achieves an F-score of 0.7820 at 13.2fps at 720p resolution.

1. Introduction

Scene text detection is important but existing multi-stage pipelines remain costly and insufficiently accurate. EAST simplifies detection with a two-stage FCN-based pipeline that directly predicts text regions and achieves strong accuracy and speed on standard benchmarks.

  • Text detection is a critical prerequisite for extracting and understanding textual information in natural scenes.
  • Existing conventional and deep-learning methods often use several stages and components that are potentially sub-optimal and time-consuming.
  • EAST uses only an FCN and NMS, directly producing word- or text-line-level predictions as rotated rectangles or quadrangles.
  • 13.2fps at 720p resolution accompanies an F-score of 0.7820 on ICDAR 2015 for the best-performing model.The method also reports F-scores of 0.8072 on ICDAR 2015 with multi-scale testing, 0.7608 on MSRA-TD500, and 0.3945 on COCO-Text.
  • The pipeline eliminates most intermediate steps while significantly outperforming state-of-the-art methods in both accuracy and speed.

2. Related Work

Scene text detection has progressed from manually engineered features to deep neural networks, but many systems retain complex candidate-generation and refinement stages. EAST instead directly targets word- or text-line-level detection with a lightweight FCN pipeline.

  • Conventional methods use manually designed features and generally lag deep neural networks in accuracy and adaptability on challenging scenes.Examples include low resolution and geometric distortion.
  • Deep-learning approaches became mainstream through pipelines combining candidate extraction, dense scanning, classification, and refinement.
  • EAST directly targets final word- or text-line-level detection, abandoning unnecessary intermediate components and steps.
  • A single lightweight neural network enables end-to-end training and optimization while surpassing previous methods in performance and speed.

3. Methodology

EAST is a fully convolutional text detector that predicts text-instance existence and geometry directly from full images. This design removes intermediate proposal, region-formation, and word-partition stages.

  • The key model is trained to predict text-instance existence and geometries directly from full images.
  • The fully convolutional network produces dense per-pixel predictions for words or text lines.
  • The architecture eliminates candidate proposal, text-region formation, and word-partition steps before thresholding and NMS.

3.1. Pipeline

The pipeline converts an image into dense score-map and geometry predictions, then thresholds valid regions and applies NMS to produce final detections. It supports rotated boxes and quadrangles.

  • An image is fed into the FCN, which generates pixel-level text score maps and geometry maps.
  • The score map contains pixel values from 0 to 1, while geometry channels encode the word enclosing each pixel.
  • The detector supports rotated-box and quadrangle geometries through corresponding loss functions.
  • Thresholding retains geometries above a predefined score threshold, and NMS produces the final output.

3.2. Network Design

The network merges features from multiple levels while keeping the upsampling branches small, then predicts score and geometry maps for RBOX or QUAD text regions.

  • 3.2. Network Design: Multi-level features address the contrasting requirements of detecting large words and accurately enclosing small word regions.Late-stage features help determine large-word existence, while early-stage information supports small-region geometry prediction.
  • 3.2. Network Design: The model comprises a feature-extractor stem, a feature-merging branch, and an output layer.The stem extracts four feature-map levels at 1/32, 1/16, 1/8, and 1/4 of the input size.
  • 3.2. Network Design: The feature-merging branch gradually upsamples, concatenates, bottlenecks, and fuses feature maps from successive levels.Unpooling doubles spatial size; 1×1 convolutions reduce channels before 3×3 convolutions fuse information.
  • 3.2. Network Design: The output layer projects features into one score-map channel and a multi-channel geometry map, supporting either RBOX or QUAD output.RBOX uses four AABB distance channels plus one rotation-angle channel; QUAD uses eight coordinate-shift channels.
  • 3.2. Network Design: Keeping branch convolutions narrow adds only a fraction of computation overhead over the stem.This design targets computation efficiency while retaining feature maps from different levels.

3.3. Label Generation

Label generation shrinks quadrangle score regions and constructs geometry targets for either rotated rectangles or quadrangles at positive-score pixels.

  • 3.3. Label Generation: The positive score area is designed as a roughly shrunk version of the original quadrangle.The method computes a reference length for each vertex before shrinking the quadrangle.
  • 3.3. Label Generation: The reference length for each quadrangle vertex uses the L2 distance between that vertex and its opposite vertex.Vertices are represented in clockwise order.
  • 3.3. Label Generation: The two longer opposing edges are shrunk first, followed by the two shorter opposing edges.Each edge endpoint moves inward by 0.3 times the corresponding reference length.
  • 3.3. Label Generation: For RBOX targets, each positive-score pixel stores distances to the four boundaries of a minimum-area rotated rectangle and its rotation angle.For QUAD targets, each positive-score pixel stores coordinate shifts to the quadrangle’s four vertices in eight geometry channels.

3.4. Loss Functions

The loss design combines score-map and geometry objectives while avoiding sampling-based complexity and promoting scale-invariant regression for varied text sizes.

  • 3.4. Loss Functions: Balanced cross-entropy simplifies score-map training without balanced sampling or hard-negative mining.Those techniques can improve performance but add a non-differentiable stage, tuning parameters, and pipeline complexity.
  • 3.4. Loss Functions: Scale-invariant regression is pursued because text sizes in natural scenes vary tremendously.The method uses IoU loss for AABB regression and scale-normalized smoothed-L1 loss for QUAD regression.
  • 3.4. Loss Functions: The AABB IoU loss computes overlap from predicted and ground-truth rectangle geometry, including intersection and union areas.The four distances represent pixel-to-top, right, bottom, and left boundary distances.
  • 3.4. Loss Functions: The rotation-angle loss is Lθ(ˆθ, θ∗) = 1 − cos(ˆθ − θ∗).ˆθ is the predicted angle and θ∗ is the ground-truth angle.
  • 3.4. Loss Functions: The overall geometry loss weights AABB and angle losses as Lg = LAABB + λθLθ, with λθ set to 10 in the experiments.Computing LAABB regardless of rotation angle approximates quadrangle IoU and can provide a correct learning gradient when angle prediction is imperfect.
  • 3.4. Loss Functions: QUAD regression adds normalization by the quadrangle’s shortest edge and considers equivalent vertex orderings.Vertex-order permutations address inconsistent quadrangle annotation order in public training datasets.

3.5. Training

The network is trained end-to-end with ADAM using cropped minibatches, scheduled learning-rate decay, and training until performance stops improving.

  • 3.5. Training: ADAM trains the network end-to-end with uniformly sampled 512x512 crops and minibatches of size 24.The learning rate starts at 1e-3 and decays by one-tenth every 27300 minibatches until reaching 1e-5.
  • 3.5. Training: Training continues until performance stops improving.This stopping condition follows the scheduled optimization procedure.

3.6. Locality-Aware NMS

Locality-aware NMS addresses the quadratic cost of merging dense geometries by exploiting spatial correlation. It merges and averages nearby predictions, providing practical speed and video stability under the locality assumption.

  • O(n^2) naive NMS is impractical for tens of thousands of geometries from dense predictions.
  • The proposed technique merges geometries row by row, iteratively combining each geometry with the last merged one.
  • The method assumes geometries from nearby pixels are highly correlated and runs in O(n) in best scenarios.
  • Weighted merging averages quadrangle coordinates using the scores of the two input geometries.
  • Unlike standard NMS, the procedure averages rather than selects geometries, acting as a voting mechanism that stabilizes video predictions.

4. Experiments

Experiments on three public benchmarks evaluate the proposed detector across datasets, base networks, geometries, challenging scenes, and efficiency. The method achieves strong detection accuracy while retaining high processing speed, with performance affected by receptive field and text-line requirements.

  • Datasets: Experiments cover ICDAR 2015, COCO-Text, and MSRA-TD500 using qualitative and quantitative evaluation.The datasets include arbitrary orientations, quadrangle or rotated-box annotations, and English and Chinese text.
  • Qualitative Results: The detector handles non-uniform illumination, low resolution, varying orientation, and perspective distortion in qualitative examples.The voting mechanism in NMS also yields stable video detections, with frames processed independently.
  • ICDAR 2015: 0.7820 F-score is achieved on ICDAR 2015 at original scale, rising to 0.8072 with multi-scale testing.The multi-scale comparison uses relative scales of 0.5, 0.7, 1.0, 1.4, and 2.0; the latter exceeds the cited best method by 0.16 in absolute F-score.
  • COCO-Text: 0.0614 F-score and 0.053 recall improvements over the previous top performer are reported on COCO-Text.All three tested settings achieve higher accuracy than the previous top performer.
  • MSRA-TD500: 0.0208 F-score and 0.0428 precision improvements over Zhang et al. are reported by Ours+PVANET2x on MSRA-TD500.The best performer slightly exceeds [41] in F-score, while the VGG16 setting performs worse than PVANET and PVANET2x because of its smaller effective receptive field.
  • Efficiency: 16.8 FPS is obtained by the fastest setting, while the best-performing Ours+PVANET2x model runs at 13.2 FPS.The slowest setting runs at 6.52 FPS, and the reported speeds are measured at 720p resolution on a Titan-X GPU.

5. Conclusion and Future Work

The paper presents a scene text detector that directly predicts word- or line-level regions from full images, with geometry tailored to application needs. It reports superior accuracy and efficiency, and identifies curved text, recognition integration, and general object detection as future directions.

  • The detector directly produces word- or line-level predictions from full images using a single neural network.
  • It predicts either rotated rectangles or quadrangles for text regions through suitable loss functions.
  • Experiments on standard benchmarks show substantially better accuracy and efficiency than previous methods.
  • Future work includes direct detection of curved text, integration with a text recognizer, and extension to general object detection.
Loading 1704.03155v2…