PaperlayerView exact revision on arXiv

Source-linked AI summary

TextBoxes++: A Single-Shot Oriented Scene Text Detector

Minghui Liao, Baoguang Shi, Xiang Bai

arXiv:1801.02765v3cs.CV

TL;DR

Scene text detection must handle arbitrary orientations, small sizes, and highly variable aspect ratios in natural images. TextBoxes++ addresses these challenges with an end-to-end single-shot detector and recognition-aware refinement. Across evaluated benchmarks, it reports higher localization accuracy and runtime than competing methods, while achieving state-of-the-art word spotting and end-to-end recognition performance.

  • Problem

    Scene text detection is challenging because natural images contain arbitrary orientations, small sizes, and significantly varying aspect ratios.

  • Method

    TextBoxes++ is an end-to-end trainable detector that predicts arbitrary-oriented word boxes in a single network pass and can combine detection with CRNN recognition.

  • Results

    TextBoxes++ outperforms competing methods in localization accuracy and runtime across evaluated datasets, while its CRNN combination achieves state-of-the-art word spotting and end-to-end recognition performance.

  • Takeaways & Limitations

    The paper presents a fast, accurate, and relatively simple pipeline for arbitrary-oriented scene text detection and text reading.

Abstract

from arXiv · show

Scene text detection is an important step of scene text recognition system and also a challenging problem. Different from general object detection, the main challenges of scene text detection lie on arbitrary orientations, small sizes, and significantly variant aspect ratios of text in natural images. In this paper, we present an end-to-end trainable fast scene text detector, named TextBoxes++, which detects arbitrary-oriented scene text with both high accuracy and efficiency in a single network forward pass. No post-processing other than an efficient non-maximum suppression is involved. We have evaluated the proposed TextBoxes++ on four public datasets. In all experiments, TextBoxes++ outperforms competing methods in terms of text localization accuracy and runtime. More specifically, TextBoxes++ achieves an f-measure of 0.817 at 11.6fps for 1024*1024 ICDAR 2015 Incidental text images, and an f-measure of 0.5591 at 19.8fps for 768*768 COCO-Text images. Furthermore, combined with a text recognizer, TextBoxes++ significantly outperforms the state-of-the-art approaches for word spotting and end-to-end text recognition tasks on popular benchmarks. Code is available at: https://github.com/MhLiao/TextBoxes_plusplus

I. INTRODUCTION

TextBoxes++ addresses the difficulty of detecting scene text with arbitrary orientations, small sizes, varied aspect ratios, and complex backgrounds. It introduces a single-shot, end-to-end detector and combines it with recognition for broader text-reading tasks.

  • Scene text detection is difficult because natural images contain arbitrary orientations, small text, varied aspect ratios, lighting variation, and complex backgrounds.
  • TextBoxes++ directly predicts word bounding boxes with quadrilateral or oriented-rectangle representations in one end-to-end trainable network.The detector uses default boxes, multiple output layers, and non-maximum suppression to produce final detections.
  • The detector uses long convolutional kernels and densely predicts multi-scale text boxes, supporting efficient single-pass inference.The paper also reports higher accuracy when using multiple passes on multi-scale inputs.
  • Combining TextBoxes++ with CRNN yields state-of-the-art performance on word spotting and end-to-end text recognition tasks.The recognizer contributes semantic-level awareness that regularizes text detection.
  • The paper extends the earlier horizontal TextBoxes detector to arbitrary-oriented text and improves network structure, training, evaluation, and detection-recognition score fusion.
  • The paper contributes comparative studies of bounding-box representations, model configurations, and evaluation methods, alongside recognition-based detection refinement.

B. Text detection

The paper situates TextBoxes++ among text detectors that differ in primitive targets and bounding-box shapes. Its approach favors direct word detection with a simpler pipeline and recognition-aware refinement.

  • Scene text reading combines detection, which localizes word boxes, with recognition, which transcribes cropped words into character sequences.
  • Classification strategy based on primitive detection targets: Text detectors can use characters or text parts as primitives, then group or link them into words.
  • Classification strategy based on the shape of target bounding boxes: Shape-based detectors include methods for horizontal or nearly horizontal text and methods designed for oriented text.
  • TextBoxes++ detects word boxes directly with one end-to-end network, avoiding the multi-step pipelines used by several competing approaches.
  • Unlike SSD, TextBoxes++ adapts default-box detection to extreme word aspect ratios and arbitrary orientations through specialized text-box layers.
  • TextBoxes++ combines horizontal default boxes, quadrilateral regression, and detection-recognition scores to simplify matching and refine results.

A. Overview

TextBoxes++ is a fully convolutional, SSD-inspired detector that predicts text presence and multiple box representations from default boxes across feature maps. Its design supports multi-scale, arbitrary-oriented word detection with efficient post-processing.

  • TextBoxes++ uses an end-to-end fully convolutional network inspired by SSD to detect arbitrary-oriented text.
  • The architecture retains VGG-16 layers, converts its final fully connected layers to convolutional layers, and appends additional convolutional stages.
  • Text-box layers attached to multiple feature maps predict text presence and bounding-box offsets for associated default boxes.
  • The detector can output oriented rectangles or quadrilaterals together with minimum horizontal rectangles containing them.
  • Training matches ground-truth boxes to default boxes using overlap of their minimum horizontal rectangles and uses aspect-ratio-specific defaults.
  • Vertical offsets expand default-box coverage for dense text regions where ordinary boxes may miss nearby or vertically displaced words.

3) convolutional layers:

TextBoxes++ replaces the preliminary model’s elongated filters with rectangular 3 × 5 filters better suited to oriented text.

  • The preliminary horizontal detector used irregular 1 × 5 convolutional filters because natural text lines are often long.
  • For oriented text, TextBoxes++ uses inception-style 3 × 5 filters whose rectangular receptive fields fit large aspect ratios while avoiding square-filter noise.

C. Adapted training for arbitrary-oriented text detection

TextBoxes++ adapts ground-truth representations for arbitrary-oriented text using quadrilaterals or rotated rectangles, with vertex ordering chosen to align corresponding points.

  • The rotated-rectangle representation explicitly defines the first two ground-truth vertices and the rectangle height.
  • The horizontal reference box is the minimum enclosing rectangle, with vertices ordered clockwise from the top-left corner.
  • Quadrilateral ground truths represent oriented text using four vertices ordered clockwise.
  • The quadrilateral vertex ordering minimizes the summed Euclidean distances between corresponding vertices of horizontal and oriented boxes.Equivalent cyclic orderings are compared, with the minimum-distance shift selected.
  • Rotated rectangles use an alternative representation instead of the conventional (x, y, w, h, θ) form because angle distributions can be uneven and dataset-dependent.

2) Loss function:

Training combines confidence and localization losses while using hard-negative mining and object-coverage-aware cropping to address text-like distractors and small text.

  • Loss function: The loss averages confidence and localization terms, weighting localization by α.The confidence term uses a 2-class soft-max loss, while localization uses smooth L1 loss.
  • Loss function: α is set to 0.2 for quick convergence.
  • Loss function: Hard-negative mining increases the negative-to-positive ratio from 3:1 in the first training stage to 6:1 in the second.The strategy targets textures and signs that resemble text.
  • Data augmentation: Object coverage is added to Jaccard overlap because overlap constraints alone are unsuitable for small text.For small objects, accepted crops can become disproportionately enlarged after resizing.
  • Data augmentation: Cropping thresholds for both overlap and coverage are randomly selected from 0, 0.1, 0.3, 0.5, 0.7, and 0.9.A threshold of 0 removes the corresponding minimum constraint.

5) Multi-scale training:

TextBoxes++ uses multi-scale inputs and efficient post-processing, then combines detection with CRNN recognition to refine word-level predictions.

  • Multi-scale training: Randomly cropped training regions are resized to a fixed image size, while larger inputs are used late in training for multi-scale text.The fully convolutional architecture permits arbitrary input sizes at inference.
  • Multi-scale training: Inference resizes six multi-scale prediction outputs to the original image size before fusing them into one confidence map for NMS.
  • Recognition refinement: CRNN recognition is connected to TextBoxes++ for word spotting and end-to-end recognition.CRNN uses CTC to estimate sequence probabilities from the input image.
  • Recognition refinement: Lexicons are used only for fair comparison and are not required by the proposed method.
  • Recognition refinement: The combined score S integrates detection score s_d and recognition score s_r after exponentiation and harmonic averaging.This avoids the severe bias caused by directly combining scores with very different thresholds.

IV. EXPERIMENTS

Experiments evaluate TextBoxes++ across oriented and horizontal text datasets using recognition-aware metrics and implementation variants.

  • Datasets and evaluation: Recognition-aware hits require both sufficient IOU and correct recognition, unlike ordinary detection hits.Detection hits are generally defined using an IOU threshold of 0.5.
  • Datasets and evaluation: Fig. 6 compares detection boxes with equal IOU by showing the ground truth, detections, and recognized yellow text.The example motivates evaluating detection through downstream recognition as well as IOU.
  • Datasets and evaluation: The evaluation includes oriented-text IC15 and COCO-Text datasets plus horizontal-text IC13 and SVT datasets.SynthText is used to pre-train the model.
  • Implementation details: Table I records learning rate, input size, negative ratio, and training-iteration settings for implementation stages.
  • Implementation details: Training uses SynthText pre-training, dataset-specific continuation, and a final lower-learning-rate stage with larger inputs and more negatives.The SynthText pre-training stage is fixed at 60k iterations.
  • Implementation details: Table II compares four TextBoxes++ variants across IOU thresholds using rotated-rectangle or quadrilateral boxes and single- or multi-scale inputs.

C. Quadrilateral VS Rotated Rectangle

TextBoxes++ uses quadrilateral boxes to represent arbitrary-oriented text and reports stronger localization across oriented and horizontal benchmarks. Its design also preserves efficiency while handling varied word shapes and scales.

  • C. Quadrilateral VS Rotated Rectangle: Quadrilaterals represent arbitrary-oriented text more flexibly than rotated rectangles, especially after image resizing.A rotated rectangle can become a parallelogram when resized, causing mismatch with the text.
  • C. Quadrilateral VS Rotated Rectangle: At least 2.5 percent improvement was achieved by the quadrilateral variant over rotated rectangles on ICDAR 2015 Incidental Text.The quadrilateral version remained better with multi-scale inputs, particularly at an IOU threshold of 0.7.
  • Oriented Text Dataset: 3.5 percent improvement over the state-of-the-art was achieved by single-scale TextBoxes++ on ICDAR 2015 Incidental Text.The method was also described as more robust across oriented text and varied scales.
  • Oriented Text Dataset: At least 16 percent improvement over competing methods was achieved by single-scale TextBoxes++ on COCO-Text.Multi-scale inputs further boosted performance by 2.81 percent.
  • Horizontal Text Dataset: At least 1.0 percent improvement over other methods was achieved on ICDAR 2013 except for a cascaded method that required 1.36 seconds per image.TextBoxes++ also handled large-aspect-ratio words better than a straightforward SSD adaptation.
  • Horizontal Text Dataset: TextBoxes++ matched the preliminary TextBoxes at single scale and performed better with multi-scale inputs on ICDAR 2013.This confirmed no performance loss despite extending the detector to arbitrary-oriented text.

2) Runtime:

TextBoxes++ combines efficient detection with recognition-oriented evaluation. It reports a strong runtime–accuracy balance and substantial gains in word spotting and end-to-end recognition across datasets and lexicon settings.

  • 2) Runtime: 0.817 F-measure at 11.6 fps was achieved by TextBoxes++ on ICDAR 2015 Incidental Text.The authors report a better runtime–performance balance than competing methods.
  • 1) Word spotting and end-to-end recognition: 6 percent improvement was achieved over the best competing method for both word spotting and end-to-end recognition under a strong lexicon.Under weak lexicons, gains were 6 percent for word spotting and 4 percent for end-to-end recognition.
  • 1) Word spotting and end-to-end recognition: 2.7 percent and 1.2 percent improvements were reported for word spotting and end-to-end recognition, respectively, with a generic lexicon.The authors describe these gains as less significant than those under strong and weak lexicons.
  • 1) Word spotting and end-to-end recognition: At least 2 percent improvement was achieved over the best competing method across all listed evaluation protocols on ICDAR 2013.The method also outperformed the state of the art on SVT and SVT-50 by at least 8 percent.
  • 1) Word spotting and end-to-end recognition: At least 8 percent improvement was achieved over the state-of-the-art method on both SVT and SVT-50.The authors attribute this mainly to robustness on low-resolution images from training with relatively low-resolution data.

2) Refining detection with recognition:

TextBoxes++ combines detection with recognition to refine detection scores and improve word-spotting performance, while remaining limited on vertical and low-resolution text. The paper also identifies broader difficult cases, including occlusion, large character spacing, false detections, and curved text.

  • Recognition-based refinement: Recognition scores are integrated into detection scores, using CRNN to add semantic awareness and regularize detection.The combined system targets word spotting and end-to-end text recognition.
  • Recognition-based refinement: 0.5 percent and 1.3 percents: recognition without a lexicon improves TextBoxes++ detection on ICDAR 2013 and ICDAR 2015 Incidental Text, respectively.With a specified lexicon, the improvements increase to 0.8 percent and 1.9 percents on the same datasets.
  • Limitations: The recognizer still struggles with vertical text and low-resolution text.These limitations constrain the current recognition-based refinement.
  • Limitations: TextBoxes++ fails on object occlusion, large character spacing, some vertical text, text-like false detections, and curved text.The paper attributes vertical-text failures partly to insufficient vertical training data and curved-text failures to quadrilateral representation.

1) Simplicity:

TextBoxes++ uses direct default-box regression rather than segmentation-based processing, yielding a simpler and faster detection pipeline. Its design choices also support stable training and strong benchmark performance, including a reported advantage over DMPNet.

  • Simplicity: TextBoxes++ directly classifies and regresses default boxes on convolutional feature maps, avoiding EAST’s time-consuming pyramid-like deconvolution.With VGG-16, TextBoxes++ runs at 11.6fps versus 6.52fps for EAST’s VGG16 RBOX version.
  • Simplicity: TextBoxes++ avoids reliance on a segmentation score map, directly regressing bounding boxes from convolutional feature maps.This avoids the difficulty of separating close words when segmentation merges their text regions.
  • Design choices: Horizontal default boxes use fewer boxes per region than oriented defaults, while matching the horizontal-rectangle receptive fields.The paper argues that target quadrilaterals can still be well regressed from these defaults.
  • Design choices: Simultaneously regressing maximum horizontal rectangles and quadrilateral boxes makes training more stable than DMPNet’s approach.The paper also studies small-text handling and combines detection and recognition scores.
  • Performance: 11 percents: TextBoxes++ outperforms DMPNet in F-measure on ICDAR 2015 with a single-scale setting.DMPNet did not report runtime, although the paper argues it is slower based on architectural analysis.
  • Performance: TextBoxes++ achieves state-of-the-art performance with high efficiency across horizontal and oriented text datasets.The conclusion bases this claim on evaluations covering text detection, word spotting, and end-to-end scene text recognition.
Loading 1801.02765v3…