PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-Attentional Convolutional Neural Networks for Scene Text Detection

Tong He, Weilin Huang, Yu Qiao, Jian Yao

arXiv:1510.03283v2cs.CV

TL;DR

Existing CNN-based text component filters rely mainly on global representations and binary labels, which can underrepresent text amid cluttered backgrounds. The paper proposes Text-CNN with multi-level supervision and CE-MSERs, achieving a 0.82 F-measure on ICDAR 2013.

  • Problem

    Existing deep text filters compute global component features and commonly use binary text/non-text labels, limiting text-focused representation in complicated backgrounds.

  • Method

    The system combines a multi-task Text-CNN trained with text-region masks, character labels, and binary labels, with CE-MSERs that enhances text-background contrast.

  • Results

    0.82 F-measure was achieved on ICDAR 2013, while full-system evaluations reported substantial improvements over recent results.

  • Takeaways & Limitations

    Rich multi-level supervision improves Text-CNN discrimination of ambiguous text and robustness to complicated backgrounds, while CE-MSERs improves detection of difficult patterns.

Abstract

from arXiv · show

Recent deep learning models have demonstrated strong capabilities for classifying text and non-text components in natural images. They extract a high-level feature computed globally from a whole image component (patch), where the cluttered background information may dominate true text features in the deep representation. This leads to less discriminative power and poorer robustness. In this work, we present a new system for scene text detection by proposing a novel Text-Attentional Convolutional Neural Network (Text-CNN) that particularly focuses on extracting text-related regions and features from the image components. We develop a new learning mechanism to train the Text-CNN with multi-level and rich supervised information, including text region mask, character label, and binary text/nontext information. The rich supervision information enables the Text-CNN with a strong capability for discriminating ambiguous texts, and also increases its robustness against complicated background components. The training process is formulated as a multi-task learning problem, where low-level supervised information greatly facilitates main task of text/non-text classification. In addition, a powerful low-level detector called Contrast- Enhancement Maximally Stable Extremal Regions (CE-MSERs) is developed, which extends the widely-used MSERs by enhancing intensity contrast between text patterns and background. This allows it to detect highly challenging text patterns, resulting in a higher recall. Our approach achieved promising results on the ICDAR 2013 dataset, with a F-measure of 0.82, improving the state-of-the-art results substantially.

I. INTRODUCTION

Scene text detection remains challenging because text varies greatly and appears amid complicated backgrounds. The paper proposes Text-CNN and CE-MSERs, combining rich supervision with contrast enhancement to improve component filtering and detection.

  • Natural-image text detection remains an open problem because text can be small, low-quality, low-contrast, distorted, blurred, or strongly affected by perspective and lighting.
  • Low-level ER/MSER detectors generate vastly more non-text than true text components, making accurate component filtering critical.Examples include over 10^6 ERs and 10^3 MSERs per image versus dozens of characters.
  • Hand-crafted low-level features have limited generality for challenging text and reduced robustness against text-like objects such as bricks, windows, and leaves.
  • Text-CNN focuses on deep text features using text-region masks, character labels, and binary text/non-text supervision, improving discrimination of ambiguous components and robustness to background.
  • Multi-task learning treats each supervision level as a task, allowing shared features and low-level supervision to facilitate convergence of text/non-text classification.
  • CE-MSERs enlarges local contrast between text and background, detects ambiguous patterns missed by MSERs, and forms a system reporting state-of-the-art results on ICDAR 2011 and 2013.

II. RELATED WORK

Related scene-text detectors use sliding-window or connected-component strategies, with classifier design and candidate management as recurring challenges. Deep CNNs improve representation, while this work extends an MSER-CNN pipeline with Text-CNN and CE-MSERs.

  • Scene text detection methods are broadly categorized into sliding-window and connected-component approaches.
  • Sliding-window methods scan multi-scale locations and apply classifiers, but must design discriminative features while controlling the number of scanning windows.
  • Connected-component methods detect text pixels and group them into candidates, offering O(N) speed but producing many noisy components that require sophisticated post-processing.
  • A powerful text/non-text classifier is important for both method families because low-level features struggle with challenging text and text-like non-text components.
  • Deep CNNs provide high-level image representations and have been used for component filtering, sliding-window detection, recognition, and joint detection with bounding-box regression.
  • Relative to Huang et al.'s MSER-CNN system, this work introduces Text-CNN and CE-MSERs to improve text-feature computation and detection performance.

III. PROPOSED APPROACH

The proposed system combines Text-CNN component filtering with CE-MSERs candidate generation. Text-CNN uses multi-level supervision in a multi-task framework to focus representation on text regions and character information.

  • The system has two components: Text-CNN for text-component filtering and CE-MSERs for generating component candidates.
  • Text-CNN is motivated by the value of text-region segmentation and character information for separating text from cluttered backgrounds.
  • A. Text-Attentional Convolutional Neural Network: The model is trained with text-region segmentation, character labels, and binary text/non-text information, progressing from where and what to whether a character is present.
  • A. Text-Attentional Convolutional Neural Network: Multi-task learning treats each supervision level as an independent task while sharing CNN features, addressing different learning difficulties and convergence rates.
  • A. Text-Attentional Convolutional Neural Network: The training formulation uses N examples denoted as {(x_i, y_i)}^N.

1) Problem Formulation:

The Text-CNN formulates text/non-text detection as multi-task learning with shared inputs but distinct main and auxiliary outputs. Its architecture combines classification tasks with pixel-level text-region regression while preserving spatial information for small patches.

  • Problem Formulation: The model uses one main text/non-text classification task and two auxiliary tasks: character-label classification and text-region regression.The main and character-label tasks are classification problems, whereas text-region learning is regression.
  • Problem Formulation: All three tasks share the same 32×32×3 image patch, differing primarily in their output labels.The shared input is xi ∈ R32×32×3.
  • Problem Formulation: The text-region task predicts a 32×32 probability map from a binary pixel-level text mask, and it is used only during training.The model minimizes L2 distance between the ground-truth mask and estimated probability map; testing uses only the main classifier.
  • Problem Formulation: The architecture uses three convolutional layers and two 1024D fully connected layers, with max pooling after the second convolutional layer.An additional pooling layer reduced accuracy because the feature map was already only 2×2.
  • Problem Formulation: The last fully connected layer outputs 2D text/non-text predictions and 62D character-label predictions.The character-label set contains digits and uppercase and lowercase letters.

2) Training the Text-CNN:

Text-CNN training jointly uses auxiliary text-region and character-recognition supervision before optimizing the main classifier. These auxiliary tasks produce more informative features and improve responses to ambiguous text and complicated background components.

  • Training the Text-CNN: The three tasks share learned features, allowing low-level supervision to facilitate convergence of the main text/non-text classification task.The learning mechanism formulates each supervised-information level as a task within multi-task learning.
  • Training the Text-CNN: Training begins with joint text-region regression and character recognition, embedding pixel-level and character-level information into shared features.The text-region task regresses a binary mask, while joint auxiliary learning captures information shared across multiple characters.
  • Training the Text-CNN: After approximately 30,000 iterations, text-region regression is stopped while character recognition continues with the main task until optimization finishes.The region task is stopped after low-level knowledge is embedded, because continuing it could overfit the low-level task.
  • Training the Text-CNN: The character-label task supplies character-level information that helps the main task discriminate highly confused components and supports robust text/non-text decisions.The paper describes the auxiliary tasks as functioning like pre-training for parameter optimization.
  • Training the Text-CNN: Figure 3 shows that auxiliary tasks positively affect first-layer filter learning, while binary supervision alone produces heavily noise-distorted filter maps.The comparison includes a conventional CNN, Text-CNN with only the label task, and Text-CNN with both auxiliary tasks.
  • Training the Text-CNN: Text-CNN produces high confident scores for ambiguous text components and very low scores for complicated text-like background components.Figure 4 compares Text-CNN, Text-CNN with character labels only, and a conventional CNN using reshaped feature maps and confident scores.

B. Contrast-Enhanced MSERs

CE-MSERs enhance regional contrast before applying MSER detection, aiming to recover challenging text components while reducing low-level distortions. The resulting detector improves character recall and boosts final detection recall.

  • B. Contrast-Enhanced MSERs: CE-MSERs address MSER distortions caused by complicated backgrounds and low-level detector limitations.The motivation is to improve local stability and preserve text regions that conventional MSERs may miss.
  • B. Contrast-Enhanced MSERs: CE-MSERs apply contrast enhancement to natural-image regions before running MSER detection on contrast maps and the original image.The method uses a two-step mechanism for global and small-region contrast enhancement.
  • B. Contrast-Enhanced MSERs: 91.7% character-level recall is achieved by CE-MSERs on ICDAR 2011, compared with 88.3% for original MSERs.Recall is computed using at least 0.5 overlap with the ground truth.
  • B. Contrast-Enhanced MSERs: The CE-MSERs detector generates candidate components that the Text-CNN filters before text-line construction.The pipeline groups remaining components using geometric similarity and orientation.

IV. EXPERIMENT AND RESULTS

The experiments evaluate the proposed components individually and assess the complete system on four scene-text detection benchmarks.

  • IV. EXPERIMENT AND RESULTS: The study evaluates Text-CNN and CE-MSERs separately, then tests the complete system on four scene-text detection datasets.The benchmarks are ICDAR 2005, ICDAR 2011, ICDAR 2013, and MSRA-TD500.

A. Experimental Setup and Datasets

The experiments train Text-CNN on synthetic and cropped character data using mask, character-label, and text/non-text supervision. Inputs are enlarged CE-MSER regions, while training losses and parameters are examined across stages.

  • A. Experimental Setup and Datasets: CharSynthetic provides 80,141 synthetic character images with masks and labels for jointly training text-region and character-label tasks.CharTrain adds 17,733 text samples and 85,581 non-text samples with binary and 62-class labels.
  • A. Experimental Setup and Datasets: Text-CNN training changes from CharSynthetic to charTrain after an early stop at 30K iterations, while character-label training continues throughout.Figure 8 reports the training-loss schedule.
  • A. Experimental Setup and Datasets: The study evaluates λ1 and λ2 across two stages using character accuracy, mask L2 distance, and text/non-text classification.Stage one fixes λ1 = 1 and varies λ2; stage two varies λ1.
  • A. Experimental Setup and Datasets: CharTest contains 5,751 character images and 11,502 non-text components cropped or sampled from the ICDAR 2011 test images.Character images are manually cropped, while non-text components are randomly selected from image backgrounds.
  • A. Experimental Setup and Datasets: Each Text-CNN input is a 32×32 patch formed by enlarging a CE-MSER region to a square based on its long side.The enlarged patch can include neighboring characters or partial characters, providing surrounding context for classification.
  • A. Experimental Setup and Datasets: Evaluation uses ICDAR 2005, ICDAR 2011, ICDAR 2013, and MSRA-TD500, with the listed train/test image counts and varied text orientations.The datasets include both competition benchmarks and multi-orientation text-line images.

B. Results and Comparisons

The study compares the proposed components and full detection system with prior CNN-based approaches and recent benchmark results.

  • B. Results and Comparisons: The experiments first measure Text-CNN classification efficiency, then isolate the contributions of Text-CNN and CE-MSERs to final detection.The complete system is subsequently compared with recent results on four benchmarks.
  • B. Results and Comparisons: Figure 11 compares conventional CNN, CNN-W, another prior CNN, and Text CNN using the same CE-MSERs detector.The shared detector isolates differences among the component-classification models.

1) Text-CNN on text component classification:

Text-CNN improves text/non-text component classification by combining character masks, character labels, and binary supervision. These signals reduce errors on ambiguous characters and background components, while mask pre-training also benefits recognition.

  • Text-CNN classification: 6.7% error rate is achieved by Text-CNN, versus 9.8% for conventional CNN using only binary text/non-text supervision.Character mask and character label information improve the CNN by 1.4% and 2.0%, respectively.
  • Text-CNN classification: 17.5% character-subset error rate is obtained with character labels, down from 28.3%, because labels distinguish true stroke structures from text-like backgrounds.Mask information reduces background components misclassified as characters, lowering false alarms.
  • Character recognition: 12.7% recognition error is achieved by the mask pre-trained CNN, compared with 14.2% for conventional CNN on 5,751 character images.The result supports mask regression as useful pre-training for character recognition.
  • Detector contribution: 74% final detection recall is obtained with CE-MSERs, improving from 68% with original MSERs after an approximately 3% character-level recall gain.Enhanced candidate detection helps retain text lines or words containing challenging patterns.
  • Classifier contribution: 91% precision is achieved by Text-CNN, compared with 85% for conventional CNN, with the single-character task improving by 3%.More robust filtering of background components also increases recall when those components would otherwise disrupt word detection.

3) Evaluation on full text detection:

Across three benchmarks, the proposed system improves recent scene-text detection results through enhanced recall and precision. On ICDAR 2013, it achieves 0.93 precision, 0.73 recall, and 0.82 F-measure, while failures remain concentrated in extremely ambiguous text.

  • Benchmark results: 0.82 F-measure is achieved on ICDAR 2013, with 0.93 precision and 0.73 recall.The system is reported to improve recent results considerably across the evaluated benchmarks.
  • Benchmark results: 4% F-measure improvement over MSERs-CNN is reported on the ICDAR 2005 and 2011 datasets.CE-MSERs mainly contributes recall by identifying more distorted text patterns, while Text-CNN boosts precision by reducing false alarms.
  • Qualitative analysis: Successful detections remain robust across multiple text variations and cluttered backgrounds, while failures involve extremely ambiguous text that can also challenge human detection.The figure passage characterizes the remaining failure cases rather than quantifying them.

4) Evaluation on multi-orientation and multi-language text:

The method is evaluated on multi-orientation and multi-language text using MSRA-TD500, with synthetic Chinese and English character training data. It reaches 0.69 F-measure and shows larger gains on near-horizontal text relative to the cited comparison.

  • Training setup: 30,000 synthetic Chinese character images and masks are generated for training, covering 2,238 Chinese and 62 English character classes.The CharTrain contains 2,300 character classes in total.
  • MSRA-TD500 evaluation: 0.69 F-measure is achieved on MSRA-TD500, compared with 0.61 for and 0.66 for.The result approaches the 0.71 F-measure achieved by Yin et al..
  • Near-horizontal text: 8-9% improvements over Yin et al. are obtained on near-horizontal text in ICDAR 2011 and 2013.The paper presents these results as evidence of the method’s efficiency and generality.
  • Implications: The system is designed to provide more accurate or reliable character candidates that can be incorporated with more sophisticated text-line construction methods.The conclusion identifies this integration as a route toward better performance.
Loading 1510.03283v2…