PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene Text Image Super-Resolution in the Wild

Wenjia Wang, Enze Xie, Xuebo Liu, Wenhai Wang, Ding Liang, Chunhua Shen, Xiang Bai

arXiv:2005.03341v3cs.CV

TL;DR

Low-resolution scene text is difficult to recognize, and prior SR methods trained on synthetic down-sampling do not generalize well to real images. The paper introduces the real paired TextZoom dataset and the text-oriented TSRN, whose experiments show clear gains over prior SR methods while indicating that the task remains unresolved.

  • Problem

    Prior scene-text SR methods rely on synthetic down-sampled images, which do not generalize well to real text images.

  • Method

    The paper introduces paired real LR-HR TextZoom data and TSRN with sequential residual, boundary-aware, and central-alignment components.

  • Results

    TSRN clearly outperforms seven SR methods on TextZoom and demonstrates the superiority of real data over synthetic data.

  • Takeaways & Limitations

    Low-resolution text recognition in the wild remains far from solved and requires more research effort.

  • Takeaways & Limitations

    The authors plan to expand language coverage and avoid extremely large and small images in future data collection.

Abstract

from arXiv · show

Low-resolution text images are often seen in natural scenes such as documents captured by mobile phones. Recognizing low-resolution text images is challenging because they lose detailed content information, leading to poor recognition accuracy. An intuitive solution is to introduce super-resolution (SR) techniques as pre-processing. However, previous single image super-resolution (SISR) methods are trained on synthetic low-resolution images (e.g.Bicubic down-sampling), which is simple and not suitable for real low-resolution text recognition. To this end, we pro-pose a real scene text SR dataset, termed TextZoom. It contains paired real low-resolution and high-resolution images which are captured by cameras with different focal length in the wild. It is more authentic and challenging than synthetic data, as shown in Fig. 1. We argue improv-ing the recognition accuracy is the ultimate goal for Scene Text SR. In this purpose, a new Text Super-Resolution Network termed TSRN, with three novel modules is developed. (1) A sequential residual block is proposed to extract the sequential information of the text images. (2) A boundary-aware loss is designed to sharpen the character boundaries. (3) A central alignment module is proposed to relieve the misalignment problem in TextZoom. Extensive experiments on TextZoom demonstrate that our TSRN largely improves the recognition accuracy by over 13%of CRNN, and by nearly 9.0% of ASTER and MORAN compared to synthetic SR data. Furthermore, our TSRN clearly outperforms 7 state-of-the-art SR methods in boosting the recognition accuracy of LR images in TextZoom. For example, it outperforms LapSRN by over 5% and 8%on the recognition accuracy of ASTER and CRNN. Our results suggest that low-resolution text recognition in the wild is far from being solved, thus more research effort is needed.

1 Introduction

Low-resolution scene text is difficult to recognize because optical degradation blurs character shapes, while real low-resolution images are more challenging than synthetic ones. The paper introduces TextZoom and TSRN to improve recognition through real paired data and text-oriented super-resolution.

  • Optical degradation blurs character shapes, causing modern recognizers’ performance to drop sharply on low-resolution text images.
  • The evaluation super-resolves TextZoom inputs and tests the outputs with ASTER, MORAN, and CRNN recognition models.
  • TextZoom provides paired real low-resolution and high-resolution scene-text images captured at different focal lengths, with annotations and three difficulty subsets.
  • TSRN surpasses seven representative SR methods on TextZoom, supporting the use of real scene-text data and recognition-focused super-resolution.
  • TSRN combines a Sequential Residual Block, boundary-aware gradient profile loss, and central alignment module for text-oriented super-resolution.The modules model relationships between neighboring characters, sharpen character boundaries, and address paired-image misalignment.

2 Related work

Prior scene-text SR and recognition methods largely rely on synthetic, uniformly down-sampled images. Such training learns a simple interpolation-reversal mapping that does not generalize well to real text images.

  • Super-Resolution: Traditional super-resolution treats reconstruction as regression from a low-resolution image to a high-resolution target, often using interpolation-based methods.
  • Text Recognition: Scene text recognition methods include character-based, whole-word classification, and sequence-recognition approaches such as CRNN with recurrent features.
  • Scene Text Image Super-Resolution: Earlier scene-text SR work targeted recognition or image-quality metrics but trained on down-sampled text images.
  • Scene Text Image Super-Resolution: Because synthetic low-resolution images are generated by simple, uniform down-sampling, models learn an inverse-bicubic or inverse-bilinear mapping that generalizes poorly to real text images.

3 TextZoom Dataset

TextZoom is constructed from camera-captured paired images and organized to reflect variation in focal length, distance, image size, and recognition difficulty. Its annotations preserve text content and acquisition information for real scene-text SR.

  • Data Collection & Annotation: TextZoom is derived from the paired LR-HR camera datasets RealSR and SR-RAW.
  • Data Collection & Annotation: RealSR uses four focal lengths and two cameras, while SR-RAW uses seven focal lengths from 24–240mm with shorter focal lengths serving as LR inputs.
  • Data Collection & Annotation: Cropping corresponding text regions across focal lengths makes misalignment unavoidable, so the dataset records text strings, bounding-box types, and original focal lengths.
  • Dataset Organization: Text images are not patched because cropping them could break the completed shapes of characters, and image height is used when organizing the data.
  • Dataset Organization: Images are up-sampled to standardized heights of 16 or 32 pixels to form a 2X training pair.
  • Allocation of TextZoom: TextZoom is divided into easy, medium, and hard subsets according to source dataset, focal length, and observed recognition difficulty.ASTER accuracy was 52.1% for SR-RAW images at 100mm and 75.0% for RealSR images at 105mm.

4 Method

TSRN adapts super-resolution to scene text by combining alignment, sequential feature modeling, and gradient-based boundary supervision. Its pipeline uses RGBM inputs and trains the output with complementary reconstruction and gradient-profile losses.

  • Pipeline: The RGBM input combines a binary mask with RGB channels, is rectified, and produces a super-resolved RGB image.The outputs are supervised by L2 loss, while their RGB channels also receive LGP supervision.
  • Pipeline: TSRN modifies SRResNet by adding a central alignment module and replacing basic blocks with Sequential Residual Blocks.The input concatenates a binary mask with RGB channels before alignment and feature extraction.
  • Sequential Residual Block: Sequential Residual Blocks add bidirectional LSTM mechanisms to model horizontal and vertical recurrent information in text features.The recurrent formulation uses hidden layers and input features connected along horizontal and vertical directions.
  • Central Alignment Module: The central alignment module uses an end-to-end spatial transformer with thin-plate-spline transformation to reduce pixel misalignment between paired images.Alignment makes pixel-wise losses more effective and can relieve artifacts caused by mismatched text and background pixels.
  • Gradient Profile Loss: Gradient Profile Loss compares the gradient fields of high-resolution and super-resolved images with an L1 difference over the text region.The loss uses high-resolution gradients as supervision to sharpen character boundaries and distinguish text from backgrounds.

5 Experiments

Experiments evaluate SR as a preprocessing step for low-resolution text recognition, test TSRN components, and compare TSRN with established SR methods on TextZoom.

  • Experimental Setup: Experiments evaluate SR methods on TextZoom’s easy, medium, and hard subsets, using ASTER, MORAN, and CRNN recognition accuracy.LR images are up-sampled to 64×16 and HR images to 128×32; models are trained for 500 epochs.
  • Is SR Necessary for Text Recognition?: 5.4%: re-implemented recognition training raises TextZoom average accuracy from 47.2% to 52.6% on low-resolution images.CommonLR accuracy also improves by 5%, but the re-implemented model remains below TSRN with the released ASTER recognizer.
  • Is SR Necessary for Text Recognition?: Fine-tuning on TextZoom overfits its small recognition dataset, achieving higher TextZoom accuracy but the lowest CommonLR accuracy.The fine-tuned model also loses more than 10.0% points on other testing sets such as IC13 and IC15.
  • Ablation Study on TSRN: SRBs boost average accuracy by 4.9%, while central alignment and gradient profile loss add 1.5% and 0.5%, respectively.Central alignment reduces artifacts and twisted characters; gradient profile loss also improves visual results despite its smaller accuracy gain.
  • Comparison with State-of-the-Art SR Methods: 10.7% ∼14.6%: TSRN improves average recognition accuracy over BICUBIC, compared with 2.3% ∼5.8% for seven other SISR methods.TSRN outperforms all seven compared SISR methods across ASTER, MORAN, and CRNN recognition evaluation.

6 Conclusion and Discussion

The paper introduces TextZoom and TSRN for real scene text super-resolution, while concluding that low-resolution text recognition remains far from solved. It identifies broader data coverage and recognition-aware methods as future directions.

  • Conclusion: TextZoom is presented as the first real paired scene text image super-resolution dataset, divided into easy, medium, and hard subsets.The dataset is described as well annotated and allocated.
  • Conclusion: TSRN clearly outperforms seven SR methods on the TextZoom task.
  • Discussion: Low-resolution text super-resolution and recognition remain far from solved, motivating further research.
  • Future Work: Future data collection will seek more appropriately distributed text images and additional languages such as Chinese, French, and German.Extremely large and small images are also to be avoided.

A.1 Synthetic LR vs. TextZoom LR

Experiments compare synthetic bicubic LR data with real TextZoom LR data and assess recognition accuracy and computational cost. Training on real LR data improves recognition, while TSRN adds limited overhead for attention-based recognizers.

  • Synthetic LR vs. TextZoom LR: Synthetic LR images are generated by bicubic down-sampling, whereas TextZoom provides real LR images for training and evaluation.The comparison uses SRResNet, LapSRN, and TSRN trained under both settings.
  • Synthetic LR vs. TextZoom LR: Nearly 9.0% higher ASTER and MORAN accuracy is achieved by TSRN trained on real LR than by TSRN trained on synthetic LR.
  • Evaluation: Recognition accuracy is treated as the most important evaluation metric for scene text super-resolution.
  • Computation and Speed: TSRN’s recognition speed is nearly equal to direct recognition for ASTER and MORAN, while CRNN becomes slower but gains considerable accuracy.The comparison concerns recognition with and without super-resolution.

A.4 Discussion about SRB

The SRB ablations examine sequence-block width and depth, while related alignment analysis explains how spatial correction affects reconstruction quality and visual artifacts.

  • Hidden Units: The best SRB configuration uses 32 hidden units, while too many hidden units reduce performance after sequence dependence is sufficiently modeled.The ablation compares 0, 16, 32, 64, and 128 hidden units.
  • Block Number: Five SRBs provide the best performance; adding more blocks does not improve results, and seven blocks decrease accuracy.
  • Image Quality Metrics: PSNR and SSIM can undervalue TSRN because central alignment introduces slight pixel shifts, making recognition accuracy more relevant here.
  • Central Alignment: The central alignment module uses TPS transformation to flexibly rectify spatial variation and place text regions more centrally.It is based on a spatial transformation network.
  • Central Alignment: On manually enlarged misalignment, central alignment can sharply improve accuracy, while unaligned training produces severe double shadows and artifacts.The module performs better on more misaligned text-image pairs.

B.2 Plugged into Other SR methods

Central alignment is evaluated as a plug-in for multiple SR architectures under ordinary and enlarged misalignment settings. The reported comparisons support its portability across SR methods.

  • Cross-Method Comparison: Central alignment is compared with and without alignment across SRResNet, LapSRN, and TSRN.The study displays six corresponding model variants.
  • Visualization: The enlarged-misalignment visualization compares recognition results and artifacts with and without central alignment.Red character strings indicate wrong recognition in the figure description.
  • Cross-Method Comparison: Central alignment improves performance across the evaluated SR methods and functions as a conveniently pluggable module.
  • Misalignment Methods: The study also compares central alignment with CoBi Loss as alternative approaches to handling misalignment.

C.1 Annotation of SR-RAW and RealSR.

The dataset combines focal-length-based image pairs from SR-RAW and RealSR, with alignment, cropping, filtering, and annotations designed for scene text super-resolution. Its statistics characterize text content, word lengths, and scene-text composition.

  • SR-RAW annotation: SR-RAW provides 500 images captured at seven focal lengths, using shorter focal lengths as LR images and longer focal lengths as ground truth after alignment.The dataset contains 450 training and 50 test images, aligned through field-of-view matching and geometric transformation.
  • RealSR annotation: RealSR uses 105mm images as HR references and 50mm, 35mm, and 28mm images as 2X, 3X, and 4X LR images, respectively.The annotation process crops the 105mm, 50mm, and 28mm images and rotates non-horizontal text for recognition.
  • Alignment: Text-image pairs are aligned using registration procedures, while the proposed central alignment module replaces contextual bilateral loss during training.RealSR uses luminance-aware pixel-wise registration; SR-RAW uses a Euclidean motion model for preprocessing.
  • Difficulty allocation: RealSR images achieve higher recognition accuracy than SR-RAW images at similar focal lengths, so RealSR is assigned to the easy subset.The difference is attributed to SR-RAW images being captured from a longer distance.
  • Height-based selection: Text images below 8 pixels in height are discarded because they are largely unrecognizable, while 8-16 and 16-32 pixel ranges form suitable 2X training pairs.Recognition accuracy is relatively good when text height reaches 16-32 pixels, and these ranges constitute the majority of groups.
  • Dataset statistics: TextZoom contains abundant characters and digits, 57.5% common English words, 12% plate text, and 18.2% uncommon words.The dataset also includes punctuation, rare phrases, compound words, and other strings across varied scene-text categories.

C.3 Task Analysis

TextZoom is challenging because camera capture introduces unavoidable misalignment and real images contain ambiguous, variably degraded text. These issues become more severe across the dataset’s difficulty levels.

  • Misalignment: Camera zoom and slight movement can shift text by tens of pixels, especially at short focal lengths, and preprocessing cannot eliminate all misalignment.The dataset lacks pixel-level word-location annotations, so misalignment varies without a specific pattern.
  • Difficulty: Misalignment and ambiguity become more severe as the difficulty increases across the three TextZoom subsets.The subsets are allocated according to difficulty, with the displayed examples illustrating their differences.
  • Center displacement: Characters in HR images tend to be centered relative to LR images because HR text boxes were manually kept centered during annotation.
Loading 2005.03341v3…