PaperlayerView exact revision on arXiv

Source-linked AI summary

TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei

arXiv:2109.10282v5cs.CLcs.CV

TL;DR

Text recognition has typically relied on CNN-based image understanding, recurrent or CTC-based text generation, and sometimes external language-model post-processing. TrOCR instead combines pre-trained image and text Transformers in an end-to-end wordpiece-based recognizer, achieving state-of-the-art results on printed, handwritten, and scene text recognition without post-processing.

  • Problem

    Existing text recognition systems commonly use CNN-based encoders with RNN or CTC-based decoders and may require external language models, while large-scale pre-trained vision and language models remain underused.

  • Method

    TrOCR uses a pre-trained image Transformer as the visual encoder and a pre-trained text Transformer as the decoder to generate wordpiece sequences end to end.

  • Results

    TrOCR achieves state-of-the-art results on printed, handwritten, and scene text recognition with a simple encoder-decoder model and no post-processing steps.

  • Takeaways & Limitations

    A convolution-free pure-Transformer recognizer can perform competitively across printed, handwritten, and scene text recognition without complex pre/post-processing.

Abstract

from arXiv · show

Text recognition is a long-standing research problem for document digitalization. Existing approaches are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on the printed, handwritten and scene text recognition tasks. The TrOCR models and code are publicly available at \url{https://aka.ms/trocr}.

Introduction

TrOCR addresses limitations in conventional text recognition by combining pre-trained image and text Transformers in an end-to-end model. It achieves state-of-the-art results across printed, handwritten, and scene text recognition without complex post-processing.

  • Motivation: Existing text recognition methods commonly use CNN encoders, RNN or CTC-based decoders, and sometimes external language models for post-processing.These approaches leave large-scale pre-trained computer-vision and natural-language models underused.
  • Proposed approach: TrOCR uses a pre-trained image Transformer encoder and pre-trained text Transformer decoder to recognize wordpiece sequences end to end.The input image is resized to 384×384 and split into 16×16 patches for the image Transformer.
  • Proposed approach: The model replaces CNN backbones with self-attention throughout the encoder-decoder architecture and generates recognized text with wordpiece units.This design avoids image-specific inductive biases and the additional language-modeling stage associated with character-based methods.
  • Results and extensions: TrOCR achieves state-of-the-art results on printed, handwritten, and scene text image datasets without complex pre-processing or post-processing steps.The paper also reports that the approach can be extended to multilingual recognition by using multilingual decoder models and expanding the dictionary.

2. TrOCR

TrOCR uses a standard Transformer-based encoder-decoder model that is convolution free and avoids complex pre- and post-processing. The models and code are publicly available.

  • Contribution: TrOCR achieves state-of-the-art results with a standard Transformer-based encoder-decoder model that is convolution free and avoids complex pre/post-processing steps.The TrOCR models and code are publicly available.

TrOCR

TrOCR uses a vanilla Transformer encoder-decoder architecture, with an image Transformer representing image patches and a text Transformer generating wordpieces. Both components are initialized from public pre-trained models, with encoder-decoder attention added to connect visual features and autoregressive decoding.

  • Architecture: The text Transformer decoder generates wordpiece sequences using visual features and previous predictions.Its encoder-decoder attention uses keys and values from the encoder output and queries from the decoder input.
  • Architecture: TrOCR represents resized images as patch sequences and uses an image Transformer encoder to obtain visual features.The encoder flattens fixed-size patches and linearly projects them into D-dimensional patch embeddings.
  • Decoder: Causal attention masking restricts each decoder position to previous outputs during training, matching autoregressive prediction.The decoder output is shifted relative to its input, so position i attends only to earlier positions.
  • Decoder: Decoder hidden states are projected to vocabulary logits, converted with softmax, and decoded using beam search.The linear projection maps the model dimension to vocabulary size V.
  • Pre-training: DeiT and BEiT initialize encoders, while RoBERTa and MiniLM initialize decoders; absent encoder-decoder-attention parameters are randomly initialized.Parameter mappings are set manually because the initialization models contain only Transformer encoders.

Task Pipeline

The text-recognition pipeline maps textline images to wordpiece sequences, using large-scale synthetic pre-training followed by downstream fine-tuning. The experiments also examine augmentation and training configurations through SROIE ablations.

  • Task Pipeline: Given a textline image, TrOCR extracts visual features and predicts wordpiece tokens conditioned on prior generated context.Training appends [EOS] to the target sequence and shifts it with [BOS] for decoder input.
  • Pre-training: Pre-training uses hundreds of millions of synthetic printed textlines, followed by smaller printed and handwritten datasets for a second stage.The two stages differ in their datasets before downstream fine-tuning.
  • Fine-tuning: Downstream fine-tuning is applied to most recognition tasks, while scene-text experiments are an exception.Outputs use BPE and SentencePiece rather than task-specific vocabularies.
  • Data Augmentation: Data augmentation includes geometric, blur, morphology, scaling, underlining, and RandAugment-based transformations.Printed and handwritten data use six transformations plus the original image; scene text uses RandAugment variants.
  • Ablations: The study reports ablations of pretrained initialization, augmentation, and two-stage pre-training on SROIE.Table 1 isolates models trained only on SROIE, while Table 2 evaluates the listed training factors.

Data

The paper constructs large synthetic datasets and evaluates TrOCR on printed, handwritten, receipt, and scene-text recognition benchmarks. Evaluation uses cropped textlines when text detection is outside the study’s scope.

  • Pre-training Data: The first-stage printed pre-training corpus contains 684M textlines extracted from two million publicly available digital-born PDF pages.Pages are rendered as images and their textlines are cropped.
  • Pre-training Data: The second-stage handwritten corpus contains 17.9M textlines synthesized with 5,427 handwritten fonts and Wikipedia text.It includes the IIIT-HWS dataset and is supplemented by approximately 53K real receipt images processed into normalized textlines.
  • Benchmarks: SROIE contains 626 training and 361 test receipt images, evaluated through textline crops derived from ground-truth bounding boxes.The work evaluates recognition rather than text detection on this dataset.
  • Evaluation: SROIE evaluation reports word-level precision, recall, and F1, with repeated ground-truth words required in predictions.The supplied benchmark passage identifies the metrics but not their numerical results.
  • Benchmarks: IAM handwriting evaluation uses 6,161 training, 966 validation, and 2,915 test lines from Aachen’s partition.The dataset consists of handwritten English text.
  • Benchmarks: Additional scene-text data include IIIT5K, SVT, ICDAR 2013, ICDAR 2015, SVTP, and CT80 benchmarks.These benchmarks cover challenging images affected by blur, occlusion, and low resolution.

Settings

The implementation uses Fairseq, public pretrained model repositories, large GPU allocations, wordpiece tokenization, and several encoder-decoder scale settings. CRNN provides the conventional convolutional-recurrent baseline.

  • Implementation: TrOCR is implemented in Fairseq and initialized from DeiT, BEiT, MiniLM, and RoBERTa public model repositories.Pre-training uses 32 V100 GPUs, while fine-tuning uses 8 V100 GPUs.
  • Tokenization: Fairseq BPE and SentencePiece tokenizers convert textlines into wordpieces.These tokenizers support the decoder’s wordpiece-level output representation.
  • Model Settings: The encoders use 384×384 inputs with 16×16 patches, while decoder depth, hidden size, and attention heads vary by model scale.The largest configuration uses 24 encoder layers and a 12-layer decoder with 1,024 hidden dimensions and 16 heads.
  • Baselines: CRNN is the baseline, combining convolutional feature extraction, recurrent sequence modeling, CTC training, and transcription.The paper describes CRNN as the dominant text-recognition paradigm for a long period.

Evaluation Metrics

The paper uses different evaluation metrics for receipt, handwriting, and scene text recognition, with receipt evaluation based on word-level matching.

  • SROIE: SROIE evaluates recognition using word-level precision, recall, and F1 score.Repeated ground-truth words must also appear in the prediction.
  • SROIE: Precision measures correct matches among detected words, while recall measures correct matches among ground-truth words.
  • SROIE: The SROIE F1 score combines precision and recall using their harmonic-mean formula.
  • Other datasets: IAM handwriting recognition is evaluated with case-sensitive Character Error Rate, whereas scene text recognition uses Word Accuracy.

Results

TrOCR consistently performs strongly across receipt, handwriting, and scene text recognition, while model comparisons show benefits from larger pre-trained Transformer components and pre-training.

  • Architecture comparison: BEiT encoders and RoBERTaLARGE decoders perform best among the compared encoder and decoder combinations.The paper defines TrOCRSMALL, TrOCRBASE, and TrOCRLARGE using different pre-trained encoder-decoder sizes.
  • Ablation study: Pre-trained initialization, data augmentation, and two-stage pre-training each substantially improve TrOCR models.
  • SROIE: TrOCR outperforms existing state-of-the-art models on SROIE with a pure Transformer structure and no complex pre- or post-processing.Compared baselines use CNN feature extractors and recurrent language-modeling components, whereas TrOCR uses image and text Transformers.
  • Handwriting recognition: TrOCRLARGE achieves a new state-of-the-art CER of 2.89 on the IAM Handwriting Database.It does so without extra human-labeled data and obtains comparable results to methods using an additional internal labeled dataset.
  • Scene text recognition: TrOCR establishes five new state-of-the-art results across eight scene-text experiments and is comparable on the remaining experiments.It underperforms on IIIT5K, where symbol handling differs between benchmark annotations and pre-training data.
  • Inference speed: TrOCRSMALL has comparable printed and handwriting recognition performance while using an order of magnitude fewer parameters and running twice as fast.The paper links this efficiency to lower computational resources and reduced user waiting time for deployment.

Related Work

Prior text-recognition systems commonly combine CNN-based visual encoders with CTC, RNN, or Transformer sequence decoders, often adding rectification or language modeling components.

  • CTC-based models: CRNN combines CNN visual-feature extraction with recurrent per-frame prediction and CTC decoding.
  • Sequence-to-sequence models: Sequence-to-sequence recognition methods increasingly use attention and Transformer architectures to generate text from visual feature sequences.
  • Evaluation context: Table 6 reports Word Accuracy on six scene-text benchmarks under synthetic-only and synthetic-plus-benchmark training settings.
  • Scene text recognition: Scene-text methods often add rectification modules to normalize perspective-distorted or irregularly shaped text regions.Examples include thin-plate spline transformations and iterative rectification networks.
  • Handwriting recognition: Handwriting recognition research has extensively explored RNN encoders, CTC decoders, attention mechanisms, and positional encodings.

Conclusion

TrOCR is an end-to-end OCR recognizer that replaces conventional CNN-based visual processing and character-level decoding with pre-trained image and text Transformers.

  • TrOCR uses an image Transformer encoder and a text Transformer decoder for end-to-end text recognition.
  • Wordpiece-level output reduces the need for additional character-level language modeling.
  • TrOCR achieves state-of-the-art results on printed, handwritten, and scene text recognition without post-processing.
Loading 2109.10282v5…