PaperlayerView exact revision on arXiv

Source-linked AI summary

OCR-free Document Understanding Transformer

Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park

arXiv:2111.15664v5cs.LGcs.AI

TL;DR

Existing VDU systems commonly depend on OCR, which adds computational cost, limits flexibility across languages and domains, and can propagate recognition errors. Donut replaces this pipeline with an end-to-end OCR-free Transformer, supported by synthetic-data pre-training, and reports higher performance and better cost-effectiveness across public and industrial evaluations.

  • Problem

    OCR-dependent VDU systems add computational cost, have limited flexibility across languages and domains, and propagate recognition errors into subsequent processing.

  • Method

    Donut directly maps document images to structured outputs with Transformer-based visual encoder and textual decoder modules, using pre-training and synthetic data through SynthDoG.

  • Results

    Donut reports higher performance and better cost-effectiveness across public benchmarks and private internal service datasets, including state-of-the-art performance claims.

  • Takeaways & Limitations

    An OCR-free, end-to-end VDU model can support multilingual training and structured document understanding without relying on separate OCR functionality.

  • Takeaways & Limitations

    The paper identifies remaining challenges in the Donut-like end-to-end approach alongside its current strengths.

Abstract

from arXiv · show

Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) methods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conceptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains. The code, trained model and synthetic data are available at https://github.com/clovaai/donut.

1 Introduction

Visual Document Understanding is challenging because document images require both text reading and holistic structural understanding. Donut addresses the cost, flexibility, and OCR-error problems of conventional pipelines with an OCR-free Transformer approach and synthetic-data pre-training.

  • Motivation: VDU extracts useful structured information from document images such as invoices, receipts, and business cards for classification, information extraction, and visual question answering.The task requires both reading document text and understanding its broader visual and semantic structure.
  • Conventional Pipeline: Conventional VDU methods separate text reading from holistic understanding and typically use OCR modules for detection, recognition, and parsing.The pipeline detects text locations, recognizes characters, and passes recognized text with locations to a downstream parser.
  • Limitations: OCR-dependent systems incur high inference and training costs, generalize poorly across languages or domains, and propagate recognition errors into later VDU processing.These issues are especially severe for complex character sets such as Korean or Chinese, where post-OCR correction adds system size and maintenance cost.
  • Proposed Approach: Donut directly maps raw document images to desired outputs without OCR using a Transformer-only architecture and a simple pre-training scheme.Pre-training teaches text reading from images and previous text contexts, while fine-tuning teaches document understanding for downstream tasks.
  • Results: Donut achieves comparable or better overall performance than prior methods, with experiments reporting state-of-the-art benchmark performance and practical cost advantages.The paper presents extensive experiments on public benchmarks and private industrial datasets.
  • Proposed Approach: SynthDoG enables Donut pre-training with synthetic data and supports extension to multilingual settings without retraining an off-the-shelf OCR engine.The contribution list identifies multilingual extension and synthetic-data use as central advantages of the approach.

2 Method

Donut is an OCR-free, end-to-end Transformer for mapping document images directly to structured outputs. It combines visual encoding, textual decoding, synthetic-data pre-training, and JSON prediction for downstream VDU tasks.

  • Encoder: The encoder converts document images into embeddings, which the decoder transforms into token sequences representing target information.The encoder may use CNNs or Transformers; this study uses Swin Transformer.
  • Document Understanding Transformer: Donut uses a Transformer-based visual encoder and textual decoder to map document images directly into structured formats such as JSON.The model avoids OCR modules and is trained end-to-end.
  • Pre-training: Donut predicts document text in reading order with cross-entropy next-token prediction conditioned on the image and previous text context.This pre-training task is interpreted as pseudo-OCR and uses document images as visual-language-model corpora.
  • Synthetic Document Generator: SynthDoG generates synthetic document samples from English, Chinese, Japanese, and Korean Wikipedia data using randomized layouts and rendering techniques.The generator combines backgrounds, document textures, text, and layouts to mimic real documents.
  • Structured prediction: The decoder is trained to generate task-specific token sequences that are one-to-one invertible to JSON outputs, using special start and end tokens for fields.Malformed sequences cause the corresponding field to be treated as lost.
  • Application stage: At application time, Donut fine-tunes document understanding as a JSON prediction problem for downstream tasks.Prompts and desired output sequences are used for each application.

3 Experiments and Analyses

Experiments across document classification, information extraction, and visual question answering show that Donut performs strongly without OCR, while analyses examine pre-training, backbones, resolution, attention, and low-resource behavior.

  • Document Classification: Donut achieves state-of-the-art classification performance on RVL-CDIP without an external OCR engine, using fewer parameters and twice the speed of LayoutLMv2.OCR-based systems also incur additional parameters and inference costs for their OCR frameworks.
  • Document Information Extraction: Donut obtains the best scores across public and private document information extraction datasets, including field-level F1 and TED-based accuracy.The results indicate both key information extraction and prediction of complex field structures.
  • Document Visual Question Answering: Donut achieves competitive DocVQA scores without OCR and remains robust on handwritten documents, where handwriting OCR is difficult.The end-to-end approach avoids adding OCR-error correction modules that increase system size and maintenance cost.
  • On Pre-training Strategy: Synthetic images suffice for document information extraction pre-training, whereas DocVQA benefits from real images whose distributions resemble its evaluation data.The analysis compares SynthDoG-only, IIT-CDIP-only, and combined pre-training strategies.
  • On Encoder Backbone: EfficientNetV2 and Swin Transformer outperform other tested backbones on CORD and DocVQA, leading the authors to select Swin Transformer.The selection reflects Swin Transformer's scalability and higher performance than EfficientNetV2 in their study.
  • On Input Resolution and Data Size: Larger input resolutions improve performance, especially for DocVQA's tiny texts, but increase computational cost; at 1280×960 on CORD, inference takes 0.7 sec./image with 91.1 accuracy.A 2560×1920 input is also more robust in the extremely low-resource setting, including with 80 training samples.

4 Related Work

Related VDU work commonly separates OCR-based text reading from document understanding, using serialized OCR outputs with language or multimodal models. This pattern appears across document classification, information extraction, and visual question answering, while introducing complexity and extractive limits.

  • Modern OCR systems typically detect text regions and then recognize characters in cropped image instances.
  • Document classification: Document classification commonly serializes OCR-extracted text and feeds it to BERT-like language models, sometimes with visual features.
  • Document Information Extraction: Document information extraction converts OCR results into final parses through several often-complex processing stages.
  • Visual Question Answering: Document visual question answering typically applies OCR followed by BERT-like transformers, making the approach extractive by nature.

5 Conclusions

The paper concludes that Donut directly maps document images to structured outputs without OCR, using SynthDoG and staged training to support multilingual, end-to-end VDU. Experiments on public and private datasets report higher performance and better cost-effectiveness, while improved pre-training remains future work.

  • Donut directly maps document images to structured outputs without OCR in an end-to-end VDU framework.
  • SynthDoG generates synthetic document images, reducing dependence on large real-document datasets and supporting multilingual extension.
  • Donut is trained progressively from reading text to understanding whole documents through the proposed training pipeline.
  • Experiments on public benchmarks and private internal service datasets show higher performance and better cost-effectiveness for Donut.
  • Enhancing the pre-training objective is identified as a future work direction.

A.1 Details of OCR Engines (MS, CLOVA, Easy, Paddle)

The compared VDU backbones rely on publicly available off-the-shelf OCR engines, including commercial APIs and open-source models. MS OCR is described by its language coverage for printed and handwritten text.

  • BROS, LayoutLM, and LayoutLMv2 use off-the-shelf OCR engines as input-feature providers.
  • The experiments compare MS OCR and CLOVA OCR APIs with Easy OCR and Paddle OCR open-source models.
  • MS OCR: MS OCR supports 164 languages for printed text and 9 languages for handwritten text until 2022/03.

MS OCR

CLOVA OCR is a document-information-extraction API specialized for English, Japanese, and Korean. In the CORD ablation, it achieved the best accuracy.

  • CLOVA OCR is specialized in document information extraction and supports English, Japanese, and Korean until 2022/03.
  • The CORD ablation reports that CLOVA OCR achieved the best accuracy.

CLOVA OCR

Easy OCR is a publicly available, downloadable OCR engine supporting more than 80 languages. Its deep-learning architecture is modified to be lighter and faster, with 27M parameters.

  • Easy OCR is publicly available on GitHub and downloadable.
  • The engine supports more than 80 languages.
  • Its architecture uses modern deep-learning OCR modules with modifications for a lighter and faster model.
  • 27M parameters are used by the model.

Easy OCR

Paddle OCR is an open-source OCR engine whose lightweight mobile model is designed for fast English and Chinese text recognition on CPUs.

  • Paddle OCR is available as an open-source engine on GitHub.
  • Its lightweight mobile model is designed for fast, light OCR of English and Chinese texts.
  • The model is served on a CPU environment and is approximately 10M in size.

Paddle OCR

The supplied passages pair an OCR documentation URL with SynthDoG examples in English, Chinese, Japanese, and Korean, emphasizing their role in Donut pre-training.

  • The cited OCR documentation is a Microsoft Azure Computer Vision OCR overview.
  • SynthDoG examples cover English, Chinese, Japanese, and Korean samples.
  • The synthetic samples play an important role in Donut pre-training.

A.2 Details of Synthetic Document Generator (SynthDoG)

SynthDoG generates varied synthetic document images, while Donut uses Transformer-based encoding and decoding to produce structured outputs without OCR. The appendix also illustrates training, predictions, and resource costs across document tasks.

  • A.2 Details of Synthetic Document Generator (SynthDoG): SynthDoG generates document images using sampled backgrounds, textures, layouts, multilingual text, fonts, and image post-processing.Backgrounds come from ImageNet; multilingual corpora come from Wikipedia, and Noto fonts support varied languages.
  • A.3 Details of Document Information Extraction: Document information extraction requires reading text, understanding meaning, and predicting relations and structures among extracted information.
  • Experiments and examples: The appendix evaluates Donut on CORD, Ticket, Business Card, and Receipt datasets, using real-like samples for private industrial data.
  • Model training and output format: Donut training follows a Transformer encoder-decoder with teacher forcing and minimizes cross-entropy loss over token classifications.At inference, the predicted token from the last step is fed to the next step, and predicted sequences are converted into JSON.
  • Prediction examples: TED Acc. 97.5 is reported for one Ticket prediction example.
  • Implementation and training: 2–3 GPU days are used for pre-training with 200K steps, 64 NVIDIA A100 GPUs, and a mini-batch size of 196.
  • Implementation and training: 2560×1920 input settings increase computational costs rapidly for larger datasets such as RVL-CDIP and DocVQA.With 64 A100 GPUs, DocVQA requires approximately one GPU day and RVL-CDIP approximately two GPU days.
Loading 2111.15664v5…