Source-linked AI summary
General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model
Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, Chunrui Han, Xiangyu Zhang
TL;DR
Traditional OCR systems rely on costly, task-specific modular pipelines, while LVLM designs are not well suited to dense pure-perception OCR. The paper proposes OCR-2.0 and its unified GOT model, which shows promising performance across diverse OCR tasks, while still requiring broader language, signal, and geometry support.
Problem
Traditional OCR systems use costly modular pipelines and separate networks for different subtasks, limiting generality and user convenience.
Method
The paper proposes OCR-2.0 and GOT, a unified end-to-end encoder-decoder model with high-compression image encoding and long-context OCR output.
Results
GOT demonstrates promising performance across various OCR tasks, including document, sheet music, geometry, and chart OCR.
Takeaways & Limitations
OCR-2.0 integrates diverse pan-OCR tasks into one model and offers a simpler direction for OCR model design, data engineering, and applications.
Takeaways & Limitations
GOT remains constrained by the need to support more languages, broader artificial signals, and more complex geometries.
Abstract
from arXiv · showhide
Traditional OCR systems (OCR-1.0) are increasingly unable to meet people's usage due to the growing demand for intelligent processing of man-made optical characters. In this paper, we collectively refer to all artificial optical signals (e.g., plain texts, math/molecular formulas, tables, charts, sheet music, and even geometric shapes) as "characters" and propose the General OCR Theory along with an excellent model, namely GOT, to promote the arrival of OCR-2.0. The GOT, with 580M parameters, is a unified, elegant, and end-to-end model, consisting of a high-compression encoder and a long-contexts decoder. As an OCR-2.0 model, GOT can handle all the above "characters" under various OCR tasks. On the input side, the model supports commonly used scene- and document-style images in slice and whole-page styles. On the output side, GOT can generate plain or formatted results (markdown/tikz/smiles/kern) via an easy prompt. Besides, the model enjoys interactive OCR features, i.e., region-level recognition guided by coordinates or colors. Furthermore, we also adapt dynamic resolution and multi-page OCR technologies to GOT for better practicality. In experiments, we provide sufficient results to prove the superiority of our model.
1 Introduction
OCR-1.0 systems rely on costly multi-module pipelines and separate models, while LVLM designs face compression and deployment challenges for dense OCR. The paper proposes OCR-2.0 and GOT as a unified, versatile, end-to-end alternative.
- Traditional OCR uses detection, cropping, and recognition modules, creating local optima, systematic errors, and high maintenance costs.
- LVLMs align image tokens to text for reasoning, but this limits optical-character compression in dense OCR scenes.
- OCR-2.0 is proposed as a unified framework intended to overcome bottlenecks in both traditional OCR and LVLM-based approaches.
- GOT targets versatility across texts, formulas, tables, sheet music, charts, and geometric shapes, with readable structured outputs controlled by prompts.
- GOT uses a high-compression encoder and long-context decoder within an end-to-end encoder-decoder architecture.
- GOT demonstrates promising performance across various OCR tasks, while remaining limited in languages, artificial signals, and geometric complexity.
2 Related Work
Traditional OCR depends on specialized pipelines, whereas LVLM-driven OCR inherits limitations from visual-token alignment and encoder coverage. These approaches motivate a more general OCR design.
- 2.1 Traditional OCR: Traditional OCR combines layout analysis, text detection, region extraction, and recognition networks, which can introduce systematic errors and high maintenance costs.
- 2.1 Traditional OCR: Traditional recognition modules often process only small image slices, requiring text detection and region cropping before recognition.
- 2.2 LVLM-driven OCR: LVLM OCR commonly relies on CLIP-based encoders, whose English scene-text bias limits performance on other languages and document images.
- 2.2 LVLM-driven OCR: Aligning image tokens to text tokens creates a low optical-character compression rate, encouraging sliding-window image decomposition for dense or high-resolution OCR.
3 General OCR Theory
The paper proposes OCR-2.0 as a general OCR theory and presents GOT together with its framework, multi-stage training strategy, and data engines.
- OCR-2.0 is proposed to promote development of a general OCR field beyond conventional OCR approaches.
- GOT operationalizes the theory through a novel OCR model whose technical design includes a framework, multi-stage training strategy, and corresponding data engines.
3.1 Framework
GOT combines an image encoder, connector, and output decoder, trained through staged optimization. Its framework uses a tiny decoder for encoder pre-training before integrating a larger language decoder.
- GOT comprises an image encoder, a linear connector, and an output decoder.The linear layer maps the channel dimension between the vision encoder and language decoder.
- Stage 1 pre-trains the vision encoder on pure text recognition using a tiny decoder to pass gradients efficiently.
- Stage 2 connects the vision encoder to Qwen-0.5B and trains it with broader OCR-2.0 knowledge.
- Stage 3 customizes GOT for new character-recognition features without modifying the vision encoder.
3.2 Pre-train the OCR-earmarked Vision Encoder
GOT trains an OCR-specific vision encoder for scene and document text across slice and whole-page inputs. The encoder uses local attention and is pretrained on about 5M image-text pairs assembled from natural scenes and PDFs.
- Encoder motivation: GOT trains its own OCR encoder because no suitable pretrained encoder was available for varied scene and document inputs.The target inputs include different shapes, slices, and whole pages.
- Encoder architecture: The encoder uses VitDet with local attention to reduce the computational cost of high-resolution images.Its final layers convert a 1024×1024×3 image into 256×1024 image tokens.
- Pretraining data: About 5M image-text pairs support encoder pretraining, including 3M scene-text and 2M document-OCR examples.The corpus combines real-scene and document-level data.
- Pretraining data: Scene data combines English images from Laion-2B and Chinese images from Wukong, with pseudo ground truth captured using PaddleOCR.The resulting 2M scene examples are evenly divided between Chinese and English.
- Pretraining data: Scene text is represented both as ordered full-image content and as cropped text-region image slices.The slice method contributes another 1M image-text pairs.
- Pretraining data: Document data from Common Crawl PDFs yields 1.2M full-page pairs and 0.8M line- or paragraph-level image slices.Fitz extracts dense text and bounding boxes used to create the document examples.
3.3 Scaling Up the OCR-2.0 Knowledge via Multi-task Joint-training
GOT scales OCR-2.0 knowledge through joint training with a compact Qwen decoder and synthetic engines for plain, formatted, and general optical characters. These engines cover formulas, tables, sheet music, geometric shapes, and charts.
- Model and training setup: GOT connects an approximately 80M-parameter vision encoder to a 500M-parameter Qwen decoder with multilingual prior knowledge.A 1024×1024 connector aligns the encoder output with Qwen-0.5B input channels.
- Data engines: The joint-training data combines plain OCR examples with formatted and more general optical-character data generated by specialized engines.The synthesis methods and data engines are designed to inject OCR-2.0 knowledge into GOT.
- Formatted OCR: Mathpix-formatted data preserves output structure for mathematical formulas and tables, including roughly 1M formula fragments extracted from LaTeX sources.Mathpix-markdown-it rendering is reported as more than 20× faster than direct LaTeX rendering for formulas.
- General optical characters: Sheet-music data uses GrandStaff Humdrum **kern transcriptions rendered with Verovio, producing about 0.5M samples.The rendering is limited to single-system sheet music and adds paper-style backgrounds and metadata.
- General optical characters: Geometric-shape data represents simple shapes with TikZ-style points, lines, and spatial relationships.The targeted shapes include circles, rectangles, triangles, and combinations.
- General optical characters: Chart OCR converts chart knowledge such as titles, axes, sources, and values into editable table or Python-dict output.Chart image-text pairs are rendered with Matplotlib and Pyecharts.
3.4 Customizing New OCR Features by Post-training the Decoder
After compressing diverse optical signals, GOT customizes its decoder for fine-grained, dynamic-resolution, and multi-page OCR. These features support targeted regions, extreme-resolution layouts, and documents whose page boundaries are difficult to separate.
- Feature customization: Post-training only the decoder adds fine-grained, multi-page, and dynamic-resolution OCR features.The perceptually trained encoder is retained while the decoder is adapted to user needs.
- Fine-grained OCR: Fine-grained OCR recognizes text within a region selected by box coordinates or text color.These prompts suppress unrelated characters outside the region of interest.
- Dynamic-resolution OCR: Dynamic-resolution OCR uses a large 1024×1024 sliding window for huge images such as horizontally stitched two-page PDFs.The high-compression encoder supports this under extreme-resolution conditions.
- Multi-page OCR: Multi-page OCR processes multiple pages directly when formatted PDF data cannot be cleanly divided by page.The feature avoids requiring page-broken ground truth for such documents.
4 Experiments
GOT is evaluated across five OCR task categories, including document, scene, formatted, fine-grained, and general-character recognition. The reported results show strong performance across these settings, with multi-crop inference improving formatted OCR and GOT outperforming Fox on fine-grained OCR and chart-specific models and LVLMs on chart OCR.
- Evaluation scope: Five OCR task categories are evaluated: plain document, scene text, fine-grained document, formatted document, and more general character OCR.The test benchmarks undergo strict text-similarity filtering to avoid overlap with training data.
- Plain document OCR: 580M-parameter GOT achieves advanced performance on Chinese and English document-level PDF OCR.Fox is used as the benchmark, with edit distance, F1-score, precision, recall, BLEU, and METEOR reported at word level.
- Scene text OCR: GOT works well on natural scene text in a benchmark of 400 manually corrected Chinese and English images.The benchmark is balanced equally between Chinese and English, with character-level metric calculation.
- Formatted document OCR: Multi-crop inference further improves formatted document OCR, especially for formulas and tables containing small text.The benchmark contains 90 manually corrected Chinese and English document pages; single-scale GOT already produces satisfactory results.
- Fine-grained OCR: GOT is overall better than Fox on both bounding-box-based and color-based fine-grained OCR tasks.These tasks evaluate interactive recognition using region references.
- More general OCR: GOT performs well on sheet music and geometry benchmarks and surpasses chart-specific models and popular LVLMs on chart OCR.The general-character evaluation uses 100 sheet-music samples, 180 geometry samples, and ChartQA and PlotQA chart benchmarks.
5 Conclusion
The paper presents GOT as a primary OCR-2.0 model that is structurally simpler than OCR-1.0 systems, focuses on pure OCR tasks, and reports superior performance. It positions OCR-2.0 as integrating diverse pan-OCR tasks into one model and as a direction for future research.
- GOT is structurally simpler than OCR-1.0 systems while focusing more on pure OCR tasks.
- GOT integrates various pan-OCR tasks into one model.
- OCR-2.0 remains a research direction spanning model design, data engineering, and application scenarios.
6 Appendix
The appendix demonstrates GOT's OCR behavior through prompted examples on formatted and plain-text documents. The examples cover full-page, table/formula-slice, and dense double-column inputs.
- The appendix provides output results and corresponding input prompts for different OCR task types.
- GOT works well on full-page texts and table/formula slice texts with formatted OCR output.
- GOT handles double-column documents with high text density in plain-text document OCR.
Prompt: OCR
The prompted examples illustrate GOT's interactive, resolution-adaptive, multi-page, and general OCR capabilities across several optical character types. They also show a language-scope caveat: recognition beyond Chinese and English is not guaranteed without fine-tuning.
- Prompt examples include plain OCR, formatted OCR, and coordinate-guided OCR requests.
- GOT can output OCR results for regions of interest based on prompts in scene and fine-grained OCR tasks.
- Dynamic resolution adapts GOT to high-resolution images and removes limitation by image size in dual-page reading mode.
- Multi-page OCR supports continued training with multi-page PDF-text pairs, such as Arxiv papers paired with .tex files.
- GOT processes molecular formulas, sheet music, and charts as more general OCR inputs.
- GOT's recognition quality for languages beyond Chinese and English is not guaranteed, so corresponding fine-tuning is recommended when needed.