Source-linked AI summary
OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models
Yufeng Zhong, Lei Chen, Xuanle Zhao, Wenkang Han, Liming Zheng, Jing Huang, Deyang Jiang, Yilin Cao, Lin Ma, Zhixiong Zeng
TL;DR
Existing OCR methods primarily target text in documents, leaving visually information-dense sources such as charts, webpages, and scientific plots insufficiently addressed. OCRVerse unifies text-centric and vision-centric OCR through comprehensive data engineering and two-stage SFT-RL training, achieving competitive results across both domains, including 89.23 on OmniDocBench v1.5. Its remaining gaps include layout-aware text recognition and complex table handling.
Problem
Existing OCR methods focus primarily on recognizing text in images and scanned documents, while vision-centric sources require understanding visual structures and code-level representations.
Method
OCRVerse combines comprehensive text- and vision-centric data engineering with two-stage SFT-RL multi-domain training, using mixed-domain SFT and personalized RL rewards.
Results
OCRVerse achieves competitive performance across text-centric and vision-centric scenarios, including 89.23 on OmniDocBench v1.5 and comparable vision-centric results to open-source models.
Takeaways & Limitations
OCRVerse provides an end-to-end holistic OCR framework that bridges character-level recognition and code-level representation across diverse visual information sources.
Takeaways & Limitations
OCRVerse lacks explicit layout-aware mechanisms and trails competing systems on table recognition, especially for complex multi-row, multi-column, and spanning-cell structures.
Abstract
from arXiv · showhide
The development of large vision language models drives the demand for managing, and applying massive amounts of multimodal data, making OCR technology, which extracts information from visual images, increasingly popular. However, existing OCR methods primarily focus on recognizing text elements from images or scanned documents (Text-centric OCR), neglecting the identification of visual elements from visually information-dense image sources (Vision-centric OCR), such as charts, web pages and science plots. In reality, these visually information-dense images are widespread on the internet and have significant real-world application value, such as data visualization and web page analysis. In this technical report, we propose OCRVerse, the first holistic OCR method in end-to-end manner that enables unified text-centric OCR and vision-centric OCR. To this end, we constructe comprehensive data engineering to cover a wide range of text-centric documents, such as newspapers, magazines and books, as well as vision-centric rendered composites, including charts, web pages and scientific plots. Moreover, we propose a two-stage SFT-RL multi-domain training method for OCRVerse. SFT directly mixes cross-domain data to train and establish initial domain knowledge, while RL focuses on designing personalized reward strategies for the characteristics of each domain. Specifically, since different domains require various output formats and expected outputs, we provide sufficient flexibility in the RL stage to customize flexible reward signals for each domain, thereby improving cross-domain fusion and avoiding data conflicts. Experimental results demonstrate the effectiveness of OCRVerse, achieving competitive results across text-centric and vision-centric data types, even comparable to large-scale open-source and closed-source models.
1 Introduction
OCRVerse addresses the gap between text-centric OCR and vision-centric OCR by unifying character recognition with structured visual understanding in an end-to-end model. It combines broad multimodal data coverage with two-stage multi-domain training to support diverse output formats.
- OCR applications increasingly require extracting textual information from diverse visual sources for document digitization, automated data entry, and intelligent content analysis.
- Vision-centric OCR extends beyond printed characters to semantic visual structures in charts, webpages, and scientific plots, requiring code-level representations such as HTML, Python, and LaTeX.
- OCRVerse is presented as the first end-to-end holistic OCR method unifying text-centric and vision-centric recognition.
- The two-stage SFT-RL method mixes domains during SFT, then uses personalized reward strategies during RL to address domain-specific output formats and conflicts.
- 89.23 on OmniDocBench v1.5 demonstrates competitive performance across text-centric and vision-centric scenarios, including results comparable to open-source models.
2 Related Work
Related OCR research includes pipeline, end-to-end VLM, and hybrid VLM-pipeline approaches, but these methods largely remain focused on text-centric document processing. OCRVerse is motivated by the lack of a unified framework for diverse vision-centric tasks and output formats.
- Text-centric OCR: Traditional pipeline OCR decomposes parsing into specialized layout detection and text-recognition modules, offering structured processing through a divide-and-conquer strategy.
- Text-centric OCR: End-to-end VLM methods decode document text directly from visual features, but long-sequence generation can produce hallucinations, repetition, or attention drift.
- Text-centric OCR: Hybrid VLM-pipeline methods add explicit layout priors to guide semantic reasoning and mitigate hallucinations and resolution constraints.
- Vision-centric OCR: Vision-centric OCR translates charts, webpages, scientific plots, and SVGs into executable HTML, LaTeX, or Python representations for reconstruction or data reuse.
- Research gap: Existing methods operate predominantly in silos, lacking unified cross-scenario processing for multiple information-dense visual tasks.
3 Dataset
OCRVerse’s dataset combines broad text-centric document coverage with specialized vision-centric content requiring structured representations. A multi-stage pipeline collects, cleans, annotates, and augments heterogeneous sources to provide comprehensive training data.
- Data Types: The dataset covers nine text-centric scenarios and six vision-centric scenarios, spanning everyday documents and professional structured content.
- Text-centric Data: Text-centric sources include natural scenes, books, magazines, papers, reports, slides, exam papers, notes, and newspapers with varied layouts and information density.
- Vision-centric Data: Vision-centric domains include charts, webpages, icons, geometry, circuits, and molecules that require structured and semantic representations beyond ordinary text.
- Data Construction: The construction pipeline integrates open-source datasets, real-world PDFs, and synthetic data for text-centric coverage, alongside collected chart, webpage, and SVG data for vision-centric coverage.
- Data Construction: Cleaning removes missing, corrupted, incomplete, or structurally unreliable samples, with domain-specific filtering for documents, webpages, and code-generation data.
- Data Construction: Annotations are produced through VLM re-annotation, specialized OCR and structured extraction, visualization rendering, and bootstrapped domain-specific self-annotation.
4 Method
OCRVerse trains a unified OCR model in two stages: mixed-domain SFT establishes shared knowledge, then domain-specific RL optimizes specialized output and visual requirements. Personalized rewards target text correctness, structural validity, format alignment, and visual fidelity across heterogeneous OCR domains.
- SFT Stage: SFT directly mixes data from eight domains to learn shared visual-semantic patterns while preserving domain-specific output capabilities.The domains include text-centric documents, tables, and formulas alongside charts, web pages, and scientific plots.
- RL Stage: RL uses personalized reward strategies to resolve domain conflicts and optimize specialized performance for different content types and output formats.The approach avoids conflicts associated with uniform reward signals and supports fine-grained cross-domain optimization.
- Domain-Specific Reward Design: Text-centric rewards evaluate plain text with normalized edit distance, formulas with BLEU after normalization, and tables with TEDS-S after structural normalization.The reward aggregates type-specific scores over content types present in the ground truth.
- Domain-Specific Reward Design: Vision-centric rewards measure rendered-output fidelity using DINOv2 feature cosine similarity at global thumbnail and local patch scales.Format-alignment rewards additionally encourage generated code to match the expected programming language.
- Policy Optimization: GRPO samples grouped responses, computes group-normalized advantages, and optimizes the policy with probability-ratio clipping for stable domain-specific learning.The clipping threshold constrains policy updates while the model learns from customized rewards.
- Training Overview: The two-stage method establishes cross-domain knowledge during SFT and refines domain-specific capabilities during RL to avoid conflicts from naive multi-task learning.The training data for RL is selected using domain-specific criteria, including entropy-based filtering for challenging text-centric samples.
5 Experiment
OCRVerse is evaluated across text-centric document parsing and vision-centric image-to-code tasks against pipeline tools, general VLMs, and specialized OCR systems. It achieves competitive or superior results across diverse benchmarks, while remaining limited on layout-sensitive text and complex table recognition.
- Text-centric OCR: 87.13 CDM in formula recognition surpasses Deepseek-OCR (83.37) and olmOCR-7B (86.04).The reported formula data covers single-line, multi-line, and page-level expressions across mathematics, physics, and computer science.
- Vision-centric OCR: OCRVerse reaches 84.8% execution success on ChartMimic, exceeding Qwen3-VL-8B (78.3%) and InternVL3-8B (63.3%).Its low-level score of 72.2 and high-level score of 75.4 also compare with Qwen2.5-VL-72B despite being 18× smaller.
- Vision-centric OCR: OCRVerse scores 76.3 on UniSVG and achieves competitive Design2Code results with low-level 85.7 and high-level 87.4 scores.The UniSVG score ranks second only to GPT-5 (77.3), while Design2Code assesses web layout reconstruction.
- Vision-centric OCR: OCRVerse achieves 88.7% rendering success and 63.1 EMS on Image2LaTeX-plot, surpassing GPT-5 (78.7%, 57.4).On ChemDraw, it reaches 89.1% execution success and 54.7 Tanimoto similarity, outperforming open-source alternatives.
- Overall findings: The 4B-parameter OCRVerse delivers performance comparable to or exceeding 70B-parameter models across vision-centric code-generation tasks.The evaluation spans five vision-centric benchmarks and reports competitive performance against substantially larger counterparts.
6 Conclusion
OCRVerse unifies text-centric and vision-centric OCR in an end-to-end method, combining comprehensive multi-domain data engineering with two-stage SFT-RL training. It achieves competitive performance across both scenarios, including 89.23 on OmniDocBench v1.5.
- OCRVerse is the first holistic OCR method to unify text-centric and vision-centric capabilities end to end.
- The method combines comprehensive data engineering across diverse domains with two-stage SFT-RL training to address cross-domain conflicts.
- 89.23 on OmniDocBench v1.5 demonstrates competitive text-centric performance, while OCRVerse matches open-source models on vision-centric benchmarks.
- OCRVerse provides a practical solution for data visualization, web page analysis, and intelligent content understanding.