PaperlayerView exact revision on arXiv

Source-linked AI summary

OvisOCR2 Technical Report

Shiyin Lu, Yinglun Li, Yu Xia, Yuhui Chen, An-Yang Ji, Jun-Peng Jiang, Qing-Guo Chen, Jianshan Zhao, En Lin, Haijun Li, Cheng Qin, Zhao Xu, Weihua Luo

arXiv:2607.13639v1cs.CVcs.AI

TL;DR

Document parsing is led by pipeline systems whose multiple stages complicate deployment and accumulate errors, motivating compact end-to-end alternatives. OvisOCR2 combines real and source-aligned synthetic data with staged training, achieving state-of-the-art results on public benchmarks and the highest overall score in its in-house evaluation.

  • Problem

    Pipeline document parsers complicate deployment through separate models and can accumulate errors across layout, recognition, and merging stages, motivating compact end-to-end alternatives.

  • Method

    OvisOCR2 combines filtered real-document annotations and source-aligned synthetic pages with supervised fine-tuning, reinforcement learning, on-policy distillation, and model fusion.

  • Results

    96.58 overall score on OmniDocBench v1.6 establishes state-of-the-art performance, while OvisOCR2 also achieves the highest overall scores on PureDocBench and the in-house benchmark.

  • Takeaways & Limitations

    OvisOCR2 demonstrates that a compact end-to-end parser can lead both public document-parsing benchmarks and a broader in-house comparison.

  • Takeaways & Limitations

    Robustness to degraded real-world images remains an important direction for future work.

Abstract

from arXiv · show

We introduce OvisOCR2, a 0.8B document parsing model. OvisOCR2 is designed as an end-to-end parser: given a document page image, it generates a Markdown representation in natural reading order, covering text, formulas, tables, and visual regions. We build a data engine that combines filtered real-document annotations with synthetic pages whose rendered images and Markdown targets are derived from the same HTML source. The training recipe includes supervised fine-tuning, reinforcement learning on a 4B branch with a multi-component reward design, on-policy distillation into the 0.8B model, and model fusion. On OmniDocBench v1.6, OvisOCR2 achieves a state-of-the-art overall score of 96.58, placing an end-to-end model at the top of this leaderboard previously dominated by pipeline methods and highlighting the potential of end-to-end document parsing. On PureDocBench, OvisOCR2 also achieves the highest Avg3 score of 75.06. Beyond these two public benchmarks, we evaluate OvisOCR2 on an in-house benchmark designed to cover a broader set of long-tail and challenging scenarios. OvisOCR2 obtains the best overall performance among the compared methods, providing further evidence of its generalization and robustness. OvisOCR2 is available at https://huggingface.co/ATH-MaaS/OvisOCR2.

1 Introduction

OvisOCR2 is a compact end-to-end document parser that converts page images into Markdown while preserving text, layout, and visual document elements. It achieves leading results on OmniDocBench v1.6, PureDocBench, and an in-house benchmark, surpassing pipeline-based approaches on public leaderboards.

  • Task and context: Document parsing converts visually rich page images into Markdown representations that preserve text, reading order, tables, formulas, figures, and layout elements.These structured representations support indexing, retrieval, and downstream applications.
  • Task and context: Pipeline methods separate layout analysis, region-level recognition, and page-level merging, creating deployment complexity despite strong leaderboard performance.At the time of writing, the top three OmniDocBench v1.6 methods were pipeline-based.
  • OvisOCR2: OvisOCR2 post-trains Qwen3.5-0.8B into a single compact end-to-end parser designed to achieve state-of-the-art performance with a small deployment footprint.The approach relies on a data engine combining real documents with synthetic pages and a training recipe suited to long outputs.
  • Evaluation: 96.58 is OvisOCR2’s overall score on OmniDocBench v1.6, placing an end-to-end parser ahead of the leading pipeline methods.This result establishes a new state of the art on the benchmark.
  • Evaluation: 75.06 is OvisOCR2’s Avg3 score on PureDocBench, where it ranks first.The evaluation also includes an in-house benchmark covering more than 1,000 pages and a broader range of document scenarios, where OvisOCR2 achieves the highest overall score.

2 Data engine

The data engine combines complementary real-world and synthetic pipelines because accurate end-to-end document parsing requires reliable transcription, structure, tables, formulas, and reading order. Real-world data provides visual diversity, while synthetic data supplies precise, scalable coverage of rare and difficult cases.

  • Data engine design: The data engine uses complementary real-world and synthetic pipelines because no single source provides both natural visual diversity and standardized, controllable supervision.Real documents have noisy annotations, while naive synthetic documents may lack realism.
  • Real-world data pipeline: The real-world pipeline converts large-scale document images into reliable document-to-Markdown training data spanning diverse visual and structural conditions.Coverage includes templates, scan qualities, fonts, languages, layouts, tables, formulas, figures, and other heterogeneous elements.
  • Synthetic data pipeline: The synthetic pipeline expands long-tail training coverage with controllable samples and precise ground-truth annotations for rare or difficult document structures.Examples include table topologies, dense formula-text interleaving, extreme multi-column layouts, and long Markdown outputs.

2.1 Real-world data pipeline

The real-world data pipeline converts structured outputs from specialized OCR parsers into unified Markdown, applies conservative normalization and pre-filtering, then uses subset-level manual inspection to exclude unreliable data. It preserves occasional minor errors with sample filters but removes subsets with frequent errors.

  • Conversion and filtering: OCR outputs are parsed from structured JSON and converted into unified Markdown using source-specific rules, treating parser results as candidates rather than final labels.The pipeline uses PaddleOCR-VL-1.5 or MinerU2.5-Pro and preserves parser-provided block order during serialization.
  • Conversion and filtering: Normalization validates categories, cleans text and formulas, filters malformed tables, serializes visual regions as bounding-box image tags, and handles parser-specific high-risk cases.Visual bounding boxes are normalized to [0, 1000), while unreliable seal detections and empty or malformed content are rejected.
  • Manual quality control: Manual subset inspection compares document–Markdown pairs for text, formulas, tables, visual regions, and natural reading order against the source images.Checks emphasize multi-column pages and dense mixtures of text, formulas, tables, and visual regions.
  • Manual quality control: Frequent subset-level errors cause removal from the training corpus, whereas occasional minor errors are retained alongside sample-level conversion filters.The inspection stage is conservative and avoids free-form rewriting or repair of converted annotations.

2.2 Synthetic data pipeline

The synthetic data pipeline converts real failure cases into scalable HTML-based synthesis families, generating paired document images and Markdown targets from a shared source. It expands long-tail coverage while preserving clean ground truth and faithful image-text alignment.

  • Paired image-label generation: The data engine derives Markdown directly from the corresponding HTML source and renders the same page into an image, avoiding parser-derived noise and preserving image-text alignment.Playwright rendering preserves realistic typography, spacing, tables, wrapping, colors, and page layout, while DOM geometry supplies normalized bounding boxes.
  • Hard-sample mining: The pipeline transforms observed hard-sample failures into programmatic HTML abstractions that preserve challenging factors while exposing controllable variables for scalable generation.Samples are grouped into reusable synthesis families rather than treated as isolated pages.
  • Template diversification: Agent-based diversification expands each seed template across content and structure while enforcing renderability, visual plausibility, category compliance, and valid Markdown conversion.Variations include text, numbers, formulas, terminology, tables, hierarchy, page organization, and visual-region placement.
  • Validation and scaling: The pipeline previews small batches, revises templates or rendering rules when defects appear, then scales generation with paired outputs and quality-control filtering.Filtering removes rendering, pairing, target, structural, localization, and duplication failures before dataset admission.
  • Outcome: The resulting synthetic data are controllable and precise, complement real-world data by expanding long-tail coverage, and preserve clean ground truth with faithful image-text alignment.This summarizes the pipeline’s stated contribution to training-data construction.

3 Training … 3.3 Reinforcement learning

OvisOCR2 uses staged training that combines supervised fine-tuning, reinforcement learning with structured rewards, on-policy distillation, and model fusion. Its RL stage uses GRPO, on-policy hard-case filtering, multi-component page rewards, and scalable multimodal training optimizations.

  • 3.1 Training pipeline: The training pipeline proceeds from supervised fine-tuning to reinforcement learning, on-policy distillation from a 4B teacher, and model fusion.Supervised fine-tuning initializes the parsing policy, RL improves hard-page behavior, and distillation transfers reward-aligned behavior to the deployable 0.8B student.
  • 3.2 Supervised fine-tuning: SFT combines real document pages with synthetic samples to cover natural layouts, visual noise, tables, and formula-intensive pages.Real data provide varied document conditions, while synthetic data expand controlled structural coverage.
  • 3.2 Supervised fine-tuning: The SFT stage trains Qwen3.5-0.8B for two epochs and Qwen3.5-4B for 20% of an epoch using the same document-parsing objective.The 0.8B checkpoint initializes the student, while the 4B checkpoint becomes the base policy for RL and its resulting checkpoint serves as the OPD teacher.
  • 3.3 Reinforcement learning: RL improves parsing by optimizing sampled outputs for structural correctness, addressing errors in tables, formulas, omissions, repetition, and truncation.These structural errors are weakly represented by token-level imitation loss.
  • 3.3 Reinforcement learning: GRPO samples multiple responses per page and computes group-relative advantages from verifiable rewards without training an additional value model.On-policy filtering emphasizes pages where the current model occasionally produces responses better than its average behavior.
  • 3.3.1 On-policy hard case construction: RL data rely primarily on synthetic pages with accurate ground-truth structures, while retaining a smaller high-quality real-document set for natural distributions and visual variation.This data construction supports text, formula, and table rewards while preserving realistic page conditions.
  • 3.3.2 Multi-component reward design: The page-level reward aggregates available text, table, and display-formula component scores, with validity guards zeroing affected components for truncation or unparseable structures.CDM and TEDS are normalized to [0, 1], while component availability depends on evaluable ground-truth units.
  • 3.3.3 Scalable multimodal RL training: Scalable multimodal RL uses parallel reward workers, object-store-backed visual-tensor references, and common-prefix masking to reduce computation, memory pressure, and data-transfer overhead.Shortcut reward paths and strictly on-policy single-epoch updates further reduce unnecessary processing.

3.4 On-policy distillation

On-policy distillation transfers reward-aligned parsing behavior from the more stable 4B RL model to the compact 0.8B student after direct compact-model RL proved less stable. The student learns from teacher distributions on its own generated page trajectories using a top-k reverse-KL objective, avoiding another high-variance RL stage while reducing alignment cost.

  • Motivation: Direct 0.8B RL produced larger KL drift and less stable table quality, especially on dense pages, than the 4B RL model.Later training showed higher policy divergence and declining table quality for direct 0.8B RL.
  • Teacher-student setup: The RL-enhanced 4B model serves as teacher for the 0.8B student, which generates complete page outputs and receives token-level distribution supervision on its on-policy trajectories.This transfers preferences for tables, long formulas, and dense layouts without exposing the student to another high-variance RL stage.
  • Objective: The student top-k reverse-KL objective matches teacher and student distributions over the student’s top-k token support at valid response positions.The reverse-KL direction is mode-seeking, discouraging student mass on tokens assigned low probability by the teacher.
  • Efficient alignment: Top-k support reduces the dominant distillation tensor from O(TV) to O(Tk) for response length T and vocabulary size V.The teacher scores student-generated trajectories using the same prompt-response prefixes and selected token IDs; chunked projection reduces peak memory for long responses.

3.5 Model fusion

The authors train several OvisOCR2 variants with different data mixtures and training configurations, then fuse them through weighted parameter averaging to produce the final model.

  • 3.5 Model fusion: Several candidate OvisOCR2 variants are trained before model fusion.The variants differ in their data mixtures and training configurations.
  • 3.5 Model fusion: The variants vary along two dimensions: data mixture and training configuration.
  • 3.5 Model fusion: Weighted parameter averaging is applied to produce the final OvisOCR2 model.

4 Evaluation

OvisOCR2 is evaluated against general and specialized document-parsing models on OmniDocBench v1.6, PureDocBench, and a broader in-house benchmark. It achieves leading overall performance on all three evaluations, while degraded real-world images remain a challenge.

  • Evaluation setup: The evaluation compares OvisOCR2 with general VLMs and specialized pipeline or end-to-end document-parsing models across three benchmarks.The benchmarks are OmniDocBench v1.6, PureDocBench, and an in-house benchmark targeting complex document scenarios.
  • OmniDocBench v1.6: 96.58 overall score makes OvisOCR2 state of the art on OmniDocBench v1.6 with a compact 0.8B model.It surpasses leading pipeline methods and improves over the previous best end-to-end method by 1.84 points.
  • PureDocBench: 75.06 Avg3 makes OvisOCR2 state of the art on PureDocBench, where it also ranks first on the clean and digital tracks.It remains below Gemini-3.1-Pro and Qwen3.5-122B-A10B on the real track, highlighting degraded-image robustness as a remaining challenge.
  • In-house benchmark: OvisOCR2 achieves the highest overall score on the in-house benchmark and leads consistently across text, formula, table, and reading-order metrics.Its advantage holds across the easy, medium, and hard difficulty tiers.
  • In-house benchmark subsets: On handwriting and complex-table subsets, OvisOCR2 achieves the highest overall score while leading the reported formula, text, reading-order, or table metrics.On complex tables, it also achieves the highest TEDS and lowest missing rate; GLM-OCR has a higher table TEDS score on handwriting.

5 Conclusion

OvisOCR2 is a compact 0.8B end-to-end model for page-level document parsing, supported by source-aligned data construction and a multi-stage training recipe. It establishes new state-of-the-art results on OmniDocBench v1.6 and PureDocBench across public and in-house evaluations.

  • Conclusion: OvisOCR2 is a compact 0.8B end-to-end model for page-level document parsing.It is presented as the report’s central model contribution.
  • Conclusion: The data engine combines filtered real-document annotations with source-aligned synthetic pages.The synthetic pages are aligned with their source documents.
  • Conclusion: The training recipe applies supervised fine-tuning, reinforcement learning, on-policy distillation, and model fusion.These components form the reported multi-stage training approach.
  • Conclusion: OvisOCR2 establishes new state-of-the-art results on OmniDocBench v1.6 and PureDocBench across public and in-house benchmarks.The conclusion reports new state-of-the-art performance on both named public benchmarks and evaluations conducted in-house.

Author List

The paper is authored by Shiyin Lu, Yinglun Li, Yu Xia, Yuhui Chen, An-Yang Ji, Jun-Peng Jiang, Qing-Guo Chen, Jianshan Zhao, En Lin, Haijun Li, Cheng Qin, Zhao Xu, and Weihua Luo.

  • Author List: The authors are Shiyin Lu, Yinglun Li, Yu Xia, Yuhui Chen, An-Yang Ji, Jun-Peng Jiang, Qing-Guo Chen, Jianshan Zhao, En Lin, Haijun Li, Cheng Qin, Zhao Xu, Weihua Luo.

A Qualitative examples

The appendix presents qualitative comparisons for table and handwritten documents. Figures A.1–A.4 illustrate these examples, with annotations marking insertions, recognition errors, table-structure errors, and omitted text.

  • Red highlights identify text insertions, recognition errors, and table-structure errors, while light gray bracketed text indicates omissions.
  • The appendix compares model outputs on table documents in Figures A.1 and A.2.
  • Figures A.3 and A.4 provide qualitative comparisons on handwritten documents.
Loading 2607.13639v1…