PaperlayerView exact revision on arXiv

Source-linked AI summary

OvisOCR2 Technical Report

Shiyin Lu, Yinglun Li, Yu Xia, Yuhui Chen, An-Yang Ji, Jun-Peng Jiang, Qing-Guo Chen, Jianshan Zhao, En Lin, Haijun Li, Cheng Qin, Zhao Xu, Weihua Luo

arXiv:2607.13639v1cs.CVcs.AI

TL;DR

Document parsing leaderboards remain dominated by pipeline methods, whose separate stages complicate deployment and can accumulate errors. OvisOCR2 is a compact end-to-end parser trained with real and synthetic data and a staged post-training recipe; it achieves state-of-the-art results on public benchmarks and the highest overall score in the in-house comparison.

  • Problem

    Pipeline methods lead mainstream document-parsing leaderboards, while their separate stages complicate deployment and can accumulate errors.

    Source: §1 · p. 2

  • Method

    OvisOCR2 combines filtered real-document annotations with source-aligned synthetic pages and trains a compact end-to-end parser using supervised fine-tuning, reinforcement learning, on-policy distillation, and model fusion.

    Source: §5 · p. 16

  • Results

    OvisOCR2 establishes state-of-the-art results on OmniDocBench v1.6, with an overall score of 96.58, and PureDocBench, and achieves the highest overall score in the in-house comparison.

    Source: §4.1 · p. 13, §4.2 · p. 14, §4.3 · p. 15

  • Takeaways & Limitations

    These results highlight the potential of end-to-end document parsing to achieve leading performance against pipeline methods on public benchmarks.

    Source: §4.1 · p. 13, §5 · p. 16

    Robustness to degraded real-world images remains a direction for improvement, and OvisOCR2 trails strong general VLMs on PureDocBench’s real track.

    Source: §4.2 · p. 14, §5 · p. 16

Abstract

from arXiv · show

We introduce OvisOCR2, a 0.8B document parsing model. OvisOCR2 is designed as an end-to-end parser: given a document page image, it generates a Markdown representation in natural reading order, covering text, formulas, tables, and visual regions. We build a data engine that combines filtered real-document annotations with synthetic pages whose rendered images and Markdown targets are derived from the same HTML source. The training recipe includes supervised fine-tuning, reinforcement learning on a 4B branch with a multi-component reward design, on-policy distillation into the 0.8B model, and model fusion. On OmniDocBench v1.6, OvisOCR2 achieves a state-of-the-art overall score of 96.58, placing an end-to-end model at the top of this leaderboard previously dominated by pipeline methods and highlighting the potential of end-to-end document parsing. On PureDocBench, OvisOCR2 also achieves the highest Avg3 score of 75.06. Beyond these two public benchmarks, we evaluate OvisOCR2 on an in-house benchmark designed to cover a broader set of long-tail and challenging scenarios. OvisOCR2 obtains the best overall performance among the compared methods, providing further evidence of its generalization and robustness. OvisOCR2 is available at https://huggingface.co/ATH-MaaS/OvisOCR2.

1 Introduction

OvisOCR2 is a compact end-to-end document parser designed to generate page-level Markdown while overcoming the performance gap between end-to-end and pipeline methods. It combines real and synthetic data with a multi-stage training recipe and leads the reported public and in-house benchmark results.

1 Introduction

  • Page-level document parsing converts visually rich images into Markdown in natural reading order, preserving text, layout, tables, formulas, figures, headers, and footers for downstream use [1] [2] [3] [4] [5].The paper studies the page-level image-to-Markdown setting, in which one model generates a unified Markdown representation from a document image.Source: §1 · p. 2
  • Pipeline methods lead mainstream parsing benchmarks [1] but split layout analysis, recognition, and merging [3] [6] [5]; end-to-end methods simplify deployment and use page context [8] [9], yet have lagged in performance.Errors such as missed table boundaries, imprecise formula crops, and incorrect reading order can accumulate across pipeline stages.Source: §1 · p. 2
  • OvisOCR2 post-trains the smallest Qwen3.5 model, Qwen3.5-0.8B, combining filtered real-document annotations and HTML-derived synthetic pages with SFT, 4B-branch RL, on-policy distillation, and model fusion.The real annotations are normalized to a unified Markdown schema and cleaned by rule-based filtering and subset-level spot-checking; synthetic images and Markdown targets come from the same HTML source.Source: §1 · p. 2
  • 96.58 overall score on OmniDocBench v1.6 [1] and 75.06 Avg3 score on PureDocBench [2] accompany the highest in-house overall performance across all three difficulty tiers and on handwriting and complex-table subsets.The in-house benchmark covers more than 1,000 pages across a broader range of document scenarios.Source: §1 · p. 2

2 Data engine

The data engine combines real-world and synthetic pipelines because neither source alone provides both natural document variation and reliable, scalable coverage. Together, they support supervision for transcription, structure, content recovery, and reading order.

2 Data engine

  • End-to-end parsing data must provide accurate transcription, valid serialization, faithful table and formula recovery, and globally coherent reading order.These requirements make data preparation a key part of system design.Source: §2 · p. 2
  • The data engine pairs real-world and synthetic pipelines to address the limitations of either source alone.Real documents provide natural layouts and varied visual conditions but noisy, hard-to-standardize annotations; synthetic documents provide controllable targets and scalable coverage but can lack realism.Source: §2 · p. 3, Figure 2 · p. 3
  • Real-world data exposes the model to diverse document visuals and heterogeneous elements through document-to-Markdown training data.The pipeline targets varied templates, scan qualities, fonts, languages, layouts, tables, formulas, and figures.Source: §2 · p. 3
  • Synthetic data expands long-tail coverage with controllable samples and precise ground-truth annotations for rare or difficult cases.Examples include table topologies, dense formula-text interleaving, extreme multi-column layouts, and long Markdown outputs.Source: §2 · p. 3

2.1 Real-world data pipeline

The real-world data pipeline converts OCR parser outputs into a unified Markdown schema and conservatively filters them before training. It combines deterministic, parser-specific normalization with manual checks of sampled document–Markdown pairs, rejecting unreliable subsets rather than freely rewriting annotations.

2.1 Real-world data pipeline

  • The pipeline treats OCR outputs as candidate annotations, applying deterministic conversion and conservative filtering to improve Markdown consistency and structural validity.Parser outputs are not used directly as final labels; subset-level spot-checking adds a further quality gate.Source: §2.1 · p. 4

2.1.1 Rule-based structured parsing and Markdown serialization

  • PaddleOCR-VL-1.5 [12] and MinerU2.5-Pro [6] provide structured JSON blocks that source-specific rules convert into unified Markdown.Each block carries a category, recognized content, bounding box, and optional parser-specific metadata; the converter uses block order rather than raw Markdown strings.Source: §2.1.1 · p. 4
  • The converter validates categories and normalizes text, formulas, tables, visual regions, and high-risk parser-specific blocks before serialization.It rejects unsupported categories and malformed or empty tables, filters empty or unreadable text, conservatively merges eligible adjacent text, derives heading levels from numbering patterns, standardizes math delimiters, represents visual regions as bounding-box image tags, and checks seal detections against layout confidence.Source: §2.1.1 · pp. 4–5
  • Valid blocks are concatenated in parser order, then samples with severe parser failures are prefiltered before spot-checking for image-content correspondence and global consistency.Pre-filtering removes samples lacking valid text-bearing content, duplicated source images, and obvious trailing repetitions; conversion alone cannot verify correspondence to the source image.Source: §2.1.1 · p. 5

2.1.2 Manual spot-checking and subset filtering

  • Manual spot-checks compare sampled Markdown with source images for text, formula, table, visual-region, and reading-order accuracy.Checks include dense tables and merged cells, bounding-box coverage of figures or charts, and natural reading order on multi-column pages.Source: §2.1.2 · p. 5
  • The spot-checking stage is conservative: subsets with occasional minor errors are retained with sample-level filters, while subsets with frequent errors are removed.Candidate datasets are organized by source, parser type, document domain, and conversion configuration for subset-level inspection.Source: §2.1.2 · p. 5

2.2 Synthetic data pipeline

The pipeline generates document images and Markdown targets from the same HTML source, yielding deterministic labels without parser-derived noise. It turns mined failures into diverse synthetic pages, then checks and packages aligned image-text pairs to extend long-tail coverage with clean ground truth.

2.2.1 Hard sample mining for HTML template generation

  • Mined failure cases are grouped into reusable synthesis families, and multimodal models convert representative samples into controllable HTML templates that scale challenging patterns into cleanly labeled generators.Templates preserve key layout structure and can use explicit element wrappers for tight localization.Source: §2.2.1 · p. 6

2.2.2 Agent-based HTML diversification

  • Agent-based diversification expands seed templates across content and structure while enforcing renderability, visual plausibility, category compliance, and valid Markdown conversion.Variations include semantic fields, text length, numerical values, formulas, terminology, table structure, section hierarchy, page organization, and visual-region placement.Source: §2.2.2 · p. 6

2.2.3 Markdown ground truth generation

  • Markdown targets are serialized directly from HTML, preserving element structure and reading order through type-specific formats and document-aware traversal of single- and multi-column layouts.Structured elements retain DOM order and template-defined associations.Source: §2.2.3 · p. 7

2.2.4 Document image rendering

  • Playwright renders the same HTML pages into document images, while DOM-derived bounding boxes remain aligned with screenshots and are transformed alongside geometric augmentations.The browser rendering preserves typography, spacing, table strokes, styles, wrapping, and page layout.Source: §2.2.4 · p. 7

2.2.5 Iterative quality control

  • Preview inspection guides revisions before scale-up, and quality control removes failed, degenerate, structurally invalid, mislocalized, or duplicated pairs before dataset admission.The pipeline supports multithreaded rendering and packages paired images and labels with augmented variants when applicable.Source: §2.2.5 · pp. 7–8

3 Training · 3.1 Training pipeline · 3.2 Supervised fine-tuning

OvisOCR2 is trained through staged policy improvement, beginning with SFT on a mixture of real and synthetic document pages before reward-based refinement, distillation, and model fusion.

3.1 Training pipeline

  • Document parsing must preserve text fidelity, formula renderability, table structure, reading order, and page completeness, motivating structural supervision beyond token-level imitation.Source: §3.1 · p. 8
  • SFT builds the initial parsing policy, RL improves it on hard pages with text, formula, and table rewards, OPD transfers 4B-teacher behavior to a 0.8B student, and fusion forms the final model.Source: §3.1 · p. 8

3.2 Supervised fine-tuning

  • Real pages contribute natural layouts, scan-quality variation, document types, and visual noise to the SFT data mixture.Source: §3.2 · p. 9
  • Synthetic samples broaden SFT coverage of tables, formula-intensive pages, long Markdown outputs, and other long-tail structures.Source: §3.2 · p. 9
  • Both Qwen3.5-0.8B and Qwen3.5-4B receive full-parameter SFT on the same parsing objective; the 0.8B trains for two epochs and the 4B for 20% of an epoch.Source: §3.2 · p. 9
  • The 0.8B SFT checkpoint initializes the deployable student, while the 4B checkpoint seeds RL and its resulting checkpoint becomes the OPD teacher; training uses a 16K maximum sequence length and dynamic image-resolution budget.Source: §3.2 · p. 9

3.3 Reinforcement learning

OvisOCR2’s reinforcement-learning stage uses verifiable, structure-aware rewards to improve complete document-page outputs, with on-policy case selection and scalable multimodal training infrastructure.

3.3 Reinforcement learning

  • GRPO [14] compares sampled responses using group-relative advantages, targeting structural errors in tables, formulas, and long pages that token-level imitation loss weakly captures.It uses no additional value model and favors more complete, stable, and structurally valid outputs.Source: §3.3 · p. 9

3.3.1 On-policy hard case construction

  • RL training emphasizes synthetic pages with accurate ground-truth structures, retains some high-quality real documents, and filters for cases where responses vary meaningfully in quality.The filter deprioritizes very easy, extremely hard, and reward-flat samples, selecting pages where the policy can sometimes outperform its average response.Source: §3.3.1 · p. 9

3.3.2 Multi-component reward design

  • The page reward averages only available text, table, and display-formula scores; truncation or unparseable structures zero out affected components as validity guards.Formula and table scores are averaged over evaluable ground-truth units after element matching and normalization; text is scored after page-level matching.Source: §3.3.2 · p. 10

3.3.3 Scalable multimodal RL training

  • Hierarchical reward scoring, reference-passed visual tensors, and common-prefix masking reduce computation, memory, and transfer bottlenecks for long multimodal RL outputs.Exact matches bypass formula rendering or table scoring, while timeouts guard costly reward operations; shared rollout prefixes are excluded from the actor loss, and the on-policy single-epoch update avoids an old-policy probability pass.Source: §3.3.3 · pp. 10–11

3.4 On-policy distillation

OvisOCR2 transfers reward-aligned parsing from a 4B RL teacher to the 0.8B student through on-policy, top-k reverse-KL distillation, avoiding another high-variance RL stage and making long-response training more efficient.

3.4.1 Teacher-guided policy transfer

  • Direct 0.8B RL showed higher late-training policy divergence and declining table quality than 4B RL, motivating the RL-enhanced 4B teacher.The instability was especially evident in table quality on dense pages and complex layouts.Source: Figure 4 · p. 11, §3.4.1 · p. 11
  • The student samples complete page outputs, and the 4B teacher scores tokens on those visited prefixes using student top-k reverse-KL distribution matching, transferring parsing preferences without length alignment.The reverse-KL objective is mode-seeking within the compared support, discouraging student probability on tokens the teacher assigns low probability.Source: §3.4.1 · pp. 11–12

3.4.2 Efficient OPD training

  • Top-k OPD reduces dominant alignment tensors from O(TV) to O(Tk); teacher scoring uses student-rollout token IDs, while selected-logit gathering and chunked projection limit memory.The teacher evaluates the retained token IDs under the same prompt-response prefix used by the student.Source: §3.4.2 · p. 12

3.5 Model fusion

The final OvisOCR2 model is produced by weighted parameter averaging of candidate variants trained with different data mixtures and configurations.

3.5 Model fusion

  • Candidate OvisOCR2 variants are trained with different data mixtures and training configurations.Source: §3.5 · p. 12
  • Weighted parameter averaging [20] fuses the candidates to produce the final OvisOCR2 model.Source: §3.5 · p. 12

4 Evaluation

OvisOCR2 is evaluated against general and specialized VLMs on OmniDocBench v1.6, PureDocBench, and an in-house benchmark. It leads overall on all three, while results on PureDocBench’s real track and in-house challenge subsets reveal remaining robustness challenges.

4 Evaluation

  • The evaluation compares general VLMs with specialized VLMs, distinguishing pipeline methods that parse regions from end-to-end models that generate page-level Markdown in one pass.The study also evaluates an in-house benchmark focused on complex document scenarios and provides qualitative comparisons in Appendix A.Source: §4 · pp. 12–13

4.1 OmniDocBench v1.6

  • 96.58 overall on OmniDocBench v1.6 makes OvisOCR2 the state-of-the-art model, surpassing leading pipeline methods and the previous best end-to-end method by 1.84 points.The compact 0.8B model has the lowest text edit distance and reading-order edit distance, highest formula CDM and TEDS-S, and tied-highest TEDS score.Source: Table 2 · p. 13, §4.1 · p. 13

4.2 PureDocBench

  • 75.06 Avg3 gives OvisOCR2 the state-of-the-art PureDocBench score, and it ranks first on the clean and digital tracks but trails strong general VLMs on the real track.PureDocBench uses source-traceable data across clean rendered pages, degraded images, and real-world recaptures, covering 1,475 pages and 4,425 images.Source: Table 3 · p. 14, §4.2 · p. 14

4.3 In-house benchmark

  • OvisOCR2 achieves the highest overall score on the in-house benchmark, with leading performance across text, formula, table, and reading-order metrics and an advantage across easy, medium, and hard tiers.The benchmark follows the OmniDocBench v1.6 protocol and contains more than 1,000 pages spanning varied document types.Source: Table 4 · p. 15, Table 5 · p. 15, §4.3 · p. 15
  • On handwriting, OvisOCR2 leads overall and on text, formula, and reading order despite GLM-OCR’s higher table TEDS; on complex tables, it leads overall and on TEDS with the lowest missing rate.Pipeline methods miss 13–17% of complex tables during layout parsing, an error that downstream recognizers cannot recover.Source: Table 6 · p. 15, Table 7 · p. 15, §4.3 · p. 16

5 Conclusion

OvisOCR2 is a compact 0.8B end-to-end document parser trained with a data engine combining filtered real annotations and source-aligned synthetic pages. It achieves new state-of-the-art results on two public benchmarks and the highest overall score in the in-house comparison.

5 Conclusion

  • OvisOCR2 establishes new state-of-the-art results on OmniDocBench v1.6 and PureDocBench and the highest overall score in the in-house comparison.The report evaluates the model on both public and in-house benchmarks.Source: §5 · p. 16
  • The 0.8B page-level parser combines filtered real-document annotations with source-aligned synthetic pages and uses supervised fine-tuning, reinforcement learning, on-policy distillation, and model fusion.Source: §5 · p. 16
  • Future work will target robustness to degraded real-world images and stronger performance on handwriting-heavy and complex-table documents.Source: §5 · p. 16

A Qualitative examples

The appendix presents qualitative comparisons on table and handwritten documents, using annotations to mark recognition, insertion, and table-structure issues as well as omitted text.

A Qualitative examples

Related papers

Loading 2607.13639v1…