PaperlayerView exact revision on arXiv

Source-linked AI summary

OvisOCR3 Technical Report

Yu Xia, Yinglun Li, Yuhui Chen, Jinlong Huang, Yuyang Wanyan, Shengze Shi, Yifei Song, Yue Duan, Shubo Lin, Jianshan Zhao, En Lin, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo

arXiv:2610.33066v1cs.CVcs.LG

TL;DR

OvisOCR3 addresses the challenge of achieving specialist-level performance across OCR tasks that differ in output length and supervision needs. It unifies five task families in one compact 0.8B model and combines staged training, teacher distillation, and mixed-task reinforcement learning. Across public and in-house benchmarks, it reports leading results across a broad range of OCR tasks, including an overall score of 96.74 on OmniDocBench v1.6.

  • Problem

    OCR applications need more than transcription, but compact models must handle tasks with different response lengths, output requirements, and forms of supervision.

    Source: §1 · p. 2

  • Method

    OvisOCR3 unifies five OCR-related task families in a shared 0.8B model trained with continued pretraining, domain-balanced mid-training, multi-teacher distillation, and mixed-task reinforcement learning.

    Source: §1 · p. 2, §5 · p. 24

  • Results

    OvisOCR3 achieves leading results across a broad range of OCR tasks, including an overall score of 96.74 on OmniDocBench v1.6.

    Source: §1 · p. 2, §4.1.1 · p. 16, §5 · p. 25

  • Takeaways & Limitations

    The reported benchmarks show that a compact 0.8B model can combine broad OCR task coverage with strong task-specific performance.

    Source: §1 · p. 2, §5 · p. 25

Abstract

from arXiv · show

We introduce OvisOCR3, a 0.8B model for a broad range of OCR-related tasks. OvisOCR3 unifies document parsing, general OCR, key information extraction (KIE), document figure captioning, and cross-page table reconstruction in a single model. We train the model on real and synthetic data through continued pretraining and use domain-balanced mid-training to balance supervision across tasks with different data volumes and response lengths. Multi-teacher on-policy distillation transfers expertise from five domain-specific 4B teachers into the shared model, followed by mixed-task reinforcement learning with domain-specific rewards. We compare OvisOCR3 with general-purpose vision-language models and specialized OCR models on public and in-house benchmarks covering all five task families. In these comparisons, it achieves an overall score of 96.74 on OmniDocBench v1.6 and the highest overall scores on PureDocBench, Real5-OmniDocBench, and MDPBench. On OCRBench v2, it ranks first, scoring 65.9 and 61.3 on the English and Chinese splits, respectively. It also outperforms the baselines on OCRBench and all four KIE benchmarks. On DocFigCapBench, it achieves the highest localization F1 and a caption score close to that of Qwen3.8-Max. For cross-page tables, it leads in Relation F1 on CrossTabBench and truncated-table TEDS on MPDocBench-Parse. These results demonstrate that a compact 0.8B model can achieve strong performance across diverse OCR-related tasks. OvisOCR3 is available at https://huggingface.co/ATH-MaaS/OvisOCR3.

1 Introduction

OvisOCR3 targets specialist-level performance across diverse OCR tasks in one compact 0.8B model. It combines five task families and achieves strong results across their benchmarks.

1 Introduction

  • The challenge is balancing tasks with different response lengths and correctness criteria, since balanced sample counts do not ensure balanced supervision under token-averaged loss.Full-page reconstruction can contain thousands of tokens, while field extraction may require only a few.Source: §1 · p. 2
  • OvisOCR3 unifies document parsing, general OCR, KIE, document figure captioning, and cross-page table reconstruction in one shared-parameter 0.8B model.It performs tasks according to supplied instructions and requires no separate task-specific models at inference.Source: §1 · p. 2
  • Training combines continued pretraining on real and synthetic data, domain-balanced mid-training, distillation from five domain-specific 4B teachers, and mixed-task reinforcement learning.Mid-training uses token-level and sample-level loss normalization in two branches whose weights are fused afterward.Source: §1 · p. 2
  • OvisOCR3 scores 96.74 on OmniDocBench v1.6 and leads or achieves top results across benchmarks for document parsing, OCR, KIE, figure captioning, and cross-page tables.It leads the comparisons on OCRBench and all four KIE benchmarks, and achieves the highest localization F1 on DocFigCapBench and Relation F1 on CrossTabBench.Source: §1 · p. 2

2 Model Capabilities

OvisOCR3 organizes document understanding, general OCR, and visual tasks into five task families with distinct prediction targets and output representations. Task-specific instructions specify what information to recover while all five families share model parameters.

2 Model Capabilities

  • The five task families span individual text regions through full pages and multi-page tables, with task-specific instructions defining each prediction target and representation.Table 1 summarizes task coverage and output representations.Source: §2 · pp. 2–3, Table 1 · p. 3

2.1 Document Parsing

  • Document parsing adds structured parsing and layout detection to Markdown parsing, with structured records pairing regions’ boxes and categories with content rendered as Markdown, LaTeX, or HTML.Layout detection predicts region locations, categories, and reading order without transcribing content; Markdown parsing instead produces a continuous page document.Source: §2.1 · p. 3, §2.1.1 · p. 3, §2.1.2 · p. 4

2.2 General OCR

  • General OCR includes plain transcription, word-level localization and transcription, reference-driven localization, and reading text from a supplied region.Word spotting returns text and tight boxes; reference grounding returns boxes for user-specified references, while region OCR maps a supplied location to its text.Source: §2.2.1 · p. 5, §2.2.2 · p. 5, §2.2.3 · p. 6
  • Formula recognition outputs mathematical expressions in LaTeX, while chart parsing returns chart metadata and values as structured JSON.Chart values are serialized as strings, and dual-axis charts may provide two y-axis titles in visual order.Source: §2.2.5 · p. 7, §2.2.6 · pp. 7–8
  • Other general OCR interfaces translate specified image regions, classify images using user-provided labels, answer visual questions, and localize answer text for position-aware VQA.Position-aware VQA returns an answer and a normalized bounding box around the answer text.Source: §2.2.7 · p. 8, §2.2.8 · p. 8, §2.2.9 · p. 8

2.3 Key Information Extraction

OvisOCR3 supports three KIE modes that vary in how fields and output structure are specified, and it also captions document figures and reconstructs tables across pages. Cross-page reconstruction uses a shared model in pairwise steps that can be applied recursively.

2.3 Key Information Extraction

  • KIE extracts image information into structured records through open, specified-field, or schema-based modes, with missing values represented as empty strings.Open and specified-field modes use arrays for repeated values, while schema-based extraction follows the supplied JSON structure.Source: §2.3 · p. 9

2.3.1 Open Extraction

  • Open extraction identifies key-value pairs without predefined fields or schema, leaving the model to determine the output keys and JSON organization.Repeated values use arrays.Source: §2.3.1 · p. 9

2.3.2 Specified-Field Extraction

  • Specified-field extraction returns exactly the user-provided field names as JSON keys and uses empty strings for missing values.Repeated values use arrays.Source: §2.3.2 · p. 9

2.3.3 Schema-Based Extraction

  • Schema-based extraction uses a user-provided JSON template to prescribe target fields and their organization, including nested objects and arrays.A receipt template can group item names, quantities, and prices in array elements and payment totals in a separate object.Source: §2.3.3 · p. 10

2.4 Document Figure Captioning

  • Document figure captioning returns page regions in reading order, jointly localizing and describing visual content such as photographs, charts, and flowcharts.Captions are in English and limited to 100 words; flowcharts may also include a Mermaid representation.Source: §2.4 · p. 10

2.5 Cross-Page Table Reconstruction

  • Cross-page table reconstruction detects continuations and recovers complete table content and structure from fragments across consecutive pages using the same model for parsing and reconstruction.The procedure first parses pages independently, then reconstructs from adjacent page images and their parsed representations.Source: §2.5 · p. 10, §2.5.1 · p. 11

2.5.1 Pairwise Table Reconstruction

  • Pairwise reconstruction parses each page into Markdown with HTML tables, then uses adjacent page images and parsed representations to detect continuations and produce reconstructed HTML tables.The output identifies source pages; when no continuation is detected, the model returns an explicit no-continuation result.Source: Figure 2 · p. 11, §2.5.1 · p. 11

2.5.2 Recursive Reconstruction

  • For tables spanning multiple pages, reconstruction proceeds from the sequence’s end toward its beginning, writing each merged table into earlier-page Markdown before re-rendering and pairing it with the predecessor.Each step retains the pairwise input format while accumulated content can span multiple pages.Source: §2.5.2 · p. 11, Figure 3 · p. 12

3 Training

OvisOCR3 uses staged training to build and jointly refine five OCR-related capabilities in one shared model. Domain balancing, teacher distillation, and task-specific reinforcement learning address the differing supervision and success criteria across those tasks.

3 Training

  • Training proceeds from a common five-domain foundation through domain balancing, multi-teacher distillation, and mixed-task reinforcement learning.The stages integrate domain expertise before jointly refining the shared model.Source: §3 · p. 12

3.1 Stage 1: Continued Pretraining

  • Initialized from Qwen3.5-0.8B, continued pretraining updates the vision encoder, connector, and language model on about 72 million real and synthetic examples.The mixture spans diverse genres, languages, layouts, and acquisition conditions; document parsing and general OCR receive approximately 57% and 29% of its sampling budget.Source: §3.1 · pp. 12–13

3.2 Stage 2: Domain-Balanced Mid-Training

  • Stage 2 rebalances the five domains and trains token-level and sample-level loss branches, then merges their weights for post-training.Its approximately 8.8 million examples allocate about 16–23% of the sample budget to each domain, but general OCR and KIE still contribute only 5.2% and 2.5% of supervised response tokens.Source: §3.2 · p. 13
  • Sample-level loss outperforms token-level loss on short-response tasks, including general OCR and key information extraction.Under sequence packing, sample-level loss follows original example boundaries.Source: §3.2 · p. 13

3.3 Stage 3: Multi-Teacher On-Policy Distillation

  • Five domain-specific 4B teachers supervise one shared 0.8B student on-policy, with each example guided by its corresponding domain teacher.Top-K distillation outperforms policy-gradient optimization overall, particularly on document parsing and general OCR; the authors suggest relative preferences across candidate tokens may help the capacity-constrained student absorb coordinated text and structure generation.Source: §3.3 · p. 14

3.4 Stage 4: Mixed-Task Reinforcement Learning

  • MixRL jointly optimizes the shared model with domain-specific rewards and GRPO, preserving distinct criteria for task success.Rewards distinguish, for example, field-value correctness in KIE and cross-page relation correctness before table reconstruction quality is evaluated.Source: §3.4 · pp. 14–15

4 Evaluation

OvisOCR3 is evaluated across document parsing, general OCR, KIE, figure captioning, and cross-page table reconstruction, with leading results across these task families. The benchmarks also assess robustness to degraded images and multilingual documents.

4 Evaluation

  • The evaluation compares OvisOCR3 with general-purpose vision-language models and specialized OCR models across public and in-house benchmarks covering the five task families.Document-parsing benchmarks assess general page parsing, image-degradation robustness, and multilingual parsing; OCRBench and OCRBench v2 cover recognition, localization, extraction, and visual reasoning.Source: §4.1 · p. 15, §4.2 · p. 19

4.1.1 General Document Parsing

  • 96.74 Overall on OmniDocBench v1.6 exceeds OvisOCR2 and PaddleOCR-VL-1.6, closely matches Xiaomi-OCR-0, and leads on TEDS and TEDS-S.It ranks among the top two on all five individual metrics.Source: Table 3 · p. 16, §4.1.1 · p. 16

4.1.2 Robustness to Image Degradation

  • 82.31 Overall on PureDocBench leads all three tracks; on Real5-OmniDocBench, 93.63 Overall leads, with OvisOCR3 first in four of five conditions.The Clean-to-Real drop is 7.84 points for OvisOCR3 versus 15.04 for OvisOCR2; Real5’s maximum score difference across conditions is 0.90 points.Source: Table 4 · p. 17, §4.1.2 · p. 17, Table 5 · p. 18

4.1.3 Multilingual Document Parsing

  • 88.8 Overall on MDPBench exceeds Gemini 3 Pro and Qwen3.8-Max by 2.4 points, with OvisOCR3 leading both acquisition types and both script groups.Compared with OvisOCR2, gains on non-Latin documents narrow the Latin/non-Latin gap from 23.7 to 1.5 points.Source: Table 6 · p. 18, §4.1.3 · p. 19

4.2 General OCR

  • 88.8 on OCRBench and 65.9 and 61.3 on OCRBench v2’s English and Chinese splits rank first; task-level results span recognition, extraction, question answering, and text spotting.On OCRBench v2 English, OvisOCR3 exceeds Qwen3.5-4B by 11.8 points while using one-fifth as many parameters.Source: Table 7 · p. 19, §4.2 · pp. 19–20

4.3 Key Information Extraction

  • First on all four KIE benchmarks, OvisOCR3 exceeds the strongest baselines on OCRBench and OCRBench v2 by 9.50 and 8.35 points, respectively.It scores 88.69 on schema-based CC-OCR and 77.73 on the in-house KIE Bench.Source: §4.3 · pp. 20–21, Table 10 · p. 21

4.4 Document Figure Captioning

  • Highest Localization F1 on DocFigCapBench exceeds Qwen3.8-Max by 4.30 points; Caption Score is 6.93 versus 7.30.OvisOCR3 also leads specialized OCR models by 5.65 points in Localization F1 and 2.19 points in Caption Score.Source: Table 11 · p. 23, §4.4 · p. 23

4.5 Cross-Page Table Reconstruction

  • 97.07 Relation F1 on CrossTabBench exceeds Qwen3.8-Max by 2.60 points; OvisOCR3 also scores 89.66 Reconstruction TEDS and leads MPDocBench-Parse with 92.04 truncated-table TEDS.CrossTabBench includes consecutive-page pairs requiring reconstruction as well as pairs requiring no cross-page merge.Source: §4.5 · pp. 23–24, Table 12 · p. 24

5 Conclusion

OvisOCR3 is a compact 0.8B model for five OCR-related task families, trained with a four-stage recipe. Across public and in-house benchmarks, it achieves leading results across a broad range of OCR tasks.

5 Conclusion

  • OvisOCR3 unifies document parsing, general OCR, key information extraction, document figure captioning, and cross-page table reconstruction in a compact 0.8B model.It is evaluated against general-purpose vision-language models and specialized OCR models.Source: §5 · pp. 24–25
  • The four-stage training recipe comprises continued pretraining, domain-balanced mid-training, multi-teacher on-policy distillation, and mixed-task reinforcement learning.Source: §5 · pp. 24–25
  • Across public and in-house benchmarks, OvisOCR3 achieves leading results across a broad range of OCR tasks, outperforming both general-purpose vision-language models and specialized OCR models on multiple benchmarks.Source: §5 · p. 25

A Qualitative Examples

The appendix presents qualitative examples of the five task families described in Section 2.

A Qualitative Examples

A.1 Document Parsing

Figures A.1–A.6 illustrate three document-parsing modes, with examples spanning photographed documents and a Japanese page. Each example includes the input, prompt, and complete rendered model output, including referenced image crops.

A.1 Document Parsing

  • Figures A.4–A.6 extend the parsing examples to photographed pages with deformation, uneven illumination, low light, and perspective distortion, as well as a Japanese page.The examples show the input, prompt, and complete rendered output, including image crops referenced in responses.Source: §A.1 · p. 30, Figure A.4 · p. 33, Figure A.5 · p. 34
  • Figure A.1 shows structured document parsing with predicted region boundaries, semantic categories, and rendered page content.Source: Figure A.1 · p. 30
  • Figure A.2 depicts layout detection, using box colors for predicted region categories and numeric labels for reading order.Source: Figure A.2 · p. 31
  • Figure A.3 shows Markdown parsing with text in reading order and mathematical expressions rendered from LaTeX.Source: Figure A.3 · p. 32

A.2 General OCR

Figures A.7–A.16 illustrate ten General OCR tasks through input images, prompts, and model responses. The examples include recognition, localization, translation, classification, and visual question answering.

A.2 General OCR

A.3 Key Information Extraction

The appendix illustrates three KIE modes that differ in how the extraction fields and output structure are specified. Examples show receipt extraction and form extraction, including how missing values are handled.

A.3 Key Information Extraction

  • Open extraction derives its keys from the document, while specified-field extraction uses a supplied field list and schema-based extraction follows a supplied JSON template.The examples show receipt-based open extraction, form-based specified-field extraction, and receipt-based schema extraction.Source: §A.3 · p. 46
  • In the CC-OCR receipt example, open extraction returns item amounts and payment totals using keys drawn from the receipt, without a supplied field list or JSON template.The source image is blurred.Source: Figure A.17 · p. 46
  • In the OCRBench v2 form example, the response contains exactly five requested keys, and missing telephone and fax values remain empty.The organizer’s contact details are not substituted for the applicant’s missing values.Source: Figure A.18 · p. 47
  • In the CC-OCR receipt example, schema-based extraction groups each item’s name, quantity, and price in one record and stores payment totals in a separate nested object.The source image is blurred.Source: Figure A.19 · p. 48

A.4 Document Figure Captioning

The appendix presents comparisons of OvisOCR3 with three baselines for figure localization and captioning, alongside examples of captions and Mermaid representations for diagrams. The displayed comparisons mark annotated figures and model predictions, and scores cover the entire page.

A.4 Document Figure Captioning

  • The comparisons cover figure localization and captioning against three baselines, with reported scores covering the entire page.The examples include a page with multiple visual regions, a photographed magazine page, a page with two charts, and a newspaper page with individual images and an image group.Source: §A.4 · pp. 49, 52, Figure A.20 · p. 49, Figure A.21 · p. 50
  • For flowcharts and block diagrams, the examples show OvisOCR3 generating both a caption and a Mermaid representation.The displayed full-page responses omit the other 14 records for the algorithm flowchart, 18 for the device block diagram, and 45 for the system architecture diagram.Source: §A.4 · p. 49, Figure A.24 · p. 53, Figure A.25 · p. 53
  • Red dashed boxes mark annotated figures, blue boxes mark model predictions, and each P label links a predicted region to its caption below.Caption text is from the model’s original output, with empty captions explicitly marked; OvisOCR3 and Qwen3.8-Max use the same prompt requesting English captions.Source: §A.4 · p. 52

A.5 Cross-Page Table Reconstruction

The appendix compares four-model reconstructions of two cross-page tables from MPDocBench-Parse, focusing on merged-cell structure and numerical alignment. The examples show OvisOCR3 preserving a shared label across a page boundary and attaching a currency unit to its preceding label.

A.5 Cross-Page Table Reconstruction

  • The four-model comparisons use two MPDocBench-Parse tables and highlight merged-cell structure and numerical alignment across page boundaries.The source pages and enlarged fragments accompany the reconstructions.Source: §A.5 · p. 55, Figure A.27 · p. 55, Figure A.29 · p. 57
  • OvisOCR3 preserves the shared FDI label cell across the page boundary and numerical-value placement, unlike the compared reconstructions.MinerU2.5-Pro splits the shared label cell, Qwen3.8-Max misplaces and duplicates values, and Logics-Parsing-V3 omits the second-page fragment.Source: Figure A.28 · p. 56
  • OvisOCR3 attaches the second-page currency unit to the preceding cash-flow label, avoiding the extra row produced by the three compared models.Those reconstructions leave four numerical cells empty.Source: Figure A.30 · p. 58

Related papers

Loading 2610.33066v1…