PaperlayerView exact revision on arXiv

Source-linked AI summary

FireRed-OCR Technical Report

Hao Wu, Haoran Lou, Xinyue Li, Zuodong Zhong, Zhaojun Sun, Phellon Chen, Xuanhe Zhou, Kai Zuo, Yibo Chen, Xu Tang, Yao Hu, Boxiang Zhou, Jian Wu, Yongji Wu, Wenxin Yu, Yingmiao Liu, Yuhao Huang, Manjie Xu, Gang Liu, Yidong Ma, Zhichao Sun, Changhao Qiao

arXiv:2603.01840v1cs.CVeess.IV

TL;DR

General-purpose VLMs can struggle to preserve document structure and formatting. FireRed-OCR combines a geometry-and-semantics data engine with progressive training, achieving a 92.94% overall score on OmniDocBench v1.5.

  • Problem

    General-purpose VLMs can be unstable on document-intelligence tasks and often fail to adhere to rigorous formatting constraints.

    Source: §1 · p. 3

  • Method

    FireRed-OCR pairs a Geometry + Semantics data factory with three-stage training: multi-task pre-alignment, specialized SFT, and format-constrained GRPO.

    Source: §1 · pp. 3–4

  • Results

    92.94% overall score on OmniDocBench v1.5, with state-of-the-art performance surpassing DeepSeek-OCR2 and OCRVerse across text, formula, table, and reading-order metrics.

    Source: §1 · p. 4

  • Takeaways & Limitations

    FireRed-OCR offers an open-source, reproducible paradigm for transitioning general multimodal models to specialized structured document models.

    Source: §5 · p. 17

    Single-domain GRPO optimization can generalize poorly across modalities; table GRPO underperforms the text-only baseline on text edit distance.

    Source: §4.5.1 · p. 13

Abstract

from arXiv · show

We present FireRed-OCR, a systematic framework to specialize general VLMs into high-performance OCR models. Large Vision-Language Models (VLMs) have demonstrated impressive general capabilities but frequently suffer from ``structural hallucination'' when processing complex documents, limiting their utility in industrial OCR applications. In this paper, we introduce FireRed-OCR, a novel framework designed to transform general-purpose VLMs (based on Qwen3-VL) into pixel-precise structural document parsing experts. To address the scarcity of high-quality structured data, we construct a ``Geometry + Semantics'' Data Factory. Unlike traditional random sampling, our pipeline leverages geometric feature clustering and multi-dimensional tagging to synthesize and curate a highly balanced dataset, effectively handling long-tail layouts and rare document types. Furthermore, we propose a Three-Stage Progressive Training strategy that guides the model from pixel-level perception to logical structure generation. This curriculum includes: (1) Multi-task Pre-alignment to ground the model's understanding of document structure; (2) Specialized SFT for standardizing full-image Markdown output; and (3) Format-Constrained Group Relative Policy Optimization (GRPO), which utilizes reinforcement learning to enforce strict syntactic validity and structural integrity (e.g., table closure, formula syntax). Extensive evaluations on OmniDocBench v1.5 demonstrate that FireRed-OCR achieves state-of-the-art performance with an overall score of 92.94\%, significantly outperforming strong baselines such as DeepSeek-OCR 2 and OCRVerse across text, formula, table, and reading order metrics. We open-source our code and model weights to facilitate the ``General VLM to Specialized Structural Expert'' paradigm.

1. Introduction

FireRed-OCR aims to turn Qwen3-VL into a structural document-parsing expert using balanced geometry-and-semantics data and progressive training. It targets structural hallucination and reports a 92.94% overall score on OmniDocBench v1.5.

Problem

  • Structural hallucination—such as disordered table rows, invalid formula syntax, and omitted hierarchy—makes general VLM outputs unreliable for industrial document tasks.The challenge is shifting from broad semantic interpretation to precise structural generation under logical rules.Source: §1 · p. 3

Motivation

  • Traditional OCR systems [12] [13] [52] are pixel-precise but weak at logical structure, while end-to-end models such as DeepSeek-OCR 2 [48] and OCRVerse [54] struggle with fine-grained spatial grounding.The report attributes these gaps to insufficient specialized alignment data and reinforcement mechanisms for dense-text environments.Source: §1 · p. 3

Geometry + Semantics Data Factory

  • Its “Geometry + Semantics” Data Factory clusters documents by geometry and semantic tags to balance rare layouts, synthesize scarce data, and refine hard cases.The framework pairs this data engine with progressive training as its two main pillars.Source: §1 · pp. 3–4

Three-Stage Progressive Training

  • FireRed-OCR trains Qwen3-VL [4] through pre-alignment, Markdown SFT, and format-constrained GRPO [39] to improve structural parsing.Pre-alignment teaches bounding boxes and region recognition; GRPO rewards formula validity, table closure, and content accuracy.Source: §1 · pp. 3–4

Contributions

  • 92.94% overall score on OmniDocBench v1.5 gives FireRed-OCR SOTA results, surpassing DeepSeek-OCR2 and OCRVerse.The report presents this result as evidence for the general-VLM-to-specialized-structural-model approach.Source: §1 · p. 4

2. Data

The “Geometry + Semantics” Data Factory addresses imbalanced layouts and inconsistent annotation styles by combining structural and semantic curation, standardized Markdown, synthetic examples, quality control, and expert refinement.

2. Data

  • Traditional OCR datasets [40] are limited by overrepresentation of simple layouts and conflicting Markdown conventions across sources [40].These issues motivate a focus on high-quality, structurally diverse samples rather than data volume alone.Source: §2 · pp. 4–5, §2.2 · p. 5

2.1.1. Geometric Clustering

  • Visual-feature clustering separates layout complexity from semantic content, downsampling redundant common layouts while preserving sparse long-tail structures.A lightweight image encoder extracts layout-focused vectors, which are clustered using methods such as K-Means or DBSCAN.Source: §2.1 · p. 5, §2.1.1 · p. 5

2.1.2. Multi-dimensional Tagging System

  • Semantic tags for language, layout type, source, and genre complement geometry, enabling curation of a geometrically diverse and semantically balanced dataset.A classifier assigns the four tag dimensions, supporting precise control over the training distribution.Source: §2.1.2 · p. 5

2.2. Stratified Sampling and Unified Re-annotation

  • The dual-indexed sampling strategy upsamples rare layouts and complex genres while capping simple text, producing a more uniform difficulty distribution.The pipeline reprocesses the sampled corpus with PaddleOCR-VL [12] to standardize Markdown syntax across 10 million+ samples.Source: §2.2.1 · p. 5, §2.2.2 · p. 5

2.3. Synthetic Data Generation for Structural Priors

  • Procedural HTML/CSS rendering generates high-resolution images with aligned Markdown ground truth for scarce complex tables and nested formulas.Randomized content and style parameters produce examples with spanning cells, invisible borders, color-coded rows, heavy formula nesting, and rare symbols.Source: §2.3 · p. 6

2.4. Automated Quality Control and Hard-Negative Mining

  • A bi-level quality-control process filters invalid markup and noise while retaining legible, complex or initially misrecognized hard cases for refinement.Rule-based checks test structural closure, table integrity, and garbage; a lightweight LLM audit distinguishes discard-worthy noise from hard cases, which are kept.Source: §2.4 · p. 6, §2.4.1 · p. 6, §2.4.2 · p. 6

2.5. Expert-Level Refinement via Model Distillation

  • Expert-model refinement repairs structural hallucinations and reduces bias from initial OCR, distilling frontier-model reasoning into supervision for difficult documents.Hard Cases are routed to advanced teacher models such as Gemini 3 Pro series, which correct structural errors while preserving standardized Markdown.Source: §2.5 · p. 6

3. Method

FireRed-OCR turns Qwen3-VL into a structure-aware document model through progressive pre-alignment, specialized Markdown SFT, and format-constrained GRPO. The stages move from visual grounding to structured generation and reinforcement of syntactic integrity.

3. Method

  • FireRed-OCR uses Qwen3-VL in a three-stage pipeline: multi-task pre-alignment, specialized SFT for structured Markdown, and format-constrained GRPO.Pre-alignment grounds document structure; SFT adapts base capabilities to end-to-end Markdown; GRPO applies rule-based rewards for formula syntax, hierarchical closure, table integrity, and textual accuracy.Source: §3 · p. 6, Figure 4 · p. 7

3.1. Stage 1: Multi-task Pre-alignment

  • Stage 1 jointly trains detection with OCR, region OCR, and full-page Markdown to ground fine-grained perception, yielding robust base document capability but inconsistent formatting.Detection with OCR predicts bounding boxes and text together; region OCR recognizes text from a crop or coordinate prompt; Markdown tasks bridge visual layout and logical structure.Source: §3.1 · p. 7

3.2. Stage 2: Specialized Supervised Fine-Tuning (SFT)

  • Stage 2 SFT targets long-context coherence, faithful hierarchy, canonical Markdown, and robustness to cross-lingual content and complex layouts, reducing structural hallucinations.It trains valid document structure, precise nesting of headings and lists, standardized styling and math representations, and recognition across diverse scripts and spatial topologies.Source: §3.2 · pp. 7–8
  • Coarse-to-fine training—PaddleOCR-VL v1 annotations in Stage 1 and v1.5 in Stage 2—outperforms using v1.5 in both stages.The report attributes this to coarser labels providing a smoother initial learning landscape, with finer labels introduced later as refinement.Source: §3.2 · p. 8

3.3. Stage 3: Format-Constrained GRPO

  • GRPO estimates its baseline from grouped outputs rather than a separate Value model, and rule-based rewards constrain formula syntax, structural closure, table integrity, and text fidelity.Formula checks use a parser/compiler; closure penalties count unclosed nodes; tables are rewarded for consistent columns; textual accuracy uses normalized negative Levenshtein distance against SFT pseudo-ground truth or human labels.Source: §3.3 · p. 8
  • GRPO reduces format collapse and generalizes to unseen document types; applying its format constraints only to grounding also improves the main task through a reported spillover effect.The report attributes the spillover to stricter coordinate prediction reinforcing visual-textual alignment.Source: §3.3 · p. 9
  • Iterative SFT-GRPO alternates the stages for K iterations, using SFT to preserve meaningful OCR content and regularize reward hacking while GRPO progressively tunes hard structural constraints.The report describes the cycle as a curriculum from basic tag generation toward layout corner cases that standard SFT loss may miss.Source: §3.3 · p. 9

4. Experiments

The experiments evaluate FireRed-OCR against general VLMs, traditional pipeline systems, and specialized end-to-end OCR models.

Experiments

  • General VLM baselines include Qwen3-VL-235B [4] and Gemini-3.0 Pro.These are listed as state-of-the-art general Large Vision-Language Models.Source: §4 · p. 9
  • The traditional pipeline baseline is PaddleOCR-VL-1.5 [13].It represents the traditional pipeline systems category.Source: §4 · p. 9
  • Specialized end-to-end OCR baselines include DeepSeek-OCR 2 [48] and dots.ocr [25].They represent the specialized E2E OCR model category.Source: §4 · p. 9

4.1. Data Preparation

FireRed-OCR’s training corpus combines high-quality open-source data with re-annotation to span document layouts, fonts, and domains. Preprocessing adapts specialized formula, OCR, table, and handwriting data to the model’s generation targets.

4.1. Data Preparation

  • The training dataset combines high-quality open-source sources and re-annotated data to cover diverse document layouts, fonts, and domains.Source: §4.1 · p. 9
  • Formula training uses LaTeX OCR for simple formulas and latex-formulas-80M for complex equations, with normalization to align syntax across sources and benchmarks.Source: §4.1 · p. 9
  • For general document OCR, the pipeline filters BLIP-3 OCR [49] for high-quality bounding boxes and relabels Docmatix OCR ground truth to fit training objectives.Source: §4.1 · p. 10
  • Tabular training uses PubTabNet [53], converting raw HTML ground truth into a unified Parquet format for efficient loading.Source: §4.1 · p. 10
  • Handwriting training uses English lines from the IAM Handwriting Database, converting annotations into Markdown to match text-generation targets.Source: §4.1 · p. 10

4.2. Implementation Details

FireRed-OCR uses a three-stage training pipeline with stage-specific data scales and learning rates, alongside constrained generation settings for reinforcement learning.

4.2. Implementation Details

  • Pre-alignment used approximately 1.3M samples, specialized SFT used 400k document-to-Markdown pairs, and both supervised stages used batch size 256 and a 3 × 10−5 learning rate.The stages ground visual perception and standardize structural output, respectively.Source: §4.2 · p. 10
  • Format-Constrained GRPO used 50K samples and a 5 × 10−7 learning rate for reinforcement learning.Source: §4.2 · p. 10
  • RL generation used nucleus sampling with p=0.99 and k=50, a 24,576-token context limit, and a 2,048-token completion limit; all stages used 0.05 linear warmup.Source: §4.2 · p. 10

4.3. Evaluation

The evaluation uses several OCR and table benchmarks, including a newly introduced internal benchmark for complex, non-standard layouts. It reports overall scores alongside task-specific metrics.

4.3. Evaluation

  • FireRedBench stress-tests layout robustness on distorted scans, dense multi-column papers, and embedded logic diagrams, where traditional crop-based methods often fail.The evaluation also uses OmniDocBench v1.5 [34], OCRBench [29], TEDS, and PubTabNet [53].Source: §4.3 · p. 10
  • Reported metrics include Edit Distance for text and reading order (lower is better), CDM for formula match, and TEDS for table structure.Overall scores are reported alongside these specific metrics.Source: §4.3 · p. 10

4.4. Main Results

FireRed-OCR-2B posts a 92.94 Overall score on OmniDocBench v1.5 and strong results across OCRBench, document elements, and FireRedBench. The comparisons report gains over end-to-end and general VLM baselines, including models with far more parameters.

4.4.1. Overall Performance Comparison

  • On OCRBench, 93.5 beats GPT-5.2 (83.7) and Gemini-3.0 Pro (91.9), while approaching Qwen3-VL-235B (95.0).On OmniDocBench v1.5, the 2B model scores 92.94, surpassing Qwen3-VL-235B (89.15) and Qwen3.5-397B (90.80).Source: §4.4.1 · p. 11

4.4.2. Detailed Analysis on OmniDocBench v1.5

  • Against the equivalent-size Qwen3-VL-2B, FireRed-OCR-2B improves Overall score by 11.07 points and Table TEDS by 20.54 points, validating its OCR training recipe over standard VLM pre-training.Within E2E, FireRed-OCR-2B outperforms the previous best model, DeepSeek-OCR 2 (91.09), across all metrics.Source: §4.4.2 · p. 11
  • TextEdit 0.032 beats PaddleOCR-VL-1.5 (0.035), R-orderEdit is best at 0.041, and FormulaCDM is 91.71.Pipeline systems retain a slight edge on complex formulas, but the E2E model narrows the gap while balancing accuracy with system simplicity.Source: §4.4.2 · p. 11

4.4.3. Performance on FireRedBench

  • 74.62 on FireRedBench beats the GLM-OCR pipeline (74.33) and is competitive with PaddleOCR-VL-1.5 (76.47).The model internalizes layout-variability capabilities in a single 2B-parameter architecture, avoiding cascading errors in multi-stage pipelines.Source: §4.4.3 · pp. 11–12

4.4.2. Detailed Analysis on OmniDocBench v1.5

  • 92.94 on OmniDocBench v1.5 establishes FireRed-OCR-2B as the top E2E model, ahead of DeepSeek-OCR 2 (91.09) and dots.ocr (88.41).The report says it achieves this without external detection modules.Source: §4.4.2 · p. 11, §4.4.1 · p. 11, Table 2 · p. 12

4.4.3. Performance on FireRedBench

  • +13.01 Overall over DeepSeek-OCR 2 and TableTEDS 65.63 versus 55.06 demonstrate FireRed-OCR-2B's FireRedBench E2E lead.It also surpasses dots.ocr (72.93) and parses dense tables without external layout analysis tools.Source: §4.4.3 · p. 12
  • +9.04 Overall, +15.78 TableTEDS, and +16.64 TableTEDS_s over Qwen3-VL-2B quantify FireRedBench gains over the backbone.The report attributes the gains to its supervised fine-tuning strategy.Source: §4.4.3 · p. 12

4.5. Ablation Study

The ablations test GRPO across text, table, and formula domains, finding that balanced multimodal supervision performs best while single-domain or naive joint optimization has limitations.

4.5.3. Balanced Mixture Strategy

  • 89.60 Overall is achieved by Base + Mix 1:1:1, exceeding Table GRPO (Extended) at 89.28.The authors attribute the gain to balanced multimodal supervision mitigating modality competition rather than simply accumulating data.Source: Table 4 · p. 13, §4.5.3 · p. 13

4.5.1. Single-Domain Optimization

  • Base + Table GRPO gains 2.0 points in Table TEDS, while Base + Text GRPO reaches 0.04 Text Edit, but single-domain optimization can impair other modalities.For example, Table GRPO underperforms the Text-only baseline in Text Edit.Source: Table 4 · p. 13, §4.5.1 · p. 13

4.5.2. The Challenge of Modality Interference

  • In the ablation, Base + Table + Text yields an Overall score of 88.72, lower than the pure text baseline (88.91) and single-domain Table GRPO (89.18).Source: §4.5.2 · p. 13

4.6. Qualitative Evaluation

The qualitative evaluation examines mathematical expression parsing, handwriting recognition, complex document layouts, and table reconstruction, showing how the model converts pixels into structured semantic outputs.

4.6.1. Case 1: Mathematical Structure Parsing

  • The model parses limits and nested fractions into syntactically correct LaTeX or Markdown while preserving the Product Rule’s logical derivation.Figure 5 illustrates recognition of limit notation and fraction hierarchies.Source: §4.6.1 · p. 14, Figure 5 · p. 14

4.6.2. Case 2: Robust Handwritten Text Recognition

  • On noisy, lined backgrounds, the model separates foreground text from lines and recognizes cursive strokes, with near-perfect recall on essay content.Figure 6 shows unconstrained handwriting with grid-line interference and varied writing styles.Source: §4.6.2 · p. 15, Figure 6 · p. 15

4.6.3. Case 3: Complex Layout Analysis (Newspaper)

  • For newspapers with mixed text orientations and multiple columns, the model recovers logical reading order and handles vertical CJK text.It distinguishes the main article from sidebars and parses vertical text flows.Source: §4.6.3 · p. 16, Figure 7 · p. 16

4.6.4. Case 4: Structural Table Reconstruction

  • For dense financial tables, the model reconstructs the logical table tree, identifying row spans and hierarchical headers.The row spans group stocks by category.Source: §4.6.4 · p. 16

5. Conclusion

FireRed-OCR combines a “Geometry + Semantics” data engine with “Three-Stage Progressive Training” to specialize Qwen3-VL as an open-source document parsing framework addressing structural hallucination. The authors report that even 2B-parameter models achieve SOTA performance on OmniDocBench v1.5.

5. Conclusion

  • FireRed-OCR combines a “Geometry + Semantics” data engine and “Three-Stage Progressive Training” to turn general-purpose Qwen3-VL into a pixel-precise structural expert.The framework is open-source and addresses structural hallucination in VLMs.Source: §5 · p. 17
  • Even 2B-parameter models achieve SOTA performance on OmniDocBench v1.5 with high-quality data and specialized training constraints.Source: §5 · p. 17
  • The authors present this work as a reproducible paradigm for transitioning from general multimodal models to specialized structured document models.Source: §5 · p. 17

Related papers

Loading 2603.01840v1…