PaperlayerView exact revision on arXiv

Source-linked AI summary

FireRed-OCR Technical Report

Hao Wu, Haoran Lou, Xinyue Li, Zuodong Zhong, Zhaojun Sun, Phellon Chen, Xuanhe Zhou, Kai Zuo, Yibo Chen, Xu Tang, Yao Hu, Boxiang Zhou, Jian Wu, Yongji Wu, Wenxin Yu, Yingmiao Liu, Yuhao Huang, Manjie Xu, Gang Liu, Yidong Ma, Zhichao Sun, Changhao Qiao

arXiv:2603.01840v1cs.CVeess.IV

TL;DR

General VLMs can interpret document content yet produce structurally invalid OCR outputs, motivating a specialized approach. FireRed-OCR combines geometry- and semantics-driven data curation with three-stage progressive training, achieving state-of-the-art document parsing performance and supporting the transition from general VLMs to specialized structural models.

  • Problem

    General VLMs frequently produce structural hallucinations on complex documents, including disordered tables, invalid formulas, and omitted hierarchy.

  • Method

    FireRed-OCR combines a Geometry + Semantics Data Factory with Multi-task Pre-alignment, Specialized SFT, and Format-Constrained GRPO.

  • Results

    92.94% overall score on OmniDocBench v1.5 makes FireRed-OCR state of the art against DeepSeek-OCR2 and OCRVerse.

  • Takeaways & Limitations

    FireRed-OCR shows that a general Qwen3-VL can be specialized into a pixel-precise structural document parsing expert.

Abstract

from arXiv · show

We present FireRed-OCR, a systematic framework to specialize general VLMs into high-performance OCR models. Large Vision-Language Models (VLMs) have demonstrated impressive general capabilities but frequently suffer from ``structural hallucination'' when processing complex documents, limiting their utility in industrial OCR applications. In this paper, we introduce FireRed-OCR, a novel framework designed to transform general-purpose VLMs (based on Qwen3-VL) into pixel-precise structural document parsing experts. To address the scarcity of high-quality structured data, we construct a ``Geometry + Semantics'' Data Factory. Unlike traditional random sampling, our pipeline leverages geometric feature clustering and multi-dimensional tagging to synthesize and curate a highly balanced dataset, effectively handling long-tail layouts and rare document types. Furthermore, we propose a Three-Stage Progressive Training strategy that guides the model from pixel-level perception to logical structure generation. This curriculum includes: (1) Multi-task Pre-alignment to ground the model's understanding of document structure; (2) Specialized SFT for standardizing full-image Markdown output; and (3) Format-Constrained Group Relative Policy Optimization (GRPO), which utilizes reinforcement learning to enforce strict syntactic validity and structural integrity (e.g., table closure, formula syntax). Extensive evaluations on OmniDocBench v1.5 demonstrate that FireRed-OCR achieves state-of-the-art performance with an overall score of 92.94\%, significantly outperforming strong baselines such as DeepSeek-OCR 2 and OCRVerse across text, formula, table, and reading order metrics. We open-source our code and model weights to facilitate the ``General VLM to Specialized Structural Expert'' paradigm.

1. Introduction

FireRed-OCR addresses structural hallucination in general VLMs by combining balanced structural data curation with progressive training for precise document parsing. The framework targets strict formatting and logical structure across complex OCR outputs.

  • Motivation: Structural hallucination causes disordered Markdown tables, invalid mathematical syntax, and omitted hierarchical logic in complex document parsing.These failures make otherwise semantically capable VLM outputs unusable for downstream industrial applications.
  • Framework: FireRed-OCR transforms Qwen3-VL into an industrial-grade structural expert through a high-precision data engine and progressive training strategy.
  • Data Factory: The Geometry + Semantics Data Factory clusters documents by geometric features and semantic tags to balance rare layouts and improve supervision quality.It also synthesizes scarce examples and uses expert refinement for difficult cases.
  • Training: Three-stage training progresses from physical perception to standardized Markdown generation and format-constrained reinforcement learning.The stages are Multi-task Pre-alignment, Specialized SFT, and Format-Constrained GRPO with rewards for formula validity, table closure, and content accuracy.
  • Results: 92.94% overall score on OmniDocBench v1.5 establishes FireRed-OCR as state of the art against DeepSeek-OCR2 and OCRVerse.

2. Data

The data pipeline targets layout imbalance and annotation inconsistency through geometry-aware sampling, unified Markdown re-annotation, synthesis, quality control, and expert refinement. Its design preserves difficult structural cases while standardizing supervision for training.

  • Data Challenges: Traditional OCR datasets overrepresent simple layouts and contain inconsistent Markdown and formula annotation styles.
  • Geometry Indexing: Geometry-based clustering identifies redundant common layouts and preserves sparse long-tail cases such as nested tables, irregular forms, and artistic typography.
  • Dual Indexing: Dual indexing combines geometric features with tags for language, layout type, document source, and genre to control training-distribution diversity.The resulting index supports geometrically diverse and semantically balanced dataset curation.
  • Sampling: Stratified sampling upsamples rare layouts and complex genres while capping simple text documents, producing a more uniform difficulty distribution.
  • Re-annotation: The corpus is reprocessed with PaddleOCR-VL to standardize annotations into consistent Markdown syntax across more than 10 million samples.
  • Synthetic Data: Render-based synthesis generates scarce complex layouts with aligned Markdown ground truths, including spanning tables and heavily nested formulas.
  • Quality Control: Quality control combines rule-based Markdown checks with lightweight LLM audits that discard noise but retain legible hard cases for refinement.Hard cases are routed to a specialized repository rather than discarded.
  • Expert Refinement: Expert models repair structural hallucinations and reduce bias from the initial OCR pass, distilling frontier-model reasoning into difficult training examples.

3. Method

FireRed-OCR specializes Qwen3-VL through progressive training that moves from visual grounding to structured Markdown generation and format-constrained reinforcement learning. The method combines supervised learning with rule-based structural rewards to reduce formatting failures and improve document parsing.

  • Training framework: FireRed-OCR uses Qwen3-VL as a backbone and applies Multi-task Pre-alignment, Specialized SFT, and Format-Constrained GRPO.The three stages progress from document perception to structured output and reinforcement-based format enforcement.
  • Stage 1: Multi-task Pre-alignment: Multi-task Pre-alignment jointly trains detection, region OCR, and full-page Markdown tasks to connect precise visual perception with document structure.Detection and OCR predict bounding boxes with text, region OCR improves local resolution sensitivity, and Markdown tasks bridge visual layout and logical structure.
  • Stage 2: Specialized SFT: Specialized SFT trains curated Markdown data to improve structural consistency, hierarchical expression, and standardized syntax.The stage targets valid document structure, nesting of headers and lists, and canonical Markdown formatting.
  • Stage 3: Format-Constrained GRPO: Format-Constrained GRPO uses rule-based rewards for formula validity, hierarchical closure, table integrity, and textual accuracy.The composite reward guides structurally valid generation, while group-relative optimization avoids the separate value model required by PPO.
  • Training effects: GRPO constraints applied only to grounding improve the main task, suggesting that coordinate-level structural discipline reinforces visual-textual alignment.The reported spillover benefits the primary generation task even without direct constraints on that task.
  • Iterative refinement: Iterative SFT-GRPO alternation improves the trade-off between semantic fidelity and structural rigidity while reducing format collapse.The authors report that repeated refinement between supervised tuning and GRPO yields superior performance.

4. Experiments

FireRed-OCR’s training data combines open-source datasets with re-annotation to cover varied document layouts and specialized OCR capabilities. The preparation includes sources for formulas, general documents, tables, and handwriting.

  • Dataset construction: The training dataset combines high-quality open-source data with re-annotated open-source datasets to diversify layouts, fonts, and domains.This combination is intended to support a robust and versatile model.
  • Mathematical expressions: Formula training uses LaTeX OCR and latex-formulas-80M datasets, followed by normalization of syntax, spacing, symbols, and environments.Normalization addresses discrepancies between source annotations and evaluation-benchmark LaTeX formats.
  • General document understanding and OCR: General document parsing incorporates BLIP-3 OCR and Docmatix, with filtering and re-labeling to align annotations with the training objectives.BLIP-3 OCR contributes high-quality bounding-box annotations, while Docmatix provides re-labeled VQA data.
  • Tabular data: PubTabNet supplies tabular data converted from raw HTML into a unified Parquet format for efficient loading.The conversion supports training on structural table information.
  • Handwritten text: IAM Handwriting Database lines are converted into Markdown annotations to support handwriting recognition with consistent text-generation targets.The processed data consists of English handwriting lines.

4.2. Implementation Details

Implementation uses progressively specialized data and fixed supervised-training settings across the first two stages, followed by constrained GRPO with long-context configuration.

  • Training data and stages: 1.3M samples support Multi-task Pre-alignment, while 400k high-quality document-to-Markdown pairs support Specialized SFT.The supervised stages use different data scales and optimization strategies.
  • Optimization settings: Both supervised stages use global batch size 256 and learning rate 3 × 10^-5.These settings are reported for Multi-task Pre-alignment and Specialized SFT.
  • GRPO and context configuration: The training configuration sets p=0.99, k=50, a maximum context length of 24,576 tokens, and a completion limit of 2,048 tokens.A linear warmup ratio of 0.05 is applied across all training stages.

4.3. Evaluation

Evaluation spans general OCR, formula, table, reading-order, and difficult-layout benchmarks. Results are reported with overall scores and task-specific metrics for text, formulas, and table structure.

  • Benchmarks: FireRed-OCR is evaluated on OmniDocBench v1.5, OCRBench, TEDS, PubTabNet, and the internal FireRedBench.FireRedBench targets distorted scans, dense multi-column papers, and embedded logic diagrams.
  • Metrics: Reported metrics include Overall scores, Edit Distance for text and reading order, CDM for formula matching, and TEDS for table structure.Edit Distance is lower-is-better, while CDM and TEDS measure formula and table agreement respectively.

4.4. Main Results

FireRed-OCR-2B delivers strong document-parsing performance across end-to-end, general-VLM, and pipeline comparisons. It combines high benchmark accuracy with robustness and compact model size.

  • 92.94 Overall on OmniDoc v1.5 surpasses DeepSeek-OCR 2 (91.09) and dots.ocr (88.41).
  • 93.5 on OCRBench (Text) outperforms the nearest end-to-end competitor by over 1.4 points.
  • 93.5 on OCRBench (Text) exceeds GPT-5.2 (83.7) and Gemini-3.0 Pro (91.9), while approaching Qwen3-VL-235B (95.0).
  • 90.31 TableTEDS surpasses Qwen3-VL-235B (86.21) and GPT-4o (67.07) despite FireRed-OCR-2B using less than 1% of their parameter count.
  • FireRed-OCR-2B improves Overall by +11.07 points and Table TEDS by +20.54 points over the equivalent-size Qwen3-VL-2B.
  • TextEdit of 0.032 and R-orderEdit of 0.041 are the best reported values, while FormulaCDM remains slightly stronger for pipeline systems.
  • 74.62 Overall on FireRedBench exceeds GLM-OCR (74.33) and remains competitive with PaddleOCR-VL-1.5 (76.47).

4.5. Ablation Study

The ablation study shows that GRPO improves targeted domains, but naive multimodal combination can cause interference. Balanced sampling produces the strongest overall result.

  • Single-Domain Optimization: Table GRPO gains +2.0 Table TEDS points, while Text GRPO reduces Text Edit distance to 0.04.
  • Single-Domain Optimization: Single-domain GRPO often fails to generalize across modalities, with Table GRPO underperforming the Text-only baseline on Text Edit.
  • Balanced Mixture Strategy: 88.72 Overall from combined Table and Text GRPO is below the pure-text baseline (88.91) and single-domain Table GRPO (89.18).
  • Balanced Mixture Strategy: 89.60 Overall from balanced 1:1:1 mixing exceeds Table GRPO Extended (89.28).
  • Balanced Mixture Strategy: Balanced multimodal supervision mitigates modality competition more effectively than simply accumulating datasets.

4.6. Qualitative Evaluation

Qualitative cases show FireRed-OCR bridging raw pixels and structured representations across formulas, handwriting, layouts, and tables. The examples emphasize preservation of spatial and hierarchical structure.

  • The qualitative evaluation covers mathematical expressions, handwriting, document layouts, and structural table reconstruction.
  • Mathematical Expression Parsing: The model converts limits and nested fractions into syntactically correct LaTeX or Markdown while preserving the Product Rule’s logical derivation.
  • Handwritten Text Recognition: The model separates foreground writing from grid lines and recognizes cursive strokes on noisy lined-paper backgrounds.
  • Complex Document Layout Analysis: The model recovers reading order in mixed-orientation, multi-column newspapers, including vertical Chinese text.
  • Structural Table Reconstruction: The model reconstructs financial-table hierarchy by identifying spanning cells, grouped rows, and hierarchical headers.

5. Conclusion

FireRed-OCR is presented as an open-source framework that combines a Geometry + Semantics data engine with three-stage progressive training to address structural hallucination in VLM document parsing. On OmniDocBench v1.5, the authors report state-of-the-art performance and claim that even 2B-parameter models can achieve it.

  • FireRed-OCR combines a Geometry + Semantics data engine with a Three-Stage Progressive Training strategy for structural document parsing.The framework transforms Qwen3-VL into a pixel-precise structural expert.
  • FireRed-OCR addresses structural hallucination in VLM-based document parsing through specialized data and training constraints.
  • 2B-parameter models can achieve SOTA performance on OmniDocBench v1.5 with high-quality data and specialized training constraints.
  • The authors position FireRed-OCR as a reproducible paradigm for transitioning from general multimodal models to specialized structured document models.The framework and model are presented as open source.

6. Authors •

The paper lists its core contributors and additional contributors.

  • Core contributors include Hao Wu, Haoran Lou, Xinyue Li, Zuodong Zhong, and Zhaojun Sun.
  • Additional core contributors include Phellon Chen, Xuanhe Zhou, Kai Zuo, Yibo Chen, and Xu Tang.
  • The contributor list also includes Yao Hu, Boxiang Zhou, Jian Wu, Yongji Wu, and Wenxin Yu.
Loading 2603.01840v1…