PaperlayerView exact revision on arXiv

Source-linked AI summary

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence

Daxiang Dong, Mingming Zheng, Dong Xu, Chunhua Luo, Bairong Zhuang, Yuxuan Li, Ruoyun He, Haoran Wang, Wenyu Zhang, Wenbo Wang, Yicheng Wang, Xue Xiong, Ayong Zheng, Xiaoying Zuo, Ziwei Ou, Jingnan Gu, Quanhao Guo, Jianmin Wu, Dawei Yin, Dou Shen

arXiv:2603.13398v1cs.CV

TL;DR

Existing OCR workflows face deployment complexity, error propagation, and visual-context loss when layout analysis, recognition, and understanding are separated. Qianfan-OCR unifies these capabilities in a 4B-parameter end-to-end vision-language model and optionally generates structured layout reasoning through Layout-as-Thought. The model achieves state-of-the-art results among end-to-end models while remaining competitive with pipeline systems.

  • Problem

    Two-stage OCR systems introduce deployment complexity, inter-stage error propagation, and irreversible visual-context loss, while end-to-end OCR lacks explicit layout analysis.

  • Method

    Qianfan-OCR integrates layout analysis, recognition, and semantic understanding in one 4B-parameter vision-language model with optional Layout-as-Thought reasoning.

  • Results

    Qianfan-OCR achieves state-of-the-art results among end-to-end models on OmniDocBench v1.5 and OlmOCR Bench, demonstrating competitiveness with pipeline systems on recognition accuracy.

  • Takeaways & Limitations

    Preserving visual context throughout processing offers advantages for document-intelligence tasks requiring joint visual and textual understanding.

  • Takeaways & Limitations

    Layout-as-Thought has been validated only on OmniDocBench v1.5 document parsing, leaving its effectiveness on KIE, document QA, and chart understanding unexplored.

Abstract

from arXiv · show

We present Qianfan-OCR, a 4B-parameter end-to-end vision-language model that unifies document parsing, layout analysis, and document understanding within a single architecture. It performs direct image-to-Markdown conversion and supports diverse prompt-driven tasks including table extraction, chart understanding, document QA, and key information extraction. To address the loss of explicit layout analysis in end-to-end OCR, we propose Layout-as-Thought, an optional thinking phase triggered by special think tokens that generates structured layout representations -- bounding boxes, element types, and reading order -- before producing final outputs, recovering layout grounding capabilities while improving accuracy on complex layouts. Qianfan-OCR ranks first among end-to-end models on OmniDocBench v1.5 (93.12) and OlmOCR Bench (79.8), achieves competitive results on OCRBench, CCOCR, DocVQA, and ChartQA against general VLMs of comparable scale, and attains the highest average score on public key information extraction benchmarks, surpassing Gemini-3.1-Pro, Seed-2.0, and Qwen3-VL-235B. The model is publicly accessible via the Baidu AI Cloud Qianfan platform.

1 Introduction

Qianfan-OCR addresses fragmented OCR workflows by unifying layout analysis, recognition, and document understanding in one 4B-parameter vision-language model. Its optional Layout-as-Thought phase restores explicit structural reasoning for complex documents while supporting prompt-driven OCR and understanding tasks.

  • Motivation: Traditional OCR systems trade cost, accuracy, and capability across specialized pipelines, two-stage models, and general vision-language models.Two-stage systems add deployment complexity, error propagation, and visual-context loss, while general VLMs underperform on structured parsing.
  • Motivation: Industrial applications commonly chain detection, OCR, and language models, increasing deployment cost and limiting end-to-end optimization.The fragmented workflow also requires orchestration of heterogeneous components.
  • Contributions: Qianfan-OCR integrates layout analysis, text recognition, and semantic understanding into a single vision-language model.The design retains spatial relationships, chart structures, and formatting throughout processing and can directly output results without mandatory layout preprocessing for simpler tasks.
  • Results: 93.12 ranks Qianfan-OCR first among end-to-end models on OmniDocBench v1.5.Figure 1 compares pipeline models on the left with end-to-end models on the right; the red dashed line marks Qianfan-OCR’s score for cross-category comparison.
  • Contributions: Layout-as-Thought optionally generates bounding boxes, element types, and reading order before final outputs through special think tokens.It provides structured layout results and targets challenging cases where spatial reasoning is useful; well-structured documents can omit the phase.
  • Contributions: Qianfan-OCR combines OCR-specialist-level accuracy with document understanding for chart understanding, document QA, and key information extraction.The model is controllable through prompts and extends beyond document parsing, handwriting recognition, and table extraction.

2 Related Work

Related OCR systems fall into pipeline, end-to-end, and general vision-language approaches with different trade-offs in modularity, efficiency, layout control, and document understanding. Qianfan-OCR is positioned as an end-to-end model that adds explicit layout analysis to the recognition-focused end-to-end paradigm.

  • Pipeline OCR Systems: Pipeline OCR systems decompose parsing into layout detection, element recognition, and rule-based assembly.They provide explicit layout outputs but suffer from inter-stage error propagation and irreversible visual-context loss.
  • End-to-End OCR Models: End-to-end OCR models directly map document images to structured outputs and have expanded format support and efficiency techniques.Existing approaches primarily focus on recognition accuracy or efficiency while lacking explicit layout-analysis output.
  • End-to-End OCR Models: Qianfan-OCR’s Layout-as-Thought addresses the layout-analysis gap in end-to-end OCR models.It is positioned as complementary to prior supervised fine-tuning and layout-aware reinforcement-learning approaches.
  • General Vision-Language Models: General vision-language models offer broad multimodal capabilities but incur higher inference costs and underperform specialized systems on structure-sensitive metrics.Qianfan-OCR targets OCR-specialist-level accuracy at comparable inference cost while supporting explicit layout analysis.

3 Model Architecture and Training Data

Qianfan-OCR combines a vision encoder, cross-modal adapter, and language-model backbone with large-scale synthesized data spanning OCR, layouts, tables, charts, KIE, and multilingual documents. Its training uses progressive stages, augmentation, and fine-grained layout annotations to support diverse document-intelligence tasks.

  • Architecture: Qianfan-OCR uses a vision encoder, lightweight projection adapter, and language-model backbone for multimodal encoding, alignment, generation, and reasoning.The architecture is adopted from Qianfan-VL and illustrated in Figure 3(b).
  • Architecture: The vision encoder dynamically tiles images into 448×448 patches and supports variable-resolution inputs up to 4K.With up to 16 tiles, it can represent a document using as many as 4,096 visual tokens for dense text and complex layouts.
  • Architecture: The language backbone is Qwen3-4B with 4.0B total parameters and a native 32K context window.The selected scale balances complex document understanding and layout reasoning with single-GPU serving practicality.
  • Training Data: Training data covers document parsing, key information extraction, complex tables, chart understanding, formula recognition, and multilingual OCR.Six synthesis pipelines and multilingual rendering procedures generate diverse document scenarios and writing systems.
  • Training Data: The adopted layout taxonomy contains 25 categories across text elements, headers and footers, figures and tables, and formulas.PaddleOCR-VL’s finer-grained labels and higher detection accuracy motivated its selection as the annotation engine.
  • Training Data: Layout-as-Thought training produces bounding boxes, labels, and content summaries inside layout tags before final output when users append think tokens.The intermediate layout phase focuses attention on relevant regions for complex layouts, multi-column text, and interleaved figures.
  • Training Data: Rotation augmentation significantly improves performance on KIE and table recognition tasks involving non-standard document orientations.The augmentation includes rotations of 90°, 180°, 270°, and ±15° alongside text, background, and imaging noise.

4 Training Recipe

Qianfan-OCR uses a four-stage progressive training recipe that combines broad OCR pretraining, domain-specific enhancement, and instruction tuning. Ablations show that foundational general-purpose training and mixed OCR/general data are important for achieving the best performance.

  • Training Pipeline: The four-stage recipe progresses from cross-modal alignment to foundational OCR training, domain-specific enhancement, and instruction tuning with reasoning.Stage 1 uses adapter-only training; later stages use OCR-focused data mixtures and instruction samples for document intelligence tasks.
  • Training Pipeline: OCR-centric data composition emphasizes document parsing, tables, charts, information extraction, and multilingual OCR throughout training.The training framework retains general data while increasing coverage of enterprise-critical OCR domains.
  • Ablation Findings: Stage 2 foundational pretraining is essential: the best Stage 3-only configuration reaches 80.07%, versus 83.47% for Stage 2 followed by Stage 4.Domain-specific training alone cannot substitute for the capability foundation established by large-scale general-purpose pretraining.
  • Ablation Findings: A 1:1 OCR-plus-general mixture reaches 80.07%, exceeding pure OCR data at 75.97%, while the Stage 2-inclusive comparison is 84.39% versus 84.09%.The results support mixing general-purpose data during domain specialization.
  • Ablation Findings: The complete four-stage configuration achieves 84.39%, improving 13.02 percentage points over the Stage 1+4 baseline and surpassing Qwen2.5-VL-7B at 79.30%.These ablations on Qianfan-VL-8B directly motivated the recipe used for Qianfan-OCR and suggest transfer across model scales within the architectural family.

5 Evaluation Framework

The evaluation framework spans specialized OCR, general OCR, document understanding, and key information extraction. It also compares Qianfan-OCR with pipeline systems using controlled baselines to examine architectural trade-offs.

  • Benchmark Coverage: Specialized OCR benchmarks assess diverse document parsing, end-to-end document OCR, multilingual recognition, and large-scale document processing.The framework includes Omni-Doc-Bench v1.5, OLMOCRBench, CCOCR, and BigDocs.
  • Benchmark Coverage: General OCR benchmarks cover scene text, document text, handwriting, formulas, and multilingual text across varied real-world conditions.OCRBench and OCRBench v2 test robustness beyond specialized document parsing scenarios.
  • Benchmark Coverage: Document understanding evaluation includes scene-text and document QA, academic document reasoning, chart understanding, and related comprehension tasks.The benchmark suite includes TextVQA, DocVQA, CharXiv, ChartQA, and ChartQAPro.
  • Benchmark Coverage: Key information extraction is evaluated on five public benchmarks covering receipts, invoices, forms, identity documents, business licenses, and financial documents.The suite includes English and Chinese extraction settings and Nanonets KIE F1 scoring.
  • Comparison Design: Controlled comparisons combine specialized OCR models with language models of comparable parameter count to isolate architectural effects from model scale.This setup evaluates trade-offs between end-to-end and OCR-then-LLM pipeline approaches.

6 Experimental Results

Qianfan-OCR performs strongly across specialized OCR, general OCR, document understanding, KIE, and throughput evaluations. Its results show that end-to-end processing preserves advantages on layout-sensitive tasks, while thinking mode is most useful for complex layouts.

  • OCR-Specific Benchmarks: 79.8 is Qianfan-OCR’s highest overall OlmOCR Bench score among end-to-end models, close to PaddleOCR-VL’s 80.0.It is especially strong on Base at 99.6, Headers & footers at 92.2, and Old scans at 42.0, the best end-to-end result.
  • OCR-Specific Benchmarks: 93.12 is Qianfan-OCR’s highest OmniDocBench v1.5 score among end-to-end models, exceeding DeepSeek-OCR-v2 at 91.09 and Gemini-3 Pro at 90.33.It also surpasses MinerU2.5 at 90.67 and MonkeyOCR-pro-3B at 88.85, while approaching PaddleOCR-VL 1.5 at 94.50.
  • OCR-Specific Benchmarks: 92.64 versus 93.12 is the aggregate score for thinking versus default mode, with thinking improving TableTEDs but lowering several other metrics.The comparison motivates analyzing performance by layout complexity rather than relying only on the aggregate.
  • OCR-Specific Benchmarks: Thinking provides a stable advantage on high-entropy layouts, but no-think achieves the higher cumulative total after lower-entropy samples are included.Users are advised to enable thinking for heterogeneous pages and disable it for homogeneous documents to reduce latency.
  • General OCR: 880 is Qianfan-OCR’s OCRBench score, exceeding Qwen3-VL-4B at 873, while it leads both CCOCR metrics at 76.7 multilan and 79.3 overall.On OCRBenchv2, Qianfan-OCR scores 56.0 English and 60.77 Chinese; Qwen3-VL-4B leads English at 60.68.
  • Document Understanding: Qianfan-OCR achieves the best scores on six of eight document-understanding benchmarks, including CharXiv_DQ at 94.0, ChartQA at 88.1, and ChartBench at 85.9.Qwen3-VL-4B performs better on DocVQA at 94.9 versus 92.8 and TextVQA at 81.8 versus 80.0.
  • Document Understanding: Two-stage systems score 0.0 on both CharXiv tasks, while Qianfan-OCR reaches 94.0 on CharXiv_DQ and 85.2 on CharXiv_RQ.The cited analysis links this gap to spatial and layout context discarded during text extraction.
  • Key Information Extraction: 87.9 is Qianfan-OCR’s mean across five public KIE benchmarks, exceeding Qwen3-4B-VL at 83.5 and Qwen3-VL-235B-A22B at 84.2.It leads OCRBenchv2 KIE Chinese at 82.3 and Nanonets KIE F1 at 86.5.

7 Limitations and Future Work

Qianfan-OCR’s unified end-to-end approach remains an early exploration with unresolved questions about Layout-as-Thought, performance ceilings, deployment efficiency, and broader task coverage.

  • Layout-as-Thought: Layout-as-Thought has been validated only for document parsing on OmniDocBench v1.5, leaving its effectiveness on key information extraction, document QA, and chart understanding unexplored.The current implementation also produces layout bounding boxes, labels, and brief descriptions in a relatively rigid supervised format.
  • Performance Ceiling: The performance ceiling of purely end-to-end OCR architectures remains an open question against heavily optimized pipeline systems.Future work should examine architectural innovations, training strategies, and data scaling laws.
  • Deployment Efficiency: Although W8A8 quantization provides competitive GPU throughput, the 4B-parameter footprint limits deployment on edge devices and CPU-only servers.Knowledge distillation and pruning are proposed to develop 1B–2B-parameter variants.
  • Deployment Efficiency: Qianfan-OCR also remains limited for video OCR, 3D text on curved surfaces, and highly stylized artistic handwriting.These areas are identified as directions for extending the unified architecture.

8 Conclusion

Qianfan-OCR unifies text recognition, layout analysis, and semantic understanding in one vision-language architecture. It achieves state-of-the-art results among end-to-end models while showing that preserving visual context benefits tasks requiring joint visual and textual understanding.

  • Conclusion: Qianfan-OCR is a 4B-parameter end-to-end model that unifies text recognition, layout analysis, and semantic understanding within one vision-language architecture.The model is presented as a unified architecture for document intelligence.
  • Conclusion: State-of-the-art results among end-to-end models on OmniDocBench v1.5 and OlmOCR Bench show competitiveness with pipeline systems on recognition accuracy.The conclusion attributes this comparison to end-to-end model performance on the cited benchmarks.
  • Conclusion: Layout-as-Thought integrates layout reasoning as optional chain-of-thought, allowing dynamic structural analysis for complex documents.This mechanism is introduced as a way to invoke layout analysis when needed.
  • Conclusion: Two-stage OCR+LLM pipelines degrade substantially on spatial and visual reasoning tasks, including zero accuracy on cited chart interpretation benchmarks.The conclusion connects this weakness to the loss of layout information in intermediate representations.
  • Conclusion: Preserving visual context throughout processing offers significant advantages for document intelligence tasks requiring joint visual and textual understanding.This is the paper’s supported conclusion about unified visual-textual processing.
  • Conclusion: The model is publicly accessible through the Baidu AI Cloud Qianfan platform.The conclusion provides a public access route via the Qianfan-VL repository.
Loading 2603.13398v1…