PaperlayerView exact revision on arXiv

Source-linked AI summary

OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation

Junyuan Zhang, Qintong Zhang, Bin Wang, Linke Ouyang, Zichen Wen, Ying Li, Ka-Ho Chow, Conghui He, Wentao Zhang

arXiv:2412.02592v4cs.CV

TL;DR

OCR errors and inconsistent structured-data representations can introduce noise into RAG knowledge bases, but OCR’s cascading effects across RAG components have received limited attention. OHRBench evaluates OCR solutions and controlled Semantic and Formatting Noise, finding that no current OCR solution is fully capable across scenarios and that RAG performance varies with noise type and degree.

  • Problem

    OCR-derived noise in external knowledge bases and its cascading impact on RAG components and full systems have received limited benchmark attention.

    Source: §1 · p. 1

  • Method

    OHRBench pairs diverse document images and multimodal Q&As with ground-truth structured data and controlled perturbations for two OCR noise types.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    No current OCR solution is fully capable across all scenarios, and increasing Semantic Noise is associated with declining retrieval and generation performance.

    Source: §4.3.1 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    OHRBench provides a basis for developing OCR solutions tailored to RAG and RAG systems more resistant to OCR noise.

    Source: §5 · p. 8

Abstract

from arXiv · show

Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge to reduce hallucinations and incorporate up-to-date information without retraining. As an essential part of RAG, external knowledge bases are commonly built by extracting structured data from unstructured PDF documents using Optical Character Recognition (OCR). However, given the imperfect prediction of OCR and the inherent non-uniform representation of structured data, knowledge bases inevitably contain various OCR noises. In this paper, we introduce OHRBench, the first benchmark for understanding the cascading impact of OCR on RAG systems. OHRBench includes 8,561 carefully selected unstructured document images from seven real-world RAG application domains, along with 8,498 Q&A pairs derived from multimodal elements in documents, challenging existing OCR solutions used for RAG. To better understand OCR's impact on RAG systems, we identify two primary types of OCR noise: Semantic Noise and Formatting Noise and apply perturbation to generate a set of structured data with varying degrees of each OCR noise. Using OHRBench, we first conduct a comprehensive evaluation of current OCR solutions and reveal that none is competent for constructing high-quality knowledge bases for RAG systems. We then systematically evaluate the impact of these two noise types and demonstrate the trend relationship between the degree of OCR noise and RAG performance. Our OHRBench, including PDF documents, Q&As, and the ground truth structured data are released at: https://github.com/opendatalab/OHR-Bench

1. Introduction

RAG depends on OCR-parsed structured data to build knowledge bases, but OCR errors and varying data representations can introduce noise. OHRBench evaluates how that noise affects RAG, using diverse documents, Q&As, and controlled perturbations.

1. Introduction

  • OHRBench evaluates OCR’s impact across RAG components and full systems, addressing a gap left by benchmarks that lack fine-grained assessment or account for OCR noise.Prior work also lacks documents with diverse OCR challenges, including scanned historical, multilingual, and handwritten documents.Source: §1 · pp. 1–2
  • The benchmark includes 8,561 document images from seven real-world RAG domains and Q&As covering realistic tasks and evidence from document-parsing components.The Q&As span understanding, reasoning, and multi-page questions; evidence sources include core OCR components.Source: §1 · p. 2
  • OHRBench distinguishes Semantic Noise from OCR prediction errors and Formatting Noise from varied representations, then creates perturbed structured data at varying noise levels.This setup enables analysis of the quantitative relationship between OCR noise and RAG performance.Source: §1 · p. 2
  • The best OCR solutions still show a performance gap of at least 14% against ground-truth structured data, while Semantic Noise significantly affects retrieval and generation.Formatting Noise affects specific retrievers and LLMs differently.Source: §1 · p. 2
  • The paper evaluates pipeline-based OCR systems, end-to-end OCR models, and vision-language models for OCR.These are the three solution families included in its comprehensive benchmark.Source: §1 · p. 2

2. Related Works

Prior RAG-noise work examines noise introduced during retrieval, while OCR-derived noise remains less explored. Existing OCR parsing and RAG evaluation benchmarks also leave gaps in coverage of real-world documents and OCR’s effects on knowledge-base construction.

2.1. Retrieval-Augmented Generation

  • Prior noise studies examine chunk-level retrieval noise, while GARAG considers synthetic typographical errors in plain text; OCR-derived noise across real-world applications remains unexplored.This paper analyzes noise introduced at the OCR stage and its impact on RAG systems.Source: §2.1 · p. 3

2.2. Document parsing with OCR

  • OCR parsing approaches include pipeline-based systems that decompose subtasks, end-to-end models that output recognition results, and vision-language models applied to OCR.The paper evaluates these paradigms across real-world document domains for RAG suitability.Source: §2.2 · p. 3

2.3. Benchmark and evaluation of Retrieval-Augmented Generation

  • RAG evaluation frameworks target specific components or capabilities, while document-understanding RAG work considers limited OCR solutions and omits challenging documents and noise-type analysis.OHRBench is introduced to investigate OCR noise’s impact on RAG comprehensively.Source: §2.3 · p. 3

3. OHRBench

OHRBench combines diverse PDFs, human-verified structured data, Q&As grounded in document elements, and structured data perturbed with varying OCR noise. Its construction covers data selection, Q&A quality control, and controlled Semantic and Formatting Noise.

3. OHRBench

  • OHRBench comprises seven-domain PDFs, Q&As derived from multimodal document elements, ground-truth structured data, and perturbed data with varying OCR noise.The evaluation uses current OCR outputs for benchmarking and perturbed data to assess OCR impact on RAG components and the overall system.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Data collection

  • The collection contains 1,261 PDFs and 8,561 page images across seven RAG scenarios, including complex layouts, dense text, handwriting, scanned and historical documents, and Chinese and English content.Experts revised initial Mathpix structural extractions to preserve the PDFs’ structure and content.Source: §3.1 · pp. 3–4

3.2. Q&A pairs generation

  • Q&As cover text, tables, formulas, charts, reading order, and multi-page questions, with understanding and reasoning tasks generated from ground-truth structured data.Multi-page questions combine single-page Q&As and pages sharing an entity name.Source: §3.2 · p. 4
  • Quality control filters for realistic RAG compatibility, task faithfulness, and correctness, yielding 8,498 high-quality Q&As from 15,317 candidates.LLM-based checks and manual review verify evidence contexts and answers.Source: §3.2 · pp. 4–5

3.3. Data perturbation with OCR noise

  • Semantic Noise represents OCR prediction errors; three controlled noise levels are generated using image distortions and OCR systems to capture varied real-world errors.Eight perturbation types were refined through cross-validation against naturally distorted documents.Source: §3.3 · p. 5
  • Formatting Noise captures representation inconsistencies that do not alter semantics but complicate information integration for retrievers and LLMs.Heuristic perturbations vary in proportion and include format conversions; performance is also assessed across Markdown, LaTeX, and HTML.Source: §3.3 · p. 5

4. Experiments

The experiments benchmark OCR systems across RAG stages, then test how OCR noise and table formats affect retrieval, generation, and end-to-end performance. Semantic Noise degrades RAG broadly, while Formatting Noise has more variable, often multimodal-specific effects.

4.1. Experimental settings

  • Experiments assess retrieval, generation, and overall system performance using top-2 retrieved chunks for generation in the overall evaluation.The setup uses BGE-M3 and BM25 retrievers, LCS for retrieval evidence inclusion, and F1-score for generated answers.Source: §4.1 · p. 5

4.2. Benchmarking current OCR solutions

  • Even the best OCR solutions lose 14% (5 F1-score) in overall-stage evaluation versus ground truth, with larger losses in retrieval and generation.Qwen2.5-VL-72B performs best overall among the evaluated OCR solutions, but current systems remain challenged across diverse scenarios.Source: §4.2 · p. 6

4.3. In-depth analysis of OCR noise’s impact on RAG

  • The analysis varies Semantic Noise and Formatting Noise across perturbation levels to examine their relationship with RAG performance.The benchmark uses three levels—mild, moderate, and severe—for each noise type.Source: §4.3 · p. 6, Figure 3 · p. 7

4.3.1. Fine-grained impact on retrieval and generation

  • Semantic Noise causes nearly a 50% performance decline for most retrievers and LLMs as rnoise rises above 0.6, while both BM25 and BGE-M3 decline consistently.Figure 3 compares Semantic Noise and Formatting Noise across text, multimodal elements, reading order, and combined evidence sources.Source: §4.3.1 · p. 7, Figure 3 · p. 7

4.3.2. Impact on end-to-end evaluation

  • Formatting Noise mainly harms multimodal questions, with maximum losses of 12.7% for BGE-M3 retrieval and 9.1% for Llama3.1-8B generation.Semantic Noise also degrades every RAG stage, especially multimodal questions; larger LLMs show greater resilience to Formatting Noise.Source: §4.3.1 · p. 7, §4.3.2 · p. 8

4.3.3. Impact of table format

  • HTML tables retrieve worse than Markdown and LaTeX; for end-to-end table questions, Qwen2-72B with BGE-M3 and Markdown performs best.Markdown generation performs worse because it lacks support for merged cells.Source: §4.3.3 · p. 8, Figure 4 · p. 8

4.3.4. Error analysis

  • At similar OCR-error proportions for Semantic Noise and Formatting Noise (66% vs. 61%), answers perform better with Formatting Noise.For Qwen2-7B, about half of correct responses occur despite incorrect OCR; Qwen-72B has higher OCR-error-but-correct sample percentages by 9.4% for Formatting Noise and 3.5% for Semantic Noise.Source: §4.3.4 · p. 8, Figure 5 · p. 8

5. Conclusion

OHRBench evaluates OCR’s impact on RAG using diverse PDFs and multimodal-derived Q&A pairs. The experiments find that current OCR systems are not fully capable across scenarios, Semantic Noise affects all tested retrievers and LLMs, and advanced models are more resilient to Formatting Noise.

5. Conclusion

  • OHRBench covers diverse PDFs from seven RAG application scenarios and Q&A pairs derived from multimodal document elements.The benchmark is intended to support development of OCR solutions tailored for RAG and OCR-noise-resistant RAG systems.Source: §5 · p. 8

Supplementary Material

The supplementary material reports prompt choices for question generation, RAG responses, and VLM-based OCR.

I. Instruction Prompts

  • The authors use detailed VLM OCR instructions by default because they consistently outperform simple prompts across evaluations.The prompt was selected through comparison of simple and detailed instructions.Source: §I · p. 12

II. Benchmark Construction Details

The benchmark draws documents from several existing datasets and public sources, with manual selection and review to assemble its domain coverage.

II.1. Document details

  • The corpus contains 1,261 PDFs spanning 8,561 pages, with 3,596 pages designated for Q&A generation and the remainder used for the knowledge base.Sources include DUDE, OmniDocBench, FinanceBench, CUAD, GNHK, and public resources.Source: II.1. Document details · p. 12
  • Document selection includes 450 DUDE PDFs, 289 OmniDocBench PDFs, 10 FinanceBench PDFs with large complex tables and charts, and 65 CUAD PDFs.DUDE candidates were screened for feasibility of structured-data parsing and classified into seven domains.Source: II.1. Document details · p. 12
  • The GNHK selection contains 172 handwritten-document PDFs after removing samples with low knowledge density.Primary authors manually review each document for availability for academic use.Source: II.1. Document details · p. 12

II.2. Ground truth structured data annotation

The authors build ground-truth structured data and a diverse document-based Q&A benchmark, verify the generated questions, and define controlled formatting perturbations. Their process yields 8,498 Q&As and specifies how formatting noise is introduced across text, formulas, and tables.

II.2. Ground truth structured data annotation

  • Ground-truth structured data was pre-annotated with Mathpix, rendered for review, and manually corrected by the primary authors.Tables and formulas use LaTeX; chart data is represented as LaTeX tables, and parsable image text is retained as plain text.Source: II.2. Ground truth structured data… · p. 12

II.3. Document with challenging attributes

  • The benchmark selects documents with nine OCR-challenging attributes, including structured elements, complex layouts, handwriting, distortion, scanning, dense text, and multilingual content.Structured data, dense text, and multilingual pages are identified from annotations; distortion, scanning, and handwriting are checked manually.Source: II.3. Document with challenging… · pp. 12–13

II.4. Q&A generation

  • Q&As cover text, tables, formulas, charts, and reading order, with GPT-4o prompted on ground-truth evidence and MinerU used to identify reading order and paragraph boxes.Multi-page questions are generated using shared entities across single-page Q&As or paragraphs on different pages.Source: II.4. Q&A generation · p. 13
  • Over 90% of generated Q&As were correctly aligned with their intended evidence source in verification.The authors manually corrected approximately 20% of 1,500 sampled candidate matches where MinerU predictions were inaccurate.Source: II.4. Q&A generation · p. 13
  • Some generated multi-page Q&As were answerable from one paragraph or stitched together separate questions, so the authors added a comprehensive filtering process.The text describes this as a limitation of the generation strategies.Source: II.4. Q&A generation · p. 13

II.5. Q&A verification.

  • Verification checks realistic context dependence, task faithfulness, evidence provenance, and answer correctness, including repeated answer sampling against oracle evidence.For multi-page and reading-order questions, items are excluded if GPT-4o answers correctly without context or with either single context alone.Source: II.5. Q&A verification · p. 13
  • 8,498 high-quality Q&As remained from 15,317 candidates after two filtering refinements, sampled checks, and manual review.Questions were screened for realistic RAG compatibility, task faithfulness, and correctness.Source: II.5. Q&A verification · p. 13

III.1. Rules for Formatting Noise introduction

III.2. Rules for Semantic Noise introduction

The authors construct perturbed document images using distortion strategies intended to approximate real-world distributions. They report that the resulting Semantic Noise distributions align with those of real distorted PDFs in most cases.

III.2. Rules for Semantic Noise introduction

  • Eight distortion strategies produce three datasets: one weak distortion per page, one strong distortion per page, or two randomly selected distortions per page.The strategies include background addition, salt-and-pepper noise, dirty rollers, rotation, binarization, warping, shadows, and blur; OCR parsing yields nine perturbed datasets.Source: III.2. Rules for Semantic Noise… · p. 14
  • Semantic Noise distribution is shown against edit distance, with mild, moderate, and severe levels defined using born-digital PDFs.Real-world OCR noise is also summarized as semantic-error ratios and redundant formatting commands per page.Source: Figure S2 · p. 15, Table S3 · p. 15
  • In most cases, perturbed PDFs’ Semantic Noise distributions align with those of real-world distorted PDFs.The authors report this as validation of the perturbation method’s realism.Source: III.2. Rules for Semantic Noise… · p. 15
  • The authors compare the frequency of Semantic Noise and Formatting Noise in real-world OCR by matching corresponding text, formula, and table blocks of about 130 tokens each.Source: III.2. Rules for Semantic Noise… · p. 15

IV.1. Experimental details

The experiments compare OCR-based RAG with VisRAG and examine performance after rewriting questions to resemble human queries. VisRAG is competitive on multimodal-element questions but weaker on text and reading-order questions.

IV.1. Experimental details

  • MinerU 0.9.210, Marker 0.2.1711, and Nougat 0.1.0-base (350M) were used; for all LLMs and VLMs, temperature was set to 0 with do_sample=False by default for reproducibility.Source: IV.1. Experimental details · p. 15

IV.2. Sim2Real GAP

  • Rewritten-question performance was 27.2/23.2 (GT), 20.7/18.0 (MinerU), 12.8/12.9 (GOT), and 23.1/20.0 (Qwen2.5-VL).The authors report degraded performance after rewriting, while concluding that the relative conclusions about OCR solutions still hold.Source: IV.2. Sim2Real GAP · p. 15

IV.3. Multimodal RAG

  • VisRAG is competitive on table- and chart-related Q&As but underperforms on text and reading-order tasks, including high-resolution newspapers.The comparison uses Qwen2-VL-7B as the generator for both approaches.Source: IV.3. Multimodal RAG · p. 15

IV.4. Effectiveness of robust generator

The supplementary evaluations report that stronger generators improve robustness to formatting noise, while OCR quality can remain a bottleneck. Azure OCR performs differently depending on the generator, and the paper also presents domain-level results and illustrative cases.

IV.4. Effectiveness of robust generator

  • Ext2Gen-8B-R2 consistently improves performance, while the gap between Azure OCR and ground truth increases by 1.6 compared with Qwen2-7B.The authors say stronger models exhibit greater robustness to formatting noise, but the expanding gap indicates remaining room for OCR improvement.Source: IV.4. Effectiveness of robust generator · pp. 15–16, Table S5 · p. 16

IV.5. Commercial OCR

  • Azure yields the best performance with Qwen2-7B and Ext2Gen-8B-R2, with a gap of up to 4.0 versus ground truth, but falls below MinerU with Llama3.1-8B.A manual check suggests Azure’s custom formatting tags may affect Llama3.1-8B’s generation.Source: IV.5. Commercial OCR · p. 16, Table S5 · p. 16

V. Case Study

  • The case-study figures show OCR outputs and retrieval- and generation-stage responses for examples spanning text, formulas, tables, and reading order.Examples include academic documents, textbooks, financial reports, newspapers, and manuals.Source: §V · p. 16, Figure S3 · p. 22, Figure S4 · p. 23

IV.6. Details in different domains

Related papers

Loading 2412.02592v4…