PaperlayerView exact revision on arXiv

Source-linked AI summary

AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation

Zhengren Wang, Dongsheng Ma, Huaping Zhong, Jiayu Li, Wentao Zhang, Bin Wang, Conghui He

arXiv:2602.24134v1cs.CVcs.CL

TL;DR

Visual RAG’s page-level retrieval leaves generators with irrelevant context while visual-token limits can sacrifice detail. AgenticOCR selectively parses query-relevant regions, and the paper reports improved visual-RAG efficiency and accuracy, including performance above the human-expert baseline on MMLongBench-Doc.

  • Problem

    Visual RAG’s page-level retrieval supplies irrelevant page content and compresses detailed pages into limited visual-token budgets, while instruction-following on-demand parsing remains unmet.

    Source: §1 · p. 1

  • Method

    AgenticOCR uses visual tools to locate and parse query-relevant regions on demand, transforming OCR from static preprocessing into a query-driven process.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    AgenticOCR improves visual-RAG signal-to-token ratio and accuracy; its 8B Evidence+OCR configuration reaches 66.4 overall accuracy on MMLongBench-Doc, above the 65.8 human-expert baseline.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    AgenticOCR is proposed as a potential third visual-document-RAG building block alongside Embedding and Reranking modules.

    Source: §1 · p. 2

    The current version lags top agentic frameworks on table and unanswerable subsets, with incomplete table extractions and lower retrieval precision identified as issues.

    Source: §4.2 · p. 10

Abstract

from arXiv · show

The expansion of retrieval-augmented generation (RAG) into multimodal domains has intensified the challenge for processing complex visual documents, such as financial reports. While page-level chunking and retrieval is a natural starting point, it creates a critical bottleneck: delivering entire pages to the generator introduces excessive extraneous context. This not only overloads the generator's attention mechanism but also dilutes the most salient evidence. Moreover, compressing these information-rich pages into a limited visual token budget further increases the risk of hallucinations. To address this, we introduce AgenticOCR, a dynamic parsing paradigm that transforms optical character recognition (OCR) from a static, full-text process into a query-driven, on-demand extraction system. By autonomously analyzing document layout in a "thinking with images" manner, AgenticOCR identifies and selectively recognizes regions of interest. This approach performs on-demand decompression of visual tokens precisely where needed, effectively decoupling retrieval granularity from rigid page-level chunking. AgenticOCR has the potential to serve as the "third building block" of the visual document RAG stack, operating alongside and enhancing standard Embedding and Reranking modules. Experimental results demonstrate that AgenticOCR improves both the efficiency and accuracy of visual RAG systems, achieving expert-level performance in long document understanding. Code and models are available at https://github.com/OpenDataLab/AgenticOCR.

1 Introduction

Full-document OCR is mature, but visual RAG still needs query-driven parsing because page-level retrieval supplies irrelevant context and can lose fine detail. AgenticOCR selectively extracts query-relevant regions and is presented as a potential third building block for visual document RAG.

1 Introduction

  • Full-document OCR has reached high accuracy, yet instruction-following, on-demand parsing remains an unmet need in visual RAG.The passage describes OCR as mature and reports 90–95% accuracy for several systems on OmniDocBench.Source: §1 · p. 1
  • Page-level retrieval passes irrelevant page content to generators, diluting attention and forcing high-resolution pages into limited visual-token budgets.The resulting loss of fine detail can increase hallucination risk, particularly for rotated tables, small annotations, formulas, and complex layouts.Source: §1 · pp. 1–2
  • AgenticOCR reasons over document layout to locate and selectively parse query-relevant regions, using zoom and rotation for on-demand visual decompression.The passage also describes it as an efficient pre-processor that can reduce visual-token consumption for large generators.Source: §1 · p. 2, Figure 1 · p. 2
  • AgenticOCR is proposed as a potential third visual-document-RAG building block alongside Embedding and Reranking, with possible use in key-information extraction, evidence citation, and interactive assistants.The paper reports that its experiments improve the signal-to-token ratio and yield accuracy gains in visual RAG pipelines.Source: §1 · p. 2

2 Related Work

Prior work advances visual retrieval, OCR, and agentic multimodal reasoning, but the paper identifies a gap in specialized, query-driven document evidence extraction. AgenticOCR applies agentic interaction specifically to document parsing.

Visual Retrieval-Augmented Generation

  • Visual RAG systems use page snapshots to preserve layout, but page-level retrieval introduces irrelevant content and can sacrifice detail under visual-token limits [6] [36] [25] [30].RegionRAG [14] and hierarchical or agentic systems such as DocLens [46], DocDancer [41], VRAG-RL [31], and EVisRAG [24] refine retrieval or evidence localization.Source: §2 · p. 3

OCR and Document Parsing

  • Modern OCR spans traditional pipelines, end-to-end VLMs, coarse-to-fine parsing, and visual-token compression, yet remains static despite 90–95% OmniDocBench accuracy [40] [28] [2] [33] [18].The passage contrasts these full-document systems with RAG settings where only sparse regions are query-relevant.Source: §2 · p. 3

Agentic Multimodal Models

  • Agentic multimodal models use visual tools for active reasoning, but prior frameworks largely target general visual reasoning rather than document-intelligence parsing [23] [45] [42] [43] [8].The passage distinguishes using tools as reasoning aids from using them as extraction mechanisms.Source: §2 · pp. 3–4
  • AgenticOCR specializes the agentic paradigm for document parsing, addressing the stated lack of a specialized or lightweight model for on-demand evidence extraction.This makes parsing itself query-driven rather than a static preprocessing step.Source: §2 · pp. 3–4

3 AgenticOCR: Parsing Only What You Need

AgenticOCR combines visual interaction, staged training, and a plug-and-play RAG integration to parse query-relevant document regions on demand. Its design makes the model choose where to look, how to orient a crop, and how to recognize it.

3 AgenticOCR: Parsing Only What You Need

  • AgenticOCR combines a unified visual interaction tool, SFT+RL training, and a plug-and-play integration protocol for visual RAG.The components support on-demand evidence extraction, tool-use policy learning, and deployment between retrieval and generation.Source: §3 · p. 4

3.1 The image_zoom_and_ocr_tool Primitive

  • The image_zoom_and_ocr_tool takes a page image, normalized crop box, rotation angle, and semantic type, returning a cropped image and structured recognition results.Its region mode runs layout analysis before fine-grained recognition; text, table, and equation modes recognize directly, while image mode returns a crop without OCR.Source: §3.1 · p. 4

3.2 Cold-Start via Supervised Fine-Tuning

  • SFT cold-start trajectories are distilled from Gemini-3-Pro-Preview and filtered using dual recall thresholds to produce 5k high-quality positive examples.The method retains trajectories with Recallmin at least 0.8 and ranks survivors by Recallmin + RecallEM; the SFT set also includes 1.5k negatives at a 77:23 positive-to-negative ratio.Source: §3.2 · pp. 5–6

3.3 Alignment via Reinforcement Learning

  • GRPO aligns the SFT policy for coverage, localization accuracy, and behavioral constraints, with uncertainty filtering concentrating training on ambiguous cases.The RL curriculum includes rotated or complex layouts, and penalties discourage spurious, redundant, or oversized regions.Source: §3.3 · pp. 6–7

3.4 Integration with Visual RAG Pipelines

  • In the RAG pipeline, AgenticOCR assesses each retrieved page, extracts structured evidence when needed, and passes interleaved text-image snippets to the generator.This preserves visual context while reducing irrelevant-region token use and requires no changes to existing generators beyond minor prompt engineering.Source: §3.4 · p. 7

4 Experiments

Across two visual-document benchmarks, AgenticOCR reaches strong overall accuracy, with gains in selected modalities and retrieval-pipeline efficiency. Its single-page tests show capable evidence localization, while table extraction, unanswerable questions, and retrieval precision remain challenges.

4.2 Main Results

  • 66.4 overall accuracy on MMLongBench-Doc lets AgenticOCR-8B Evidence+OCR exceed the 65.8 human-expert baseline; its FinRAGBench-V accuracy is 78.6.The MMLongBench-Doc score corresponds to 65.9 on the original 1,082 samples after dataset-size adjustment; the authors also compare it with adjusted DocLens performance of 66.1.Source: §4.2 · p. 8, Table 1 · p. 9
  • 67.4 on TXT, 68.8 on LAY, and 63.6 on FIG are reported for AgenticOCR-8B Evidence+OCR, surpassing the best DocLens configurations on those subsets.The passage attributes the pattern to localization, layout preservation, and on-demand zoom, and describes Evidence+OCR as interleaving page snapshots, cropped patches, and extracted text.Source: Table 1 · p. 9, §4.2 · p. 10
  • On MMLongBench-Doc, AgenticOCR lags top frameworks on table and unanswerable subsets; its unanswerable accuracy is 66.8 versus DocLens at up to 74.0.The authors cite incomplete table extractions and lower retrieval precision, with 8.6 pages retrieved on average versus DocLens’s 1.5, as contributing issues.Source: Table 1 · p. 9, §4.2 · p. 10

4.3.1 Performance Analysis of AgenticOCR Models

  • AgenticOCR-4B achieves Pageacc 97.3 and Recallmin 83.7 on the FinRAGBench-V subset, with RecallEM 50.1, and overall performance comparable to Gemini-3-Pro-Preview.The evaluation covers page-level relevance and element localization; the passage notes a hard-negative gap and less stable exact box matching on MVToolbench.Source: Table 2 · p. 9, §4.3.1 · p. 11

4.3.2 Performance Analysis of Visual Retrieval Pipeline

  • On FinRAGBench-V with Gemini-2.5-Pro, Evidence+OCR uses 9,090 input tokens versus 9,289 for Page+OCR, while accuracy rises from 76.5 to 78.6.The authors attribute the comparison to selective cropping and on-demand text extraction filtering irrelevant visual noise.Source: §4.3.2 · p. 12, Table 3 · p. 12
  • On MMLongBench-Doc with Qwen3-VL-32B-Thinking, Evidence+OCR reduces total input tokens from 14,517 to 13,238 relative to Page+OCR.Gemini-2.5-Pro instead incurs higher token use for evidence settings on that benchmark because its fixed per-image allocation accumulates across cropped patches.Source: §4.3.2 · p. 12, Table 3 · p. 12
  • Page-level recall reaches 93.5% on MMLongBench-Doc and 95.3% on FinRAGBench-V, while precision is 28.0% and 34.8%, respectively.The authors report that the resulting distracting pages do not severely degrade answer accuracy at roughly 10,000-token context volume, but identify precision improvement as a direction for further work.Source: §4.3.2 · pp. 12–13

C.2 Evaluation on FinRAGBench-V

FinRAGBench-V evaluates generated answers against reference answers using a holistic scoring rubric, with additional guidance for numerical equivalence and open-ended questions. The evaluation score is thresholded to a final binary result.

C.2 Evaluation on FinRAGBench-V

  • Irrelevant answers receive 1, relevant answers with factual errors or significant hallucinations receive 2–3, and relevant, fully correct answers receive 4–5.The rubric judges relevance and correctness against the reference answer.Source: §C.2 · p. 23
  • For open-ended queries, the evaluator is given separate scoring guidance, though the supplied passage does not include its full criteria.Source: §C.2 · p. 23
  • Mathematically equivalent answers count as correct across unit, format, or absolute-versus-relative differences, with mental calculations used to verify equivalence.The prompt gives examples such as 1 kilometer and 1000 meters, or 0.5 and 50%.Source: §C.2 · p. 23

D Case Studies

The case studies illustrate AgenticOCR extracting evidence through on-demand zoom and OCR, including text-mode retrieval and downstream reasoning over evidence groups. A hard case exposes errors from single-page processing and low-resolution screenshots.

Case Studies

  • Figure 6 demonstrates text-mode zoom and OCR returning plain text while localizing and retrieving a relevant evidence crop in a real-world task.Source: Figure 6 · p. 24
  • Figure 7 shows Gemini-2.5-Pro deriving the correct result through simple calculations from key table information and cross-validating evidence groups.Source: Figure 7 · p. 25
  • Figure 8 shows a failure in which single-page processing selects the wrong café table, and a low-resolution screenshot leads the generator to misidentify the café.The page contains two cafés in the table in Group 2 / Page 16.Source: Figure 8 · p. 26

Related papers

Loading 2602.24134v1…