Source-linked AI summary
ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents
Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, Feng Zhao
TL;DR
ViDoSeek addresses the lack of benchmarks for retrieval, comprehension, and reasoning over large collections of visually rich documents. ViDoRAG combines multimodal retrieval with a multi-agent reasoning framework and achieves over 10% improvement over strong baselines on ViDoSeek.
Problem
Existing VQA benchmarks typically pair queries with a single image or document, leaving large-corpus visual document retrieval and reasoning insufficiently evaluated.
Method
ViDoRAG combines visual and textual retrieval with GMM-based distribution adjustment and a multi-agent workflow using seeker, inspector, and answer agents.
Results
Over 10% improvement over strong baselines establishes ViDoRAG as a new state-of-the-art on ViDoSeek.
Takeaways & Limitations
ViDoRAG demonstrates that coarse-to-fine reasoning and multimodal retrieval can improve RAG performance on visually rich documents.
Takeaways & Limitations
Human-expert query construction may bias question types and phrasing, while iterative multi-agent interactions add computational overhead that may limit scalability under strict latency requirements.
Abstract
from arXiv · showhide
Understanding information from visually rich documents remains a significant challenge for traditional Retrieval-Augmented Generation (RAG) methods. Existing benchmarks predominantly focus on image-based question answering (QA), overlooking the fundamental challenges of efficient retrieval, comprehension, and reasoning within dense visual documents. To bridge this gap, we introduce ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich documents requiring complex reasoning. Based on it, we identify key limitations in current RAG approaches: (i) purely visual retrieval methods struggle to effectively integrate both textual and visual features, and (ii) previous approaches often allocate insufficient reasoning tokens, limiting their effectiveness. To address these challenges, we propose ViDoRAG, a novel multi-agent RAG framework tailored for complex reasoning across visual documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy to effectively handle multi-modal retrieval. To further elicit the model's reasoning capabilities, we introduce an iterative agent workflow incorporating exploration, summarization, and reflection, providing a framework for investigating test-time scaling in RAG domains. Extensive experiments on ViDoSeek validate the effectiveness and generalization of our approach. Notably, ViDoRAG outperforms existing methods by over 10% on the competitive ViDoSeek benchmark. The code is available at https://github.com/Alibaba-NLP/ViDoRAG.
1 Introduction
The paper introduces ViDoSeek to evaluate retrieval, reasoning, and generation over large visually rich document corpora, then proposes ViDoRAG to address multimodal retrieval and insufficient reasoning guidance. ViDoRAG combines hybrid retrieval with a multi-agent coarse-to-fine framework and achieves over 10% improvement over strong baselines on ViDoSeek.
- Motivation: Existing VQA benchmarks pair queries with one image or document, making them unsuitable for evaluating retrieval across large document corpora.Their answers may also lack uniqueness within the full corpus.
- ViDoSeek: ViDoSeek provides unique-answer queries with specific reference pages for visually rich document retrieval-reason-answer evaluation.It covers diverse content types and multi-hop reasoning, enabling retrieval and generation to be evaluated separately.
- ViDoRAG: ViDoRAG is a multi-agent, coarse-to-fine retrieval-augmented generation framework tailored to reasoning over large visually rich document corpora.The framework is designed to improve reasoning over retrieved visual-document evidence.
- ViDoRAG: GMM-based multimodal hybrid retrieval integrates visual and textual features while dynamically adjusting the retrieval-result distribution for each query.The approach is intended to reduce irrelevant computation and improve generation efficiency.
- ViDoRAG: Three agents—the seeker, inspector, and answer agents—perform selection, review and reflection, and final consistency checking in an iterative workflow.The seeker scans thumbnails, the inspector provides feedback and preliminary answers, and the answer agent produces the final answer.
- Results: Over 10% improvement over strong baselines establishes ViDoRAG as a new state-of-the-art method on ViDoSeek.This is the paper’s reported headline result.
2 Related Work
Prior work largely addresses visual document question answering and retrieval through single-image, multi-page, OCR-based, or multimodal methods. This paper builds on those directions by combining multimodal hybrid retrieval with a scalable coarse-to-fine multi-agent generation framework.
- Visual Document Q&A: Visual document QA research has primarily focused on questions grounded in single images, with newer work extending to multi-page documents.These settings remain distinct from retrieval over large document collections.
- Retrieval-Augmented Generation: RAG systems have traditionally relied on OCR-derived text retrieval, while recent multimodal embeddings have improved image retrieval.The literature also includes work on in-context learning for RAG.
- This Work: ViDoRAG combines multimodal hybrid retrieval with a coarse-to-fine multi-agent generation framework that integrates embedding and generation models in a scalable system.This positions the framework across both retrieval and generation components.
- Dataset Construction: The dataset-construction pipeline samples and filters documents, elicits expert queries, screens general queries, and refines unqualified queries with golden images.GPT-4 performs general-query filtering and GPT-4o assists final query refinement.
3 Problem Formulation
Given a query and a collection of multi-page documents represented as page images, the task is to retrieve relevant information efficiently and accurately and generate a final answer.
- Problem Setup: The document collection contains M documents, with each document represented as a set of page images.The formulation denotes the collection as C = {D1, D2, . . . , DM} and each document as Dm = {I1, I2, . . . , IN}.
- Objective: The objective is to retrieve the most relevant information from the collection and generate the final answer a for query q.The formulation explicitly targets both retrieval accuracy and answer generation.
4 ViDoSeek Dataset
ViDoSeek is constructed as a large-corpus benchmark for visually rich document retrieval and reasoning, using controlled query creation, automated review, and multimodal refinement. It spans diverse visual content and reasoning types and is intended to provide more realistic and scalable RAG evaluation than prior datasets.
- Dataset Motivation: ViDoSeek addresses the mismatch between single-image or single-document QA datasets and practical questions posed over large document collections.Its large-corpus design is intended to better evaluate RAG systems.
- Dataset Construction: The dataset uses a four-step construction pipeline covering document collection, query creation, quality review, and multimodal refinement.It combines a dataset built from scratch with queries refined from SlideVQA.
- Document Collection: The source corpus contains English-language slides of 25 to 50 pages across 12 domains, including economics, technology, literature, and geography.The collection includes slides containing text, charts, tables, and two-dimensional layouts.
- Query Construction: Experts construct document-specific queries in varied forms and reasoning types, while automated review identifies problematic queries for refinement.The pipeline is designed to support large-scale retrieval and generation evaluation.
- Dataset Statistics: Approximately 1.2k questions cover text, charts, tables, and layouts, with both single-hop and multi-hop reasoning types.Layout questions constitute the largest and most challenging content category.
- Dataset Analysis: Compared with existing datasets, ViDoSeek more closely mirrors large-collection retrieval scenarios and supports robust, scalable RAG evaluation.Its comparison emphasizes limitations of datasets centered on single images or documents.
5 Method
ViDoRAG combines adaptive multimodal retrieval with an iterative multi-agent generation process for visually rich documents. GMM-based recall selects retrieval depth, while Seeker, Inspector, and Answer agents progressively refine evidence and responses.
- ViDoRAG integrates Multi-Modal Hybrid Retrieval and Multi-Scale View Generation for visually rich documents.
- Multi-Modal Hybrid Retrieval: The retrieval module combines textual and visual results, then sorts merged nodes in their original sequence to preserve page correlations.Adaptive recall limits excessive hybrid contexts and reduces unnecessary computation.
- Multi-Modal Hybrid Retrieval: GMM-based adaptive recall automatically determines top-K from similarity distributions instead of relying on a fixed K.The method establishes bounds for outliers and uses expectation-maximization to estimate similarity distributions.
- Multi-Agent Generation with Iterative Reasoning: The Seeker Agent selects relevant images from coarse views, incorporating Inspector feedback to refine image selection and maintain iterative memory.The process stops when sufficient information is available or no further relevant candidate images remain.
- Multi-Agent Generation with Iterative Reasoning: The Inspector Agent reviews images in greater detail, providing feedback or drafting an answer, while typically inspecting fewer images than the Seeker.This finer inspection addresses interference from irrelevant images and supports reasoning robustness.
- Multi-Agent Generation with Iterative Reasoning: The Answer Agent verifies the Inspector’s draft against reference images before finalizing, refining responses from coarse to fine.Consistency checking is used when the reference image matches or is a subset of the Inspector’s input.
6 Experiments
The experiments evaluate retrieval and generation across multiple models and retrievers on ViDoSeek, using GPT-4o-based accuracy and recall metrics. Results indicate strong performance from ViDoRAG and hybrid retrieval.
- Evaluation setup: End-to-end accuracy is evaluated by GPT-4o scores from 1 to 5, with scores of 4 or above counted as correct.Retrieval performance is measured using recall.
- Generation models: The experiments compare GPT-4o, Qwen2.5-7B-Instruct, Qwen2.5-VL-7B-Instruct, and Llama3.2-Vision-90B-Instruct.The study includes both closed-source and open-source models.
- Generation models: Closed-source models generally outperform open-source models, while Qwen2.5-VL-7B demonstrates strong instruction-following and reasoning within ViDoRAG.The authors also report that Llama3.2-Vision requires 90B parameters for the same instructions.
- Retrieval performance: Hybrid retrieval combines visual and textual pipelines and achieves state-of-the-art performance, while smaller contexts can reduce recall.The comparison includes OCR-based and visual-based retrievers.
7 Analysis
The analysis examines ablations, dynamic retrieval, latency, query types, and scaling behavior. ViDoRAG trades modest latency and additional reasoning rounds for shorter contexts and improved performance on complex visual-document tasks.
- Ablations: Directly merging visual and textual retrieval results can create long contexts and suboptimal results, whereas adaptive recall keeps retrieval concise.The ablations separate Dynamic and Hybrid components.
- Time efficiency: GMM-based dynamic retrieval can reduce recall because of distribution bias, but shortening the generation context improves end-to-end performance.The method selects the number of retrieved documents from query–corpus similarity distributions.
- Time efficiency: Multi-agent generation increases delay because seeker, inspector, and answer agents operate sequentially, but higher answer quality makes the latency–accuracy trade-off beneficial.The latency analysis is reported in Figure 5.
- Query analysis: The vision-based pipeline outperforms the text-based pipeline across all query types, including queries involving text content.The analysis covers ViDoSeek and the refined SlideVQA datasets.
- Test-time scaling: Stronger models require fewer seeker–inspector interaction rounds, while weaker models often need additional reasoning iterations.The experiment samples 200 queries to analyze model-dependent scaling behavior.
- Test-time scaling: Predefining tasks and decomposing complex tasks into simpler ones provides an effective approach for scaling inference.This conclusion is drawn from the reported interaction-round analysis.
8 Conclusion
The paper concludes that ViDoRAG combines coarse-to-fine reasoning with multi-modal retrieval for visually rich documents and achieves new state-of-the-art performance on ViDoSeek. Future work targets efficiency and broader real-world applications.
- Conclusion: ViDoRAG is a multi-agent RAG framework that combines coarse-to-fine reasoning with a multi-modal retrieval strategy.It is tailored for visually rich documents.
- Conclusion: ViDoRAG significantly outperforms existing methods and achieves new state-of-the-art performance on the ViDoSeek benchmark.The conclusion identifies efficiency optimization and applications in education and finance as future directions.
Limitations
The authors identify query-construction bias, computational overhead, and hallucinations as limitations of ViDoSeek and ViDoRAG. They highlight future work on broader document generalization, efficiency, bias, and grounding.
- Human-expert query construction may bias question types and phrasing, limiting coverage of diverse real-world queries.
- Iterative interactions among the seeker, inspector, and answer agents add computational overhead that may limit scalability under strict latency requirements.
- Models can still produce hallucinated answers unsupported by retrieved information, causing incorrect or misleading responses when overconfident.
- Future work targets generalization across diverse document types, query-bias reduction, computational efficiency, and hallucination mitigation.
Ethical Considerations
The paper states that its dataset contains no private or sensitive information and derives all content from publicly available sources. Dataset construction and refinement were conducted with respect for copyright and intellectual-property rights, while experiments used substantial computational resources.
- The dataset contains no private or sensitive information.
- All dataset content is derived from publicly available sources.
- Dataset construction and refinement were conducted in a manner respecting copyright and intellectual-property rights.
- Experiments used a server equipped with 8 A100 GPUs and 96 CPU cores, reflecting substantial resource requirements for open-source models.
B More Details on Datasets
The appendix details ViDoSeek’s dataset scope, difficulty, construction pipeline, and agent prompts. It describes multi-document, multimodal, single-hop, and multi-hop challenges alongside automated review and multimodal refinement procedures.
- Dataset characteristics: ViDoSeek covers text, tables, charts, layouts, and both single-hop and multi-hop reasoning.
- Dataset characteristics: Its difficulty comes from multi-document contexts, intricate content types, and especially challenging layout-based questions.
- SlideVQA-Refined: SlideVQA-Refined evaluates retrieval over visually sparse and dense information, including multi-hop references distributed across pages.
- Construction pipeline: ViDoSeek construction uses document collection, expert query creation, quality review, and multimodal refinement.
- Construction pipeline: Quality review filters potentially non-unique queries with LLMs, then uses top-k retrieval and VLM judgments to verify answerable reference pages.
- Agent prompts: The appendix provides prompts for query review, multimodal refinement, and seeker, inspector, and answer agents, including few-shot examples for some open-source models.