M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

1. Introduction and Background

• Existing DocVQA methods either process single pages or use text-based RAG, which cannot handle many long documents or preserve complex visual information such as tables and charts.These limitations are especially relevant when questions require evidence across pages or documents.Source: §1 · p. 1, Figure 1 · p. 2
• M3DocRAG retrieves relevant pages with multi-modal models and answers from them with a multimodal language model across closed- and open-domain settings.Its stages include visual page embedding, top-K page retrieval, and visual question answering.Source: §1 · pp. 1–2, Figure 3 · p. 3
• M3DocVQA introduces open-domain questions spanning 3,368 PDFs and more than 41,005 pages, requiring multi-hop reasoning across text, images, and tables.It extends closed-domain question answering toward retrieval from a large, diverse document corpus.Source: §1 · pp. 2–3, Figure 2 · p. 2
• M3DocRAG with ColPali and Qwen2-VL 8B achieves superior performance over many strong baselines across M3DocVQA, MMLongBench-Doc, and MP-DocVQA, including state-of-the-art performance on MP-DocVQA.The benchmarks cover both open-domain and closed-domain DocVQA settings.Source: §1 · p. 3

2. M3DOCRAG: A Unified Framework for Multi-modal, Multi-page, Multi-document Understanding

• M3DOCRAG supports closed- and open-domain questions by retrieving the top K relevant pages from either one document or a global corpus before answering.The framework treats question answering as selecting a small relevant subset from the available page images, enabling single-document and multi-document settings.Source: §2 · pp. 3–4

2.1. Document Embedding

• Document pages and textual queries are projected into a shared multi-modal embedding space, preserving visual document information for retrieval.Each page is treated as an image and encoded with a dense visual embedding, while the query receives a corresponding text embedding.Source: §2.1 · p. 4

2.2. Page Retrieval

• Page relevance is scored with ColPali’s MaxSim late-interaction mechanism, and the highest-scoring K pages are selected for question answering.The method compares query and page embeddings through token-level maximum similarity and retrieves the pages with the largest scores.Source: §2.2 · p. 4
• Approximate indexing enables open-domain retrieval across large corpora, reducing page-search latency from 20s/query to less than 2s/query.The framework uses exact search for closed-domain retrieval and IVF-based approximate search for open-domain retrieval.Source: §2.2 · p. 4

2.3. Question Answering

• A multi-modal language model receives the query and retrieved page images, then autoregressively generates the final answer.The model’s visual encoder processes the retrieved pages, and its language model combines those visual embeddings with the query’s text embeddings.Source: §2.3 · p. 4

• Unlike closed-domain DocVQA datasets, M3DOCVQA requires models to retrieve relevant information from a large document corpus before answering.Existing datasets provide questions tied to specific content within a single document, whereas M3DOCVQA simulates open-domain retrieval across documents.Source: §3 · pp. 4–5
• M3DOCVQA contains 2,441 multi-hop questions over 3,368 PDF documents totaling 41,005 pages.Each question is supported by evidence in one or more documents and may involve text, images, or tables.Source: §3 · p. 5
• The benchmark provides 24,162 Wikipedia PDFs in its training split for future large-scale retrieval or model-training studies.Source: §3 · p. 5
• M3DOCVQA is constructed by converting Wikipedia contexts from MultimodalQA into rendered PDFs that preserve vector graphics, metadata, zooming, and hyperlinks.Source: §3 · p. 5
• The benchmark captures real-world multi-modal reasoning by distributing evidence across multiple documents and document elements such as text, images, and tables.Source: §3 · p. 5

• M3DOCRAG is evaluated on M3DOCVQA, MMLongBench-Doc, and MP-DocVQA, spanning open-domain and closed-domain document understanding.The datasets include collections exceeding 3,000 PDFs, single PDFs up to 120 pages, and single PDFs up to 20 pages.Source: §4 · p. 5
• The study reports F1 and exact match for M3DOCVQA, generalized accuracy and F1 for MMLongBench-Doc, and ANLS plus page-retrieval accuracy for MP-DocVQA.Source: §4 · pp. 5–6
• The main multimodal pipeline uses ColPali retrieval with recent multimodal language models, while the text baseline combines ColBERT v2 with Llama 3.1.Source: §4 · p. 6
• Experiments compare multimodal retrieval and language models against prior leaderboard entries using deterministic greedy decoding.Source: §4 · p. 6
• Pages are converted to RGB images, resized for each dataset, and supplied to multimodal language models with up to four visual inputs on one H100 GPU.Source: §4 · p. 6

5.1. Open-domain DocVQA

• M3DOCRAG significantly outperforms text RAG across evidence modalities, question hops, and retrieved-page counts on M3DOCVQA, with the largest gains for image evidence.Using four retrieved pages generally outperforms using one or two pages for both text RAG and M3DOCRAG.Source: §5.1 · p. 6

5.2. Closed-domain DocVQA

• ColPali retrieval with Idefics2 surpasses Idefics2 without RAG and previous multimodal entries on MMLongBench-Doc, while ColPali + Qwen2-VL 7B achieves the best overall F1 and most evidence-modality/page settings.The result supports multimodal retrieval over concatenating many low-resolution page images.Source: §5.2 · pp. 6–7
• M3DOCRAG with ColPali + Qwen2-VL 7B achieves state-of-the-art performance on MP-DocVQA.All multimodal RAG pipelines outperform their text-based counterpart in the reported comparison.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Additional analysis

• Approximate indexing with IVFFlat or IVFPQ reduces retrieval time from 21 seconds per query to under 2 seconds while retaining most accuracy.The analysis presents a speed-accuracy tradeoff for ColPali + Qwen2-VL 7B.Source: §5.3 · p. 7, Table 4 · p. 8
• Qwen2-VL 7B outperforms Idefics2, Idefics3, and InternVL2 across all three benchmarks, so it is used as the default multimodal language model.Source: §5.3 · p. 8, Table 5 · p. 8
• ColPali performs significantly better than ColQwen on M3DOCVQA, whereas ColQwen is slightly better on MP-DocVQA and MMLongBench-Doc.Source: §5.3 · p. 8, Table 6 · p. 8

5.4. Qualitative Examples

• Qualitative examples show successful answers when evidence is visually stored in logos, distributed across pages or documents, or supplemented by the language model’s knowledge.Source: §5.4 · p. 8

Document visual question answering

• Earlier DocVQA methods focused on single-document contexts, while text-based RAG pipelines retrieved extracted paragraphs rather than document visuals.Source: §6 · pp. 8, 10
• Qualitative examples show answers grounded solely in visual regions, requiring multi-page reasoning, or combining retrieved document evidence with model knowledge.Source: Figure 5 · p. 9, Figure 6 · p. 9, Figure 7 · p. 10

Retrieval-augmented generation

• M3DocRAG extends retrieval-augmented generation to textual, tabular, and visual information distributed across document pages.Source: §6 · p. 10

• M3DocRAG retrieves relevant pages with a multi-modal model and processes pixel-based documents with a multimodal language model.Source: §7 · p. 10
• M3DocVQA provides an open-domain benchmark with 2,000+ questions over 3,000+ PDF documents requiring image, text, and table evidence.Source: §7 · p. 10
• Experiments on M3DocVQA, MP-DocVQA, and MMLongBench-Doc show advantages over existing methods, including state-of-the-art performance in MP-DocVQA.Source: §7 · p. 10
• The authors analyze indexing strategies, multimodal language models, and multimodal retrieval models, and demonstrate questions answered from varied evidence modalities.Source: §7 · p. 10

• English-heavy training may limit understanding of non-English prompts or documents.Source: Ethical Considerations · p. 10
• Model components may produce false or biased information, so real-world use should include human supervision.Source: Ethical Considerations · p. 10
• The framework permits replacing components as more accurate solutions become available, while the dataset attributes source Wikipedia URLs.Source: Ethical Considerations · p. 10