SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinement

• DocVQA involves multiple pages, cross-references between pages, and multiple modalities such as text, tables, and images.Source: §1 · p. 1
• The introduction contrasts RAG's handling of multiple pages and references with approaches that preprocess modalities into text or retrieve pages as images.Source: §1 · pp. 1–2
• SimpleDoc indexes each page with a visual embedding and semantic summary, then uses summary-based reranking and a memory-guided reasoner that can retrieve again.Source: §1 · p. 2
• 3.2 absolute points over previous baselines across four datasets, with only 3.5 pages retrieved per question.Source: §1 · p. 2

• Earlier DocVQA work often targeted single-page images with OCR and multimodal models, while newer benchmarks emphasize long, multi-page, multimodal understanding.Source: §2 · p. 2
• Page-as-image methods such as ColPali, VisRAG, and VDocRAG use visual page representations to avoid missing information from separately parsing text and images.Source: §2 · p. 3
• The paper also adapts Chain-of-Notes and Plan∗RAG, representing retrieval-context improvement and query decomposition, for DocVQA evaluation.Source: §2 · p. 3

3 Method

• SimpleDoc has two stages: offline document processing and online iterative retrieval-augmented question answering, combining vector and semantic representations with memory updates.Source: §3 · p. 3

3.1 Offline Document Processing

• Each page receives a VLM embedding and a query-independent, typically 3–5 sentence summary of salient information for future questions.Source: §3.1 · p. 4

3.2 Multi-modal Question Answering

• Retrieval first selects k pages by embedding MaxSim, then an LLM reranks their summaries and chooses a dynamic subset of n < k pages for the reasoner.Source: §3.2 · p. 4
• The reasoner uses retrieved page images and text with working memory to answer, mark a question unanswerable, or issue a query update for missing evidence.Source: §3.2 · p. 4
• On a query update, SimpleDoc retrieves new pages, adds notes to memory, and repeats until an answer, an unanswerable decision, or the iteration limit.Source: §3.2 · pp. 4–5

4.1 Main Results

• 70.12% average accuracy makes SimpleDoc the highest-performing method across the four DocVQA datasets, outperforming all tested baselines.The evaluation spans four PDF document-understanding benchmarks.Source: §4.1 · p. 5, Table 1 · p. 6
• On FetaTab, SimpleDoc performs below MDocAgent, which the authors attribute to its dedicated image agent for table grids.SimpleDoc instead treats pages as images for a single reasoner agent.Source: §4.1 · pp. 5–6
• 3.5 pages per question gives SimpleDoc the best overall accuracy with fewer pages, while retrieval balances evidence coverage against conciseness.The authors report that SimpleDoc has nearly the same coverage as ColQwen-2.5 at k=2 while achieving more than double its page-level F1.Source: §4.1 · p. 6, Table 2 · p. 6
• SimpleDoc initially retrieves broadly relevant pages, then refines its query to find the missing evidence and answer the question.The example's targeted query retrieves the table showing that temperature 0.1 yields the highest alignment score, 85.9.Source: §4.1 · p. 6, Figure 3 · p. 7

4.2 Results with different models

• In the model setting, SimpleDoc achieves 50% overall accuracy, improving over MDocAgent by 6.62 points, a bigger gap than with larger models (+4.15 points).SimpleDoc also achieves much better results on unanswerable questions, which test hallucinations.Source: §4.2 · pp. 7–8

4.3 Other RAG methods

• The tested knowledge-QA RAG methods lag behind DocVQA-specific approaches; Chain-of-Note reaches 40.4% accuracy, while Plan∗RAG's inaccurate query graphs can propagate errors.The authors note that page-level summaries can miss fine details needed for precise answers.Source: §4.3 · p. 8

4.4 Additional Analysis of SimpleDoc

• Larger embedding-retrieval top-k gives the agent more candidates, but it does not lead it to select significantly more pages.The authors interpret this as the agent dynamically deciding which pages are relevant to each query.Source: §4.4 · p. 8, Table 4 · p. 8
• Across iterations, SimpleDoc's accuracy improves while query updates decline, as the system finds needed information or recognizes an unanswerable question.The example run shows a refined query retrieving a page with the exact temperature alignment scores missing from the initial results.Source: Figure 3 · p. 7, §4.4 · p. 8, Table 5 · p. 8

5 Conclusion

• SimpleDoc combines dense-vector embeddings and page summaries for efficient page retrieval with a reasoning agent that iteratively detects and remedies missing evidence.Source: §5 · p. 8
• Across four DocVQA benchmarks, SimpleDoc surpasses prior RAG-style systems and multi-agent baselines with fewer components and fewer page retrievals.Source: §5 · pp. 8–9
• The results highlight the use of modern VLMs for retrieval-augmented multimodal reasoning.Source: §5 · p. 9

6 Limitations

• Experiments cover only single-document VQA, although embedding retrieval is described as readily extensible to a whole document database.Source: §6 · p. 9

A Appendix

• Table 6 defines All-Match as retrieving every ground-truth evidence page and compares two retrieval models on MMLongBench.Source: Table 6 · p. 12

A.1 Pilot Study

• The pilot study evaluates Qwen2.5-VL-32B on MMLongBench with no evidence, ground-truth pages, and image, text, or combined page inputs.Source: §A.1 · p. 12
• Image-based input has a 25% accuracy advantage over text-based input; combining image and text further boosts performance, but not significantly.Source: §A.1 · p. 12

• An AI assistant helped debug code and build utility functions.Source: §A.2 · p. 12
• The authors also used an AI assistant to refine writing.Source: §A.2 · p. 12

• All-hit rate counts a question as covered only when every gold page is retrieved, so one missing page is a failure.Source: §A.3 · p. 12
• For each question, page-level F1 is the harmonic mean of precision and recall when their sum is positive.Source: §A.3 · p. 12
• The final PageF1 score is the macro-average of question-level F1 scores.Source: §A.3 · p. 12

• The page-indexing prompt extracts structured information from each document page.Source: §A.4 · p. 12, Figure 4 · p. 13
• The retrieval prompt uses page summaries and the query to select pages for detailed examination.Source: §A.4 · p. 14, Figure 5 · p. 14
• The question-answering prompt guides the reasoner using both extracted text and page images.Source: §A.4 · p. 12, Figure 6 · p. 15