Source-linked AI summary
DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding
Dawei Zhu, Rui Meng, Jiefeng Chen, Sujian Li, Tomas Pfister, Jinsung Yoon
TL;DR
Long visual document question answering is challenging because relevant evidence may be hard to locate across pages and within visual elements. DocLens combines tool-assisted localization with answer sampling and adjudication. On the evaluated benchmarks, it reports state-of-the-art results and performance above human experts with Gemini-2.5-Pro.
Problem
Existing approaches struggle to retrieve relevant pages and capture fine-grained visual details in long documents, a challenge the paper associates with hallucinations on unanswerable queries.
Source: §1 · p. 2Method
DocLens uses a tool-augmented Lens Module to locate relevant pages and visual elements, followed by a Reasoning Module that samples candidate answers and adjudicates among them.
Source: §1 · p. 2Results
DocLens with Gemini-2.5-Pro achieves state-of-the-art results on MMLongBench-Doc and FinRAGBench-V and surpasses human experts.
Source: §7 · p. 11Takeaways & Limitations
On MMLongBench-Doc, DocLens improves Unanswerable-subset scores by +8.2%, +13.0%, and +13.8% for Claude-4-Sonnet, Gemini-2.5-Flash, and Gemini-2.5-Pro, respectively.
Source: §4.2 · p. 5The current approach does not distinguish document domains, although specialized fields may require domain-specific expert knowledge for accurate interpretation.
Source: Limitations · p. 11
Abstract
from arXiv · showhide
Comprehending long visual documents, where information is distributed across extensive pages of text and visual elements, is a critical but challenging task for modern Vision-Language Models (VLMs). Existing approaches falter on a fundamental challenge: evidence localization. They struggle to retrieve relevant pages and overlook fine-grained details within visual elements, leading to limited performance and model hallucination. To address this, we propose DocLens, a tool-augmented multi-agent framework that effectively ``zooms in'' on evidence like a lens. It first navigates from the full document to specific visual elements on relevant pages, then employs a sampling-adjudication mechanism to generate a single, reliable answer. Paired with Gemini-2.5-Pro, DocLens achieves state-of-the-art performance on MMLongBench-Doc and FinRAGBench-V, surpassing even human experts. The framework's superiority is particularly evident on vision-centric and unanswerable queries, demonstrating the power of its enhanced localization capabilities.
1. Introduction
Long visual documents distribute information across text and visual elements, making evidence localization a central challenge. DocLens combines tool-augmented localization with sampling and adjudication; the paper reports strong benchmark results, including performance above human experts.
Motivation
- Page-level retrieval misses evidence, and full-page views obscure visual details; the authors connect these localization failures to hallucinations on unanswerable queries.On MMLongBench-Doc, Gemini-2.5-Pro recalls 68% of evidence pages, while ColBERT and ColPali achieve 55.3% Recall@10; models invent responses for over half of unanswerable queries.Source: §1 · p. 2
Approach
- DocLens uses OCR-augmented page retrieval and layout detection with cropping to localize pages and visual elements, then synthesizes the evidence into an answer.Its Reasoning Module samples candidate answers and uses an Adjudicator to assess them and select the best candidate.Source: §1 · p. 2
Results
- DocLens achieves state-of-the-art performance on MMLongBench-Doc and FinRAGBench-V, reduces hallucination, and surpasses human experts.The paper reports 97.3% evidence-page recall for the Page Navigator.Source: §1 · p. 2
2. Problem Formulation
The paper frames long visual document question answering as finding concise, relevant evidence before generating an answer. That evidence includes relevant pages and their textual and visual elements.
2. Problem Formulation
- The paper separates answering over long visual documents into finding concise relevant evidence and generating an answer from it.The document is a sequence of page screenshots, and the answer should be grounded in specific evidence pages.Source: §2 · pp. 2–3
- The extraction function f_extract maps a document and question to a concise evidence set S containing relevant pages and their visual and textual elements.The evidence set is intended to retain the material needed for answering.Source: §2 · p. 3
- The generation function f_generate infers the final answer exclusively from the condensed evidence, with both functions optimized for answer accuracy.The composite formulation maps the document and question to an answer.Source: §2 · p. 3
3. DocLens Framework
DocLens first localizes relevant pages and their text and visual elements, then reasons over the collected evidence to produce an answer. Its Lens and Reasoning Modules divide retrieval from answer synthesis.
3. DocLens Framework
- The Lens Module retrieves relevant pages and identifies key elements; the Reasoning Module analyzes that evidence to generate an answer.The framework is presented as two primary components for evidence identification and answer generation.Source: §3 · p. 3
3.1. Lens Module
- The Page Navigator uses OCR text and page screenshots as interleaved input to an LLM, repeating retrieval and taking the union of identified pages.When the context window cannot accommodate all pages, it processes page chunks in parallel and merges their predicted page sets.Source: §3.1 · pp. 3–4
3.2. Reasoning Module
- The Element Localizer uses layout detection to find visual-element bounding boxes and crops them, collecting the crops with page screenshots and text as evidence.The Answer Sampler produces multiple reasoning-answer pairs; the Adjudicator compares their reasoning paths and selects a consistent, logical conclusion.Source: §3.1 · p. 4, §3.2 · p. 4
4. Experiments
DocLens improves results on MMLongBench-Doc and FinRAGBench-V, with especially strong gains on unanswerable and visual-evidence questions. The experiments evaluate three proprietary backbones against screenshot, OCR-augmented, and agentic baselines.
4.1. Experimental Setup
- The evaluation uses MMLongBench-Doc and FinRAGBench-V, testing Gemini-2.5-Pro, Gemini-2.5-Flash, and Claude-4-Sonnet against screenshot, OCR-augmented, and agentic baselines.MMLongBench-Doc averages 49.4 pages and includes an Unanswerable subset; FinRAGBench-V evaluates visual citation by pinpointing block-level evidence.Source: §4.1 · p. 5
4.2. Main Results on MMLongBench-Doc
- +8.2%, +13.0%, and +13.8% absolute gains on the MMLongBench-Doc Unanswerable subset for Claude-4-Sonnet, Gemini-2.5-Flash, and Gemini-2.5-Pro, respectively.The authors say these gains indicate that DocLens effectively mitigates hallucination.Source: §4.2 · p. 5
- OCR text appended to page screenshots substantially improves performance over the vanilla setting across all four tested backbones.The authors attribute the improvement to OCR facilitating implicit evidence-page retrieval.Source: §4.2 · p. 5
- Gemini-2.5-Pro with DocLens surpasses the MMLongBench-Doc human baseline by ∼2%, while Claude-4-Sonnet and Gemini-2.5-Flash reach near-human performance.DocLens improves performance across all three backbones.Source: §4.2 · p. 5, Table 1 · p. 6
4.3. Main Results on FinRAGBench-V
- +3.1%, +10.2%, and +5.5% over the strongest baseline with Claude-4-Sonnet, Gemini-2.5-Flash, and Gemini-2.5-Pro, respectively, on FinRAGBench-V.The authors hypothesize that gains are larger because FinRAGBench-V contains more dense, complex visual layouts, such as newspapers.Source: Table 1 · p. 6, §4.3 · p. 6
- +10.9% and +23.5% on chart questions, and +4.2% and +6.9% on table questions, for Gemini-2.5-Pro and Gemini-2.5-Flash, respectively, over the strong OCR-augmented baseline.The authors report that the performance boost is primarily driven by better handling of visual evidence.Source: Table 1 · p. 6, §4.3 · p. 6
5. Analysis
The analysis tests DocLens’s core modules, examines how page and element localization affect evidence retrieval, and explores whether cheaper retrieval backbones can preserve performance.
5.4. Hybrid Backbones for Cost Efficiency
- 67.1% with Gemini-2.5-Flash and 64.4% with Flash-Lite exceed the 63.3% Gemini-2.5-Pro baseline for Page Navigator retrieval.The Page Navigator processes an average of 49.4 MMLongBench-Doc pages, versus 3.5 pages for the Reasoner, creating a cost-efficiency opportunity.Source: Table 3 · p. 7, §5.4 · p. 9
5.1. Ablation on Core Modules
- Removing the Lens Module drops Gemini-2.5-Pro performance by 4.1% on MMLongBench-Doc and 5.3% on FinRAGBench-V; removing the Reasoning Module hurts unanswerable-query performance.Lens removal also degrades Gemini-2.5-Flash by 7.6% and 1.4% on those benchmarks, respectively.Source: §5.1 · p. 7
5.2. Analysis on Page Navigator
- 97.3% evidence-page recall with Gemini-2.5-Pro on MMLongBench-Doc leaves final accuracy only 1.5% below using oracle pages.MDocAgent retrieves 13.6 pages on average with low evidence recall and the poorest final accuracy; SimpleDoc scores 64.4% but still trails Page Navigator even with its weakest backbone.Source: §5.2 · pp. 7–8, Table 3 · p. 7
- OCR raises evidence-page recall by 10.0% on MMLongBench-Doc and 40.5% on FinRAGBench-V with Gemini-2.5-Pro, while sampling also improves recall and final accuracy.Gemini-2.5-Flash recall gains are 15.9% and 50.1%, respectively; the authors note VLMs are more adept at textual than visual retrieval.Source: §5.2 · p. 8, Table 4 · p. 8
5.3. Analysis on Element Localizer
- 4.9% precision, 9.3% recall, and 6.7% F1 gains show Element Localizer improves block identification, with substantial benefits when evidence includes visual elements.Cases include identifying a trend in a small bar chart and extracting and ordering values from a line plot.Source: §5.3 · pp. 8–9, Figure 3 · p. 9, Figure 4 · p. 9
6. Related Work
Related work spans long-document comprehension, evidence retrieval, and agentic approaches to long-context modeling. DocLens extends retrieval-based agents by using document-parsing tools to localize visual elements within relevant pages.
Visual Document Understanding
- Single, long-document comprehension is a distinct setting from multi-document retrieval, requiring models to handle cohesive but extensive visual documents.The paper situates DocLens in this single-document setting.Source: §6 · p. 10
Evidence Localization in Visual Document Understanding
- Evidence localization methods commonly retrieve top-K pages with vector-based models, while SimpleDoc adds LLM selection from summaries and iterative query refinement.The paper contrasts these approaches with OCR-augmented long-context VLM retrieval and element-level localization.Source: §6 · p. 10
Agentic Frameworks for Long Context Modeling
- Long-context agentic frameworks commonly use retrieval-augmented generation or compress context into summaries or abstract memory before answering.DocLens extends retrieval-based frameworks through document-parsing tools that localize relevant elements within the document.Source: §6 · pp. 10–11
7. Conclusion
DocLens addresses evidence localization in long visual document understanding through Lens and Reasoning Modules. On two benchmarks, it improves VLM performance and, with Gemini-2.5-Pro, achieves state-of-the-art results while surpassing human experts.
Conclusion
- DocLens is a tool-augmented multi-agent framework addressing evidence localization in long visual document understanding.It uses a Lens Module for evidence retrieval and a Reasoning Module for analysis.Source: §7 · p. 11
- DocLens significantly improves VLM performance on MMLongBench-Doc and FinRAGBench-V.Source: §7 · p. 11
- With Gemini-2.5-Pro, DocLens achieves state-of-the-art results and surpasses human expert performance.Source: §7 · p. 11
Limitations
DocLens’s limitations include difficult visual-element cases that simple zooming cannot handle and a current approach that does not distinguish document domains. Its prompts also specify how agents inspect evidence and judge answer consistency and answerability.
Limitations
- Some challenging chart, table, and figure cases remain beyond simple zooming; the authors suggest specialized agents or stronger backbone perception as directions for handling them.The proposed alternatives are either dedicated agentic frameworks for particular visual-element types or advances in backbone LLM perception.Source: Limitations · p. 11
- The current approach does not distinguish document domains, although specialized fields may require domain-specific expert knowledge for accurate interpretation.Automatically constructing expert-level agents tailored to different domains is identified as future work.Source: Limitations · p. 11
A. Prompt Templates for our Agentic Framework.
- The Page Navigator prompt prioritizes inclusive page recall, while resolving explicit page references to document indices and printed page numbers.It also directs agents to locate supporting pages and handle numerical answers by finding or calculating the requested quantity.Source: A. Prompt Templates for our Agentic… · p. 16
- The Answer Sampler prompt can inspect zoomed-in figures or tables alongside page images and extracted Markdown, but requires original images for layout questions.It also instructs the sampler to answer “Not answerable” when the supplied pages lack sufficient information.Source: A. Prompt Templates for our Agentic… · pp. 18–19
- The Adjudicator aggregates multiple agents’ answers by assessing evidence rather than answer frequency, and selects “Not answerable” when the pages do not support an answer.Its instructions also call for common-sense interpretation rather than overly literal readings.Source: A. Prompt Templates for our Agentic… · p. 20
B. Dataset Statistics
This material describes the benchmarks’ domain coverage, implementation choices for DocLens, a SimpleDoc reproduction setup, and examples and answer-extraction rules for MMLongBench-Doc.
B. Dataset Statistics
- MMLongBench-Doc spans seven document domains, whereas FinRAGBench-V focuses solely on financial reports.The listed MMLongBench-Doc domains include academic papers, financial reports, guidebooks, and administration or industry files.Source: B. Dataset Statistics · p. 22
C.1. Implementing DocLens
- DocLens uses MinerU for OCR, layout detection, and cropping; page chunking improves Claude-4-Sonnet’s FinRAGBench-V evidence-page recall from 72% to 87%.For Claude-4-Sonnet, the method processes 50 pages per iteration; Gemini-2.5-Pro and Gemini-2.5-Flash do not require chunking in these experiments.Source: §C.1 · pp. 22–23
C.2. Reproducing SimpleDoc
- The SimpleDoc reproduction returns 30 pages for vector retrieval, substitutes Gemini-2.5-Flash, Gemini-2.5-Pro, and Claude-4-Sonnet as backbones, and follows the original evaluation protocol.For MMLongBench-Doc, the authors report the original metric for comparison, noting DocLens with Gemini-2.5-Pro scores 75 under LLM-based evaluation and 67.6 under the original metric.Source: §C.2 · p. 23
C.3. Evaluation on MMLongBench-Doc
- MMLongBench-Doc evaluation manually corrected some evidence-page annotations and extracts answers into integer, float, string, or list formats, with “Not answerable” and “Fail to answer” distinguished.The extraction instructions reserve “Fail to answer” for analyses that only report inability to read or understand the document.Source: §C.3 · p. 24
C.4. Evaluation on FinRAGBench-V
The evaluation material defines page- and element-level retrieval metrics and describes how model answers are scored against ground truth on FinRAGBench-V.
C.4. Evaluation on FinRAGBench-V
- Answer evaluation compares the prediction’s final conclusion with the ground truth and assigns a score of 1.0 for a match or 0.0 otherwise.The evaluator is instructed to ignore reasoning steps when identifying the final answer and to allow flexible formatting equivalence.Source: §C.4 · p. 27
- Page-level precision, recall, and F1 are calculated from predicted and ground-truth page sets using their intersection and unmatched pages.True positives are pages in both sets, false positives are predicted pages absent from ground truth, and false negatives are ground-truth pages absent from predictions.Source: §C.4 · p. 27
- Element-level evaluation matches predicted bounding boxes to ground-truth boxes using IoU, counting a match as a true positive at IoU ≥0.5 and matching each ground-truth box at most once.Precision, recall, and F1 are then calculated from true positives, false positives, and false negatives.Source: §C.4 · p. 28
D.1. Test-time Scaling of Page Navigator and Answer Sampler
Test-time scaling benefits the Page Navigator more than the Answer Sampler. Navigator recall rises with sampling before gains diminish, while answer performance plateaus quickly.
D.1. Test-time Scaling of Page Navigator and Answer Sampler
- Over 90% Page Navigator recall is reached with 8 samples, up from approximately 78% with one sample; gains beyond 8 samples fall below 1 percentage point.The figure passage identifies the test-time scaling comparison for the Page Navigator and Answer Sampler.Source: Figure 6 · p. 28, §D.1 · pp. 28–29
- Adjudicated Answer Sampler performance stays around 68–69% across sample counts, while best-of-N gains are limited beyond 2 samples.Increasing samples from 1 to 2 produces an approximately 5-percentage-point gain.Source: §D.1 · p. 29
E. Hard Cases
DocLens fails on some vision-centric cases. The reported errors span chart comparisons, opinion changes, map counting, and retrieval-node overlap.
E. Hard Cases
- For the adviser’s organic growth rate, DocLens reports 4 years above 2011 versus a ground-truth answer of 1.Source: E. Hard Cases · p. 29
- For China opinions, DocLens selects ages 30–49 rather than the ground-truth group aged 50 and older as having the greatest change.Source: E. Hard Cases · p. 30
- On the last-page map, DocLens counts 9 “WC” labels against the ground-truth count of 19.Source: E. Hard Cases · p. 31
- For the two Figure 4 questions, DocLens retrieves nodes [1, 11, 16, 17, 20] instead of the ground-truth [16, 19, 25].Source: E. Hard Cases · p. 32