Source-linked AI summary
VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval
Lexiang Hu, Yanzhao Zhang, Mingxin Li, Dingkun Long, Yikang Li, Fuwei Zhang, Yisen Wang, Zhouchen Lin
TL;DR
VisDocAgentBench addresses the underexplored problem of ranking documents during iterative evidence acquisition, rather than evaluating only one-shot matching or downstream answers. It introduces a closed-corpus benchmark with shared ranked outputs and reviewed compositional queries, showing that visual retrieval and capable planning are central to agentic document retrieval.
Problem
Existing evaluation emphasizes one-shot query–page matching or downstream question answering, leaving closed-corpus document ranking under iterative evidence acquisition underexplored.
Method
The paper introduces VisDocAgentBench, a reviewed 2,375-page benchmark with 120 balanced queries, relation-preserving construction, and a shared ranked-output contract for static retrievers and tool-using agents.
Results
Visual retrieval outperforms OCR-text across planners, while agents recover much of static retrieval’s loss on queries requiring distributed evidence.
Takeaways & Limitations
Reliable retrieval requires jointly developing modality-preserving representations and evidence-directed search policies, because tool access alone does not ensure strong ranking.
Takeaways & Limitations
The benchmark uses English-language scientific papers, fixed corpora, page-level targets, and small topic and cross-document subsets, limiting transfer and broader analysis.
Abstract
from arXiv · showhide
Visually rich documents encode relevance through language, layout, structured visual elements, and corpus context, yet retrieval is typically evaluated by one-shot query--page matching. Agentic-search benchmarks usually score downstream question answering or report generation, leaving document ranking under iterative evidence acquisition underexplored. We introduce VisDocAgentBench, a closed-corpus benchmark comparing static and agentic retrieval under a shared ranked-output contract. It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures. Relation-preserving construction yields semantic, relational, and visual queries, followed by full-document review and hard-negative validation. A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context. Agents recover much of this loss, but planner choice and retrieval representation remain decisive. Every planner performs better with visual retrieval, whose best R@1 reaches 67.50% versus 37.50% for OCR-text. Ablations identify iterative search and page inspection as consequential capabilities, and providing the complete support context improves ranking on both routes. Trace analysis localizes the remaining losses to target discovery, candidate examination, and evidence-role integration. These findings motivate retrieval agents that combine modality-preserving discovery with evidence-directed verification.
1 Introduction
VisDocAgentBench addresses the missing evaluation of agentic search for ranked retrieval over closed visual-document corpora. It introduces a reviewed benchmark and compares static retrievers with tool-using agents under a shared ranked-output contract.
- Visually rich documents require connecting local text–layout composition with definitions, results, or conditions elsewhere in the collection.
- Existing visually rich-document work largely uses fixed-query page ranking or retrieval for question answering, while many agentic visual-search benchmarks focus on natural images.
- The benchmark balances direct, one-bridge, and two-bridge evidence structures while hiding construction annotations and gold labels from evaluated systems.
- The benchmark targets closed-corpus agentic visual-document retrieval with ranked retrieval as the endpoint and a shared output contract.
- VisDocAgentBench spans 2,375 pages and 120 balanced, unique-target queries grounded in reviewed full-document evidence.
- Its construction pipeline preserves evidence roles and relations through automatic screening, full-document review, constrained authoring, and hard-negative validation.
- Static retrievers and tool-using agents are evaluated across visual and OCR-text routes with controlled diagnostics and auditable traces.
2 Related Work
Related work broadens multimodal retrieval and agentic interaction, but VisDocAgentBench connects these lines through auditable, compositional retrieval over visual-document corpora. Its benchmark emphasizes relation validation, full-document review, and hard-negative auditing.
- Vision–language pretraining and instruction-aware retrievers broaden open-vocabulary retrieval across tasks, intents, and media.
- Visual-document retrieval preserves language, layout, tables, figures, and fine-grained regions through embeddings, late interaction, query conditioning, region supervision, and visual refinement.
- Search agents interleave reasoning with query reformulation, iterative retrieval, and evidence accumulation, while multimodal variants add visual queries and observations.
- Process-oriented benchmarks expose intermediate visual-search behavior through checkpoints, cropping, browsing, and multimodal evidence traces.
- Existing document benchmarks cover question answering, chart reasoning, long-context understanding, parsing, and direct page retrieval, whereas agentic benchmarks span browser actions and multimodal answers.
- VisDocAgentBench connects these lines through a shared ranked-page endpoint and exposes search, inspection, modality-conversion, and final-ranking trajectories.
- The benchmark adapts shortcut control and postconstruction verification by validating relation paths, reviewing full-document context, constraining query clues, and auditing hard negatives.
3 Task Formulation
The task defines visual-document retrieval as ranking a unique target under language, relational, visual, and corpus-context relevance. It contrasts fixed pairwise scoring with budgeted agentic evidence gathering that conditions later searches and final rankings on observations.
- 3.1 Closed-Corpus Visual-Document Retrieval: A visual document unit x_i may combine embedded text, layout, tables, diagrams, formulas, screenshots, or interface components.
- 3.1 Closed-Corpus Visual-Document Retrieval: The fixed corpus contains visual units, and each natural-language query has a unique target a_q whose rank should be maximized.
- 3.1 Closed-Corpus Visual-Document Retrieval: Relevance may jointly depend on language, concept relations, and visual-layout properties, while evaluation concerns target ranking.
- 3.2 Context-Dependent Compositional Relevance: Target-local evidence appears on a candidate, whereas corpus-context evidence resides in other units and can resolve semantic or relational conditions.
- 3.2 Context-Dependent Compositional Relevance: When support context is present, a locally plausible unit may remain incorrect unless the complete corpus-grounded relation is satisfied.
- 3.3 Static and Agentic Retrieval: A static retriever scores each query–unit pair independently and produces one fixed ordering evaluated at its first K positions.
- 3.3 Static and Agentic Retrieval: Agentic retrieval is a budgeted action–observation process in which a policy gathers evidence from the corpus and appends actions and observations to interaction history.
- 3.3 Static and Agentic Retrieval: After interaction ends, an output policy maps accumulated history to a ranking, allowing later searches and final ordering to depend on observed corpus evidence.
4 Benchmark Construction and Quality Control
VisDocAgentBench is constructed through relation-preserving evidence paths, constrained query authoring, full-document review, and hard-negative validation. The resulting benchmark uses rendered scientific-paper pages with hidden construction annotations and audited unique targets.
- Corpus: 2,375 rendered pages from 100 scientific documents form the retrieval corpus, with 2,324 content-bearing pages used for query construction.Each visual document unit is a full page with document and page provenance.
- Role-Specific Descriptors and Directed Relations: Role-specific descriptors represent query anchors, semantic bridges, and visual target descriptors connected by directed intra-unit relations.Anchors provide non-identifying semantic starts, bridges align across units, and target descriptors capture observable layout or structured evidence.
- Relation-Preserving Path Construction: 1,522 validated evidence paths remain after semantic alignment, coherence, evidence-necessity, visual-necessity, and grounding checks.The retained paths comprise 364 L1, 1,058 L2, and 100 L3 paths.
- Quality Control: Every validated path receives independent human and AI full-document scoring before queries undergo corpus-wide alternative checks and hard-negative inspection.Queries are revised or replaced when retrieved alternatives prevent natural uniqueness.
- Query Authoring: Final queries expose anchors, directed relations, and abstracted visual targets while hiding semantic bridges, alignments, identifiers, captions, and construction terminology.Visual specificity is calibrated so the visual phrase is non-unique alone but identifying in conjunction with semantic and relational constraints.
5 Experimental Setup
The evaluation compares static retrievers with tool-using agents under a shared top-10 ranked-output contract across visual and OCR-text routes. Agents can search iteratively, inspect pages and regions, and receive a fixed interaction budget, with a controlled support-context intervention.
- Evaluation Protocol: 2,375 corpus units and one gold target per query are evaluated using ten distinct opaque identifiers under a shared ranked-output contract.Systems are scored with Recall@k for k ∈ {1, 3, 5, 10} and MRR@10.
- Evaluation Protocol: R@1 is the primary metric, R@10 measures top-10 coverage, and MRR@10 summarizes target ordering within the submitted list.Results are reported both overall and by evidence-path level.
- Static Retrieval: Static baselines rank page images or OCR records by similarity, with BM25 and reciprocal-rank fusion added for OCR retrieval.A token-level late-interaction model provides a stronger visual-document baseline, using published defaults without benchmark tuning.
- Tool-Using Retrieval: Both agent routes support multiple searches, while visual retrieval adds visual search and on-demand OCR and OCR-text retrieval uses text search and cached OCR.Both routes expose full-page inspection, batched regional cropping, and ranked submission.
- Planner Models and Interaction Budget: Agents receive up to 12 interaction steps and, when needed, a ranking-only finalization call under the same top-10 contract as static baselines.The study evaluates multiple closed and open-weight planner models and ablates iterative search and page inspection for GPT-5.6-sol.
- Controlled Evidence Intervention: A controlled intervention supplies every annotated support page as an unlabeled initial observation while keeping the query, corpus, tools, and 12-step budget fixed.This creates a paired comparison with the standard agent run for L2 and L3 queries.
6 Results and Analysis
The benchmark shows that agentic interaction improves visually rich document retrieval beyond static matching, but performance remains shaped by planner, representation, evidence-path, and verification choices. Visual retrieval consistently outperforms OCR-text retrieval, while iterative search, page inspection, and complete support context provide distinct benefits.
- Main Results: 97.50% visual R@1 on direct items falls to 2.50% on two-bridge items, exposing static matching’s evidence-path gap.The strongest static retriever reaches 40.00% visual R@1 overall, while BM25 reaches 6.67% static R@1 on OCR-text.
- Main Results: 67.50% visual R@1 versus 37.50% OCR-text R@1 marks the strongest agentic performance, with planner choice producing wide variation.Across planners, visual R@1 ranges from 19.17% to 67.50%, whereas OCR-text R@1 ranges from 1.67% to 37.50%; Claude Opus 5 leads R@1 on both routes.
- Main Results: Every planner performs better with visual retrieval, whose advantage ranges from 12.50 to 33.33 R@1 points over OCR-text.The evidence suggests that visual representations preserve layout and structured evidence during global search, while targeted text extraction remains useful for verifying selected pages.
- Main Results: Longer evidence paths remain difficult: GPT-5.6-sol visual R@1 declines from 85.00% to 40.00%, and OCR-text R@1 from 42.50% to 27.50%.Claude Opus 5 shows the same pattern, declining from 92.50% to 47.50% on the visual route.
- Ablation Studies: Removing iterative search lowers R@1 by 8.34 visual points and 9.17 OCR-text points, while removing page inspection causes 15.84- and 21.67-point drops.The fixed candidate set in the no-iterative-search ablation prevents observations from guiding subsequent retrieval; page inspection is identified as the key verification step.
- Evidence Intervention and Trace Diagnostics: Complete support context improves every reported metric across both routes and path depths, with MRR@10 gains of 9.28–12.50 points and a 22.50-point visual L3 R@10 gain.Trace analysis localizes route differences to discovery and examination: visual search discovers 83.3% of targets versus 58.3% for OCR-text, and agents examine 85.0% versus 72.9%.
7 Conclusion
VisDocAgentBench evaluates visually rich document retrieval under static and agentic paradigms using relation-preserving evidence paths and full-document audits. Its results show that iterative, modality-preserving retrieval helps, but planner variation and evidence integration remain decisive.
- The benchmark compares static and agentic retrieval under a shared ranked-output contract, using relation-preserving evidence paths and full-document audits.
- A strong late-interaction retriever is nearly perfect on direct items yet rarely solves two-bridge queries, revealing the difficulty of distributed evidence.
- Iterative search improves retrieval, but wide planner variation shows that tool access alone does not yield reliable retrieval.
- Visual search supports corpus-wide discovery, while page inspection and targeted text access provide complementary verification.
- Providing complete support context further improves ranking, while remaining failures involve target discovery, candidate examination, and evidence-role integration.
Limitations
The benchmark’s controlled English-language scientific-paper setting supports reproducibility and full-document verification, but narrows the scope of its claims. Its fixed corpus, page-level targets, and limited topic subsets leave broader transfer and finer-grained analysis untested.
- The benchmark uses English-language scientific papers, which may limit transfer to forms, manuals, slides, interface captures, and multilingual documents.
- Fixed corpus membership and page-level targets leave collection updates and finer retrieval granularities untested.
- Small topic and cross-document subsets allow only descriptive analysis of those settings.
- These choices enable rigorous path and hard-negative auditing while narrowing the scope of the claims.
A Benchmark Specification
The benchmark defines visually grounded retrieval around a unique target page, explicit evidence paths, and hidden construction annotations. It standardizes query exposure, corpus access, validation, document selection, and final balancing across 120 items.
- Task formulation: Each query targets one rendered page and records intermediate evidence pages when its relation-preserving path is bridged.
- Evidence-path structure: Direct items have no evidence pages, while bridged items require corpus context connecting intermediate evidence to the target.
- Evidence-path structure: The construction vocabulary includes query anchors, semantic bridges, visual target descriptors, directed relations, and verified semantic alignments.
- Evidence-path structure: The three evidence levels are determined by the number of relation-bearing evidence units and their path template.
- Query constraints: Queries expose natural-language relations and a verifiable visual phrase while withholding bridge identities, titles, page numbers, captions, target identifiers, and construction terminology.
- Inference-time visibility: Standard episodes provide only the final query and an opaque image-handle space; evidence-path level and topic remain hidden until post-prediction analysis.
- Corpus construction: The corpus comprises 2,375 rendered pages from 100 documents selected through topic-specific search, metadata scoring, licensing records, and rendering and text-extraction procedures.
- Validation and selection: The construction funnel reduced 3,328 composed paths to 1,522 automatically validated paths before full-document review and final balanced selection.
D.4 Ablation Configurations
The ablation control fixes candidate pages and verification resources to test whether iterative observations redirect retrieval. Resource matching makes the comparison focus on search adaptivity rather than substantially different exposure or call budgets.
- Ablation configurations: The no-iterative-search control fixes candidates to the 30 highest-ranked pages and aggregates judgments from ten three-page batches.
- Ablation rationale: The comparison tests whether intermediate observations can redirect subsequent retrieval while holding the verification budget comparable.
- Resource matching: The fixed pipeline uses 30 page observations and 11 model calls, matching or exceeding the full agent’s average resources on both routes.
- Execution environment: Visual and OCR-text routes run on separate NVIDIA A100-SXM4-80GB devices with cached rendering, OCR, and embedding indices.
E.1 Level-Wise Main Results
Level-wise results show that longer evidence paths sharply weaken static query–page matching, while tool-using agents recover some performance through iterative search and inspection. Visual retrieval consistently outperforms OCR-text, but cross-document evidence remains difficult.
- Main results: Late interaction is nearly perfect on direct items but drops sharply on one- and two-bridge evidence paths, while tool-using agents provide the strongest results on longer paths.The benchmark distinguishes direct query–page matching from retrieval requiring one or two semantic bridges.
- Topic-level results: Visual retrieval matches or exceeds OCR-text in every GPT-5.6-sol topic at R@1, while OCR-text ranges from 63.64% on T10 to 7.69% on T8.The T8 result is associated with spatial diagrams and view-dependent layouts that OCR text alone represents poorly.
- Ablations: Iterative search and page inspection generally improve both routes, with inspection producing the largest losses when removed.Text access benefits L1 and L2 but has mixed effects on L3, while cropping has no stable effect across routes or levels.
- Main results: Visual R@1 decreases from 85.00% on single-unit paths to 52.70% on same-document paths and 16.67% on cross-document paths.OCR-text R@1 decreases from 42.50% to 36.49% and 0.00%, respectively.
F.2 Two Audited Level-3 Trajectories
The audited trajectories contrast successful evidence-directed retrieval with a failed episode caused by source anchoring. Inspection and follow-up search can transform partial evidence into a grounded rank-one decision, but failure to follow a cross-document relation leaves relevant pages undiscovered or unexamined.
- Successful trajectory: The successful trajectory uses inspection to diagnose incomplete evidence, then retrieves the remaining path pages and verifies the target before ranking it first.Subsequent OCR resolves the broader family and verifies its namesake error on the target page.
- Failed trajectory: The failed trajectory increasingly treats DragOn as the destination, although Support 2 and the target appear only at ranks 25 and 34 and remain unexamined.The final submission instead ranks a DragOn prose page first and omits the requested grouped example strip.
- Failed trajectory: The failure reflects mistaken source anchoring rather than successful relation-following to AndroidWorld-Generalization.The audit localizes the error to relation-following and candidate selection before visual verification.