PaperlayerView exact revision on arXiv

Source-linked AI summary

Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

Łukasz Borchmann, Jordy Van Landeghem, Michał Turski, Shreyansh Padarha, Ryan Othniel Kearns, Adam Mahdi, Niels Rogge, Clémentine Fourrier, Siwei Han, Huaxiu Yao, Artemis Llabrés, Yiming Xu, Dimosthenis Karatzas, Hao Zhang, Anupam Datta

arXiv:2603.12180v2cs.CLcs.AI

TL;DR

MADQA evaluates multimodal agents on complex, multi-stage retrieval and reasoning over heterogeneous PDFs, where standard retrieval assumptions are insufficient. The benchmark and accuracy-effort analysis show that strong agents can perform well but may process information inefficiently, motivating more calibrated reasoning and improved exploration policies.

  • Problem

    Existing benchmarks often lack the complexity needed to evaluate planning and multi-step retrieval in realistic document automation workflows.

  • Method

    The paper introduces MADQA for multi-stage document retrieval and reasoning, constructs principled dataset splits, and uses extractive LLM-based Accuracy to assess grounded answers.

  • Results

    The best-performing system achieves 82.2% accuracy versus 78.6% for its optimized non-agentic counterpart, while unconstrained reasoning can process over 270 million input tokens at a cost of $850 without matching its BM25 counterpart.

  • Takeaways & Limitations

    The findings motivate calibrated, efficient reasoning and suggest episodic memory and reinforcement learning with search-tool feedback as directions for improving exploration policies.

  • Takeaways & Limitations

    MADQA is limited to English-language documents and is predominantly U.S.-centric, so performance may not transfer to non-English settings or regions with distinct document practices.

Abstract

from arXiv · show

Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.

1. Introduction

MADQA is designed to evaluate agentic document reasoning over heterogeneous PDFs, combining visual understanding, multi-step retrieval, and grounded evidence aggregation. Its benchmark and evaluation framework address limitations in prior document QA by testing planning, multi-hop reasoning, and efficiency.

  • Benchmark motivation: MADQA evaluates multimodal agents on complex, multi-stage retrieval and reasoning tasks over heterogeneous documents.The benchmark targets workflows resembling enterprise document automation.
  • Benchmark design: MADQA addresses prior benchmark gaps by combining heterogeneous PDF layouts, broad scope, multi-step reasoning, fresh documents, and fully human-authored questions.Prior benchmarks often use plain text, narrow domains, single-step metrics, generated questions, or recycled documents.
  • Task formulation: The task requires answers grounded in a minimal evidence set, with extractive, closed-world, multi-hop, agentic, and visual-answering properties.Evidence may span disjoint pages or documents, and a single retrieval query should not suffice to retrieve the complete evidence set.
  • Task formulation: The agentic formulation necessitates decomposing queries, iterating on intermediate findings, and synthesizing partial answers across retrieved evidence.These requirements distinguish the benchmark from single-step retrieval and standard document QA.
  • Benchmark design: The benchmark contains 2,250 questions over 800 heterogeneous, fresh PDFs and uses Classical Test Theory to create a discriminative, efficient test split.The split preserves strong rank correlation with the full benchmark while reserving difficult items for long-term relevance.
  • Evaluation and comparison: The evaluation protocol measures answer correctness, evidence attribution, and effort calibration, enabling comparisons among static RAG, unconstrained RLMs, tool-augmented agents, and humans.The study specifically examines accuracy, search efficiency, and differences in human–agent research behavior.

2. Dataset Construction and Validity

MADQA is a rigorously constructed benchmark of heterogeneous PDFs and human-authored questions designed to test multi-hop, visually grounded document reasoning. Its validation checks show that questions require semantic retrieval, corpus grounding, and visual understanding rather than simple lexical matching or parametric guessing.

  • Corpus construction: MADQA contains 800 PDFs spanning 63 fine-grained categories across 13 high-level domains, from single-page summaries to 800+ page filings.The corpus was manually curated from related-document clusters to support realistic cross-document questions.
  • Question annotation: 17.3% of the 2,250 questions are multi-hop: 8.3% require multiple pages within one document, while 9.0% require cross-document reasoning.This distribution closely matches the approximately 20% multi-hop target.
  • Construct validity: Unigram retrieval finds approximately 4k pages per query at 0.03% precision, whereas trigram retrieval reaches approximately 1 page but only 51% recall.The contrast supports the need for semantic understanding beyond lexical overlap.
  • Construct validity: Parametric-only answer guessing averages 11.2% across six frontier models, with 3% attributable to chance and 8% to memorized public facts.The benchmark therefore distinguishes document-grounded answering from unsupported guessing.
  • Visual validity: 58% of questions benefit from structured layouts, tabular data, or visual artifacts, while only 42% can be answered from free text alone.Some questions require interpreting relationships among forms, tables, figures, or other visual elements.
  • Principled splits: Classical Test Theory produces a 500-item test set and 200-item development set while reserving 100 unsolved test items to preserve headroom.The test set correlates with the complete benchmark at Spearman’s ρ > 0.85.

3. Evaluation Protocol

The evaluation protocol measures answer correctness, evidence grounding, and whether agents allocate effort appropriately. It combines calibrated answer judging and page/document attribution with a Kuiper-based accuracy–effort analysis.

  • Answer correctness: LLM-based Accuracy accepts semantically correct answers while requiring concrete, corpus-grounded responses suitable for downstream automation.Human calibration produced quadratic-weighted Cohen’s κ = 0.88 after excluding exact-match cases.
  • Evidence attribution: Page F1 measures overlap between agent-cited pages and the minimal human evidence set, penalizing both missing necessary pages and irrelevant citations.Doc F1 relaxes attribution to the document level, allowing diagnosis of last-mile page-navigation failures.
  • Efficiency and calibration: The protocol uses discrete tool-call steps as an effort proxy and evaluates whether accuracy changes systematically with increasing effort.Step counts and token-based effort measures correlate at Spearman ρ > 0.85, so step counts are reported by default.
  • Efficiency and calibration: The Kuiper statistic measures the total range of cumulative accuracy deviation across effort levels, with lower values indicating more stable effort-invariant performance.High values identify regimes where agents spend substantial effort on difficult queries they ultimately fail to solve.
  • Main evaluation: Table 3 compares Accuracy, X-Page and X-Doc performance, Page F1, Doc F1, and Kuiper calibration across agentic and non-agentic systems.Kuiper is excluded for non-agentic systems because they use fixed computational budgets.

4. Baseline Approches

The benchmark compares multimodal, semantic, visual, collaborative, recursive, and managed-retrieval systems alongside human searchers. Human trajectories and oracle retrieval provide reference points for separating search efficiency from retrieval limitations.

  • Agentic baselines: The BM25 MLLM Agent iteratively formulates text queries, retrieves pages, and analyzes rendered page images with an MLLM.It combines textual retrieval with visual reasoning in an agentic loop.
  • Agentic baselines: The Claude Agent with Semtools uses PDF parsing, semantic search, bash commands, and query reformulation for flexible document exploration.Its CLI-based tools support iterative retrieval and exploration.
  • Collaborative baselines: MDocAgent coordinates parallel text and image retrieval through specialized agents before consolidating findings with a summarizing agent.Its fixed pipeline includes General, Critical, Text, and Image agents.
  • Visual retrieval baselines: M3DocRAG encodes document pages as images for vision-aware retrieval, while HEAVEN uses multi-vector retrieval followed by ColQwen2.5 reranking.Both pass retrieved pages to an MLLM for final answering.
  • Non-agentic baselines: ColBERTv2 + LLaMA provides a text-only late-interaction baseline by retrieving relevant pages and passing them to Llama 3 8B.Managed RAG services provide an additional black-box retrieval reference point.
  • Human baselines: Human annotators use the same search engine as the BM25 baseline, with complete queries, page views, and timestamps logged for accuracy and efficiency comparisons.A separate oracle-retrieval condition removes imperfect search from the human results.

5. Results and Analysis

Results show that constrained agentic systems can outperform static retrieval and reach human-level accuracy, but accuracy, calibration, attribution, and failure modes remain distinct. Agents still trail oracle-assisted humans and often spend effort inefficiently, especially when retrieval or reformulation fails.

  • Accuracy and Cost: 82.2% accuracy makes Gemini 3 Pro BM25 MLLM Agent outperform Gemini 3 Pro File Search at 78.6%.The improvement over static RAG is reported across GPT-family models, with Gemini 2.5 Flash as the exception.
  • Accuracy and Cost: 270 million input tokens and $850 of cost did not let Claude Sonnet 4.5 RLM match its constrained BM25 MLLM counterpart.The comparison illustrates the efficiency penalty of unconstrained information processing without corresponding performance gains.
  • Calibration and Attribution: 77.7% accuracy with Kuiper 52.6 for GPT-5 BM25 Agent contrasts with 82.2% and Kuiper 25.8 for Gemini 3 Pro BM25 Agent.Kuiper therefore captures an effort-calibration axis that is not monotonic with answer accuracy.
  • Failure Modes: 35.7% of 3,273 agent errors are retrieval failures, 28.8% comprehension failures, 23.0% navigation failures, and 12.6% refusals.Failure profiles vary by model: stronger systems shift toward comprehension errors, while weaker systems show more refusals and retrieval failures.
  • Failure Modes: 38 percentage points of accuracy are lost when multi-hop evidence spans conceptually dissimilar contexts, regardless of physical page distance.Successful agents also reformulate queries more aggressively after failure; Claude Sonnet 4.5 averages drift 0.38 versus GPT-4.1 Nano at 0.10.
  • Human-Agent Comparison: Humans with oracle retrieval perform at least 18% better than frontier agents, while humans and Gemini 3 Pro using BM25 achieve comparable performance.The result places the remaining benchmark headroom primarily at the retrieval bottleneck.
  • Human-Agent Comparison: Humans and Gemini 3 Pro both reach approximately 82% accuracy but agree on only κ = 0.24 of item outcomes.Humans also achieve Kuiper 14.6, below every agent system, whose scores range from 22.9 to 73.2.

6. Conclusion

The conclusion argues that frontier agents can answer difficult document-grounded questions but often spend effort without recognizing when further exploration helps. MADQA and its open implementations support evaluating more efficient strategies, including memory and search-feedback learning.

  • Conclusion: Frontier MLLM agents expend substantial effort without reliably recognizing whether additional exploration is beneficial.This motivates the paper’s shift from brute-force retrieval toward calibrated, efficient reasoning.
  • Conclusion: MADQA, open-source agentic baselines, and architectural toggles support evaluation of diverse strategies beyond those explicitly studied.The release is intended to help the community evaluate alternative agent designs.
  • Conclusion: The authors identify episodic memory and reinforcement learning with search-tool feedback as two directions for improving exploration policies.Memory could capture corpus-specific terminology and document structure across queries.

7. Limitations

MADQA’s limitations concern representativeness, measurement, evaluation scope, and deployment assumptions. These constraints limit how broadly its scores and conclusions should be interpreted.

  • Coverage and Representativeness: MADQA is English-only, predominantly U.S.-centric, and sourced from public repositories, limiting transfer to other languages, regions, and proprietary enterprise settings.Its corpus may underrepresent private, regulated, or highly domain-specific workflows.
  • Annotation Granularity and Task Scope: Page-level evidence supports scalable annotation but cannot diagnose fine-grained grounding errors within correctly retrieved pages.The benchmark also does not directly evaluate editing, external data entry, long-horizon multi-task planning, or real-world side effects.
  • Evaluation Methodology and Residual Noise: LLM-as-a-judge scoring is calibrated to human judgments but remains vulnerable to residual formatting, ambiguity, and attribution-measurement noise.Answer correctness and citation quality are scored separately, allowing correct answers with incomplete citations and penalizing imperfect localization.
  • Effort as a Proxy for Uncertainty: Step-count effort is system-dependent, so Kuiper scores are most interpretable under consistent tooling and alongside tokens, latency, or dollar cost.A step may bundle different amounts of computation, retrieved context, or latency.
  • Sensitivity of Ranking to Budget: Fixed step budgets can invert model rankings, making leaderboard results asymptotic and potentially unsuitable for real-time constrained inference.The paper recommends budget-aware evaluation curves rather than scalar accuracy alone for such applications.
  • Robustness, Safety, and Responsible Use: MADQA evaluates benign documents and does not establish robustness against prompt injection, deceptive layouts, or malicious formatting in deployed document workflows.Public records may also contain personal or sensitive information requiring appropriate handling.

A.1. Dataset Card

MADQA is a closed-world benchmark for multimodal agents performing multi-step, grounded question answering over heterogeneous PDF collections. It combines human-authored questions, formal task properties, diverse documents, and evaluation choices designed to test navigation, visual comprehension, and evidence grounding.

  • Dataset Summary: MADQA evaluates agentic systems, RAG pipelines, and multimodal models on multi-step reasoning over visually rich PDF collections.The closed-world setup requires navigation, retrieval, and reasoning across multiple pages or documents.
  • Dataset Summary: The benchmark contains 2,250 human-authored questions over 800 heterogeneous, fresh PDF documents and uses Construct Validity to certify benchmark integrity.Its splits preserve difficult questions while supporting leaderboard evaluation, development, and RL-based optimization.
  • Curation Rationale: Questions were designed to bridge simple chat-with-PDF tasks and realistic document automation involving heterogeneous layouts, long contexts, and multi-step retrieval.Documents were manually curated from related clusters across Financial, Legal, Government, Commercial, and Personal domains.
  • Annotation Process: The annotation pipeline used over 1,200 hours of professional work, human-generated questions and answers, restricted external knowledge, and multi-stage verification.The supplied passage describes pilot evaluation, GPT-5 oracle-context checks, and manual review.
  • Task Definition: Agentic Document Collection VQA requires producing an answer and minimal evidence from a corpus of multi-page documents.Its formal properties include extractive answers, multi-hop evidence, closed-world grounding, attribution, iterative retrieval, and visual comprehension.
  • Desired Properties: Multi-hop evidence may span disjoint pages within one document or pages across multiple documents, requiring aggregation rather than single-page lookup.The task explicitly distinguishes cross-page and cross-document evidence.
  • Desired Properties: The closed-world property requires answers to be derived solely from the corpus, with external facts treated as hallucinations.The formal expression excludes parametric knowledge outside the corpus.
  • Desired Properties: The agentic property requires planning, iterative navigation, and aggregation because no single retrieval query should surface all evidence.Visual properties additionally require information encoded in layout, tables, figures, or graphical elements beyond linearized text.

C.3. Sample Questions

MADQA combines heterogeneous document collections with multi-hop, visually grounded questions designed to resist simple retrieval and measure item difficulty and discrimination. Its analyses show substantial lexical mismatch and limited evidence-free guessability, supporting the benchmark’s focus on semantic search and grounded reasoning.

  • Sample Questions: MADQA requires identifying relevant documents among hundreds of distractors and combining handwriting, color-coded tables, and information across disjoint files.Examples include cross-year comparisons, cross-document product tables, dense newspaper layouts, and handwritten financial entries.
  • Item Analysis: Item difficulty is the mean accuracy across models, while item discrimination measures how well an item separates strong from weak models.The item-discrimination calculation uses corrected total scores to avoid autocorrelation.
  • Test Selection: Items with p ≤0.1 form a Sentinel Pool, while remaining items are stratified into nine difficulty bins to balance frontier headroom and discriminative signal.Sentinel items are typically hard and low-variance, making them useful for measuring frontier capability.
  • Lexical Solvability: N-gram analysis finds no effective Ctrl+F strategy: shorter n-grams match too many pages, whereas trigrams miss many gold evidence pages.The analysis searches 18,619 pages and measures matched pages, precision, and recall.
  • Parametric Knowledge vs. Grounding: Approximately 3% of questions are answered correctly by chance alone, while models exceed the 50% random baseline on binary question types.Table 9 separates random contributions from memorization and other correct guesses.
  • Parametric Knowledge vs. Grounding: Public reports, court filings, and annual reports in the benchmark may have entered model training data through web indexing.This motivates measuring evidence-free guessability before attributing performance to document understanding.

E.3. Visual Perception

MADQA’s visual-perception analysis classifies questions by the visual or structural information needed for accurate answering. Most questions require more than unstructured text, with document structure and visual artifacts often carrying essential information.

  • Visual Taxonomy: The taxonomy distinguishes free text, tabular relationships, structured forms, spatial layout, and visual artifacts such as handwriting, checkboxes, and charts.Categories are assigned according to whether visual modality would help answer the question, with secondary categories allowed when multiple modalities are genuinely required.
  • Classification Procedure: The classifier uses question text, gold answers, and rendered gold-evidence images to assign primary and genuinely necessary secondary modalities.Its guidance prioritizes the actual task over keyword presence and asks whether clean OCR would suffice.
  • Visual Taxonomy: 42.8% of questions can be answered from unstructured free text alone, leaving the majority dependent on visual or structural comprehension.Table 10 reports the distribution of primary visual-modality requirements.
  • Multiple Modalities: 87.7% of questions require only one visual modality, while 12.3% benefit from multiple modalities.Structured plus Other Visual is the most common combination at 7.0%, and only 1.5% require both tabular and structured comprehension.
  • Implications: Over 57% of questions require understanding document structure, often through implicit visual layout rather than explicit text.This makes structural inference from linearized text an important alternative to direct visual encoding.
  • Implications: PDF-to-markdown conversion can substitute for end-to-end visual encoding, but it may lose information needed for layout-sensitive questions.The benchmark supports measuring this trade-off between conversion-based and visual approaches.
  • Implications: Providing gold evidence pages isolates reading performance from retrieval, while visual retrieval remains a complementary research direction.Visual document representations may help identify relevant pages missed by text-based retrieval.
  • Category Boundaries: Table questions require row-column correlation, whereas structured-data questions extract values from label-value fields in forms.Spatial layout is reserved for cases where position matters beyond table or form structure.

F. Formal Definition of Metrics

The paper formalizes answer, retrieval, and effort-calibration metrics for document agents. The evaluation combines calibrated semantic accuracy with page- and document-level grounding, then measures whether computational effort tracks performance consistently.

  • F.1. Accuracy Metric: Answer accuracy combines exact or alternative-gold matching with LLM semantic judgment when no exact match is found.The judge evaluates refusal, content, critical errors, format, and verbosity, assigning Correct, Partial, or Incorrect scores.
  • F.1. Accuracy Metric: Human calibration improved human–LLM agreement from 82% to 90% on held-out samples before final sensitivity and specificity measurement.The review targeted list formatting, verbosity, and unit-qualifier edge cases.
  • F.1. Accuracy Metric: The Rogan–Gladen correction uses sensitivity q1 = 0.980 and specificity q0 = 1.000 to adjust raw LLM judgment scores.Confidence intervals include both test-sample variance and calibration uncertainty.
  • F.2. Retrieval and Attribution Metrics: Retrieval quality is measured with F1 at page and document granularity using cited evidence units and minimal gold evidence units.Page F1 penalizes wrong-page citations, while Doc F1 isolates file-level retrieval from intra-document navigation.
  • F.2. Retrieval and Attribution Metrics: If an agent cites nothing, precision and F1 are zero, while every valid benchmark question has at least one gold evidence unit.This evaluation assumes benchmark solvability through nonempty minimal evidence sets.
  • F.3. Efficiency and Calibration Metrics: Effort is the number of discrete steps, outcome is binary correctness, and samples are ordered by ascending effort before cumulative deviations are computed.The Cumulative Difference sequence compares local performance with global mean accuracy.
  • F.4. Sensitivity of Kuiper Statistic to Effort Measure: Effort definitions are robust: steps and token measures correlate at ρ = 0.72 to ρ = 0.95, and Kuiper values vary by at most 20%.These findings apply to the six systems achieving at least 60% accuracy.

G. Baseline Implementation Details

The baseline implementations span iterative search agents, managed retrieval services, semantic CLI tools, and recursive-context approaches. Their analyses also show that multi-hop difficulty depends on semantic relationships and that human performance is imperfect under benchmark cognitive load.

  • BM25 MLLM Agent: The BM25 MLLM agent iteratively searches OCR-indexed pages, receives rendered images, and produces answers with exact file-and-page citations.It supports boolean, phrase, and wildcard queries, with T = 10 maximum iterations and k = 5 results per search.
  • Managed Retrieval: Managed Gemini and OpenAI file-search services provide single-shot or automatic chunk retrieval over the complete PDF collection without task-specific tuning.They establish an out-of-the-box comparison point for commercial RAG systems.
  • Claude Agent with Semtools: The Claude Semtools agent combines PDF-to-Markdown parsing, semantic search, embedding workspaces, and bash-based document exploration.It returns answers, page citations, and search history through structured JSON.
  • Claude Agent with Semtools: Semtools supports flexible iterative exploration but depends on effective bash-pipeline formulation and interpretation of unstructured search results.This dependence is an explicit constraint of the baseline’s operating paradigm.
  • Recursive Language Models: Recursive Language Models handle long contexts by programmatically examining, decomposing, and recursively calling themselves over the document corpus.This differs from fixed chunking and embedding strategies used by conventional retrieval-augmented generation.
  • Same-Document Multi-Hop: 72.4% accuracy on semantically similar same-document evidence falls to 34.8% on dissimilar evidence, while semantic distance predicts difficulty more strongly than page distance.The reported correlations are r = -0.26 for semantic distance and r = -0.06 for physical page distance.
  • Cross-Document Multi-Hop: 75.7% cross-document accuracy exceeds 61.2% same-document accuracy, with weaker dependence on semantic distance across documents.Explicit document boundaries may provide clearer retrieval structure for cross-document comparisons.
  • Human Baseline: Human participants occasionally failed despite identical tools and documents, reflecting genuine cognitive load and benchmark complexity rather than annotation errors.The human baseline therefore captures imperfect performance under the task’s realistic conditions.

H.3. Error Decomposition

Error analysis separates retrieval, navigation, comprehension, and refusal failures, showing that harder evidence structures and iterative search expose distinct bottlenecks across systems.

  • System Profiles: Claude Sonnet 4.5 has 4.0% retrieval failure but 8.6% comprehension failure, indicating effective retrieval alongside answer-interpretation errors.
  • System Profiles: Human searchers have 2.2% retrieval and 4.6% navigation failure, but 9.4% comprehension failure when the correct evidence is available.
  • Capability Trends: As model capability increases, errors shift from refusals to retrieval failures and eventually comprehension failures.The strongest systems are described as largely solving retrieval while answer extraction remains challenging.
  • Evidence Complexity: Cross-page questions raise error rates from ∼13% for single-evidence questions to ∼35%, with retrieval failures forming a larger share.The pattern suggests that navigating within long documents is harder than finding separate documents.
  • Cross-System Agreement: Humans and agents show low agreement despite similar accuracy: Human–agent κ ranges from 0.06–0.24, and Human and Gemini 3 Pro both reach ∼82% accuracy with κ = 0.24.Among 107 disagreements, human-specific failures are 64% comprehension, while model-specific failures split evenly between retrieval and comprehension.
  • Search Trajectories: Humans find the gold document on their first query ∼80% of the time versus ∼70% for Gemini 3 Pro, while recovery after failed first queries varies sharply across systems.Claude Sonnet 4.5 and Gemini 3 Pro recover in >90% of cases, compared with 56% for Gemini 2.5 Pro and 12% for GPT-4.1 Nano.

I. Solution Space

The solution-space analysis compares document-agent designs across efficiency, visual representation, indexing, and memory, identifying trade-offs between constrained control and broader capabilities.

  • Efficiency: Unconstrained Recursive Language Models incur catastrophic computational overhead compared with constrained Iterative architectures such as the BM25 MLLM Agent.The analysis links unconstrained reasoning with diminishing returns in efficiency calibration curves without robust guardrails.
  • Visual Representation: 57.2% of questions require structural or visual comprehension, limiting systems that rely solely on textual representations.The construct-validity analysis supports using full page images or visual crops for tables and forms.
  • Indexing and Scope: Retrieval failures account for 39.1% of all errors, making indexing and scope choices central to benchmark performance.The reported hierarchy favors Late Interaction or hybrid strategies over simple Sparse indexing for high recall.
  • Memory: Episodic memory is identified as a future optimization because stochastic search produces wasteful exploratory loops.The proposed role is learning corpus-specific terminology across queries to reduce repeated exploration.

J.1. Related Works Assessment

The related-work assessment compares document benchmarks by domain diversity, layout diversity, and annotation method, positioning the proposed benchmark as broad, multimodal, and fully human-curated.

  • Assessment Axes: Prior benchmarks are assessed along document diversity, annotation provenance, and task framing, including single-document VQA, document RAG, and agentic research.
  • Assessment Axes: The taxonomy distinguishes domain diversity from single-topic to open-domain, layout diversity by visual structural complexity, and annotation from synthetic to fully human-curated.
  • Prior Benchmarks: MMLongBench-Doc provides high domain and layout diversity with expert-annotated questions requiring evidence from multiple pages and modalities.
  • Prior Benchmarks: Several benchmarks combine high domain and layout diversity with mixed annotation, including M2RAG, MMDocIR, and DocBench.These datasets use human labels alongside inherited, generated, bootstrapped, or LLM-assisted components.
  • Our Benchmark: The proposed benchmark uses heterogeneous real-world PDFs with high domain and layout diversity, and its questions and document selection are fully human-curated.
Loading 2603.12180v2…