Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

1. Introduction

• MADQA targets complex, multi-stage retrieval and reasoning over heterogeneous PDFs, requiring agents to iteratively retrieve pages, interpret visual and textual content, and synthesize attributed evidence.The task produces an answer and minimal evidence set from a corpus of multi-page documents.Source: §1 · p. 1, Figure 1 · p. 1, §1.2 · p. 2

1.1. Motivation and Related Works

• Existing benchmarks miss visual comprehension, broad multi-step scope, or unbiased provenance, whereas MADQA combines heterogeneous fresh PDFs, broad domains, and fully human-authored questions.Prior datasets may use HTML or plain text, narrow PDF domains, single-step metrics, generated questions, or recycled documents.Source: §1.1 · pp. 1–2, Table 1 · p. 2

1.2. Guiding Principles

• MADQA defines six properties: extractive, multi-hop, closed-world, minimally grounded, agentic, and visual answering, with design and construction enforcing genuine corpus use.The agentic property requires planning, navigation, and aggregation because no single retrieval query should surface all evidence.Source: §1.2 · pp. 2–3

1.3. Contributions

• The benchmark releases 2,250 human-authored questions over 800 heterogeneous, fresh PDFs and uses Construct Validity to certify benchmark integrity.Classical Test Theory creates a cost-effective test split strongly rank-correlated with the full benchmark while reserving hard items.Source: §1.3 · p. 3
• The evaluation measures correctness, evidence attribution, and effort calibration, comparing humans with static RAG, Recursive Language Models, and tool-augmented agents.The study reports that constrained agency outperforms static RAG while avoiding RLMs’ catastrophic effort overhead, alongside a critical human–agent efficiency gap.Source: §1.3 · p. 3

2.1. Sourcing and Filtering Documents

• The corpus comprises 800 PDFs across 63 fine-grained categories and 13 domains, intentionally grouped to support realistic cross-document multi-hop questions.Documents range from single-page summaries to filings exceeding 800 pages, with domain-specific variation in tables, figures, text, and lists.Source: §2.1 · p. 3

2.2. Question Annotation and Quality Assurance

• MADQA contains 2,250 QA pairs, including 17.3% multi-hop questions, with 8.3% requiring multiple pages and 9.0% requiring cross-document reasoning.The resulting multi-hop share closely matches the approximately 20% target; Table 2 describes a corpus with median 5-page documents, mean 23.3 pages, and maximum 859 pages.Source: §2.2 · p. 4, Table 2 · p. 4

2.3. Construct Validity Analysis

• Lexical matching is insufficient: trigrams retrieve approximately 1 page with 51% recall, while unigram precision is only 0.03%.Unigram matching returns a median of approximately 4k pages per query, and bigram matching returns approximately 24 pages at 2.6% precision.Source: §2.3 · p. 5
• Models guessing from question text achieve 9.1%–15.2% accuracy, averaging 11.2%, whereas 80%+ accuracy with document evidence indicates genuine comprehension.Approximately 3% reflects chance on yes/no or binary-choice questions, while the remaining 8% reflects recalled facts from public documents.Source: §2.3 · p. 5
• Only 42% of questions are answerable from free text alone, while 15% involve visual artifacts and 7% require both artifacts and structured layout.Overall, 58% benefit from understanding structured layouts, tabular data, or visual artifacts, requiring relationships between document elements to be considered.Source: §2.3 · p. 5, Figure 4 · p. 5

2.4. Principled Splits Creation

• Classical Test Theory yields non-overlapping Test, Development, and Train sets of 500, 200, and 1,550 items, with Test rank correlation above 0.85 and 100 Sentinel items.The Sentinel Pool reserves items unsolved by current models, while remaining questions are selected using difficulty and discrimination to preserve statistical power and headroom.Source: §2.4 · p. 5

3.1. Answer Correctness (Property: Extractive)

• LLM-based Accuracy accepts semantically correct, corpus-grounded concrete answers, achieving quadratic-weighted Cohen’s κ = 0.88 with human judgments.Answers are represented as string lists, and calibration addressed list formatting, verbosity tolerance, and unit qualifiers; ∼50% exact-match cases were excluded.Source: §3.1 · p. 6

3.2. Retrieval and Attribution (Property: Grounded)

• Page F1 measures overlap with the minimal evidence pages, while Doc F1 relaxes evaluation to documents to diagnose last-mile navigation failures.Page-level scoring penalizes missing necessary pages and irrelevant citations, and annotations are made at page rather than bounding-box level.Source: §3.2 · p. 6

3.3. Efficiency and Calibration (Property: Agentic)

• Kuiper range quantifies effort–accuracy dependence: high values reveal poor calibration when agents spend substantial budget on queries they ultimately fail to solve.Step counts are reported by default because step- and token-based measures correlate strongly (Spearman ρ > 0.85), with Kuiper values varying by less than 20%.Source: §3.3 · p. 6, Figure 6 · p. 6
• 18% oracle gap remains, while agentic systems consistently outperform static RAG counterparts, indicating retrieval—not reasoning—remains the primary bottleneck.Table 3 reports Accuracy, X-Page and X-Doc performance, Page F1 and Doc F1, and Kuiper calibration across MADQA systems.Source: Table 3 · p. 7

Proposed agents

• The BM25 MLLM Agent iteratively combines text-based search with rendered-page visual reasoning, allowing the MLLM to formulate queries and analyze retrieved pages.Its interaction design supports direct comparison with humans using the same search engine.Source: §4 · pp. 7–8
• The Claude Agent with Semtools uses PDF-to-Markdown parsing, semantic search, and Unix-style commands to support flexible query reformulation and document exploration.The system combines Claude Agents SDK tools with regular bash commands.Source: §4 · p. 7

Related baselines

• Recursive Language Models programmatically expose document collections to LLM-written code that recursively processes them through sub-LLM calls.This task-agnostic approach performs context decomposition through programmatic examination.Source: §4 · p. 7
• MDocAgent coordinates General, Critical, Text, Image, and Summarizing agents in a fixed five-stage pipeline using parallel textual and visual retrieval (Han et al., 2025).Other retrieval baselines include visual page encoding in M3DocRAG (Cho et al., 2024), hybrid multi-vector retrieval in HEAVEN (Kim et al., 2025), managed RAG services, and ColBERTv2 + LLaMA late interaction.Source: §4 · pp. 7–8

Human performance

• Human baselines cover all test questions, with logged queries, page views, and timestamps enabling accuracy and search-efficiency comparisons against agents.A separate oracle-retrieval condition removes imperfect-search effects and reduces task cognitive load.Source: §4 · p. 8

5. Results and Analysis

• 82.2% accuracy from Gemini 3 Pro BM25 MLLM Agent exceeds Gemini 3 Pro File Search at 78.6%, while M3DocRAG and MDocAgent surpass 60%.These specialized 8B systems outperform ColBERTv2 + Llama 3 at 40.2% and Qwen3-VL 8B BM25 MLLM Agent at 47.3%.Source: §5 · p. 8
• Over 270 million input tokens and $850 cost did not let Claude Sonnet 4.5 RLM match its BM25 MLLM counterpart, while agents provide stronger page-level attribution than managed RAG.Managed RAG can achieve high Doc F1 while masking lower Page F1; Kuiper also separates calibration from accuracy, with Gemini 3 Pro at 82.2% and 25.8 versus GPT-5 at 77.7% and 52.6.Source: §5 · p. 8, Figure 7 · p. 8

5.1. Search Dynamics and Error Taxonomy

• Retrieval failures comprise 35.7% of 3,273 errors, comprehension 28.8%, navigation 23.0%, and refusals 12.6%, with error profiles varying sharply across models.Claude Sonnet 4.5 has 4.0% retrieval and 8.6% comprehension failures, whereas GPT-4.1 Nano has 48.2% refusals; stronger agents also reformulate queries more aggressively, and semantic discontinuity reduces accuracy by 38 percentage points.Source: §5.1 · pp. 8–9
• Only 5% of correct predictions are over-verbose, while 87.4% of errors involve retrieval, navigation, or comprehension failures, indicating that the benchmark primarily tests information-seeking ability.This analysis concerns the BM25 MLLM Agent family, which contains the most evaluated models.Source: §5.1 · pp. 8–9

5.2. Human-Agent Comparative Analysis

• Humans with oracle retrieval outperform frontier agents by at least 18%, while humans and Gemini 3 Pro both reach approximately 82% accuracy but agree on only κ = 0.24 of items.Humans solve 54 disagreement items uniquely and the model 53; human-specific failures are 64% comprehension, whereas model-specific failures split 43% retrieval and 43% comprehension.Source: §5.2 · pp. 9–10
• Humans calibrate effort better, with Kuiper 14.6 versus agent scores from 22.9 to 73.2, and accuracy falls from 86% under one minute to 68% over ten minutes.Perfect retrieval nearly eliminates human negation, temporal, and role-confusion errors; humans also change strategy faster after failed initial queries.Source: §5.2 · p. 10, Figure 9 · p. 10

• Frontier MLLM agents expend substantial effort without reliably recognizing whether additional exploration is beneficial, even when they answer challenging document-grounded questions.Source: §6 · p. 10
• The authors release MADQA and open-source implementations of strong agentic baselines to support calibrated, efficient reasoning and evaluation of diverse strategies.Appendix I also characterizes a broad design space of architectural “toggles.”Source: §6 · p. 10
• Episodic memory could help agents learn corpus-specific terminology and document structure across queries.Source: §6 · p. 10
• Reinforcement learning with search tool feedback could significantly improve agents’ exploration policies.Source: §6 · p. 10
• Future evaluation campaigns will target new bottlenecks so the benchmark remains a discriminative signal for frontier capabilities.Source: §6 · p. 10

7. Limitations

• MADQA is predominantly U.S.-centric and English-language, while public repositories may underrepresent private, proprietary, and highly specialized enterprise documents.Performance may not transfer to regions, languages, document genres, layouts, or administrative practices absent from the corpus.Source: §7 · p. 11
• Public documents may overlap with model pretraining data, so reported scores mix document-grounded competence with limited prior exposure that may advantage some model families.Answer-guessability analysis bounds but cannot eliminate memorization effects.Source: §7 · p. 11
• Page-level evidence supports scalable annotation but cannot diagnose fine-grained grounding failures, and attributed question answering omits planning, editing, data entry, and end-to-end automation.The benchmark may miss errors such as extracting the wrong field from a correctly retrieved page.Source: §7 · p. 11
• LLM-judge scoring remains imperfect, while separate answer and attribution metrics can reward correct answers with noisy citations or penalize imperfect localization.Rare formatting edge cases and ambiguous surface forms can create residual measurement error without systematic human verification of every prediction.Source: §7 · p. 11
• Effort measured as step count is system-dependent, so Kuiper scores are most interpretable under consistent tooling and should be paired with tokens, latency, or dollar cost.A step may bundle different computation, retrieved context, or latency across interfaces.Source: §7 · p. 11
• Model rankings depend on the step budget: Gemini 3 Pro dominates at convergence but underperforms humans and Claude Sonnet 4.5 when N ≤3.The benchmark therefore reflects asymptotic performance under a typically N = 10 budget, which may not predict real-time constrained inference.Source: §7 · p. 11
• MADQA does not test document-based attacks or establish deployment security, and its public records may contain sensitive information requiring responsible handling.It omits adversarial prompt injection, deceptive layouts, and malicious formatting; users should follow policies for redaction and redistribution.Source: §7 · p. 11

A.1. Dataset Card

• The benchmark evaluates systems that navigate, retrieve, reason over, and aggregate information from heterogeneous, visually rich PDF collections under a closed-world assumption.Its corpus spans related document clusters and domains including Financial, Legal, Government, Commercial, and Personal records, with questions grounded solely in the provided PDFs.Source: §A.1 · p. 16, §A.2 · p. 17
• The dataset contains 500 test, 200 development, and approximately 1,550 training samples, with Classical Test Theory preserving difficult, discriminative test questions.Over 1,200 hours of professional annotation produced fully human-generated, PDF-grounded questions and answers, followed by automated verification, expert review, and corrections.Source: §A.1 · p. 16

A.2. Desired Properties

• MADQA formalizes answers with minimal page-level evidence sets and requires extractive, multi-hop, grounded, agentic, and visual reasoning rather than single-pass text retrieval.Questions may require cross-page or cross-document aggregation, while necessary information can depend on layout, tables, figures, or other non-textual structure.Source: §A.2 · pp. 17–18

A.3. Benchmark Design Decisions

• Page-level evidence annotations were chosen because pilot agreement was high and the granularity matches human navigation and retrieval-system operation without adding finer-localization noise.The benchmark’s design space emphasizes strict PDF grounding, multi-hop reasoning, and efficiency-aware evaluation.Source: §A.3 · pp. 18–19, Table 5 · p. 18

B.1. Domains and Categories

• The corpus uses real-world DocumentCloud PDFs with substantial variation in layout, OCR quality, visual density, document length, and category.This variability ranges from single-page posters to an 859-page expense document and is intended to test robustness across diverse document structures.Source: §B.1 · p. 20, Table 6 · p. 20

B.2. Layout Element Density Heatmap

• Layout heterogeneity is quantified by extracting page elements with Granite-Docling and comparing per-category densities using element-wise z-score normalization.The resulting heatmap distinguishes below-, average-, and above-mean densities, with categories differing in picture, table, and text-block prevalence.Source: §B.2 · p. 21, Figure 11 · pp. 22–23

C.1. Annotation Guidelines

• Annotation guidelines require document-dependent, unambiguous questions answerable from PDF text or visuals, with minimal evidence pages recorded for each answer.Approximately 20% of questions require multiple evidence pieces involving bridging, comparison, or common-property reasoning.Source: §C.1 · p. 24

C.2. Human Baseline Collection

• Human baselines answer all 500 test questions through the same keyword-based retrieval interface as agents, enabling controlled comparisons of accuracy, search strategy, and efficiency.The system logs queries, retrieved pages, navigation, timestamps, and selected evidence, supporting analyses of search counts, time, retrieval precision, and reformulation behavior.Source: §C.2 · pp. 24–25

C.3. Sample Questions

• MADQA questions require locating relevant documents among distractors, switching modalities, and aggregating evidence across disjoint files.Examples involve color-coded tables, checkboxes across years, product brochures, dense newspaper layouts, and handwritten financial statements.Source: §C.3 · pp. 26–27

D. Principled Splits Creation

• The CTT split divides 2,250 questions into 500 test and 200 development items, using predictions from 16 diverse models to avoid architecture-specific bias.The remaining rejected questions form the training set.Source: D. Principled Splits Creation · p. 28, Table 7 · p. 28
• Item selection balances difficulty and discrimination by reserving p ≤0.1 items as a Sentinel Pool and filling the remaining budget from nine difficulty bins using highest rpbis.The strategy increases the density of hard and sentinel items in the selected test subset to prevent saturation.Source: D. Principled Splits Creation · p. 28, Figure 13 · p. 28

E.1. Lexical Overlap vs. Reasoning

• No n-gram size enables effective Ctrl+F retrieval: unigrams and bigrams have low precision, while trigrams have low recall.The analysis searches 18,619 corpus pages and measures matched pages, precision, and recall, confirming that semantic understanding is required to bridge vocabulary gaps.Source: Table 8 · p. 29, §E.1 · p. 29

E.2. Parametric Knowledge vs. Grounding

• Approximately 3% of questions are answered correctly by chance, while constrained-answer accuracy above 50% indicates additional knowledge from training data.The constrained categories are yes/no and other binary-choice questions, each with a 50% random baseline.Source: §E.2 · p. 29, Table 9 · p. 29
• Models exhibit approximately 8% total knowledge-based answering, with GPT-5 reaching 11.8% contamination after subtracting its 3.4% random component from 15.2% total guessability.Thus, model-dependent guessability ranges from 9–15% and represents an upper bound on performance obtainable without document understanding.Source: §E.2 · p. 30
• The guessability analysis indicates that public government reports, court filings, and annual reports were likely indexed online and incorporated into model training data.Correctly guessed examples include specific emissions, establishment dates, official names, fees, and acronym expansions.Source: §E.2 · p. 30

E.3. Visual Perception

• The taxonomy distinguishes Free Text, Tabular, Structured, Other Spatial, and Artifacts requirements, with mutually exclusive primary assignments and optional secondary categories.Tabular questions require row-column relationships, Structured questions require form fields, Other Spatial questions require positional understanding, and Artifacts cover handwriting, signatures, checkboxes, stamps, charts, and images.Source: §E.3 · p. 31
• 42.8% of questions can be answered from unstructured free text alone, while the majority require visual or structural comprehension.Table 10 reports the distribution of visual modality requirements across MADQA questions.Source: Table 10 · p. 32, §E.3 · p. 32

Question Modality Classifier Prompt

• The prompt uses the question, answer, and rendered gold evidence-page images, assigning one primary category in most cases and additional categories only when modalities genuinely combine.Its guidance prioritizes table structure, then structured form extraction, and adds spatial layout only when position matters beyond table or form structure.Source: §E.3 · p. 32, Question Modality Classifier Prompt · p. 34

E.3. Visual Perception

• 87.7% of questions require one visual modality, whereas 12.3% benefit from multiple; Structured + Other Visual is the most common combination at 7.0%.Only 1.5% require both Tabular and Structured modalities, indicating distinct table and form challenges.Source: §E.3 · p. 32
• Over 57% of questions require understanding document structure, often through implicit visual layout rather than explicit text.The analysis therefore treats text as structured and requires systems to comprehend layout or infer relationships from linearized text.Source: §E.3 · p. 32
• The analysis isolates reading by supplying gold evidence pages, leaving visual effects on retrieval accuracy as a complementary research direction.PDF-to-markdown conversion may substitute for end-to-end visual encoding but can lose information; the taxonomy complements the text-versus-image ablation in Figure 4.Source: §E.3 · p. 32

Question Modality Classifier Prompt

• The classifier prompt defines eight categories, including table structure, charts or diagrams, text formatting, images, spatial layout, structured data extraction, and other visual artifacts.It distinguishes tables by row-column correlation, forms by label-value fields, and other visual cases by artifacts such as handwriting, signatures, checkboxes, stamps, and seals.Source: Question Modality Classifier Prompt · p. 33

F.1. Accuracy Metric

• Answer accuracy combines exact string matching with LLM semantic judgment, applying the latter when no gold-answer variant matches exactly.The semantic judge follows G-Eval-style chain-of-thought assessment across refusal, content, critical-error, format, and verbosity criteria.Source: §F.1 · p. 35
• The judge assigns Correct = 1.0, Partial = 0.5, or Incorrect = 0.0, with Gemini 2.5 Flash producing structured outputs.Partial answers preserve correct content but have format or verbosity issues, whereas incorrect answers contain wrong content, missing answers, or critical errors.Source: §F.1 · p. 35, LLM Judge Prompt · p. 36
• The calibrated accuracy pipeline improved human-LLM agreement from 82% to 90% on held-out samples and applies Rogan-Gladen bias correction using q1 = 0.980 and q0 = 1.000.Confidence intervals incorporate both test-sample variance and calibration uncertainty.Source: §F.1 · p. 35

F.2. Retrieval and Attribution Metrics

• Retrieval and attribution are measured with Page-level and Document-level F1 over cited evidence units versus the minimal ground-truth evidence set.Page F1 penalizes citing the wrong page, while Doc F1 ignores intra-document navigation errors and their comparison isolates last-mile failures from retrieval failures.Source: §F.2 · p. 35, LLM Judge Prompt · p. 37

LLM Judge Prompt

• The LLM judge evaluates refusal, semantic content matching, critical errors, formatting, and verbosity before returning a structured final judgment.The prompt treats correct content with list-format or irrelevant-addition issues as Partial, while wrong entities, missing scale qualifiers, and invalid answer types are Incorrect.Source: LLM Judge Prompt · p. 36

F.3. Efficiency and Calibration Metrics

• Kuiper calibration orders samples by effort and summarizes cumulative accuracy deviations, with lower K indicating effort-invariant performance and higher K indicating effort-dependent failure regimes.Positive cumulative-difference slopes indicate above-mean local accuracy, while negative slopes commonly occur in high-step regimes where effort does not yield success.Source: §F.3 · p. 37

F.4. Sensitivity of Kuiper Statistic to Effort Measure

• For models achieving ≥60% accuracy, steps and token measures correlate at ρ = 0.72 to ρ = 0.95, largely preserving sample orderings.These correlations are reported in Table 11 across systems.Source: §F.4 · p. 38, Table 11 · p. 38
• Kuiper values vary by at most 20% across effort definitions for six systems, confirming robust calibration assessment.Table 12 compares steps/calls, total tokens, generated tokens, and execution time.Source: §F.4 · p. 38, Table 12 · p. 38

G.1. BM25 MLLM Agent.

• The BM25 MLLM agent iteratively retrieves OCR-indexed pages and receives rendered images for VLM reasoning over layout-sensitive documents.Its search supports Boolean, phrase, and wildcard queries, with T = 10 maximum iterations and k = 5 results per search.Source: §G.1 · p. 39

G.2. Managed RAG Services

• Managed Gemini File Search and OpenAI Assistants File Search provide single-call or automatic vector-store retrieval over the complete collection without task-specific fine-tuning.Both services were prompted to produce structured answers with page-level citations, representing an out-of-the-box comparison point.Source: §G.2 · pp. 39–40

G.3. HEAVEN

• HEAVEN uses the original encoder configurations from Kim et al. (2025), with DocLayout-YOLO for title-region extraction and paper-default hyperparameters.The cited encoders are DSE (MrLight/dse-qwen2-2b-mrl-v1) and ColQwen2.5 (vidore/colqwen2.5-v0.2).Source: §G.3 · p. 40, Table 13 · p. 41

G.4. Claude Agent with Semtools

• The Claude agent with Semtools composes PDF parsing, semantic search, and persistent embedding workspaces while producing answers, page citations, and search histories.This flexible iterative paradigm depends on the agent formulating effective bash pipelines and interpreting unstructured search results.Source: §G.4 · pp. 40–41

G.5. Recursive Language Models

• Recursive Language Models expose the full OCR-preserved corpus in a code-accessible REPL, enabling programmatic chunking and recursive sub-LLM calls before structured citation extraction.The implementation uses RLM4 with explicit [PAGE N] ... [END PAGE N] markers to support source-location recovery.Source: §G.5 · pp. 41–42

Claude Agent CLAUDE.md

• The Claude configuration instructs the agent to use parse, search, and workspace utilities and return short answer lists with essential information and exact document wording.The configuration warns that verbose answers score poorly and requires answer and citation fields, with search history included in the structured output.Source: Claude Agent CLAUDE.md · pp. 43–44

RLM Prompt

• The RLM prompt directs the model to inspect truncated REPL outputs, divide the corpus into smart chunks, query sub-LLMs, buffer their answers, and synthesize a final response.Each sub-LLM query can process approximately 500K characters, or about 10 documents.Source: RLM Prompt · p. 45

H.1. Multi-Hop Question Complexity Analysis

• 72.4% accuracy for semantically similar evidence fell to 34.8% for dissimilar evidence, a 38 percentage point decline; semantic distance predicted difficulty more strongly than physical distance.The correlations were r=-0.26 for semantic distance and r=-0.06 for physical page distance.Source: §H.1 · p. 46, Table 14 · p. 46
• 75.7% cross-document accuracy exceeds 61.2% same-document accuracy, with semantic distance weakly related to performance (r=-0.09).Accuracy ranged from 72.4% to 80.0% across semantic-distance bins, suggesting explicit document boundaries may help structure retrieval.Source: §H.1 · p. 46
• Cross-document multi-hop is easier because explicit comparisons provide clearer structure, whereas same-document reasoning requires subtle integration across distant sections.The comparison is counter-intuitive because cross-document questions can be more structured despite spanning multiple documents.Source: §H.1 · p. 46

H.2. Qualitative Analysis of Human Baseline Errors

• Human participants occasionally failed despite using the same tools and documents as agents, reflecting genuine cognitive-load and benchmark-complexity failures rather than annotation errors.These were observed during establishment of the human baseline and were not caused by data annotation noise.Source: §H.2 · p. 47
• Human errors primarily involved attention to detail, cross-page navigation, and semantic precision, underscoring the benchmark’s reasoning difficulty.The baseline study categorized these recurring failure modes as common human errors.Source: §H.2 · p. 47

H.3. Error Decomposition

• The error cascade deterministically separates retrieval, navigation, comprehension, and refusal failures, using citation metadata and an LLM judge only for the correctness boundary.Retrieval failure means the gold document is absent; navigation failure means the document is found but the gold page is not; comprehension failure occurs after retrieving the correct page.Source: §H.3 · p. 47
• As capability increases, failures shift from refusals to retrieval and finally comprehension, indicating that stronger systems increasingly solve search but still struggle with answer extraction.Weaker models are dominated by no-answer refusals, whereas the strongest models’ remaining errors are primarily comprehension failures.Source: §H.3 · p. 48, Figure 15 · p. 48
• Systems form distinct failure profiles: top models occupy the low-retrieval/low-comprehension region, while mid-tier systems are retrieval-limited or comprehension-limited.Claude Sonnet 4.5 has 4.0% retrieval failure but 8.6% comprehension failure; smaller models show higher retrieval failure, including GLM-4.6V Flash at 36.0%.Source: §H.3 · pp. 47–48, Figure 16 · p. 49

H.4. Cross-System Item Agreement

• Cross-system agreement is moderate within capability tiers but low between humans and agents despite comparable accuracy, showing that they solve fundamentally different benchmark subsets.Human–agent agreement ranges from κ = 0.06–0.24; Human and Gemini 3 Pro both achieve ∼82% accuracy yet share κ = 0.24.Source: §H.4 · p. 50, Figure 20 · p. 52
• The benchmark’s difficulty is mostly discriminating and increases monotonically with capability: 66.8% of items are discriminating, while only 1.0% are anomalously easy for weak models but hard for strong ones.Among 500 items, 23.8% are universally easy and 8.4% universally hard; only 5 items are anomalous.Source: §H.4 · p. 50, Figure 21 · p. 52

H.5. Search Trajectory Analysis

• Humans retrieve evidence earlier, finding the gold document on ∼80% of first queries versus ∼70% for Gemini 3 Pro, while strong agents recover from misses at human-like rates.Claude Sonnet 4.5 and Gemini 3 Pro recover in >90% of failed-first-query cases, compared with 56% for Gemini 2.5 Pro and 12% for GPT-4.1 Nano.Source: §H.5 · p. 51, Figure 22 · p. 53, Figure 23 · p. 53

H.6. Query Reformulation Analysis

• More aggressive query reformulation correlates with higher accuracy: Claude Sonnet 4.5 has median drift 0.38, whereas GPT-4.1 Nano has median drift 0.10.Drift is measured as the mean cosine distance between consecutive embedded queries using Snowflake Arctic Embed.Source: §H.6 · p. 51, Figure 24 · p. 54, Figure 25 · p. 54
• Every system performs better on single-query questions, but GPT-4.1 Nano drops from 46.1% to 13.2% on multi-query questions, a 33 percentage point gap.Human performance declines from 86.4% on single-query questions to 79.5% on multi-query questions.Source: §H.6 · p. 51

I. Solution Space

• Unconstrained programmatic flows such as Recursive Language Models incur catastrophic computational overhead relative to constrained iterative architectures such as the BM25 MLLM Agent.The analysis links this overhead to diminishing returns in efficiency calibration and the need for robust guardrails.Source: §I · p. 55
• 57.2% of questions require structural or visual comprehension, limiting systems that rely only on textual Markdown or JSON representations.Full-page images or visual crops are therefore necessary for tables, forms, and other visually structured content.Source: §I · p. 55
• 39.1% of errors arise from retrieval failures, making indexing and scope central and exposing limitations of simple sparse retrieval.Late-interaction or hybrid strategies such as HEAVEN are positioned as better suited to the benchmark’s high-recall demands.Source: §I · p. 55
• Stochastic search makes memory a critical optimization target, with episodic stores proposed to learn corpus-specific terminology and reduce wasteful exploratory loops.This recommendation addresses the behavioral inefficiency observed in current frontier models.Source: §I · p. 55

J. Landscape of Agentic Approaches to Documents

• Document understanding has progressed from OCR and layout-aware transformers (Xu et al., 2020; Powalski et al., 2021) to OCR-free models and MLLMs (Kim et al., 2022; Lee et al., 2023; Liu et al., 2023a), while evaluation remains mostly single-document.The architectural progression has not yet been matched by broad evaluation of document-collection reasoning.Source: J. Landscape of Agentic Approaches to… · p. 56
• Benchmarks have expanded from single-page documents (Mathew et al., 2021b;a) to longer and massive PDFs (Tito et al., 2023; Chia et al., 2024), but many newer datasets use semi-automated annotation or single-step retrieval (Cho et al., 2024; Faysse et al., 2025).These designs fail to capture the iterative planning required for complex reasoning.Source: J. Landscape of Agentic Approaches to… · p. 56
• Agentic frameworks interleave reasoning, retrieval, and self-correction (Shinn et al., 2023), but prevailing evaluations operate mainly over HTML pages or plain text.Thus, existing agentic benchmarks do not adequately test multimodal reasoning over document collections.Source: J. Landscape of Agentic Approaches to… · p. 56

• The taxonomy evaluates benchmarks along domain diversity, layout diversity, and annotation method, using coarse low, medium, and high categories.Annotation is distinguished as synthetic, hybrid, or fully human-curated, while diversity reflects breadth of subjects and visual structures.Source: §J.1 · p. 56, Figure 26 · p. 56
• Some benchmarks preserve human annotation while restricting scope, including DocVQA, InfographicVQA, SlideVQA, FinRAGBench-V, and BRIGHT.Their limitations or emphasis differ: DocVQA and SlideVQA use narrower document families, FinRAGBench-V is financial-domain focused, and BRIGHT is text-centric retrieval.Source: §J.1 · pp. 56–58
• Prior benchmarks span narrow single-domain settings, broad heterogeneous collections, and mixed annotation pipelines, revealing substantial variation in benchmark scope and curation.Examples include low-diversity TAT-DQA, high-diversity DUDE and DocBench, and mixed-annotation M-LongDoc, MuRAR, ViDoRE, and MMDocIR.Source: §J.1 · pp. 56–58
• Several newer benchmarks provide high domain and layout diversity but rely partly on generated, inherited, or LLM-assisted questions and labels.This pattern applies to M2RAG, ViDoRE, ViDoSeek, and MMDocIR, whereas MMLongBench-Doc retains high diversity with expert-annotated questions.Source: §J.1 · pp. 57–58
• Our benchmark achieves high domain and layout diversity through heterogeneous multi-page PDFs, while questions and document selection are fully human-curated.The collection includes technical reports, standards, slide decks, and research documents, supporting challenging multi-hop, agentic reasoning.Source: §J.1 · p. 58