PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of VLM Pipelines for Long-Document QA

Kenan E. Ak, Jay Mohta, Gwang Gook Lee, Yan Xu, Dimitrios Dimitriadis

arXiv:2609.29933v1cs.CLcs.AIcs.CVcs.IR

TL;DR

Long-document QA combines text and visual evidence across pages, but the joint accuracy/cost trade-offs of input modality, retrieval, and agentic access have not been measured at frontier-VLM scale. This study evaluates these choices on two benchmarks and finds that a six-tool agent is strongest under the official scorer on both, with its advantage varying by benchmark and reader scale.

  • Problem

    Long-document QA can require finding and combining text and visual evidence across pages, while the joint behavior and accuracy/cost trade-offs of input modality, retrieval, and agentic access remain unmeasured at frontier-VLM scale.

    Source: §1 · p. 1

  • Method

    The study compares input modalities, retrieval approaches, and agentic designs on two long-document QA benchmarks using frontier API and open-weight VLM readers.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    The six-tool agent is highest under the official scorer on both benchmarks, leading same-reader top-5 retrieval by 11.7 pp on MMLongBench-Doc and tying the best retrieval pipeline on LongDocURL.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Agentic access is not uniformly advantageous: on MMLongBench-Doc its benefit appears only with larger readers, while on LongDocURL it leads or ties static input across the evaluated readers.

    Source: §4.3 · p. 6, §5 · p. 7

    The study evaluates long-document QA only, and whether its pipeline rankings transfer to summarization, extraction, or classification is left to future work.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Vision-Language Models (VLMs) are increasingly used for long-document processing, where the inputs combine text with charts, tables, figures, and complex layouts. Deploying them means choosing how to feed the document to the model, which retriever to use when only a subset of pages is sent, and whether to run the model agentically or as a static pipeline. We study these choices on two long-document QA benchmarks with both frontier API and open-weight VLMs. First, on MMLongBench-Doc our six-tool agent with page, table, figure, and search calls pays off only once the answering VLM is large enough: with Qwen3.5-4B and 9B it trails static page input, with Qwen3.5-27B it draws level, and with Sonnet 4.5 it leads. On LongDocURL it is level with or ahead of static input at every reader. Its lead over the strongest static pipeline is clearest with the frontier reader on MMLongBench-Doc and narrows to within noise on LongDocURL. Second, retrieval modality matters more than the specific retriever: the strongest image retriever leads the strongest text pipeline, and on the text side a single off-the-shelf cross-encoder rerank essentially matches a much heavier multi-stage LLM pipeline. Top-k image retrieval is also the most token-efficient input at every reader we paired it with, at roughly a seventh to a quarter of the tokens of sending every page. Third, cutting across all three choices, three of our strongest pipelines succeed on different questions, and an oracle that picks the best pipeline per question gains roughly thirteen points over the best single pipeline, though evidence-type routing recovers almost none of it.

1 Introduction

Long-document QA requires finding and combining evidence across pages containing prose, charts, tables, and figures, capabilities short-context benchmarks rarely test together. This study jointly evaluates input modality, retrieval, and agentic access on two benchmarks, finding that its six-tool agent is strongest under the official scorer on both.

1 Introduction

  • Long-document questions may require locating a page, interpreting a chart or table, and combining evidence scattered across pages—skills short-context benchmarks rarely test together.Real-world documents span tens to hundreds of pages and interleave prose with visual elements under inconsistent layouts.Source: §1 · p. 1
  • The study jointly measures input modality, retrieval, and agentic access, including their accuracy/cost trade-offs at frontier-VLM scale, which had not been measured together.Each axis had been studied on its own.Source: §1 · p. 1
  • The six-tool agent is the highest-scoring configuration under the official scorer on both benchmarks, leading same-reader top-5 retrieval by 11.7 pp on MMLongBench-Doc and tying the best retrieval pipeline on LongDocURL.The study compares frontier Sonnet 4.5 and open-weight Qwen3.5 readers.Source: §1 · p. 1
  • The evaluation re-scores a close prior system with each benchmark’s official scorer, lowering its judge-scored number by roughly 9 pp.The paper emphasizes comparable evaluation rather than proposing a novel agent design.Source: §1 · pp. 1–2

2 Related Work

Prior work spans long-document multimodal QA, VLM architectures, page retrieval, and tool-using agents, but evaluations have generally committed to one access mode or compared agentic systems with one another. The paper situates its study as a head-to-head examination of these choices and their cost/accuracy trade-offs.

2.1 Long-Document Multimodal QA

  • MMLongBench-Doc covers shorter, more visually varied PDFs, while LongDocURL contains longer documents and spans Understanding, Reasoning, and Locating tasks.MMLongBench-Doc has 135 PDFs and 1,082 questions; LongDocURL’s public split has 396 documents and 2,325 questions.Source: §2.1 · p. 2

2.2 Vision-Language Models

  • VLMs and document-specific models encode text and visual structure in different ways, but evaluations often use smaller contexts or commit to one access mode rather than comparing choices head to head.Document-specific models include layout-aware systems requiring OCR and OCR-free page-image encoders.Source: §2.2 · p. 2

2.3 Page Retrieval over Long Documents

  • Retrieval work spans lexical, dense, vision-based, and reranking methods, but the contribution of each step relative to the underlying text source is rarely isolated.The paper’s retrieval ablation examines those contributions.Source: §2.3 · p. 2

2.4 Agentic and Tool-Using Document QA

  • Multimodal agent systems range from single-VLM toolkits to multi-model stacks, and most use per-document preprocessing; comparisons have largely been against other agentic systems, not strong static retrieval-and-read pipelines.Examples include specialized multi-agent roles and a separate reasoner/answerer design.Source: §2.4 · pp. 2–3

3 Method

The method treats long-document QA as a deployment problem with three separable choices: what input reaches the reader, how pages are ranked, and whether access is static or tool-driven. It evaluates these choices across readers and benchmarks using a shared scoring protocol, and defines a per-question oracle as a hindsight upper bound.

3 Method

  • The method decomposes a QA system into input modality, page retrieval when only top-k pages are sent, and an agentic policy that is either a static pass or a multi-turn tool loop.Questions may require evidence from one page, non-adjacent pages, or none of the document.Source: §3 · p. 3
  • The study evaluates five input modalities, multiple retriever and agent families, and Sonnet 4.5 and Qwen3.5-4B/9B/27B on MMLongBench-Doc and LongDocURL.Configurations include page images, plain and layout-aware text, retrieved images, and function-calling, gist-based, and iterative-refinement agents.Source: §3 · p. 3
  • A shared evaluation pipeline normalizes free-form answers with Claude Haiku 4.5 before each benchmark’s official deterministic scorer, and reports per-question accuracy and mean input tokens.Token counts include access-mode and agent-loop costs but exclude one-time preprocessing and retrieval-time compute.Source: §3 · p. 3
  • The per-question oracle takes the best score among fixed pipelines for each question, providing a hindsight upper bound on what a per-question router could add.Unless otherwise stated, it uses three strong configurations.Source: §3 · p. 3

3.1 Six-Tool Function-Calling Agent

  • The function-calling agent explores the document on demand using a structural catalog and six tools for catalog lookup, page rendering, figure and table access, text search, and final answers.The loop allows up to eight calls per question and uses a median of two in practice.Source: §3.1 · pp. 3–4
  • Unlike gist-based agents, the toolkit preserves actual page images; one model can combine search with direct table access, while element tools provide higher resolution than full-page rendering.For example, it can search for a section and then retrieve a specific table.Source: §3.1 · p. 4

4 Experiments

Across the experiments, the strongest access strategy depends on reader scale and benchmark, while image-based retrieval generally outperforms text retrieval and a single rerank matches a heavier LLM pipeline. The six-tool agent’s advantage is reader-dependent, and different strong pipelines answer different questions.

4.1 Input Modality

  • On LongDocURL, raw-PDF accuracy falls 42.6 pp between the 50–75-page and 125–150-page bins, while retrieval, the agent, and all-images at a local reader vary by at most 5 pp across the full range.The Sonnet agent is highest in every bin; raw-PDF failures above the API cap account for most of its collapse.Source: §4.1 · p. 5, Figure 1 · p. 5
  • Top-k image retrieval uses 7.6–9k tokens for accuracy of 0.51–0.56; the six-tool agent reaches 0.616–0.625 at 20–33k tokens, while raw PDF uses 80k tokens for 0.522.Only top-k retrieval and the agent lie on the Pareto frontier; the Sonnet agent dominates all-images at 0.625 accuracy and 20k tokens versus 0.588 and 30k.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 Retrieval

  • Image retrievers reach 87.2 and 91.4 average Hit@5, while no text retriever exceeds 82.4; the strongest image retriever leads the strongest text pipeline by about 9 pp.With the same reader, five retrieved page images match whole-document extracted text on MMLongBench-Doc (0.508 vs. 0.480) using 7.6k vs. 27k tokens.Source: §4.2 · p. 6, Table 2 · p. 6
  • A single Cohere Rerank 3.5 rerank is statistically indistinguishable from SIRA’s multi-stage pipeline (82.0 vs. 82.4 Hit@5), without an LLM in the loop.Adding Cohere Rerank 3.5 lifts BM25 and BGE-M3 by 8 to 12 pp and renders them indistinguishable afterward.Source: §4.2 · p. 6
  • Scout reranking Nem-CE gains 2.4 pp on MMLongBench-Doc but loses 11.3 pp on LongDocURL, with its deficit widening from −5.7 pp on 50–75-page documents to −22.0 pp on 125+-page documents.The passage attributes the long-document loss to Scout increasingly demoting evidence pages that Nem-CE had ranked in its top five.Source: §4.2 · p. 6

4.3 Agentic Access

  • The six-tool agent is the highest-scoring single-VLM configuration at frontier scale on both benchmarks, beating the best retrieve-then-generate pipeline on MMLongBench-Doc and tying it on LongDocURL.On MMLongBench-Doc it leads same-reader top-5 retrieval by 11.7 pp; on LongDocURL it is level with the best retrieval pipeline.Source: Table 1 · p. 4, §4.3 · p. 6
  • The agent’s advantage on MMLongBench-Doc emerges only at larger readers: it trails static image modes at Qwen3.5-4B and 9B, draws level at 27B, and leads with Sonnet 4.5.On LongDocURL it leads top-k at 4B and 9B and ties the best static pipeline at 27B and Sonnet 4.5.Source: §4.3 · p. 6
  • Qwen3.5-4B spends 77k tokens versus 33k for 27B because it often continues calling finish after its answer is recorded, adding 9.1 versus 1.5 calls on MMLongBench-Doc.The same pattern appears on LongDocURL: 4B adds 4.9 further calls versus 0.3 for 27B.Source: §4.3 · p. 7

5 Conclusion

The study frames long-document QA as a deployment choice among input modality, retrieval, and agentic access. Its findings favor different configurations by reader scale and leave substantial question-level gains beyond any single pipeline.

5 Conclusion

  • With a frontier API, the six-tool function-calling agent is the highest-accuracy configuration measured, using roughly 1.5–7× fewer input tokens per question than sending all pages as images or raw PDF.The conclusion reports it as highest accuracy on both benchmarks.Source: §5 · p. 7
  • With Qwen3.5-9B and smaller readers, static image input is the better default: on MMLongBench the agent trails all-images, while top-5 retrieval matches all-images at a fraction of the tokens.On LongDocURL, the agent’s lead over top-5 retrieval is only a few points and it spends 4–9× the tokens.Source: §5 · p. 7
  • When images cannot be sent, MinerU with BM25 or BGE-M3 and one cross-encoder rerank matches a heavier multi-stage design’s text-side accuracy at a fraction of its compute.Source: §5 · p. 7
  • Three strong pipelines make different mistakes, leaving roughly 13 pp of per-question oracle headroom; evidence-type routing recovers under one point, while observable-feature routing performs worse than always running the agent.The conclusion attributes the complementarity to differences within evidence types.Source: §5 · p. 7

Limitations

The findings are limited to long-document question answering on two benchmarks, the tested VLM families, and English-language documents. Fine-grained score differences and some API results also have specific uncertainty and measurement boundaries.

Limitations

  • The study measures QA only; whether the rankings transfer to summarization, extraction, or classification is left open, and the authors make no generality claim beyond QA.Source: Limitations · p. 8
  • The two benchmarks contain 3,407 questions over 531 documents, and differences below about 3 pp on MMLongBench-Doc or 2 pp on LongDocURL are within noise.Source: Limitations · p. 8
  • The unified scorer’s answer-extractor LLM may slightly understate the agent’s lead when a system already returns a short structured answer.Source: Limitations · p. 8
  • Results assume English documents and the tested VLM families; transfer to other languages or smaller on-device VLMs is open, and a different evidence mix may reorder the leaderboard.Source: Limitations · p. 8

Ethical Considerations

The appendices document ethical safeguards and examine evidence-source accuracy, raw-PDF behavior, text extraction, and image resolution. Together, these sections specify how the evaluated inputs and measurements are bounded.

Ethical Considerations

  • The work uses public QA benchmarks without sensitive personal data, evaluates released models or commercial APIs, and collects no new human-labeled data or human-subject experiments.The authors recommend surfacing source pages in deployed systems so incorrect answers are auditable.Source: Ethical Considerations · p. 8

A Per-Evidence-Source Accuracy

  • Table 3 breaks accuracy out by evidence-source labels on both benchmarks, with questions carrying multiple labels counted in each applicable subset.MMLongBench-Doc labels include pure text, layout, chart, table, and figure; LongDocURL uses all but chart.Source: A Per-Evidence-Source Accuracy · p. 11, Table 3 · p. 12

B How the Claude API Processes Raw PDFs

  • Claude raw-PDF input converts pages to images with extracted text, but request limits can downsample pages or fail outright, driving many long-document zero scores.Logged token counts confirm substantial per-page token reductions as documents grow longer.Source: B How the Claude API Processes Raw PDFs · p. 11

C Text-Extractor Sweep for Plain-Text Input

D Resolution Sweep for Top-k Retrieved Images

E Scaling with Document Length

Raw-PDF accuracy collapses as LongDocURL documents grow, while selective access modes remain comparatively stable. The Scout reranker’s LongDocURL retrieval deficit also grows with document length, unlike plain Nem-CE.

E Scaling with Document Length

  • −42.6 pp: raw-PDF accuracy falls from 0.574 to 0.148 across LongDocURL’s 50–75- and 125–150-page bins, while other modes drop only −4 to +1 pp.Most of the raw-PDF decline comes from outright request failures; failure rates rise from 8% in the shortest bin to 68% in the longest.Source: Table 7 · p. 13, E Scaling with Document Length · p. 13
  • On MMLongBench-Doc, raw PDF drops −6.3 pp across dense page-count bins, while the six-tool agent remains comparatively stable, including Sonnet’s 0.644 → 0.591.The longest-bin estimates are noisier because few documents fall in those bins.Source: E Scaling with Document Length · p. 13, Table 8 · p. 14

F Scout Reranker vs. Document Length

  • −5.7 to −22.0 pp: the Nem-CE⊕Scout LongDocURL Hit@5 deficit grows from the shortest to longest bin, while plain Nem-CE stays flat at 0.92–0.95.The candidate set still contains the evidence page; Scout increasingly ranks it outside its final top five as document length grows.Source: F Scout Reranker vs. Document Length · pp. 13–14, Table 9 · p. 14

G Effect of Thinking Mode

The six-tool Sonnet agent is level with or ahead of systems scored under the official scorer while using one VLM, a median of two tool calls, and no LLM in preprocessing. The appendix also shows that reasoning-budget gains are modest, retrieval depth trades accuracy for tokens, and the agent’s structured tools differ from gist-based lookup.

G Effect of Thinking Mode

  • Thinking improves five of six tested configurations by about 4–6 pp, while Qwen3.5-9B top-k gains 2.0 pp and is reported as a tie.Gains are larger for all-images than top-k on both open-weight readers; modality ordering is unchanged.Source: G Effect of Thinking Mode · p. 14, Table 10 · p. 14

I Retrieval-Depth Sweep

  • Accuracy rises steeply to about k=10 and then flattens; k=5 captures most of the peak at roughly half the tokens.Sonnet declines by k=20 as extra pages add distractors, while Qwen readers gain at most 0.8 pp from k=10 to k=20 for twice the input tokens.Source: I Retrieval-Depth Sweep · p. 15, Table 12 · p. 16

J.1 Six-Tool Function-Calling Agent

  • The function-calling agent gives the VLM a structural catalog and tools for reading pages, fetching figure or table content, searching text, and committing an answer.Table access includes structured HTML, which preserves cell boundaries for numeric reasoning.Source: §J.1 · p. 15

H Retriever Cost and Architecture

  • Image retrievers lead text retrievers by about 9 pp best-to-best, while a Cohere cross-encoder rerank matches SIRA within 0.3 pp on MMLongBench.The image-based Nem-CE result requires no LLM in the loop; SIRA spends about 11k LLM tokens per question.Source: Table 11 · p. 16

J.2 ReadAgent Baseline

  • ReadAgent selects pages from 60-word, LLM-generated gist memories and then answers after reading those pages, whereas the six-tool agent can fetch structured content and search during interaction.The paper describes gist compression as losing fine-grained list and table information, which the structural catalog and targeted tools preserve.Source: §J.2 · p. 17

K Architectural Cost of the Comparison Agents

  • The six-tool agent is level with or ahead of every system scored under the official scorer despite using one VLM, a median of two tool calls, and preprocessing without an LLM.SimpleDoc uses two large models per question, while ReadAgent incurs about 5.3 extra LLM calls per question after amortizing gist generation.Source: K Architectural Cost of the Comparison… · p. 17

L Tool Usage Analysis for the 6-Tool FC Agent

The Sonnet agent usually finishes with few tool calls, and its tool use varies with evidence type. Table questions are routed to table access more often than chart or figure questions are routed to figure access.

Analysis scope

Tool usage by evidence type

  • Table-evidence questions invoke get_table in 51% of cases, while chart and figure questions invoke get_figure in 34% and 25%, respectively.The table reports the percentage of questions invoking each tool at least once, broken down by evidence type.Source: Table 14 · p. 18

Tool-call budget

  • The agent uses a median of 2 tool calls and a mean of 2.68 across 1,082 questions; 76% finish within 3 calls and 86% within 4.Only 3.6% issue more than 6 calls, making a tighter six-call budget a possible cost-saving operating point.Source: L Tool Usage Analysis for the 6-Tool… · pp. 17–18

M Statistical Significance

Paired-bootstrap tests show that the Sonnet agent’s lead is significant on MMLongBench-Doc but its small margin over the best static pipeline on LongDocURL is within noise. Retrieval is the cheapest input mode and among the fastest, though preprocessing costs are amortized rather than included in inference-time latency.

M Statistical Significance

  • On MMLongBench-Doc, the Sonnet agent beats same-reader Top-5 ColQwen by +11.7 pp and the strongest static row, Qwen3.5-27B all-images, by +3.7 pp; on LongDocURL, its +1.5 pp over Top-5 Nem-CE is within noise.The paired bootstrap calls a difference significant when its 95% interval excludes zero.Source: M Statistical Significance · p. 18, Table 16 · p. 20
  • ColQwen image retrieval exceeds BM25 text retrieval by +10.0 pp on MMLongBench and +8.7 pp on LongDocURL; SIRA and a single rerank are statistically indistinguishable in Hit@5 against both Cohere baselines at a 2 pp noise floor.Source: M Statistical Significance · pp. 18–19
  • With token matching, the Sonnet agent beats static ColQwen k=20 by +11.6 pp, while the Qwen3.5-27B agent’s +2.1 pp margin is within noise.The shared answer extractor lifts static pipelines and ReadAgent by 47.1–49.7 pp but changes the agent by only 0.7 pp; extraction therefore does not inflate the agent’s lead.Source: M Statistical Significance · p. 19, Table 15 · p. 19

N Latency

  • Top-k retrieval is the cheapest mode and among the fastest; on MMLongBench-Doc only image-free MinerU text is faster, while the agent is competitive with all-images and faster than raw PDF.Latency figures are inference-time only; one-time preprocessing is computed once per document and amortized across its questions.Source: N Latency · pp. 19–20, Table 17 · p. 20

O Per-Question Routing Analysis

The three-pipeline per-question oracle gains roughly 13 points over the best single pipeline, but evidence-type routing captures almost none of that headroom.

O Per-Question Routing Analysis

  • The oracle uses a fixed three-pipeline member set and only questions all members answer, avoiding gains that mechanically grow with more members.The table’s four-pipeline comparison uses a shared subset of 1,064 MMLB and 2,287 LDURL questions.Source: O Per-Question Routing Analysis · pp. 20–21, Table 18 · p. 21
  • +13.1 pp on MMLongBench and +12.9 pp on LongDocURL separate the three-pipeline oracle from the best single pipeline on each benchmark.The oracle takes the highest score among its fixed members for each question they all answer.Source: O Per-Question Routing Analysis · pp. 20–21
  • +0.2 pp on MMLongBench and +0.6 pp on LongDocURL are the gains of the gold-type ceiling router over simply running the agent, both within noise.The gold-type router uses the gold evidence label and is an upper bound on type-conditioned routing.Source: O Per-Question Routing Analysis · p. 21, Table 18 · p. 21
  • The observable-feature router trails the agent by 0.9 pp on MMLongBench and 7.1 pp on LongDocURL, sometimes routing away from it.It conditions on question keywords and answer format rather than gold evidence labels.Source: O Per-Question Routing Analysis · p. 21

P Chunked Map-Reduce for Long Documents

Chunking long documents to avoid the all-images request cap does not restore accuracy; targeted retrieval and agent modes remain the recommended options for long documents.

P Chunked Map-Reduce for Long Documents

  • The chunking test covers 817 LongDocURL questions across 115 documents averaging 121 pages, with chunks of at most 100 pages.Pages were rendered at 144 DPI with a 1024 px cap, and answers were synthesized from per-chunk candidates.Source: P Chunked Map-Reduce for Long Documents · p. 21, Table 19 · p. 22
  • 0.193 for chunked map-reduce versus 0.204 for single-request all-images is a wash, and both trail the 0.444 for all-images on documents under 100 pages.The comparison uses LongDocURL documents of at least 100 pages under Sonnet 4.5 and the official scorer.Source: P Chunked Map-Reduce for Long Documents · p. 21, Table 19 · p. 22
  • Chunking removes API stitching and downsampling without restoring accuracy; splitting also loses cross-chunk context that the reduce step cannot recover.The paper attributes most of the long-document degradation to whole-document image access at this length, rather than the per-request cap.Source: P Chunked Map-Reduce for Long Documents · p. 21
  • Selective retrieval and agent modes send only a handful of targeted pages per call regardless of document length, sidestepping both failure modes.The paper recommends these selective modes for long documents.Source: P Chunked Map-Reduce for Long Documents · p. 22

Related papers

Loading 2609.29933v1…