PaperlayerView exact revision on arXiv

Source-linked AI summary

Locate, Judge, Read: Role-Specialized Tool Agents over a Knowledge-Point Index for Open-Domain Multimodal Document QA

Wenlin Lin, Fanzhe Wei

arXiv:2610.13128v1cs.AIcs.CLcs.IR

TL;DR

Document QA requires both finding relevant passages and reading them, but current agents let one model schedule both kinds of tools. KnowForge separates those jobs with disjoint-tool agents and code-scheduled hand-offs; it outperforms one-shot retrieval and published systems across the evaluated benchmarks.

  • Problem

    Document QA requires both open-ended passage search and bounded reading, while existing retrieval-augmented agents give one model both tool types and let it schedule itself.

    Source: §1 · p. 1

  • Method

    KnowForge uses a hybrid knowledge-point index, disjoint-tool Locator and Deep Reader agents, a code-scheduled bridge judge, and a typed evidence pool.

    Source: §1 · p. 2

  • Results

    Across full benchmarks, KnowForge beats one-shot retrieval by 7 F1 on M3DocVQA and published systems on both multi-document benchmarks.

    Source: §9 · p. 11

  • Takeaways & Limitations

    The measured value of reading tracks corpus visuality, while locating and a forced second hop matter across the evaluated settings.

    Source: §9 · p. 11

    The claims apply to models that can follow a multi-round tool protocol; smaller open models were tested only on a 300-question subset and did not beat their own one-shot results.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Question answering over large heterogeneous document collections requires first finding the right passages and then reading them, yet current retrieval-augmented agents give one model both kinds of tools and let it schedule itself. We present KnowForge, a pipeline separated by job: a hybrid-retrieval knowledge-point index; a Locator agent that can only search; a code-scheduled bridge judge that forces the missing second hop; a typed, size-bounded evidence pool that merges every search's ranking; a Deep Reader agent that can only read text, images and tables; and a synthesizer with a format checker. With a self-hosted Qwen3.6-35B-A3B (3B active), KnowForge reaches F1 47.5 on the full open-domain M3DocVQA (2,441 questions, 3,368 PDFs), +11.0 over the published M3DocRAG system and +7.1 over one-shot retrieve-and-answer on the same index and model, and 53.5 with DeepSeek-V4-Flash. On MultiHop-RAG it reaches 73.5 accuracy, above the strongest published RAG and GraphRAG results (71.2); on the single-document MMLongBench-Doc it lifts the same model from 34.8 to 47.0, level with GPT-4o reading page images (46.3). Per-job ablations on the full benchmarks attribute the gain: the Deep Reader is worth 3.8, 9.8 and 0 points on the three corpora, tracking how visual each is; merging both agents into one with all tools and twice the rounds loses 1.9 F1 because it keeps searching instead of reading; a round-budget sweep shows a step at four Locator rounds and a plateau beyond; query latency scales with model calls rather than corpus size. We serve the three indexed benchmarks as read-only knowledge bases (free registration) and release all prompts and the runner code: https://github.com/metask-ai/knowforge-bench

1 Introduction

KnowForge treats document QA as separate locating and reading jobs, because one agent given both tool sets tends to search again instead of reading evidence. It combines disjoint-tool agents, a code-scheduled second-hop check, and a typed evidence hand-off.

Motivation

  • When one agent shares search and reading tools, it searches again rather than reading deeper, reflecting the opposing open-ended search and closed reading tasks.In the controlled comparison, the all-tools agent searched 34% more, read half as often, and scored 1.9 F1 lower than two agents with disjoint tools and four rounds each.Source: §1 · p. 1

Results

  • On full open-domain M3DocVQA, the self-hosted 35B mixture-of-experts model (3B active) scored F1 47.5, versus 36.5 for M3DocRAG and 40.4 for one-shot retrieve-and-answer on the same index and model.Source: §1 · p. 2

Contribution

  • KnowForge separates Locator and Deep Reader tools, forces the second hop with a code-level bridge judge, and hands off through a typed, size-bounded evidence pool.The pool merges search rankings so figures and tables are not ranked out by text.Source: §1 · p. 2

2 Related Work

Related systems differ in whether they retrieve once or iterate, how they divide agent roles, and whether they target open-domain or single-document QA. KnowForge's job-based split and full-benchmark evaluation distinguish its design from prior approaches.

Document QA benchmarks

  • M3DocVQA is an open-domain, multi-hop benchmark over 2,441 questions and 3,368 PDFs, while MMLongBench-Doc is its single-long-document complement and MultiHop-RAG is text-only.The reported GPT-4o score on MMLongBench-Doc is 46.3; the strongest published RAG/GraphRAG numbers on MultiHop-RAG are 70–71.Source: §2 · pp. 2–3

Agentic and multi-stage RAG

  • Single-pass systems such as M3DocRAG retrieve once, while iterative systems such as Search-o1 and WebThinker separate reasoning from retrieval or exploration.DeepRead gives one agent both Retrieve and ReadSection tools; MDocAgent splits by modality, while several systems iterate or split roles on a single long document.Source: §2 · p. 3
  • KnowForge separates roles by job, uses disjoint tools, forces the second hop in code, and ablates each job on open-domain corpora.The related-work passage says several recent systems lack released code and reports no full-benchmark evaluation of each role for the compared multi-agent designs.Source: §2 · p. 3

Graph-structured RAG

  • KnowForge's knowledge-point graph draws on proposition retrieval, similarity edges, and bounded expansion, but its graph ablation agrees with evaluations finding no advantage over strong embedding RAG on factual QA.The paper tests graph structure as a retrieval aid even though it was also developed for browsing and summarisation.Source: §2 · p. 3

3 Method

KnowForge builds a hybrid-retrieval index and passes search results through a forced bridge check and typed evidence pool before a search-restricted reader and answer synthesizer use them. Each stage has a distinct job and tool boundary.

3.1 Knowledge construction and hybrid retrieval

  • Documents are split into typed fragments, images and tables receive VLM descriptions, and an LLM extracts linked atomic knowledge points for BM25 and dense retrieval with bounded graph expansion.For M3DocVQA, the index contains 332K nodes and 299K edges, including 133K typed edges, over roughly 200K fragments.Source: §3.1 · p. 4

3.2 Locator

  • The Locator can search and enumerate corpus material for up to L rounds but has no reading tools.The separate tool boundary addresses the observed tendency of an all-tools agent to keep searching rather than read figures.Source: §3.2 · p. 4

3.3 Bridge judge

  • After locating, a fixed bridge check identifies missing question hops and issues one or two targeted searches; on M3DocVQA it fires on 48% of questions and averages 1.1 follow-up queries.A variant allowed to discard evidence lost 3.1 F1 on the 300-question subset, so the judge that only adds evidence was retained.Source: §3.3 · p. 4

3.4 Multi-list ranked merge

  • All search rankings enter a 30K-token typed pool, whose separate text, image, table, and example sub-pools protect media from being displaced by higher-scoring text.In simulation, the typed pool kept the answer visible for 93.7% of questions versus 86% for top-5-per-search; the 30K cap kept it visible in 99.0%.Source: §3.4 · pp. 4–5

3.5 Deep Reader

  • The Deep Reader can read fragments, images, and tables from the evidence pool but cannot search, and its reading results are appended for the synthesizer.Its media tools read one asset at a time and return an answer to an explicit question; the reader does not receive the Locator's conversation.Source: §3.5 · p. 5

3.6 Answer

  • A synthesizer answers from the evidence pool and reader notes, then a format checker rewrites the answer to the benchmark convention.The final-layer convention varies by benchmark, while the tools remain identical.Source: §3.6 · p. 5

4 Experimental Setup

The experiments evaluate KnowForge on three document-QA benchmarks with a shared set of pipeline variants and report paired-comparison procedures. The main setup uses a self-hosted Qwen3.6 model, with a separate model-axis experiment.

Benchmarks

  • The evaluation covers M3DocVQA, MMLongBench-Doc, and MultiHop-RAG, with their full question and document counts and benchmark-specific scoring protocols.M3DocVQA has 2,441 questions over 3,368 PDFs; MMLongBench-Doc has 1,082 questions over 135 PDFs, including 223 unanswerable questions; MultiHop-RAG has 2,556 questions over 609 news articles.Source: §4 · p. 5

Models

  • Unless otherwise stated, text steps use self-hosted Qwen3.6-35B-A3B with 3B active parameters, while DeepSeek-V4-Flash is used for the model-axis experiment.Image reading uses the same Qwen model's vision input.Source: §4 · p. 5

Systems

  • The system comparisons include one-shot search-and-answer, the full pipeline, an all-tools single agent, and an iterative-retrieval baseline without a Deep Reader.Retrieval variants test dense-only search and disabling graph expansion.Source: §4 · p. 5

Runs and noise

  • Full-run differences above 1.4 F1 are outside the reported noise, and paired comparisons use win-loss counts, exact sign tests, and 95% paired bootstrap intervals.The paired bootstrap uses 2,000 resamples; mid300 paired standard error is 2.0 F1.Source: §4 · p. 5

Systems

  • Table 2 compares role ablations on full M3DocVQA using search calls, read calls, and the share of questions with no reading tool call.Its caption states that P0 finds documents slightly better than M5 but answers worse.Source: Table 2 · p. 6

5 Results

KnowForge outperforms published systems on three benchmarks, while ablations examine the contributions of reading, evidence pooling, retrieval, and round budgets. Transfer and efficiency results show how performance varies across corpora and what drives query cost.

5.1 Main result and comparison with published systems

  • M3DocVQA F1 47.5 vs 36.5 for M3DocRAG; MultiHop-RAG 73.5 vs 71.2 for the strongest published result; MMLongBench-Doc 47.0 vs 34.8 for the same model.The pipeline also gains 7.1 F1 over one-shot retrieve-and-answer on the same index and model.Source: §5.1 · p. 6

5.2 Which job earns the gain?

  • Removing the Deep Reader from M5 costs 3.8 F1 (CI [2.3, 5.3], p < 10−5); P0 scores 1.9 F1 below M5 despite finding slightly more documents (86.0 vs 84.6).Source: §5.2 · p. 6

5.3 The evidence pool: cap and typing

  • The bridge judge and typed pool add 1.9 F1 over M4; the pool lets the reader see up to 30K tokens instead of the first 12K.On capped questions, the typed pool scores 30.6 F1 versus 23.3 for a single ranked list with cosine eviction.Source: §5.3 · p. 6

5.4 Round budget: a step, then a plateau

  • Answer quality is flat at 46.2–46.7 for caps 1+1 through 3+3, steps to 50.9 at 4+4 across three repeats (range 0.4), then ranges 49.2–50.6 at 5+5 and 6+6.Locator 4→3 costs about 2 F1, whereas Reader 4→3 costs nothing; calls and latency continue growing beyond 4+4 (+25% time from 4+4 to 6+6).Source: §5.4 · pp. 6–7

5.5 Index factors inside the pipeline

  • Dense-only retrieval lowers F1 to 45.0 (−2.5); disabling typed graph expansion yields 47.0 (−0.5, p = 0.56), and graph expansion costs 45 ms per round.Source: §5.5 · p. 7

5.6 Model axis

  • On mid300, one-shot→M5 gains 7.8 F1 with Qwen3.6 and 6.1–6.9 with DeepSeek-V4-Flash.Qwen2-VL-7B does worse than its one-shot result (37.4 vs 40.6); the protocol assumes a model that can follow tool-call formats across rounds.Source: §5.6 · p. 7

5.7 Transfer to other corpora

  • The Deep Reader is the largest single factor on chart-heavy MMLongBench-Doc but contributes nothing on text-only MultiHop-RAG, where P1 ties M5.Source: Table 4 · p. 8

5.8 Efficiency

  • The pipeline takes 27.3 s, with about nine model calls of 3–4.6 s each; growing the corpus 42× raises one retrieval round by 7% (0.113→0.121 s).Index construction for M3DocVQA is a one-time cost of 138M model tokens, 202K embeddings, and 17K OCR pages, taking about 12 hours.Source: §5.8 · p. 9

6 Analysis

The analysis shows that errors persist even when retrieval finds the supporting document, particularly when answers require reading visual evidence. It also identifies false refusals and cross-page questions as remaining boundaries.

Where the remaining errors are

  • 31 of 100 sampled questions with the supporting document as the first hit were misread or had an image left unopened; 23 were penalized for answer granularity.The classification had 88% agreement between the first reviewer and an independent second reviewer.Source: §6 · p. 9

A worked example

  • The iterative baseline retrieved the second document but still answered incorrectly because nobody read its table; the full pipeline read the image table and answered “Rick Mears.”The single-agent configuration also solved this example by reading the table after its first search.Source: §6 · pp. 9–10

False refusals

  • 9.8% of MultiHop-RAG questions and 274 of 859 MMLongBench-Doc questions were false refusals; relaxing the conservative convention trades the unanswerable bucket one for one.Multi-round configuration reduced false refusals from 18.1% to 9.8% on MultiHop-RAG and from 475 to 274 on MMLongBench-Doc.Source: §6 · p. 10

Honest boundaries

  • Cross-page MMLongBench-Doc questions remain low at 10–22% of 358 across configurations, leaving evidence merging across pages a shared weakness.Source: §6 · p. 10
  • Graph expansion is neutral in these tests, while the reported round-cap plateau comes from one model whose loop displays the remaining budget.The reader stops early when little budget remains rather than being cut off.Source: §6 · p. 10
  • Closed-book M3DocVQA scores are non-trivial for strong models because the corpus is Wikipedia, so gains are reported against each model’s one-shot result.The passage reports closed-book scores of 43.0 for Flash and 32.2 for Qwen3.6.Source: §6 · p. 10

7 Discussion

The discussion identifies model protocol-following, round-budget behavior, and graph retrieval as qualifications on the pipeline’s results. It also describes deployment trade-offs and a possible direction for future work.

Which model can drive this pipeline

  • Below the protocol-following threshold, the design’s gain is retrieval-only; Qwen2-VL-7B and Qwen3-VL-8B did not reliably follow the multi-round protocol.The authors did not tune prompts per model; prompt choice affected results differently across models.Source: §7 · p. 10

Round budgets are upper bounds

  • A 3+3 to 4+4 round-cap increase adds 4.7 F1, but raising both caps further increases reads without increasing F1.With the other cap at four, the Locator’s fourth round is worth 2.3 F1 and the Reader’s 0.6; the threshold may vary by model and loop design.Source: §7 · p. 10

What the graph layer is for

  • Typed graph expansion is neutral (+0.5 F1, p = 0.56), whereas hybrid lexical-plus-dense retrieval adds 2.5 F1 inside the pipeline and 0.5 in one shot.Graph expansion slightly increases single-hop answer visibility but can displace BM25 hits from a fixed-size list.Source: §7 · p. 10

Deployment

  • The deployed question-answering path uses the same six-job chain, but its 27-second latency suits a research assistant rather than a chat interface.The first visible event arrives at 1.2 seconds, and streamed synthesis contributes to usability.Source: §7 · pp. 10–11

Outlook

  • The authors propose testing whether smaller or specialized models for individual jobs can match a large general model on this task.The released runner supports per-stage model overrides for that experiment.Source: §7 · p. 11

8 Open Verification

The authors provide read-only access to the three benchmark indexes and release materials that support third-party runs and scoring.

Open Verification

  • The released indexes, fetch and scoring scripts, prompts, and reference runner let third parties recompute document recall, run their own model, or score an agent.On the 299-question MultiHop-RAG subset, the released runner scored 73.9 versus 71.9 for the in-process harness, within the run-to-run band.Source: §8 · p. 11

9 Conclusion

The conclusion attributes KnowForge’s gains to job-based separation and reports that different components matter across benchmarks.

Conclusion

  • Splitting document QA by job beats one-shot retrieval by 7 F1 on full M3DocVQA and published systems on both multi-document benchmarks.The conclusion says the Reader matters where evidence is visual, while the Locator and forced second hop matter across benchmarks; extra rounds and the graph layer do not add value there.Source: §9 · p. 11

Limitations

The reported scope is bounded by model, benchmark, and evaluation conditions: the full claims are tested mainly on models that follow multi-round protocols, with several measurements tied to one judge or setup.

Limitations

  • The multi-round claims are supported for models that follow the tool protocol; smaller open models were tested only on a 300-question subset and did not beat one-shot.The round-cap plateau was measured only on Qwen3.6.Source: Limitations · p. 11
  • M3DocVQA is Wikipedia-derived and partially memorised by large models, motivating one-shot rather than closed-book as the comparison baseline.MMLongBench-Doc uses a Qwen3.6 judge, and judge choice moves scores by 2–3 points.Source: Limitations · p. 11
  • Latency figures describe one hardware configuration and one serial client, and prompts were not tuned per benchmark beyond a rule paragraph and format checker.A prompt variant that helped on a smoke test hurt on the full set.Source: Limitations · p. 11

Ethics Statement

This material documents benchmark-use conditions, prompt and agent procedures, and selected operational results. It also reports where prompt variants, round budgets, and evidence-pool limits affected outcomes.

Ethics Statement

  • The benchmarks are used under their licences, the read-only indexes require free registration and are rate-limited, and no private documents were used.MMLongBench-Doc is CC BY-NC 4.0 for research use; MultiHop-RAG is ODC-BY.Source: Ethics Statement · p. 11

A Agent prompts

  • The Locator searches and stops when materials suffice, while the Deep Reader has no search tool and reads figures, tables, and surrounding text as needed.The prompts also direct the Locator to locate both sides of multi-part comparisons before stopping.Source: A Agent prompts · pp. 13–14
  • The bridge judge flags a missing second hop when a mentioned bridge entity’s own document has not been retrieved, with follow-ups capped at two deduplicated queries.Agent-side prompts are written in Chinese for the deployed system, while answer-side prompts are in English for the English benchmarks.Source: A Agent prompts · p. 13
  • On the 35B model, replacing the synthesizer with the official M3DocRAG template cost 1.4 full-set F1, while positive phrasing cost 4.6 F1 on mid300.The official template was best among the tested prompts on the 7B model, indicating prompt choice was model-dependent.Source: A Agent prompts · p. 15

B Full M3DocVQA tables

  • M5 improved over M4 on both pool groups: untrimmed questions rose from 53.9 to 56.0 and trimmed questions from 28.9 to 30.6.811 of 2,441 questions exceeded the 30K-token pool; the pool lets the reader see up to 30K rather than the first 12K.Source: B Full M3DocVQA tables · p. 15

C Round-cap sweep: usage

  • Cutting both Locator and Reader from four to three rounds costs 4.7 F1, exceeding the 2.9 sum of the individual three-round cuts.The reported interaction means each role’s fourth round matters more when the other role is also short of rounds.Source: C Round-cap sweep: usage · p. 15

D Index construction cost

  • The M3DocVQA index contains 332K knowledge-point nodes and 299K edges, while preprocessing accounts for 30–41% of PDF indexing cost.PDF corpora with figures cost three to four times more to index per Markdown character than clean text.Source: D Index construction cost · p. 15

E Transfer benchmark details

  • On MultiHop-RAG, prompt variants that reduced refusals by forcing conclusions lost more on comparison questions than they gained.The checker maps refusals to “Insufficient information”; the pipeline predicted that response on 705, 612, 515, and 532 questions across S4, 1+1, M5, and P1.Source: E Transfer benchmark details · p. 15

F Worked example

The worked example shows why reading an image-stored table matters after retrieval: the prose alone points to the wrong driver, while reading the table supports the correct answer. The error sample then characterizes recurring failure types among difficult retrieved cases.

F Worked example

  • Reading the image-stored results table revealed the +1 Lap rows and led KnowForge to answer Rick Mears, correcting the prose-based Al Unser answer.The four table-listed drivers were checked against the Indy 500 winners list already in the evidence pool; Rick Mears had four wins.Source: F Worked example · pp. 15–16
  • The single agent also answered this example correctly and faster, but it reads half as often as the Deep Reader on average and never reads on 74% of questions.This individual success contrasts with the reported aggregate pattern for the all-tools agent.Source: F Worked example · p. 16

G Error attribution sample

  • Among 100 sampled low-F1 questions whose supporting document was the first hit, image questions were dominated by unopened or misread images, while table errors clustered in wrong rows or fields.The two LLM review passes agreed on 88 cases; the sample came from 414 such questions.Source: G Error attribution sample · pp. 16–17
  • Table 11 reports document-level evidence-article recall in the first search and across all searches, which is not directly comparable to the benchmark’s chunk-level Hits@k.The caption notes that inference questions have low first-search recall because they typically require three or four articles.Source: Table 11 · p. 17
Loading 2610.13128v1…