PaperlayerView exact revision on arXiv

Source-linked AI summary

DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding

Zhuowen Liang, Xiaotian Lin, Yuyu Luo, Haixun Wang, Nan Tang

arXiv:2610.21327v1cs.AIcs.CLcs.CV

TL;DR

Long-document QA requires finding sparse evidence across pages and modalities, while existing agents can be limited by fixed workflows or coarse actions. DocPilot combines adaptive evidence seeking, multimodal reasoning, and verification, and trains compact models through a curriculum from individual capabilities to orchestration. Across four benchmarks it reports gains across strong VLM backbones, while its 4B model is competitive with substantially larger models.

  • Problem

    Long-document QA requires connecting heterogeneous evidence across pages and modalities, but existing agents may use fixed workflows or coarse actions that limit flexibility and granularity.

    Source: §1 · p. 2

  • Method

    DocPilot adaptively composes evidence seeking, multimodal understanding, and iterative refinement, transferring these capabilities through curriculum training from single abilities to multi-ability orchestration.

    Source: §1 · p. 3, §2.2 · p. 4, §2.3 · p. 5

  • Results

    Evaluations across four benchmarks show substantial gains across strong VLM backbones, and DOCPILOT-4B is competitive with substantially larger proprietary models.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Learned document interaction offers a route to competitive performance at compact model scale while coordinating evidence seeking and reasoning.

    Source: §3.2 · p. 7, §5 · p. 9

    Generated answers may contain errors and require source-evidence verification, particularly in high-stakes finance and law settings.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Multimodal long-document question answering requires models to seek sparse evidence across pages and modalities, understand heterogeneous document content, and reason over the collected evidence. Even advanced vision-language models (VLMs) can overlook sparse evidence and struggle with cross-modal reasoning in a single pass. Existing document agents further introduce multi-step interaction, but some still rely on predefined workflows or coarse-grained actions, limiting the flexibility and granularity of document reasoning. We introduce DocPilot, an agentic document framework that supports expressive orchestration and makes complex document solving learnable by small language models (SLMs). DocPilot orchestrates evidence seeking through page-level navigation and fine-grained evidence localization, and integrates local visual inspection, code-based numerical reasoning, and cross-modal reasoning for multimodal understanding. To transfer these capabilities to small models, we propose curriculum-based agentic training that progresses from single-ability learning to adaptive orchestration through supervision from data-grounded reasoning traces and executable interaction trajectories. Across MMLongBench-Doc, DocBench, FinRAGBench-V, and MADQA, DocPilot achieves strong gains with frontier VLM backbones, while DocPilot-4B delivers competitive performance against substantially larger models, demonstrating strong effectiveness and generalization across diverse long-document reasoning settings. The code is available at https://github.com/HKUSTDial/DocPilot.

1 INTRODUCTION

Long-document QA requires finding and connecting heterogeneous evidence spread across pages and modalities. DocPilot addresses limits in static retrieval and coarse workflows with adaptive evidence seeking, multimodal reasoning, and curriculum training for compact models.

1 INTRODUCTION

  • Existing document agents can be constrained by predefined workflows and coarse actions, while small models face a difficult orchestration-learning problem.Fine-grained actions make capabilities more explicit for supervision but increase the complexity of adaptive orchestration.Source: §1 · p. 2
  • DocPilot combines adaptive page-level evidence seeking, fine-grained multimodal understanding, and iterative answer verification in a trainable document agent.Its operations include local visual inspection and code-based numerical reasoning.Source: §1 · p. 3
  • Curriculum-based training progresses from single-ability fine-tuning to multi-ability orchestration, using data-grounded reasoning traces and executable interaction trajectories.The two stages establish core interaction capabilities before teaching models to compose them based on intermediate observations.Source: §1 · p. 3
  • Experiments across MMLongBench-Doc, DocBench, FinRAGBench-V, and MADQA show gains across diverse baselines, with DOCPILOT-4B competitive against substantially larger proprietary models.The introduction also notes analyses of evidence localization and training ablations.Source: §1 · p. 3

2 DOCPILOT

DOCPILOT represents heterogeneous documents in a shared multimodal environment and adaptively composes evidence seeking, multimodal reasoning, and verification. Its curriculum transfers these capabilities to compact models by progressing from individual abilities to interaction-level orchestration.

2 DOCPILOT

  • DOCPILOT unifies adaptive evidence seeking, multimodal understanding, and iterative refinement in an agent that dynamically composes actions from intermediate observations.The orchestration framework and curriculum-based training share a multimodal document environment.Source: §2 · p. 3

2.1 MULTIMODAL DOCUMENT ENVIRONMENT

  • The multimodal environment aligns page images, OCR-derived structured representations, thumbnails, and a document-level overview for interaction with heterogeneous evidence.Representations share document and page identifiers for aligned access to textual, structural, and visual content.Source: §2.1 · pp. 3–4

2.2 AGENT ORCHESTRATION FRAMEWORK

  • DOCPILOT uses a closed loop that updates its evidence state and selects actions according to unresolved information needs, rather than following a fixed operation sequence.Navigate scans thumbnails for relevant pages, while Extract obtains structured evidence from selected pages or regions; Focus supports local visual inspection, Solve supports code-based numerical reasoning, and Analyze checks answer grounding.Source: §2.2 · p. 4

2.3 CURRICULUM-BASED AGENTIC TRAINING

  • Curriculum training first fine-tunes single abilities on teacher-generated reasoning traces, then trains adaptive orchestration on synthesized multi-ability interaction trajectories.The synthesized supervision includes reasoning traces for individual capabilities and interaction traces for composing them; environment observations condition training, while only agent-generated outputs are supervised.Source: §2.3 · p. 5

3 EXPERIMENTS

Experiments evaluate DocPilot across long-document QA benchmarks, ablations, evidence localization, interaction behavior, and multi-document reasoning. Results show gains across backbones and benchmarks, while analyses examine tool contributions and adaptive evidence seeking.

3.1 EXPERIMENTAL SETUP

  • The evaluation covers two general-purpose QA benchmarks, a financial benchmark with fine-grained evidence annotations, and MADQA for cross-document generalization, using each benchmark’s official protocol.MMLongBench-Doc and DocBench use rule-based accuracy/F1 and LLM-as-a-judge evaluation, respectively; MADQA includes X-Page, X-Doc, Page F1, and Kuiper metrics.Source: §3.1 · p. 6

3.2 MAIN RESULTS

  • DocPilot improves vanilla backbones by 10.1, 14.7, 10.8, and 13.4 points on MMLongBench-Doc, and by 3.7, 9.3, 6.2, and 10.0 points on DocBench; it also improves FinRAGBench-V by 9.8, 18.5, 11.5, and 18.7 LasJ points.The respective backbones are Qwen3.5-397B-A17B, Gemini-2.5-Pro, Gemini-3.1-Pro, and GPT-5.4.Source: §3.2 · p. 6, Table 1 · p. 7
  • With Gemini-2.5-Pro, DocPilot raises MMLongBench-Doc accuracy from 67.6 with DocLens to 72.8, FinRAGBench-V LasJ from 70.4 to 73.4, and DocBench LasJ to 84.1 versus DocDancer’s 79.9.Source: §3.2 · p. 6, Table 1 · p. 7
  • With only 4B parameters, DocPilot-4B surpasses vanilla Gemini-2.5-Pro on all three benchmarks and GPT-5.4 on FinRAGBench-V and DocBench, nearly matching Gemini-3.1-Pro on DocBench (80.3 vs. 80.6).Its absolute gains over vanilla Qwen3.5-4B are 4.2%, 7.5%, and 11.0% on MMLongBench-Doc, FinRAGBench-V, and DocBench, respectively.Source: §3.2 · p. 7, Table 1 · p. 7

3.3 ABLATION STUDIES

  • Without <Analyze>, accuracy falls from 73.5 to 64.9, while removing <Focus> reduces chart and figure accuracy by 7.2 and 11.9 points, respectively.Removing <Solve> reduces table accuracy by 4.6%; the analysis says the richer tool set requires adaptive orchestration aligned with intermediate observations.Source: Figure 5 · p. 7, §3.3 · p. 8

3.4 BEHAVIOR AND ROBUSTNESS ANALYSIS

  • DocPilot reaches 73.2 precision and 91.8 recall while retrieving 3.0 pages on average, improving final QA accuracy from 67.6 to 72.8; on FinRAGBench-V, it exceeds DocLens by +6.8 precision, +0.8 recall, and +4.5 F1 points.Page localization with DocLens reaches 97.3 recall at 55.1 precision while retrieving 3.5 pages on average.Source: §3.4 · p. 8, Table 3 · p. 8, Figure 6 · p. 8
  • Navigate averages 2.6–3.0 calls across backbones; Gemini-3.1-Pro has the shortest trajectories (7.9 actions), while Qwen3.7-397B-A 17B has the longest (11.4).GPT-5.4 rarely invokes <Solve> (0.1), favoring <Focus>; repeated calls reflect iterative evidence acquisition, inspection, and verification.Source: §3.4 · p. 8, Figure 7
  • On MADQA’s 200-sample development set with relevant documents provided, GPT-5.4-backed DocPilot reaches 86.5 accuracy, comparable to Codex (86.7), and the best X-Doc score of 89.5.The setting isolates evidence localization from document retrieval; reported results also include 59.5 X-Page and 75.0 Page F1.Source: §3.4 · p. 8, Table 4 · p. 9

4 RELATED WORK

Prior work spans OCR or parser representations, direct VLM processing, retrieval-augmented methods, and interactive document agents. The paper positions DocPilot as combining fine-grained actions with curriculum training to address constraints in orchestration and supervision.

4.1 MULTIMODAL LONG-DOCUMENT REASONING

  • Prior multimodal long-document approaches include text or structural extraction, direct VLM processing, retrieval of text or pages, and agentic interaction, but static retrieval can miss reasoning evidence and prescribed workflows or limited action spaces constrain adaptive orchestration.Source: §4.1 · p. 9

4.2 TRAINABLE DOCUMENT AGENTS

  • Document-agent training uses trajectories and environment feedback, while synthetic trajectories can scale supervision; DocPilot addresses weak multimodal supervision from coarse actions and difficult adaptive composition with data-grounded synthesis and curriculum training.Source: §4.2 · p. 9

5 CONCLUSION AND FUTURE WORK

The conclusion presents DocPilot as a shared-environment framework for adaptive evidence seeking, multimodal understanding, and iterative refinement, trained through a curriculum of synthesized trajectories. Future work proposes learning from accumulated interaction trajectories and consolidating domain knowledge across tasks.

5 CONCLUSION AND FUTURE WORK

  • Evaluations on four benchmarks show gains across strong VLM backbones and improved evidence localization, while ablations highlight the importance of coordinating complementary tools.Source: §5 · p. 9
  • Future work will explore self-evolution by distilling successful strategies and failure lessons from accumulated trajectories, and an evolving semantic layer for consolidating domain knowledge across tasks.Source: §5 · p. 9

AI USE STATEMENT

The statement says LLMs were used for auxiliary tasks, not for the paper’s research design, methodology, or core writing.

AI USE STATEMENT

  • LLMs supported language refinement, minor code debugging, synthetic data construction, and experiment evaluation, but did not contribute to research design, methodology, or core writing.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement reports use of publicly available document QA datasets and flags answer verification, high-stakes use, and privacy safeguards as deployment concerns.

ETHICS STATEMENT

  • Experiments use publicly available document QA datasets.Source: ETHICS STATEMENT · p. 10
  • Generated answers may contain errors and should be checked against source evidence, especially in high-stakes finance and law settings.Source: ETHICS STATEMENT · p. 10
  • Deployments involving sensitive documents should include appropriate access controls and privacy safeguards.Source: ETHICS STATEMENT · p. 10

A DATASET STATISTICS

The evaluation spans four benchmarks covering general, financial, and cross-document reasoning, while the agent environment aligns structured OCR, page visuals, and adaptive actions for evidence seeking and reasoning.

A DATASET STATISTICS

  • Four benchmarks cover general-purpose document understanding, financial analysis, and cross-document reasoning.MMLongBench-Doc and DocBench provide general-purpose evaluation; FinRAGBench-V covers financial documents; MADQA tests document-collection reasoning.Source: A DATASET STATISTICS · p. 14

B.1 DETAIL OF ENVIRONMENT DESIGN

  • The environment represents each page with structured OCR elements, a high-resolution screenshot, and a thumbnail, then arranges thumbnails into indexed overviews.The OCR representation retains content, type, reading order, bounding box, and page index; overviews preserve document organization at lower visual cost.Source: §B.1 · p. 15

B.2 AGENT–ENVIRONMENT INTERACTION PROTOCOL

  • A ReAct interaction loop selects actions from observations and history, while state-conditioned availability constrains which actions are valid.The six actions are Analyze, Navigate, Extract, Focus, Solve, and Answer; successful execution updates persistent state for subsequent decisions.Source: §B.2 · p. 15
  • The interaction verifies candidate answers against accumulated evidence, continues seeking or reasoning when gaps remain, and returns insufficient evidence at the step limit.Source: §B.2 · pp. 15–16

B.3 MODULE DESIGN DETAILS

  • Navigate locates candidate pages, Extract retrieves relevant evidence, Focus inspects visual regions, and Solve executes Python over collected evidence.The tools use structured arguments and are selected adaptively rather than in a fixed sequence.Source: §B.3 · p. 16

C TRAJECTORY SYNTHESIS DETAILS

DocPilot synthesizes data-grounded traces to train individual document-solving abilities and their adaptive composition, filtering trajectories for correctness, validity, and evidence grounding.

C TRAJECTORY SYNTHESIS DETAILS

  • Data-grounded trajectory synthesis addresses the lack of intermediate-decision supervision in document QA and the cost of manually collecting interactive traces.Existing document QA datasets predominantly provide question-answer triplets; annotations constrain teacher generation to produce executable, evidence-grounded traces.Source: C TRAJECTORY SYNTHESIS DETAILS · pp. 16–17
  • Single-ability traces supervise navigation, extraction, code-based solving, and multimodal answering, while multi-ability datasets support complete interaction trajectories.Sources include DocSeeker, HotpotQA, DeepAnalyze, FinQA, ChartQAPro, InfographicVQA, LongDocURL, DUDE, MADQA, and MMDocRAG.Source: C TRAJECTORY SYNTHESIS DETAILS · p. 17
  • The teacher generates capability-specific reasoning traces and full environment interaction trajectories, revisiting tools as evidence gaps emerge rather than following a fixed action sequence.Generated code is executed for validation, and other traces are checked against reference answers or evidence annotations.Source: C TRAJECTORY SYNTHESIS DETAILS · p. 17
  • Trajectory quality control checks answer correctness, interaction validity, and evidence grounding, and discards malformed traces, repeated loops, and runs without useful evidence acquisition.Source: C TRAJECTORY SYNTHESIS DETAILS · p. 17
  • The filtered DOCAGENT-INSTRUCT-40K corpus contains about 40K examples, including 33,938 single-ability traces and 4,752 multi-ability traces.Maximum text-token lengths are 8K and 20K for the respective stages, excluding visual tokens; Figure 8 shows token-length distributions across both stages.Source: Figure 8 · p. 17, C TRAJECTORY SYNTHESIS DETAILS · p. 18

D.1 EVIDENCE RECALL ACROSS NAVIGATION ROUNDS

Navigation accumulates evidence over successive rounds, with substantial early gains, diminishing returns at greater depth, and final coverage that varies by backbone.

D.1 EVIDENCE RECALL ACROSS NAVIGATION ROUNDS

  • By round three, cumulative recall reaches 80.29%–88.06%, up 33.34–41.04 percentage points from first-round recall of 45.67%–50.71%; final recall reaches 90.04%–94.63%.Source: §D.1 · p. 18
  • All backbones reach 96.9%–99.0% of final recall by round five, while navigation beyond round six adds only 0.34–1.60 percentage points.Individual questions may still require further evidence seeking despite aggregate saturation.Source: §D.1 · p. 18
  • Gemini-3.1-Pro exceeds GPT-5.4 in recall from rounds two through six, but their final coverage converges to 94.55% versus 94.63%.Qwen3.5-397B-A17B improves more gradually, reaching 90.04% by round 24.Source: §D.1 · p. 18

D.2 TOOL USAGE ANALYSIS OF DOCPILOT-4B

DOCPILOT’s results span document types, while tool-use analysis shows how curriculum training redistributes interactions toward finer-grained evidence processing. The gains vary by document type and backbone.

D.2 TOOL USAGE ANALYSIS OF DOCPILOT-4B

  • 16.41 vs. 16.43 total calls: curriculum training and direct multi-ability fine-tuning use nearly identical numbers of tool calls.<Extract>, <Analyze>, and <Answer> usage remains largely unchanged.Source: §D.2 · p. 18

D.3 PERFORMANCE ACROSS DOCUMENT TYPES.

  • 10.1–13.4 percentage points: DOCPILOT improves overall accuracy over corresponding backbones and achieves the best results in every MMLongBench-Doc document category.The reported improvements span seven document types.Source: Table 9 · p. 19, §D.3 · p. 19

D.2 TOOL USAGE ANALYSIS OF DOCPILOT-4B

  • 3.75 to 2.87: curriculum training reduces <Navigate> calls, while <Focus> rises from 0.74 to 1.46 and <Solve> from 0.71 to 0.85.This shifts interactions from broad page exploration toward local visual inspection and code-based reasoning; call frequencies alone do not establish whether invocations are appropriate or successful.Source: §D.2 · p. 19

D.3 PERFORMANCE ACROSS DOCUMENT TYPES.

  • The strongest backbone varies by document type: Gemini-3.1-Pro has the highest overall accuracy (75.6), while other backbones lead on specific categories.On brochures, Gemini-3.1-Pro and GPT-5.4 improve by 1.8 and 5.0 points, respectively, while Qwen3.5-397B declines from 62.0 to 58.3.Source: §D.3 · p. 19
  • 54.9 to 59.1: DOCPILOT-4B improves overall accuracy, with gains across six of seven document types.Its largest gains are industry (+9.8), finance (+6.8), and research (+6.4).Source: §D.3 · p. 19

E CASE STUDY

The case studies show DOCPILOT-4B composing navigation, extraction, visual inspection, and reasoning in response to different document questions. The system prompts assign distinct evidence-seeking, inspection, computation, answering, and verification roles.

E CASE STUDY

  • 245: DOCPILOT-4B combines dataset statistics with a distribution chart, locating evidence with <Navigate> and <Extract>, inspecting it with <Focus>, then computing with <Solve>.Vanilla Qwen3.5-4B returns unknown for this case.Source: E CASE STUDY · pp. 19–20, Figure 10 · p. 20
  • 15:00: Two successive <Focus> operations identify the timer-chart bar’s label, avoiding a nearby textual example that led vanilla Qwen3.5-4B to answer incorrectly.The case illustrates that page-level relevance alone may not resolve questions about small visual details.Source: E CASE STUDY · p. 20, Figure 11 · p. 21
  • After an unsuccessful <Solve>, <Analyze> redirects the cross-page comparison to <Answer>, which selects Figures 4 and 5 as conveying a message similar to the reference figure.The agent gathers comparison evidence across the main paper and appendices.Source: E CASE STUDY · p. 20, Figure 12 · p. 21
  • The cross-document case links the 2023 settlement year to an official’s title and signature date, identifying Alex Martin after <Analyze> prompts re-navigation to the TN Board signature page.Intermediate observations guide document switching and role disambiguation.Source: E CASE STUDY · p. 20, Figure 13 · p. 21

F SYSTEM PROMPT DESIGN

  • The navigation prompt selects potentially relevant pages, prioritizes recall on new pages, revisits prior candidates for precision, and requests more pages when evidence is incomplete.It also forbids selecting pages outside the supplied page set.Source: F SYSTEM PROMPT DESIGN · p. 22
  • The extraction and focus prompts retain actionable evidence across modalities and plan localized visual operations such as subfigure splits, crops, and zooms.The Focus Planner performs visual operations rather than answering the question directly.Source: F SYSTEM PROMPT DESIGN · pp. 22–23
  • The solver prompt requires deterministic Python computation over supplied inputs, while the Answer Agent grounds concise answers in evidence and applies explicit fallback rules.Textual analysis checks both answer support and search coverage before accepting a candidate.Source: F SYSTEM PROMPT DESIGN · pp. 23–25

Related papers

Loading 2610.21327v1…