PaperlayerView exact revision on arXiv

Source-linked AI summary

RouteDRAG: Learning to Route and Control Agentic Execution for Multimodal Document RAG

Zekai Chen, Hao Zhu, Yu Liu, Xunkai Li, Zejian Kang, Rui Qian, Quanjiang Li, Rong-Hua Li

arXiv:2610.36444v1cs.AI

TL;DR

Long multimodal document questions can require combining evidence across pages, tables, and figures, while Agentic execution adds computation and may not be necessary for every query. RouteDRAG learns both whether to invoke an Agent and how to control its subsequent operations. Across two benchmarks, it improves accuracy over uniform Agent execution while requiring fewer answering calls.

  • Problem

    Long multimodal document questions can require combining information across pages, tables, and figures, while Agentic execution is not always necessary and can add computation or incorrect answers.

    Source: §1 · p. 1

  • Method

    RouteDRAG combines a graph-backed document index, learned routing between shorter paths and Agent, adaptive Agent operation control, and sequential policy learning.

    Source: §1 · p. 2, §5.1 · p. 4

  • Results

    RouteDRAG improves accuracy and reduces answering calls relative to uniform Agent execution on both multimodal document benchmarks.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Agent execution’s query-dependent benefits and non-monotonic operation effects motivate controlling both Agent invocation and subsequent operations.

    Source: §1 · p. 2

Abstract

from arXiv · show

Web-based applications increasingly rely on understanding long multimodal documents, where answering questions often requires combining information across pages, tables, and figures. Agentic retrieval-augmented generation addresses these tasks through iterative retrieval and reasoning. However, Agentic execution is not always necessary, introducing additional computation and potentially incorrect answers. Our empirical analysis reveals that Agentic execution provides query-dependent benefits, while additional operations may improve, preserve, or degrade answer accuracy. These findings motivate controlling both Agent invocation and subsequent execution. To address these challenges, we propose RouteDRAG, a learning-based framework comprising four components. Unified Document Index organizes multimodal content into a graph-backed index and derives query representations from initial retrieval. Adaptive Path Routing preserves exact answering and predicts the relative benefit of Agent over shorter execution paths. Dynamic Agent Execution maintains an evolving workspace and refines Planner proposals to control tool use and termination. Sequential Policy Learning first trains operation control from observed outcomes and then trains path routing using the resulting Agent's performance. Across two benchmarks, RouteDRAG achieves an average accuracy improvement of 3.47 percentage points over shorter-path-only execution while assigning only 79.3% of queries to Agent. Notably, it also surpasses uniform Agent execution on both benchmarks while requiring fewer answering calls. Our code is now available at https://github.com/Ereshkigal1/RouteDRAG.

1 Introduction

Long multimodal document questions often require combining information across pages, tables, and figures, but Agentic retrieval is not always necessary. RouteDRAG addresses when to invoke an Agent and how to control its later operations, motivated by query-dependent benefits and non-monotonic operation outcomes.

1 Introduction

  • Questions over long, visually rich documents can require locating and combining information distributed across pages, tables, and figures, which a single retrieval step may not fully provide.Agentic RAG addresses this with iterative retrieval, visual understanding, and multi-step reasoning.Source: §1 · p. 1
  • Agent improves accuracy by 1.00 and 3.12 percentage points on DocBench and MMLongBench-Doc, respectively, but uses 1.65× and 1.81× as many answering tokens as Standard.The evidence also reports query-dependent benefits and cases where additional operations improve, preserve, or degrade correctness.Source: §1 · p. 2
  • RouteDRAG combines a graph-backed Unified Document Index, Adaptive Path Routing, Dynamic Agent Execution, and Sequential Policy Learning to adapt Agent invocation and operations to individual queries.Path routing predicts Agent’s benefit over a shorter path; sequential learning trains operation control first, then path routing using the resulting Agent’s performance.Source: §1 · p. 2
  • 75.86% and 53.80% accuracy on DocBench and MMLongBench-Doc, respectively, with 16.4% and 14.9% fewer answering calls than uniform Agent execution.The evaluation compares RouteDRAG with nine RAG methods.Source: §1 · p. 2

2 Related Work

Prior work develops multimodal document representations and more flexible Agentic information acquisition, while adaptive retrieval research selects strategies or controls operations. RouteDRAG focuses on jointly deciding whether Agent execution is needed and controlling its subsequent operations.

2.1 Multimodal and Agentic Document RAG

  • ReAct [28] established a reasoning-and-acting paradigm that interleaves language-model reasoning with external actions.Source: §2.1 · p. 2
  • Graph-based and visual document methods organize multimodal evidence or explore documents, while RouteDRAG instead targets whether Agent execution is needed and how its later operations are controlled.Examples include LightRAG [13], RAG-Anything [12], GraphReader [17], G2-Reader [8], VisRAG [29], M3DocRAG [6], and AgenticOCR [26].Source: §2.1 · p. 2

2.2 Adaptive Retrieval and Execution Selection

  • Adaptive methods select retrieval strategies or acquire evidence dynamically, including Self-RAG [1], Adaptive-RAG [14], Active Retrieval Augmented Generation [16], and MBA-RAG [24].Source: §2.2 · p. 2
  • Multi-step methods organize query-specific workflows or learn retrieval-operation policies; RouteDRAG connects query-level Agent-benefit estimation with operation refinement based on an evolving workspace.Examples include MAO-ARAG [4] and SPARKLE; sequential learning aligns routing with the behavior of the resulting Agent.Source: §2.2 · p. 2

3 Problem Definition

The problem is to choose among four execution paths for a multimodal document query while controlling whether an Agent is invoked and, if so, what it does next. The objective is to maintain answer accuracy while avoiding unnecessary Agent invocation and operations.

3 Problem Definition

  • For document D and query q, the framework considers Exact, Standard, Direct VLM, and Agent paths using the unified index K_D.Source: §3 · p. 2
  • Exact, Standard, and Direct VLM provide shorter procedures, whereas Agent supports iterative retrieval and reasoning.Source: §3 · p. 2
  • Routing decides whether Agent is necessary; after invocation, an execution policy selects legal actions from the evolving workspace, including continuing or terminating.Source: §3 · p. 2
  • The objective is to maintain answer accuracy while avoiding unnecessary Agent invocation and subsequent operations.Source: §3 · p. 2

4 Empirical Study

The empirical study examines whether uniform Agent execution justifies its cost, whether its benefits vary across queries, and whether additional operations consistently improve answers. Results show modest aggregate gains at higher token use, query-level correctness changes in both directions, and non-monotonic operation effects.

4 Empirical Study

  • The study distinguishes the overall cost comparison, query-level benefits, and effects of one additional operation before termination.Source: §4 · pp. 2–3

O1: Limited Overall Gains

  • Agent improves accuracy by 1.00 and 3.12 percentage points on DocBench and MMLongBench-Doc, respectively, while consuming 1.65× and 1.81× the answering tokens.Figure 2(a) compares Standard and Agent across five DocBench domains and seven MMLongBench-Doc document types.Source: §4 · p. 3, Figure 2 · p. 3

O2: Query-Dependent Benefits

  • Correctness is unchanged for 94.83% of DocBench queries and 89.73% of MMLongBench-Doc queries; Agent recovers 3.09% and 6.69% but harms 2.09% and 3.57%, respectively.The paired comparison supports query-dependent Agent benefits rather than uniformly beneficial execution.Source: §4 · p. 3, Figure 2 · p. 3

O3: Non-Monotonic Operation Gains

  • Across 917 matched operation comparisons, 84.51% leave correctness unchanged, while additional operations can improve or degrade answers.VISUAL operations degrade correctness more often than they improve it, motivating adaptation to the evolving workspace.Source: Figure 2 · p. 3, §4 · p. 4

5 Methodology

RouteDRAG learns two linked decisions from execution outcomes: when to invoke an Agent and how to control its subsequent operations. Its shared index, adaptive routing, dynamic execution, and sequential policy learning support these decisions.

5.1 Overview

  • RouteDRAG learns when to invoke an Agent and how to control its subsequent operations from observed execution outcomes.Its four components are Unified Document Index, Adaptive Path Routing, Dynamic Agent Execution, and Sequential Policy Learning.Source: §5.1 · p. 4

5.2 Unified Document Index

  • The shared index links multimodal document content to original locations, while query-conditioned representations combine the query with initial retrieval results.The representation uses retrieved text chunks and graph entities, with validity masking for unavailable entries.Source: §5.2 · p. 4

5.3 Adaptive Path Routing

  • After an Exact gate, routing selects the Agent only when its predicted improvement over the designated shorter path exceeds a development-set threshold.The shorter path is chosen by a fixed policy from Standard and VLM; routing predicts relative benefit rather than each path’s correctness independently.Source: §5.3 · p. 4

5.4 Dynamic Agent Execution

  • The Agent scores legal operations against immediate termination, refines valid Planner proposals, and commits results to an evolving workspace until FINISH or the execution limit.The workspace retains retrieved content, source references, and operation history; mandatory actions follow native rules.Source: §5.4 · pp. 4–5

5.5 Sequential Policy Learning

  • Sequential training first fits the Agent controller from observed operation outcomes, then trains the router using the updated Agent’s execution results.Operation learning compares termination with one additional operation at evaluated states; path learning compares the updated Agent with the fixed shorter path.Source: §5.5 · pp. 5–6

6 Experiments

RouteDRAG is evaluated on two multimodal document benchmarks against document-RAG baselines and controlled execution strategies. The experiments examine overall accuracy, component contributions, robustness, and efficiency.

6.1 Experimental Setup

  • Experiments use DocBench and MMLongBench-Doc, compare four RAG baseline categories and controlled execution strategies, and use the same Qwen3-VL-8B-Instruct answering model.Accuracy is assessed with a fixed Qwen-based judge; the main comparison reports means and standard deviations over five runs.Source: §6.1 · p. 6

6.3 Ablation Study (Q2)

  • RouteDRAG reaches 75.86%/53.80% accuracy with 78.49%/80.02% Agent use, while RouteDRAG-Efficient reaches 74.05%/50.87% with 11.52%/16.41% Agent use.Removing Agent execution yields 73.41%/49.31%; Uniform Agent yields 74.68%/52.52%.Source: §6.3 · p. 6, Table 3 · p. 7

6.2 Overall Performance (Q1)

  • 75.86% on DocBench and 53.80% on MMLongBench-Doc; RouteDRAG also exceeds Uniform Agent by 1.18 and 1.28 percentage points while reducing answering calls by 16.4% and 14.9%.The evaluation reports mean accuracy and standard deviations over five independent runs.Source: §6.2 · p. 6, Table 2 · p. 7, Table 1 · p. 7

6.4 Robustness Analysis (Q3)

  • Accuracy varies modestly across displayed DocBench settings, while MMLongBench-Doc is more sensitive when strong regularization coincides with a positive threshold offset.Stronger regularization compresses the prediction scale, allowing the same offset to exclude more Agent executions; the figure reports accuracy and percentage-point changes from its reference setting.Source: Figure 4 · p. 6, §6.4 · p. 8
  • At the highest noise ratio, RouteDRAG exceeds TAP-RAG by 5.9/13.5 points under Retrieval Noise and 8.5/12.4 points under Document Distractors; gains over Uniform Agent vary across backbones.With Qwen3-VL-4B, RouteDRAG reaches 70.15%/46.65% accuracy and reduces answering calls by 13.8%/14.2% versus Uniform Agent.Source: §6.4 · p. 8, Figure 5 · p. 8, Table 4 · p. 8

6.5 Efficiency Analysis (Q4)

  • RouteDRAG uses 6.24/7.02 answering calls and 46.04k/50.99k tokens per query, while improving accuracy over All Agent Routing and reducing calls and tokens.Figure 6 compares accuracy with inference latency and compares peak GPU memory across methods.Source: §6.5 · p. 8, Figure 6 · p. 8

7 Conclusion

Agentic execution has query-dependent benefits, and additional operations can improve, preserve, or degrade answer correctness. RouteDRAG selectively routes queries and controls subsequent Agent operations, improving accuracy with fewer answering calls than uniform Agent execution on two benchmarks.

7 Conclusion

  • Agentic execution benefits depend on the query, while additional operations yield gains, ties, and degradations in answer correctness.The study focuses on single-turn document question answering; extending selective execution to multi-turn interactions is future work.Source: §7 · p. 9
  • RouteDRAG combines adaptive path routing and dynamic Agent execution through sequential policy learning from observed execution outcomes.It selectively invokes Agent and refines subsequent operations.Source: §7 · p. 9
  • Experiments on two multimodal document benchmarks show improved accuracy with fewer answering calls than uniform Agent execution.The conclusion identifies multi-turn document interactions as a direction for future research.Source: §7 · p. 9

A.1 Datasets, Readers, and Execution Paths

The appendix specifies the evaluation datasets and execution paths, then gives comparison-specific configurations for several baselines.

A.1 Datasets, Readers, and Execution Paths

  • Evaluation covers all 1,102 DocBench questions from 229 documents and 1,091 MMLongBench-Doc questions from 135 documents.All paths share the document index and frozen BGE-M3 representations, and use Qwen3-VL-8B-Instruct with an FP8 backend.Source: §A.1 · p. 11
  • The designated short path uses Standard or eligible Direct VLM, while the Actor is the Agent with its native Planner, Workspace, legalaction filtering, and learned intervention.Controlled allocation comparisons reuse recorded short-path and Actor answers, keeping Exact Specialist outputs fixed.Source: §A.1 · p. 11

A.2 Baseline Configurations

  • Baseline configurations include native retrieval procedures, document-local retrieval settings, and a shared Qwen3-VL-8B-Instruct answering model and evaluation protocol for most baselines.Budgeted Direct MLLM reads all pages up to 128; M3DocRAG selects three pages rendered at 144 DPI.Source: §A.2 · p. 11

A.3 Robustness Evaluation Protocol

The protocol tests robustness to retrieval and document distractors, measures inference efficiency, and specifies the routing, validation, indexing, execution, and learning procedures. Its evaluation uses document-disjoint folds, while Agent execution proceeds through a shared workspace and legal actions.

A.3 Robustness Evaluation Protocol

  • At noise ratios ρ∈{0, 0.1, 0.2, 0.3, 0.4, 0.5}, retrieval noise appends irrelevant units while document distractors add unrelated pages before retrieval.At ρ=0.5, retrieval noise adds half the original retrieved units; document distractors add half the original page count, preserving original page order.Source: §A.3 · p. 11

A.4 Inference Efficiency Measurement

  • Efficiency measurement reports mean end-to-end latency and peak GPU memory for batch-size-one runs on one NVIDIA A100 GPU.Each method receives five warm-up queries and 100 timed queries; latency includes online retrieval through answer generation but excludes offline parsing and index construction.Source: §A.4 · p. 11

A.5 Routing Parameters

  • The router invokes Agent when its predicted benefit over the designated short path exceeds threshold τ=0.02; otherwise, it executes that path.The router is trained on paired Agent and short-path outcomes, with configurations and thresholds selected using development data.Source: §A.5 · p. 11

A.6 Document-Disjoint Cross-Validation

  • Five-fold cross-validation assigns each source PDF to one outer fold, with an inner development split and training split drawn from the remaining documents.The approximate document proportions are 64% training, 16% development, and 20% test; pooled out-of-fold predictions provide one routing decision per query.Source: §A.6 · p. 11

B.1 Shared Index and Source References

  • The shared index combines vector retrieval, graph relations, and source references that link visual descriptions to original pages and image assets.A learned projection maps frozen 1,024-dimensional vectors to 64 dimensions; source references retain document-local identifiers and locations.Source: §B.1 · p. 12

B.2 Exact and Short-Path Interfaces

  • Exact returns answers only for supported structured requests passing completeness checks; otherwise, a fixed policy selects Standard or eligible Direct VLM.Standard uses graph-backed retrieval and answering, while Direct VLM uses a text precheck and invokes visual reading when needed.Source: §B.2 · p. 12

B.3 Workspace, Legal Actions, and Tool Arguments

  • The evolving Workspace stores retrieved content, provenance, tool outcomes, and action history, while the Controller scores legal operations against the Planner’s proposal.Legal actions can include text search, graph expansion, structured lookup, visual inspection, and FINISH; the original synthesis procedure produces the final answer from accumulated content and references.Source: §B.3 · p. 12

B.4 Learning Signals

  • Learning uses correctness before and after an additional operation for operation-level signals, and paired Actor and short-path correctness outcomes for the routing target.Planner imitation supplies an auxiliary signal preserving the Planner’s operation preferences.Source: §B.4 · p. 12

B.5 Training and Policy Selection

Training proceeds in two stages, followed by per-query execution using a selected routing threshold. The accompanying analyses describe tool use across document-length groups and the benefit predictor’s validation objective.

B.5 Training and Policy Selection

  • Sequential training first fits the shared projection and operation Controller from operation outcomes, then freezes them while routing heads learn from paired Actor and short-path correctness.Development objectives select checkpoints and the routing threshold; during inference, the Controller refines a Planner proposal only when a legal alternative exceeds its score by the fixed margin.Source: §B.5 · p. 12

C.1 Natural Document-Length Profile

  • RouteDRAG uses fewer tool actions than uniform execution across displayed document-length groups, while action counts also vary within a length range.The longest MML bin contains five questions from one PDF.Source: §C.1 · p. 13, Figure A1 · p. 13

C.2 Validation-Loss Trace

  • The benefit predictor’s validation objective combines signed-benefit regression with a ranking term weighted by 0.25, and validation selects epoch 47.The trace describes optimization of the routing objective; workload and document-disjoint allocation results are reported separately.Source: §C.2 · p. 13, Figure A2 · p. 13

C.3 Per-Query Routing Operating Points

The routing configurations expose different accuracy and Agent-usage operating points, with the highest-accuracy RouteDRAG setting also using Agent most often.

C.3 Per-Query Routing Operating Points

  • RouteDRAG-Efficient achieves 74.05%/50.87% accuracy with 11.52%/16.41% Agent usage and 2.58/3.20 calls per query on DocBench/MMLongBench-Doc.The table compares four routing configurations with thresholds calibrated on development data.Source: §C.3 · p. 13, Table A1 · p. 13
  • Increasing routing intensity improves accuracy at the cost of higher resource use across the reported operating points.RouteDRAG-Medium and RouteDRAG-High lie between RouteDRAG-Efficient and RouteDRAG in accuracy and Agent usage.Source: §C.3 · p. 13, Table A1 · p. 13
  • 75.86% DocBench accuracy and 53.80% MMLongBench-Doc accuracy are RouteDRAG’s highest reported routing-configuration results, with 78.49%/80.02% Agent usage.RouteDRAG also has the most calls per query, at 6.24/7.02 on the two benchmarks, respectively.Source: Table A1 · p. 13, §C.3 · p. 13
Loading 2610.36444v1…