Source-linked AI summary
D2-ScaleAgent: Dual-Dimensional Scaling for Long Document Understanding
Hao Zhang, Longrong Yang, Lunhao Duan, Ziyang Wang, Qing-Guo Chen, Shanshan Zhao
TL;DR
D2-ScaleAgent addresses evidence insufficiency in visually rich long-document understanding, where existing methods rely on fixed workflows. It dynamically scales retrieval and reasoning through verifier-driven routing, achieving logical closure and state-of-the-art performance on complex benchmarks.
Problem
Existing visually rich long-document methods rely on fixed workflows and lack test-time computation scaling, leaving evidence insufficiency insufficiently addressed.
Method
D2-ScaleAgent uses a Verifier-driven closed loop and continuously updated Evidence Bank to route outward for retrieval scaling or inward for reasoning scaling.
Results
D2-ScaleAgent achieves state-of-the-art performance on multiple complex visually rich long-document benchmarks, outperforming traditional RAG and static-agent workflows.
Takeaways & Limitations
The framework delivers logical closure over evidence chains for complex long-document understanding benchmarks.
Takeaways & Limitations
The dynamic routing architecture increases inference latency and computational overhead, limiting suitability for real-time or severely latency-constrained deployment.
Abstract
from arXiv · showhide
Multi-modal retrieval-augmented generation (RAG) is a key technique for visually rich long document understanding. Existing multi-modal RAG methods are progressively advancing toward multi-agent systems: they first retrieve relevant pages based on a query, and then iteratively understand information within those pages. However, these methods typically rely on fixed workflows and lack the ability to dynamically scale computation at test time, often leading to insufficient evidence. To address this, we propose D2-ScaleAgent, an agentic framework that introduces a dual-dimensional scaling paradigm for retrieval and reasoning. The core of D2-ScaleAgent is a Verifier agent-driven dynamic routing loop based on the intrinsic difficulty of the query, centered around a continuously updated evidence bank that serves as the agent's dynamic working memory: when retrieval needs to be expanded, the agent routes outward (retrieval scaling), decomposing the query into attributes and performing parallel page retrieval, followed by adaptive pruning to ensure comprehensive evidence coverage. When fine-grained reasoning is required, the agent routes inward (reasoning scaling), dynamically selecting sub-agents with varying granularity and count to extract evidence from pages. Finally, D2-ScaleAgent achieves logical closure over the evidence chain. Extensive experiments demonstrate that D2-ScaleAgent is effective on long and visually rich document benchmarks like MMLongBench-Doc, LongDocURL, etc.
1 Introduction
D2-ScaleAgent frames long-document understanding as an evidence-insufficiency problem and replaces static retrieve-then-read workflows with dual-dimensional, verifier-driven scaling. Its Evidence Bank coordinates retrieval and reasoning, achieving state-of-the-art performance on visually rich long-document benchmarks.
- Motivation: LVLMs struggle with visually rich long documents because fine-grained visual detail needs conflict with limited context windows.This mismatch often causes performance drops on long documents.
- Problem formulation: Long-document understanding primarily requires finding sufficient and adequate evidence within limited computational resources, not merely processing document length.The introduction identifies evidence insufficiency as the core challenge.
- Prior approaches: Existing approaches combine multimodal RAG for relevant-page filtering with multi-agent or iterative mechanisms for multi-step document analysis, but typically follow static workflows.These strategies first retrieve relevant pages or navigate complex structures through iterative reasoning.
- Proposed framework: D2-ScaleAgent introduces dual-dimensional scaling through a Verifier agent-driven closed loop centered on an Evidence Bank as dynamic working memory.The bank tracks acquired pages, regions, and atomic evidence, including whether evidence supports, conflicts with, or remains missing for the answer.
- Evaluation: D2-ScaleAgent achieves state-of-the-art performance across complex, visually rich long-document benchmarks, outperforming traditional RAG and static-agent workflows.The reported evaluation includes MMLongBench-Doc and is summarized in Figure 1(b).
2 Related Work
Related work advances multi-modal RAG through improved visual and textual retrieval and applies multi-agent collaboration to complex long-document reasoning. However, predetermined workflows and fixed budgets can limit dynamic allocation of reasoning depth, causing incomplete evidence resolution for fine-grained visual details.
- Multi-modal Retrieval-Augmented Generation: Multi-modal RAG improves document question answering by supplying models with externally retrieved visual or textual contexts.Recent work advances retrieval mechanisms, including end-to-end visual retrieval over PDF pages and fine-grained retrieval.
- Multi-Agent Systems: Multi-agent systems use specialized sub-agents with distinct divisions of labor to iteratively reason over complex long documents.Examples include multi-role collaboration for mining text-image details and outline navigation, interactive reading, and reviewer agents for long-document processing.
- Multi-Agent Systems: Predetermined workflows and fixed budgets struggle to dynamically allocate reasoning depth for fine-grained visual details, causing incomplete evidence resolution.Prior systems include procedural test-time scaling and iterative seeker-inspector-answer collaboration, but their rigidity produces Depth Insufficiency.
3 Methodology
D2-ScaleAgent reformulates long-document understanding as evidence-driven, on-demand computation routing through a continuously updated Evidence Bank. A Verifier dynamically alternates retrieval scaling for breadth insufficiency and reasoning scaling for depth insufficiency until logical evidence closure supports answer generation.
- Evidence-driven state: The Evidence Bank is the unified epistemic state, tracking accumulated page-, region-, and atomic-level evidence alongside completeness and evidence-gap signals.These signals guide computational-budget allocation across retrieval and reasoning scaling.
- Retrieval scaling: Retrieval scaling decomposes queries into weighted attribute perspectives, accumulates complementary candidate pages, fuses rankings, and adaptively prunes evidence until cross-round stability indicates convergence.Its stopping condition is an evidence-stability state rather than a fixed number of rounds or rigid Top-K budget.
- Reasoning scaling: Reasoning scaling invokes cost-stratified sub-agents from coarse to fine: a Surveyor scans pages, a Locator focuses regions, and an Extractor reads verifiable atomic facts.The mechanism targets depth insufficiency and dynamically generates extraction specifications from current evidence gaps.
- Logical closure: Once the strict evidence-closure condition is satisfied, the loop stops and generation uses the logically saturated Evidence Bank rather than incomplete information, avoiding unsupported conclusions.Multi-granularity evidence is continuously appended to globally accumulated sets during reasoning.
- Verifier routing: The Verifier assesses Evidence Bank completeness, identifies logical discontinuities or cross-modal contradictions, updates global state, and routes inward for missing detail or outward for missing source evidence.This makes the Verifier both the stopping mechanism and the trigger for expansion.
4 Experiments
Experiments across six multimodal long-document benchmarks show that D2-ScaleAgent outperforms fixed-workflow baselines in end-to-end QA and retrieval. Ablations, profiling, and a cross-page case study attribute its gains to Verifier-driven coordination of retrieval and reasoning scaling.
- End-to-End QA Performance: D2-ScaleAgent achieves the best average end-to-end QA performance across six benchmarks, outperforming multimodal RAG and fixed-workflow multi-agent baselines.On GPT-4o, it scores 63.7 versus MDocAgent’s 58.3.
- Retrieval Performance: D2-ScaleAgent consistently improves Recall, Precision, nDCG, and MRR over MoLoRAG on MMLongBench-Doc and LongDocURL.The gains are attributed to query-attribute decomposition and iterative evidence expansion beyond static Top-K retrieval.
- Ablation Study: On MMLongBench-Doc, the full model scores 52.0 versus 45.0 for naive GPT-4o and 54.9 with ground-truth evidence.Removing the Verifier loop lowers accuracy to 44.1, while removing retrieval-side scaling or reasoning agents also degrades performance.
- Compute-Cost Analysis: Computation scales with query difficulty: MMLongBench-Doc uses 21.4K tokens and 16.22s, while ViDoSeek uses 15.9K tokens and 11.89s.Costs are averaged per query, with latency measured end-to-end through the GPT-4o API.
- Case Study: In a cross-page population comparison, Verifier-driven routing retrieves missing evidence and applies fine-grained reasoning to produce the correct answer.Multimodal RAG retrieves both pages but fails at cross-page composition, while fixed multi-agent reasoning remains constrained by incomplete evidence.
5 Conclusions
D2-ScaleAgent addresses fixed workflows and evidence insufficiency in long document understanding through dual-dimensional scaling, using Verifier-driven routing and an updated evidence bank for retrieval and reasoning.
- 5 Conclusions: D2-ScaleAgent uses a Verifier agent-driven dynamic routing loop centered on a continuously updated evidence bank.The framework performs outward retrieval scaling and inward reasoning scaling, and achieves logical closure.
Limitations
D2-ScaleAgent mitigates evidence insufficiency through dual-dimensional scaling but inherently trades computational efficiency for reasoning accuracy, increasing inference latency and computational overhead.
- The dual-dimensional scaling paradigm inherently trades computational efficiency for reasoning accuracy.
- Dynamic routing increases inference latency and computational overhead through multi-round attribute-guided retrieval and on-demand invocation of cost-stratified cognitive sub-agents.
Ethical Considerations
The study uses publicly available datasets without personally identifiable or sensitive information and follows applicable copyright and intellectual property guidelines during data preparation.
- Ethical Considerations: The datasets contain no personally identifiable or sensitive information because all materials came from publicly available domains.Data curation, processing, and refinement also adhered strictly to applicable copyright laws and intellectual property guidelines.
The Use of AI assistants · A Detailed Algorithmic Workflow · B Case Study for Execution Traces
The paper uses AI assistants only for manuscript grammar correction, while D2-ScaleAgent follows a Verifier-driven, evidence-bank workflow that adaptively scales retrieval and reasoning until logical closure. A cross-page execution trace illustrates this process by expanding retrieval and invoking specialized reasoning to recover missing evidence.
- The Use of AI assistants: ChatGPT is used to correct potential grammatical inaccuracies in the manuscript, not for research ideation.
- A Detailed Algorithmic Workflow: D2-ScaleAgent initializes a query, document space, and Evidence Bank, then retrieves and stores initial evidence before iterative processing.
- A Detailed Algorithmic Workflow: A Verifier-driven closed loop evaluates evidence completeness and identifies gaps rather than relying on fixed retrieval budgets such as rigid Top-K selection.
- A Detailed Algorithmic Workflow: The system routes outward to expand search for breadth deficiencies and inward to extract fine-grained visual details when depth is insufficient.
- A Detailed Algorithmic Workflow: Execution terminates only when logical chain closure is achieved, producing an answer grounded in a rigorously verified and sufficient set of multi-granularity evidence.
- B Case Study for Execution Traces: In a cross-page compositional QA trace, initial retrieval identifies Page 3, while the Surveyor finds partial party-distribution evidence and the Verifier detects missing “Hispanic” and “Male” attributes.
- B Case Study for Execution Traces: The gap triggers retrieval-side scaling to Page 22, followed by reasoning-side scaling in which the Locator finds the demographic table and the Extractor obtains missing atomic evidence.
- B Case Study for Execution Traces: Evidence is continuously written into the Evidence Bank, and the Verifier re-evaluates accumulated support until logical closure is achieved.
C Extended Ablation Study on Diverse Benchmarks · D Details
The extended ablation study evaluates component effectiveness across PaperTab and UniDoc-Bench, covering table-centric and comprehensive multi-modal documents. PaperTab lacks oracle-evidence results because it provides no ground-truth evidence-page annotations.
- C Extended Ablation Study on Diverse Benchmarks: The study extends component ablations to two additional document benchmarks.The benchmarks are PaperTab and UniDoc-Bench.
- C Extended Ablation Study on Diverse Benchmarks: PaperTab is a table-centric benchmark included in the extended evaluation.
- C Extended Ablation Study on Diverse Benchmarks: UniDoc-Bench is a comprehensive multi-modal benchmark included in the extended evaluation.
- C Extended Ablation Study on Diverse Benchmarks: The extension tests component effectiveness across different document types and evidence distributions.
- D Details: Results from the extended ablation study are summarized in Table 4.
- C Extended Ablation Study on Diverse Benchmarks: PaperTab has no oracle-evidence result because it lacks ground-truth evidence-page annotations.
D.1 Implementation Details
D2-ScaleAgent uses ColQwen2-v1.0 for visual embedding and a plug-and-play agent architecture implemented with smolagents, defaulting to GPT-4o as the reasoning engine across its functional modules.
- ColQwen2-v1.0 provides the visual embeddings for D2-ScaleAgent.
- GPT-4o is the primary reasoning engine for the Surveyor, Locator, Extractor, and Verifier modules.Gemini-3-flash-preview, Qwen2.5-VL-7B-Instruct, and Qwen3-VL-8B-Instruct are also tested as foundational models.
- The entire agentic routing is implemented using the smolagents framework.
D.2 Metrics Details · D.3 Hyperparameters · E Prompt Templates for D2-ScaleAgent
The paper specifies dataset-specific generation and retrieval metrics, default hyperparameters, and prompt templates that operationalize D2-ScaleAgent’s retrieval, image analysis, extraction, and evidence verification stages.
- D.2 Metrics Details: MMLongBench-Doc and LongDocURL use GPT-4o to extract answers scored by Exact Match (EM) and Accuracy.These metrics evaluate generation performance.
- D.2 Metrics Details: PaperTab, FetaTab, and UniDoc-Bench use GPT-4o for Binary Correctness (0/1), while ViDoSeek uses GPT-4o scoring from 1 to 5, with scores of 4 or higher considered correct.Retrieval quality is assessed with Recall, Precision, and Normalized Discounted Cum...
- D.3 Hyperparameters: D2-ScaleAgent uses default hyperparameter values throughout experiments, summarized in Table 5.The supplied passage identifies Table 5 as the summary of these defaults, but does not include their individual values.
- E Prompt Templates for D2-ScaleAgent: Query Attribute Decomposition generates multi-perspective retrieval queries by defining dynamic angles, using concrete keyword anchors, and assigning confidence scores summing exactly to 1.00.The template gives examples of visual, entity, comparative, and numerical angles and requires confidence values with two decimal places.
- E Prompt Templates for D2-ScaleAgent: The Global Surveyor analyzes all initialized candidate images to produce a coarse-grained answer, whereas the Region Locator selects relevant images and summarizes their query-relevant content.Both prompts accept the question and images; the Locator returns selected image indices and summaries.
- E Prompt Templates for D2-ScaleAgent: The Fine-grained Extractor reads Locator-selected images to obtain requested details such as numbers, text, names, locations, dates, times, and prices.Its JSON response uses image keys as dictionary keys for detailed extracted information.