Source-linked AI summary
Visual Document Understanding and Reasoning: A Multi-Agent Collaboration Framework with Agent-Wise Adaptive Test-Time Scaling
Xinlei Yu, Chengming Xu, Zhangquan Chen, Yudong Zhang, Shilin Lu, Cheng Yang, Jiangning Zhang, Shuicheng Yan, Xiaobin Hu
TL;DR
Monolithic scaling for document understanding and reasoning yields diminishing returns, while document analysis requires procedural reasoning and reliable factual grounding. MACT assigns the workflow to four specialized agents and adaptively allocates test-time computation; across 15 benchmarks, its variants rank among the top three on average and improve on their base models.
Problem
Monolithic scaling yields diminishing returns for document-based VQA, whose procedural reasoning, specialized skills, and sensitivity to small errors challenge a single-model approach.
Source: §1 · pp. 1–2Method
MACT decomposes document analysis into planning, execution, judgment, and answer agents, with agent-wise adaptive test-time scaling and judgment-based self-correction.
Source: §1 · p. 2Results
Across 15 benchmarks, MACT variants rank in the top three and achieve the best performance on 13; they improve over their base models by 10.3%, 9.9%, and 11.5% on average.
Source: §1 · p. 2, §4.1 · p. 7Takeaways & Limitations
The results present adaptive test-time scaling in a multi-agent framework as an alternative to monolithic scaling for document tasks without compromising general capabilities.
Source: §1 · p. 2Excessive corrections may confuse the agent and obscure correct answers.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
The dominant paradigm of monolithic scaling in Vision-Language Models (VLMs) is failing for understanding and reasoning in documents, yielding diminishing returns as it struggles with the inherent need of this domain for document-based procedural reasoning, cognitive complexity, and factual accuracy. To this end, we introduce MACT, a Multi-Agent Collaboration framework with agent-wise adaptive Test-time scaling that pioneers a paradigm shift to procedural scaling, adapting dynamically to the functional entities of visual documents understanding and reasoning. MACT decomposes the visual document processing flow into four specialized agents, i.e., planning, execution, judgment, and answer, to resolve cognitive overload and introduce a critical self-correction loop for factual grounding. This collaborative architecture is amplified by an agent-wise adaptive test-time scaling strategy that intelligently allocates computational resources based on the complexity and redundancy of each functionality. Evaluated on multiple visual document understanding benchmarks, MACT achieves superior performance with a smaller parameter scale, adapting effectively to various document scenarios without compromising its general or mathematical reasoning capabilities. The three variants of MACT consistently attain top-three average performance rankings, with average performance enhancements of 9.9-11.5% over the base models. The source code will be released publicly.
1. Introduction
Document questions have advanced from simple lookups to semantic and multi-hop reasoning, while monolithic scaling yields diminishing returns for document-based VQA. MACT decomposes document reasoning across specialized agents with adaptive test-time scaling, and its three variants achieve top-three average performance rankings.
1. Introduction
- Document benchmarks and questions have evolved from simple documents and lookups toward multi-page reports, complex charts, webpages, and semantic or multi-hop reasoning.Visual question answering serves as a quantifiable benchmark of comprehension of complex layouts, dense text, and structured data.Source: §1 · p. 1
- Document analysis requires multistep reasoning and distinct skills, while monolithic models lack internal verification, allowing small extraction mistakes to cascade.Errors such as truncating a paragraph or misaligning a table row can invalidate an answer.Source: §1 · pp. 1–2
- MACT shifts from monolithic to procedural scaling through planning, execution, judgment, and answer agents, with adaptive resource allocation and judgment-based self-correction.The framework allocates scrutiny according to each function and uses the judgment agent as an internal verification loop.Source: §1 · p. 2
- MACT variants rank in the top three and achieve the best performance on 13 of 15 benchmarks, with average increases of 3.2% to 5.6% over comparable-scale and larger models.The comparison includes larger-scale models; Figure 1 compares MACT variants with their base models and larger models in the same families.Source: §1 · p. 2, Figure 1 · p. 2
2. Methodology
MACT organizes document analysis as a collaborative, multistage workflow, then adapts inference-time scaling and reward signals to the agents’ different functions.
2.1. Multi-Agent Collaboration Framework
- Four specialized agents plan, execute, judge, and answer; the judgment agent flags mistakes and routes them to earlier agents for correction, with at most three corrections.Correct or corrected plan-process pairs are forwarded to the answer agent.Source: §2.1 · pp. 2, 4
- Planning produces high-level plans focused on step targets and requirements, while execution carries out plan steps sequentially and passes their outputs onward.The planner can generate distinct execution plans for relevant examples; the executor uses selected tools and concatenates step outputs.Source: §2.1 · p. 3
2.2. Agent-Wise Adaptive Test-Time Scaling
- MACT tailors test-time scaling to agent roles: it generates parallel plans, selects top-scoring execution candidates per step, and uses budget forcing for judgment.The answer agent receives only marginal improvement from test-time scaling.Source: §2.2 · pp. 4–5
2.3. Mixed Reward Modeling
- Mixed reward modeling combines agent-specific process and outcome rewards with a global reward for the selected path, reinforcing correct paths and moderating agent self-interest.Process rewards provide step-wise feedback, while outcome rewards provide a single signal for judgment and answer outputs.Source: §2.3 · p. 5
3. Experiments
The experiments train three MACT variants through a two-stage SFT and RL pipeline, then evaluate them on document and non-document benchmarks. Evaluation compares them with other methods using shared and benchmark-specific scoring procedures.
3. Experiments
- The experiments section directs readers to appendices for detailed training pipelines, datasets, evaluations, and agent prompts.Source: §3 · p. 5
3.1. Training Pipeline
- The two-stage pipeline first fine-tunes VLMs for visual understanding and reasoning, then trains judgment and answer LLMs before RL optimization.The VLMs serve as planning and execution agents; the LLMs serve as judgment and answer agents.Source: §3.1 · pp. 5–6
3.2. Benchmarks
- The benchmark suite covers 15 datasets across four document categories and general and mathematical tasks, using original train and test splits.Document categories include text, webpages, charts, and tables; the non-document sets include two general and three mathematical benchmarks.Source: §3.2 · pp. 6–7
3.3. Evaluations
- Most benchmarks are evaluated with GPT-4o as judge through LMMs-Eval, while others use original metrics such as ANLS and F1.Table 1 compares MACT and other methods across 15 benchmarks; its colors indicate the top three values, and its symbols identify MACT base models.Source: Table 1 · p. 6, §3.3 · p. 7
4. Results and Discussions
Across 15 benchmarks, the three MACT variants rank among the top performers, with the 28B MiMo-VL variant leading the average results. Additional analyses report gains from agent collaboration, adaptive test-time scaling, and mixed reward modeling, while noting that more corrections do not continually improve performance.
4.1. Main Results
- 5.6% and 5.9% are the average-score gains of MACT-MiMo-VL-Series-28B over the top-performing open-source and closed-source models, respectively.The three variants collectively place in the top positions on 13 of 15 benchmarks; MiMo-VL-28B leads on seven.Source: §4.1 · p. 7
- 10.3%, 9.9%, and 11.5% are the average gains across 15 benchmarks for the MiMo-VL, InternVL3, and Qwen2.5-VL MACT variants over their respective base models.Qwen2.5-VL-24B and InternVL3-28B also gain at least 6.6% and 3.7%, respectively, over their corresponding larger monolithic models.Source: §4.1 · p. 7
4.2. Additional Quantitative Analysis
- The ablations report an 8.6% advantage for document-centric procedural scaling over the monolithic system, while adaptive test-time scaling and mixed reward modeling each add at least 3% average accuracy.The study uses MACT-Qwen2.5-VL-Series-24B and examines three low-performing reasoning- or long-context benchmarks alongside the overall average.Source: §4.2 · pp. 7–8, Table 2 · p. 7
- With Aplan and Aexe, the system averages a 3.9% increase over the base model; adding Ajudg further boosts performance by 5.5%, and Aans adds 0.9% on average.The proposed judgment strategy outperforms two counterparts by at least 2.6% on average while requiring an average of 0.3 fewer corrections.Source: §4.2 · p. 8
- Performance does not improve continuously as the maximum correction count rises, because excessive corrections may confuse the agent and obscure correct answers.Source: §4.2 · p. 8
- Agent-wise adaptive scaling outperforms four existing test-time strategies by at least 1.8% on average, and global reward adds 1.3% over agent-wise reward alone.Higher numbers of generated plans and candidate executions increase the likelihood of finding correct answers; the figure describes these settings and correction-count analyses.Source: §4.2 · p. 8, Figure 4 · p. 8, Table 4 · p. 8
5. Conclusion
MACT combines four-agent collaboration with adaptive test-time scaling for visual document understanding and reasoning. The authors report stronger judgment-agent performance with fewer corrections and validate the framework through comparative experiments and supplementary analysis.
5. Conclusion
- MACT combines four collaborative agents with adaptive test-time scaling to shift document understanding from monolithic to procedural scaling.The agents divide and collaborate across the document-analysis workflow.Source: §5 · p. 8
- The judgment-agent strategy outperforms existing self-correction mechanisms while requiring fewer corrections.Source: §5 · p. 8
- Agent-wise adaptive test-time scaling and mixed reward modeling further reduce cognitive overload adaptively.Source: §5 · p. 8
- Comparative experiments and supplementary analysis validate MACT's superiority for visual document understanding and reasoning.Source: §5 · p. 8