PaperlayerView exact revision on arXiv

Source-linked AI summary

MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understanding

Siwei Han, Peng Xia, Ruiyi Zhang, Tong Sun, Yun Li, Hongtu Zhu, Huaxiu Yao

arXiv:2503.13964v1cs.LG

TL;DR

DocQA systems struggle to integrate textual and visual evidence in long, complex documents, despite the need to answer questions spanning both modalities. MDocAgent addresses this gap with dual text-image RAG and five collaborating specialized agents. Across five benchmarks, it reports a 12.1% average improvement over the current SOTA method, while its ablations support the value of both modality-specific agents.

  • Problem

    Existing DocQA methods often retrieve or process text and images separately, limiting integrated reasoning for questions requiring both modalities.

  • Method

    MDocAgent uses parallel text-based and image-based RAG with general, critical, text, image, and summarizing agents to retrieve, refine, analyze, and synthesize evidence.

  • Results

    12.1% average improvement over the current SOTA method is reported across five DocQA benchmarks.

  • Takeaways & Limitations

    The framework provides a collaborative architecture for combining textual and visual information in long, complex DocQA tasks.

  • Takeaways & Limitations

    Existing RAG techniques remain limited because retrieval often occurs independently rather than integrating and reasoning over text and image information.

Abstract

from arXiv · show

Document Question Answering (DocQA) is a very common task. Existing methods using Large Language Models (LLMs) or Large Vision Language Models (LVLMs) and Retrieval Augmented Generation (RAG) often prioritize information from a single modal, failing to effectively integrate textual and visual cues. These approaches struggle with complex multi-modal reasoning, limiting their performance on real-world documents. We present MDocAgent (A Multi-Modal Multi-Agent Framework for Document Understanding), a novel RAG and multi-agent framework that leverages both text and image. Our system employs five specialized agents: a general agent, a critical agent, a text agent, an image agent and a summarizing agent. These agents engage in multi-modal context retrieval, combining their individual insights to achieve a more comprehensive understanding of the document's content. This collaborative approach enables the system to synthesize information from both textual and visual components, leading to improved accuracy in question answering. Preliminary experiments on five benchmarks like MMLongBench, LongDocURL demonstrate the effectiveness of our MDocAgent, achieve an average improvement of 12.1% compared to current state-of-the-art method. This work contributes to the development of more robust and comprehensive DocQA systems capable of handling the complexities of real-world documents containing rich textual and visual information. Our data and code are available at https://github.com/aiming-lab/MDocAgent.

1. Introduction

DocQA must handle textual, visual, and cross-modal questions in long documents, where existing models and single-modal retrieval struggle. MDocAgent combines dual-modality retrieval with specialized agents and reports improved performance across five benchmarks.

  • DocQA questions may target text, document visuals, or information requiring both modalities.
  • LVLMs can struggle with text-dominant or nuanced cross-modal questions, while processing entire documents burdens computation and relevance selection.
  • Single-modal RAG systems often fail to synthesize textual and visual evidence when questions require both.
  • MDocAgent combines text-based and image-based RAG with five specialized agents for retrieval, critical information extraction, modality-specific analysis, and answer synthesis.
  • 12.1% average improvement over the current SOTA method is reported across MMLongBench, LongDocURL, PaperTab, PaperText, and FetaTab.

2. Related Work

Related work shows that LVLMs and RAG broaden document understanding, but independently retrieved or processed modalities remain difficult to integrate for complex DocQA.

  • LVLMs in DocQA Tasks: LVLMs combine language-model semantics with document-image processing for increasingly long and visually rich DocVQA tasks.
  • Retrieval-Augmented Generation: Text RAG supplies textual context, while image RAG retrieves visual content for LVLMs.
  • Retrieval-Augmented Generation: Existing RAG methods often retrieve text and images independently, limiting integrated reasoning over both modalities in complex DocQA.
  • Multi-Agent Systems: Multi-agent systems assign different task aspects to specialized agents, but DocQA requires integration across diverse modalities.

3. Multi-Modal Multi-Agent Framework for Document Understanding

MDocAgent processes documents through dual text-image representations, multimodal retrieval, critical-information extraction, specialized modality analysis, and final answer synthesis. Its agents progressively refine evidence before producing the answer.

  • Document Pre-processing: OCR and PDF parsing extract text while original pages remain as images, creating parallel textual and visual document representations.
  • Multi-modal Context Retrieval: ColBERT retrieves top-k text segments and ColPali retrieves relevant page images as parallel multimodal context.
  • Initial Analysis and Key Extraction: The general agent combines retrieved text and visual context to generate a preliminary answer.
  • Initial Analysis and Key Extraction: The critical agent analyzes the question, retrieved contexts, and preliminary answer to identify crucial information guiding specialized agents.
  • Initial Analysis and Key Extraction: Critical outputs include selected textual information and a textual description of important visual content.
  • Specialized Agent Processing: Text and image agents separately analyze their modality-specific contexts, producing text-based and visually grounded answers.
  • Answer Synthesis: The multi-agent pipeline synthesizes the general, text-agent, and image-agent answers into a final response.

4. Experiments

Experiments evaluate MDocAgent across five benchmarks, agent ablations, evidence modalities, retrieval backbones, and a cross-modal case study. Results consistently favor the full multi-agent framework over the compared baselines and variants.

  • Main Results: MDocAgent outperforms all comparison methods across five document-understanding benchmarks.The evaluation includes MMLongBench, LongDocURL, PaperTab, PaperText, and FetaTab.
  • Main Results: 51.9% average improvement over the best LVLM and 23.7% over the text-RAG baseline occur with top-1 retrieval.On PaperText, MDocAgent scores 0.399, 16.7% above M3DocRAG; on FetaTab, it scores 0.600, 21.0% above the second-best method.
  • Main Results: With top-4 retrieval, MDocAgent exceeds M3DocRAG by 10.9% on average and ColBERTv2+Llama-3.1-8B by 6.9%.The reported comparison suggests the framework benefits from additional retrieved context.
  • Ablation Studies: Removing either the text agent or image agent causes a noticeable performance drop, while the full method remains highest across benchmarks.The ablation supports using specialized agents for both modalities.
  • Ablation Studies: Removing the general and critical agents reduces performance across datasets by weakening multimodal integration and the identification of crucial information.The general agent integrates modalities initially, while the critical agent focuses specialized analysis.
  • Fine-Grained Performance Analysis: MDocAgent outperforms LVLM baselines across evidence modalities and is strongest in Chart, Table, and Text categories under top-1 retrieval.Under top-4 retrieval, performance improves across categories, particularly Figure evidence.
  • Compatibility Analysis: ColPali and ColQwen2-v1.0 achieve the same overall average score of 0.407 within MDocAgent.Their marginal dataset-level differences indicate limited overall sensitivity to the image-based RAG backbone.
  • Case Study: In the case study, MDocAgent correctly compares 795 foreign-born Latinos with 1,051 cellphone-interviewed Latinos after both baselines fail.The text and image agents extract the two values, and the summarizing agent integrates them.

5. Conclusion

The conclusion presents MDocAgent as a dual-RAG, specialized multi-agent framework for integrating textual and visual information in DocQA. Experiments report improvements over LVLMs and multimodal RAG methods, while future work targets inter-agent communication and external knowledge.

  • 5. Conclusion: MDocAgent integrates text and visual information through specialized agents and a dual RAG approach.Its agents handle text processing, image analysis, critical information extraction, and answer synthesis.
  • 5. Conclusion: Experimental results demonstrate improvements over LVLMs and multi-modal RAG methods.The conclusion associates these results with the collaborative multi-agent architecture.
  • 5. Conclusion: Future work will explore more advanced inter-agent communication and integration of external knowledge sources.

A. Experimental Setup

The experimental setup lists several vision-language model backbones used in document understanding comparisons. These include Qwen2-VL, Qwen2.5-VL, LLaVA-NeXT, and Phi-3.5-vision-instruct.

  • Model Backbones: Qwen2-VL-7B-Instruct is a vision-language model designed to handle multiple images as input.
  • Model Backbones: Qwen2.5-VL-7B-Instruct is described as an enhanced version of Qwen2-VL-7B-Instruct for processing multiple images.
  • Model Backbones: LLaVA-NeXT and Phi-3.5-vision-instruct are additional vision-language model backbones listed for comparison.

A.2. Evaluation Benchmarks

The evaluation uses multiple document-understanding benchmarks and evaluates multimodal reasoning through text and image evidence. The setup also specifies retrieval, generation, and image-processing prompts for the agents.

  • Evaluation Benchmarks: The benchmarks include MMLongBench, LongDocURL, PaperTab, PaperText, and FetaTab, covering diverse document-understanding scenarios.MMLongBench contains 1,091 questions across 135 documents averaging 47.5 pages, while LongDocURL covers over 33,000 pages and 2,325 question-answer pairs.
  • Agent Evaluation Setup: The multimodal prompts instruct agents to extract image text, analyze visual content, and combine textual and visual information for answering questions.
  • Agent Evaluation Setup: The critical-information prompt requires separate text and image outputs in a Python-dictionary format.

A.5. Evaluation Metrics

All benchmarks use average binary correctness evaluated by GPT-4o.

  • Evaluation Metrics: Average binary correctness evaluated by GPT-4o is the metric used for all benchmarks.A Python script extracts the result provided by GPT-4o.

B. Additional Results

On LongDocURL, MDocAgent achieves the highest average accuracy across categories with top-4 retrieval and performs strongly across layout, text, and table categories.

  • LongDocURL Results: With top-4 retrieval, MDocAgent reaches the highest score in all LongDocURL categories.
  • LongDocURL Results: With top-1 retrieval, MDocAgent achieves the highest average accuracy and performs strongly on Layout, Text, and Table.M3DocRAG performs slightly better on Figure, while ColBERTv2+Llama3.1-8B performs slightly better on Others.

B.2. Experiments on different model backbones in MDocAgent

Backbone choice materially affects MDocAgent performance: Qwen2.5-VL favors MMLongBench, while GPT-4o performs best across benchmarks. The framework benefits from stronger LVLM backbones.

  • Backbone Comparison: Qwen2.5-VL performs worse than Qwen2-VL on PaperTab, PaperText, and FetaTab under both retrieval settings.
  • Backbone Comparison: Qwen2.5-VL improves markedly over Qwen2-VL on MMLongBench, possibly reflecting stronger visual-question-answering performance.The passage also suggests weaker performance on textual tasks.
  • Backbone Comparison: GPT-4o significantly outperforms Qwen2-VL and Qwen2.5-VL across all benchmarks.GPT-4o’s top-1 performance surpasses both Qwen models’ top-4 results in almost all cases.

B.3. Additional case studies

Two case studies show MDocAgent answering questions that challenge retrieval and multimodal reasoning, while baseline systems fail despite partial retrieval success.

  • Figure 4: In Figure 4, MDocAgent identifies “Most Beautiful Campus” as the only reason without a corresponding image containing people.ColBERT fails to retrieve the evidence page, while ColPali retrieves it but its downstream system answers incorrectly.
  • Figure 5: In Figure 5, MDocAgent correctly identifies Professor Lebour’s “M.A. degree” after ColPali fails retrieval and ColBERT + Llama adds an incorrect designation.The critical agent identifies “M.A.” in both text and image, guiding the text and image agents before answer synthesis.
  • Overall observation: These cases demonstrate resilience to imperfect retrieval through collaborative multimodal processing and general-critical agent guidance.The examples include both insufficient evidence-page retrieval and ambiguous answer interpretation.
Loading 2503.13964v1…