PaperlayerView exact revision on arXiv

Source-linked AI summary

HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generation

Pei Liu, Xin Liu, Ruoyu Yao, Junming Liu, Siyuan Meng, Ding Wang, Jun Ma

arXiv:2504.12330v1cs.CLcs.AI

TL;DR

HM-RAG addresses the difficulty of answering complex multimodal queries that require coordinated reasoning across heterogeneous data sources. It uses hierarchical agents for query decomposition, parallel multi-source retrieval, and answer voting with refinement. The framework achieves state-of-the-art zero-shot performance on ScienceQA and CrisisMMD, including a 12.95% answer-accuracy improvement over baseline RAG systems.

  • Problem

    Conventional single-agent RAG is limited for complex queries requiring coordinated reasoning across heterogeneous data ecosystems.

  • Method

    HM-RAG coordinates decomposition, vector-, graph-, and web-based retrieval, and decision-agent voting with expert-guided refinement.

  • Results

    HM-RAG achieves state-of-the-art zero-shot performance on ScienceQA and CrisisMMD, with a 12.95% absolute improvement over vector-based baselines on ScienceQA.

  • Takeaways & Limitations

    The modular framework supports dynamic knowledge synthesis across structured, unstructured, and graph-based data for multimodal reasoning and knowledge synthesis.

Abstract

from arXiv · show

While Retrieval-Augmented Generation (RAG) augments Large Language Models (LLMs) with external knowledge, conventional single-agent RAG remains fundamentally limited in resolving complex queries demanding coordinated reasoning across heterogeneous data ecosystems. We present HM-RAG, a novel Hierarchical Multi-agent Multimodal RAG framework that pioneers collaborative intelligence for dynamic knowledge synthesis across structured, unstructured, and graph-based data. The framework is composed of three-tiered architecture with specialized agents: a Decomposition Agent that dissects complex queries into contextually coherent sub-tasks via semantic-aware query rewriting and schema-guided context augmentation; Multi-source Retrieval Agents that carry out parallel, modality-specific retrieval using plug-and-play modules designed for vector, graph, and web-based databases; and a Decision Agent that uses consistency voting to integrate multi-source answers and resolve discrepancies in retrieval results through Expert Model Refinement. This architecture attains comprehensive query understanding by combining textual, graph-relational, and web-derived evidence, resulting in a remarkable 12.95% improvement in answer accuracy and a 3.56% boost in question classification accuracy over baseline RAG systems on the ScienceQA and CrisisMMD benchmarks. Notably, HM-RAG establishes state-of-the-art results in zero-shot settings on both datasets. Its modular architecture ensures seamless integration of new data modalities while maintaining strict data governance, marking a significant advancement in addressing the critical challenges of multimodal reasoning and knowledge synthesis in RAG systems. Code is available at https://github.com/ocean-luna/HMRAG.

1 INTRODUCTION

HM-RAG addresses limitations in multimodal and modality-isolated retrieval by coordinating specialized agents across heterogeneous data sources. Its architecture combines query decomposition, multi-source retrieval, and expert-guided refinement, with state-of-the-art results reported on ScienceQA and CrisisMMD.

  • Multimodal systems often fail to establish coherent correlations between visual elements and textual context in question answering.
  • Modality-specific systems risk information loss because textual precision and visual-spatial understanding remain insufficiently aligned.
  • HM-RAG coordinates query decomposition, parallel retrieval, and expert-guided refinement through specialized agents.The framework retrieves from vector, graph, and web-based databases and synthesizes candidate responses.
  • HM-RAG reports state-of-the-art performance on the ScienceQA and CrisisMMD benchmarks.

2 RELATED WORK

Related RAG research progressed from text and image retrieval toward graph-based, agent-based, active, and dynamic approaches. However, multimodal integration and coherent cross-modal reasoning remain unresolved challenges.

  • Text-based RAG integrates external textual knowledge with language models, while image-based RAG emerged to address visually rich documents.
  • Graph-based RAG models inter-modal and intra-modal relationships using vector embeddings and topological structure.
  • Agent-based RAG improves modularity and operational flexibility over static pipelines that struggle with multimodal query processing.
  • Active and dynamic RAG methods introduce anticipatory retrieval and entity-aware augmentation, but multimodal integration challenges persist.

3 METHODOLOGY

HM-RAG is a multi-agent, multimodal architecture that combines specialized information extraction and multi-source retrieval for grounded question answering. The framework is organized around coordinated processing of a question and reference document.

  • HM-RAG uses specialized agents for information extraction and multi-source retrieval in a multimodal architecture.
  • Given a question and reference document, RAG retrieves semantically relevant content and integrates it with a generative language model to produce grounded answers.
  • The framework is presented as an advance for multimodal question answering and multi-agent RAG capabilities.

3.1 Multimodal Knowledge Pre-Processing

HM-RAG preprocesses multimodal inputs into textual and graph-based knowledge representations. Visual descriptions are refined with textual context, then integrated into multimodal knowledge graphs for retrieval and reasoning.

  • Multimodal representation: VLMs convert visual information into textual representations that are integrated with original text to construct vector and graph databases.
  • Multimodal representation: BLIP-2 provides open-vocabulary visual-to-language conversion, while contextual refinement addresses overly condensed descriptions lacking visual specificity.
  • Multimodal representation: Hierarchical visual encoding, cross-modal interaction, and context-aware generation produce refined textual outputs with reduced descriptive ambiguity and lexical sparsity.
  • Knowledge integration: The multimodal textual knowledge base integrates original textual inputs with generated textualizations.
  • Knowledge graph construction: LightRAG constructs multimodal knowledge graphs from refined visual descriptions and optional external text for multi-hop reasoning and dynamic knowledge integration.
  • Knowledge graph construction: Hybrid extraction represents entities and relations, while global and local retrieval target thematic clusters and entity-specific connections.
  • Knowledge graph construction: The framework embeds visual data locations in the graph to support cross-modal grounding and bidirectional knowledge enhancement.
  • Hierarchical retrieval pipeline: HM-RAG’s three-stage pipeline decomposes questions, retrieves top-k documents from vector, graph, and web sources, and votes and refines the final answer.

3.2 Decomposition Agent for Multi-intent Queries

The Decomposition Agent identifies complex multi-intent queries and breaks them into coherent, executable sub-tasks for targeted retrieval or reasoning.

  • Decomposition Agent: The Decomposition Agent breaks complex, multi-intent queries into coherent and executable sub-tasks.It uses hierarchical parsing to identify query structure and form atomic units targeting specific modalities or retrieval tasks.
  • Decomposition Necessity Judgment: A binary judgment first classifies each question as single-intent or multi-intent before decomposition.Single-intent questions pass through directly, while multi-intent questions proceed to sub-question generation.
  • Intent Decomposition: For multi-intent questions, the agent generates 2 to 3 logically connected sub-questions while retaining keywords from the original query.The structured prompt bases decomposition on the question’s intent and reasoning steps.

3.3 Multi-source Plug-and-Play Retrieval Agents

The framework uses modular vector, graph, and web retrieval agents with standardized interfaces to retrieve heterogeneous evidence. Each modality applies specialized retrieval and answer-generation procedures, including semantic ranking, graph traversal, and API-based web search.

  • Architecture: Three specialized agents provide vector-based, graph-based, and web-based retrieval through unified communication protocols.The modular design supports interoperability across diverse search scenarios.
  • Vector retrieval: The vector agent embeds the query, retrieves relevant textual contexts, and generates answers with deterministic decoding.It uses top-p = 1.0 and temperature 0 to select highest-probability tokens.
  • Graph retrieval: The graph agent constructs context-aware subgraphs for multi-hop queries using LightRAG traversal and cross-modal attention.Its hierarchical search prioritizes local one-hop neighbors before expanding through global cross-modal paths.
  • Graph retrieval: Graph-vector retrieval decomposes queries into local and global keywords, matches them against entities and relations, and expands the retrieved subgraph.One-hop expansion preserves structural integrity while capturing broader contextual relationships.
  • Web retrieval: The web agent queries Google Search through the Serper API and returns ranked results containing titles, snippets, URLs, and positional metadata.The configured search uses k results, English language, and web search type.

3.4 Decision Agent for Multi-answer Refinement

The Decision Agent compares answers from vector, graph, and web retrieval using consistency metrics, then either fuses consistent answers or invokes expert refinement for conflicts.

  • Consistency voting: Consistency voting compares vector, graph, and web answers using ROUGE-L and BLEU over generated summaries.ROUGE-L captures shared factual information, while BLEU detects precise terminology and numerical matches.
  • Consistency voting: A weighted fusion of ROUGE-L and BLEU balances macro-level semantic alignment with micro-level detail consistency.Answers exceeding a predefined similarity threshold are refined with a Lightweight Language Model.
  • Expert Model Refinement: Conflicting answers are synthesized by LLMs, Multimodal LLMs, or chain-of-thought language models using the original query and retrieved evidence.Expert Model Refinement produces the final answer from multi-source evidence.

4 EXPERIMENTS

HM-RAG is evaluated on ScienceQA and CrisisMMD using zero-shot comparisons, qualitative case analysis, and component ablations. Across these evaluations, it achieves strong multimodal performance, with results highlighting multi-source reasoning and decision refinement.

  • Results on ScienceQA: 93.73% average accuracy on ScienceQA establishes HM-RAG as state of the art, exceeding vector-, graph-, and web-based baselines by 12.95%, 12.71%, and 12.13%.It also surpasses LLaMA-SciTune by 4.11%, GPT-4o by 2.82%, and human expert performance by 6.03%.
  • Results on CrisisMMD: 58.55% average accuracy on CrisisMMD establishes state-of-the-art performance, improving over GPT-4o by 2.44% and Qwen2.5-72B by 3.44%.The result is achieved with 7B parameters, while the method exceeds its text-only variant by 2.3% and a comparable VLM by 4.79%.
  • Results on CrisisMMD: 5.7% and 2.01% average-accuracy gains over text-only and graph-only variants show the contribution of multimodal integration on CrisisMMD.Task 1 reaches 72.06% accuracy, outperforming GPT-4o by 3.86%.
  • Ablation Studies: 10.82% is the largest ablation decline when the Decision Agent is removed, with sharper losses on image-based and social reasoning tasks.Removing the web-based agent decreases average performance by 5.63%, while the fully integrated system reaches 93.73%.
  • Ablation Studies: 2.64% higher accuracy on grade 7-12 problems indicates stronger handling of complex queries.The reported outcomes support the architectural effectiveness of coordinating specialized agents for multimodal reasoning.

5 CONCLUSION

HM-RAG addresses complex multimodal query processing and knowledge synthesis through specialized agents coordinating query decomposition, multi-source retrieval, and decision refinement. Experiments on ScienceQA and CrisisMMD report state-of-the-art accuracy for multimodal question answering and classification, with improvements over baseline methods.

  • HM-RAG integrates specialized agents for query decomposition, multi-source retrieval, and decision refinement.The framework supports dynamic knowledge synthesis across structured, unstructured, and graph-based data.
  • HM-RAG performs multimodal knowledge synthesis across structured, unstructured, and graph-based data.
  • HM-RAG achieves state-of-the-art accuracy on multimodal question answering and classification across ScienceQA and CrisisMMD.The reported performance shows significant improvements over all categories of baseline methods.

A PREDICTED EXAMPLES

The paper presents predicted examples for evaluating language reasoning with and without image context, alongside comparisons for single-modal and multimodal question answering. HM-RAG consistently produces accurate answers in the reported examples.

  • Predicted examples assess models’ language reasoning capabilities with and without image context.
  • HM-RAG consistently produces accurate answers in the representative examples.
  • Figure 4 compares systems on single-modal question answering.
  • Figure 5 compares systems on multimodal question answering.
Loading 2504.12330v1…