Source-linked AI summary
Hypergraph-based Multimodal Retrieval-Augmented Generation with Incremental Refinement
Shenao Chen, Yidan Xu, Xiangmin Han, Rundong Xue, Duanpo Wu, Yuhan Gao, Chenggang Yan, Yue Gao
TL;DR
Multimodal RAG often loses visual, layout, and cross-modal relationships while exhaustive refinement adds redundancy. Hyper-M2RAG uses multimodal hypergraphs and anchor-driven local refinement, with ablations confirming that their combination improves multimodal document understanding.
Problem
Text-centric and pairwise RAG representations can discard visual, layout, and joint cross-modal evidence, while exhaustive multimodal processing introduces computational redundancy.
Method
Hyper-M2RAG represents textual, visual, and tabular entities in a multimodal hypergraph and selectively refines one-hop neighborhoods around boundary-crossing anchors.
Results
Ablation results confirm that multimodal representation and anchor-driven refinement are both essential, with their combination enabling more effective multimodal document understanding.
Takeaways & Limitations
Localized anchor refinement captures long-range dependencies while avoiding unnecessary processing of irrelevant regions.
Abstract
from arXiv · showhide
Modern Multimodal Retrieval-Augmented Generation (M-RAG) systems are fundamentally limited by the binary connectivity paradigm of traditional simple graphs, which fails to capture the intricate, high-order correlations among heterogeneous entities, such as the N-ary relationships between a visual chart, its scattered textual descriptions, and underlying numerical data. Furthermore, existing refinement strategies often rely on exhaustive, full-page reconstruction to align cross-modal information, leading to prohibitive computational redundancy and the introduction of contextual noise in long-form document processing. In this paper, we propose Hyper-M2RAG, a novel framework that redefines multimodal document retrieval through High-order Hypergraph Representation Learning. We first formalize the document structure as a Multimodal Hypergraph, utilizing hyperedges as unified semantic containers to encapsulate multi-way associations across text, images, and tables, thereby transcending point-to-point modeling. To mitigate semantic fragmentation caused by physical pagination, we introduce an Anchor-driven Incremental Refinement mechanism. Rather than performing a global sweep, our approach identifies boundary-crossing anchor nodes and reconstructs their local hyper-topology using one-hop neighborhood contexts. This targeted refinement effectively bridges cross-page knowledge gaps with minimal computational footprints. Extensive evaluations on multimodal benchmarking datasets demonstrate that Hyper-M2RAG significantly outperforms state-of-the-art methods in both retrieval precision and generation coherence. Our code is available at https://github.com/ShenAoChen2001/MMHRAG.
1 Introduction
The introduction identifies that text-centric and pairwise graph-based RAG representations fragment high-order multimodal evidence and overlook visual, layout, and cross-page dependencies. Hyper-M2RAG addresses these limitations with multimodal hypergraphs and anchor-driven local refinement for efficient long-document reasoning.
- Motivation: Multimodal Doc-QA requires reasoning over evidence distributed across text, figures, tables, formulas, and page layouts.Such evidence may span many document pages.
- Limitations of Existing RAG: Linearizing visual content into captions weakens spatial organization and can cause retrieval to overlook visual context, layout proximity, and cross-modal interpretation.This limitation affects evidence embedded in figures, tables, and layouts.
- Core Challenges: Pairwise graph modeling fragments high-order dependencies, losing semantic coherence when claims require joint interpretation of multiple heterogeneous elements.The introduction motivates hypergraphs because multimodal evidence can involve group-wise dependencies among text, figures, tables, and layout context.
- Proposed Framework: Hyper-M2RAG represents textual, visual, and tabular entities as vertices and connects jointly meaningful heterogeneous elements with hyperedges.This shifts RAG’s structural foundation from graphs to hypergraphs to preserve multimodal semantic units.
- Anchor-Driven Refinement: Hyper-M2RAG identifies boundary-crossing anchor entities and selectively reconstructs their local one-hop neighborhoods instead of reprocessing entire pages.The refined hyperedges are merged back into the global structure, reducing redundant long-document refinement.
2 Related Work
Prior RAG work progresses from flattened chunk retrieval to graph- and hypergraph-based modeling, but existing approaches remain limited in representing higher-order multimodal relationships. Recent multimodal systems address cross-modal retrieval and complex document layouts through parsing, alignment, reformulation, aggregation, reasoning, and evidence fusion.
- Graph-enhanced RAG: Traditional RAG uses flattened chunk retrieval, while GraphRAG and LightRAG introduce entity-centric relations for improved multi-hop reasoning but retain pairwise graph connectivity.These binary models struggle to capture long-range dependencies and higher-order associations.
- Hypergraph-based RAG: HyperRAG models higher-order group semantics with hyperedges connecting multiple vertices simultaneously.CogRAG extends this direction with cognitive-inspired hypergraph structures for adaptive reasoning.
- Hypergraph-based RAG: Despite strong text-based high-order modeling, existing hypergraph methods remain largely vision-blind on complex multimodal layouts.This limitation motivates multimodal hypergraph approaches that jointly represent visual and textual structure.
- Multimodal RAG: Recent multimodal RAG studies pursue cross-modal alignment, specialized document parsing, and multi-source retrieval for interleaved visual-textual components in complex layouts.Related work also explores query reformulation, heterogeneous source aggregation, multi-turn reasoning, and evidence fusion.
3 Task Description and Preliminaries
The task is to generate comprehensive answers to queries over heterogeneous-modality documents using a structured evidence package. Hyper-M2RAG represents documents as multimodal hypergraphs and retrieves evidence through dual alignment of symbolic structure and vector representations.
- Multi-modal long-context Doc-QA generates a comprehensive answer Y_ans from a query Q and document D containing heterogeneous modalities.
- The generation process uses S, the Structured Evidence Package retrieved from the document-indexed corpus.
- Hyper-M2RAG organizes each document as a multimodal hypergraph G = (V, E), with normalized semantic entities as vertices and multi-order relationships as hyperedges.
- Its hyperedges separate intra-page pairwise associations, intra-page group associations, and anchor-refined latent cross-page dependencies.E_low captures pairwise associations, E_high captures group associations, and E_refine captures reconstructed cross-page dependencies.
- HyperRetriever performs dual-path alignment across the symbolic hypergraph and vector space, integrating vertex-anchored evidence C_ent with hyperedge-directed context C_rel.This integration supports complex reasoning over the high-order multimodal substrate.
4 Methodology
Hyper-M2RAG’s methodology transforms fragmented multimodal pages into a unified, high-order hypergraph index. This workflow is illustrated in Fig. 1.
- Methodology: The core workflow transitions from fragmented multi-modal pages to a unified, high-order hypergraph index.The workflow is illustrated in Fig. 1.
4.1 Multi-modal Data Ingestion
The ingestion pipeline models documents as page-level multimodal units rather than linear text streams, using MinerU to parse PDFs into structured sequences. Its unified schema preserves page layout and visual evidence while supporting sparse-vision, text-only, and multimodal pages.
- PMU-based ingestion: The Page-level Multi-modal Unit (PMU) replaces the linear text stream as the fundamental atomic unit for structured knowledge extraction.PMUs preserve heterogeneous document structures at page granularity.
- PMU-based ingestion: MinerU parses raw PDFs into a structured sequence D = {p_1, p_2, ..., p_n} of page-level units.Each page unit serves as the basis for subsequent multimodal representation.
- Multimodal representation: Visual inputs are separated into macro-visual page context for layout awareness and fine-grained evidence from detected image patches such as figures and tables.The representation assigns distinct semantic roles to page-level and localized visual information.
- Robust ingestion: Semantic placeholders preserve vision-dominant pages with sparse text, while optional visual fields support both text-only and multimodal data through one interface.An example placeholder is “Please see the Figures,” and visual fields may be empty (∅).
4.2 Multi-modal Hypergraph Index Construction
HyperIndex construction combines page-level multimodal extraction and canonical hypergraph merging with anchor-driven refinement. Refinement reconstructs local anchor neighborhoods to bridge cross-page fragments while avoiding redundant full-sequence processing.
- HyperIndex construction follows hierarchical page-level extraction followed by anchor-driven structural refinement, transforming fragmented units into a cohesive semantic network.The pipeline begins with localized page processing and later reconciles cross-page structure.
- For each page, multimodal extraction captures localized entities and high-order associations from textual content and hierarchical visual evidence.Extracted elements retain their original page provenance through src = {i}.
- Canonical merging normalizes vertices, collapses duplicate hyperedges, and unifies cross-page entities while aggregating their provenance indices.The resulting initial global hypergraph remains limited by page-level receptive fields.
- Cross-page anchors are high-provenance-frequency entities whose topological neighborhoods provide multi-source contexts for targeted refinement.For each anchor, the star-expansion subgraph includes incident pairwise and high-order hyperedges and their contained vertices.
- Anchor-neighborhood refinement synthesizes or consolidates relations within localized sub-hypergraphs before integrating them into the final global multimodal hypergraph.Shifting refinement from raw page sequences to anchor neighborhoods removes redundant processing of stable local information and improves computational efficiency.
4.3 Cognitive-inspired Hybrid Retrieval
The cognitive-inspired hybrid retriever decomposes queries into fine-grained and thematic anchors, then combines vertex-anchored entity retrieval with hyperedge-directed relational retrieval. It fuses both channels into structured, synchronized evidence so generation can reason over local topology and cross-page associations.
- Query Decomposition: An MLLM decomposes each query into fine-grained entity keywords K_low and high-level thematic keywords K_high.K_low targets specific objects or technical terms, while K_high captures abstract concepts or cross-page relations.
- Vertex-anchored Entity Retrieval: Path I retrieves vertices matching K_low and expands their incident hyperedges to preserve entity-centered relational context.Semantic similarity search identifies relevant vertices using an embedding function and similarity threshold γ, after which HypergraphDB supplies topological neighbors.
- Hyperedge-directed Relational Retrieval: Path II retrieves high-order hyperedges matching K_high, then back-traces their constituent vertices and multi-page source chunks.This path targets refined associations that transcend individual page boundaries and reconciles broad conceptual queries with consolidated structural evidence.
- Evidence Fusion: The system fuses entity-centric and relation-centric contexts into a Structured Evidence Package containing normalized entities, high-order hyperedges, and synchronized multi-page source chunks.Generation is prompted to reason over the supplied topological structures and cross-page evidence rather than fragmented raw text.
- Cross-page Evidence: Cross-page refined hyperedges enable integrated retrieval beyond individual pages, while local entity details provide holistic and factually grounded document understanding.The dual-channel design reconciles high-order associations with entity-centric evidence for complex, long-form documents.
5 Experiments … 5.4 Global QA Performance
Hyper-M2RAG is evaluated on global and local multimodal retrieval tasks using standardized comparisons, synthetic global questions, and automated metrics. Across four global-QA domains, it consistently outperforms GraphRAG, HyperRAG, and MegaRAG, with especially strong multimodal gains.
- 5.1 Datasets: The experiments evaluate Hyper-M2RAG at two granularities: Global QA for holistic understanding and Local QA for granular retrieval.These tasks assess multimodal document structures from both whole-document and page- or slide-level perspectives.
- 5.1 Datasets: Global QA uses Mixed-Domain, NeurologyCorp, World History, and Sustainable Report to test long-range textual and multimodal understanding.Mixed-Domain contains approximately 0.62M tokens, while NeurologyCorp contains approximately 1.97M tokens.
- 5.1 Datasets: The automated global-question protocol generates 125 questions per textual dataset by combining five simulated professional users, five strategic tasks, and five complex questions per task.A separate modality-specific pipeline is used for vision-heavy documents.
- 5.1 Datasets: RealMMBench evaluates page- or slide-level retrieval under visual-rich layouts, table-heavy content, query rephrasing, and dense high-order structural dependencies.The benchmark supports stress testing of hypergraph construction and anchor-driven refinement while maintaining computational feasibility.
- 5.2 Baselines and Evaluation Metrics: Hyper-M2RAG is compared with GraphRAG, text-only HyperRAG, and multimodal MegaRAG across textual and multimodal benchmarks.Global QA responses use Comprehensiveness, Diversity, Empowerment, and Overall, while Local QA reports Accuracy against reference answers.
- 5.3 Implementation Details: The study standardizes response generation with Qwen3-VL-8B and uses DeepSeek-v3 for synthetic question generation and evaluation.All methods share gme-Qwen2-VL-2B-Instruct embeddings; textual corpora use 1,200-token chunks with 100-token overlap, and MinerU2.5 extracts multimodal elements.
- 5.4 Global QA Performance: Hyper-M2RAG consistently outperforms all three baselines across qualitative dimensions, with decisive advantages in multimodal scenarios.In Sustainable Report, its Overall win rate is 88.8% against MegaRAG and 77.6% against GraphRAG.
- 5.4 Global QA Performance: Hyper-M2RAG’s Overall win-rate margin over HyperRAG ranges from 22.4% to 44.8%.It also achieves a 75.2% Diversity win rate against MegaRAG in World History and a 90.4% Empowerment score in Sustainable Report.
5.5 Local QA Performance · 5.6 Ablation Study
Hyper-M2RAG achieves strong local QA accuracy on RealMMBench, particularly in visual-heavy technical slides, while ablations show that multimodal extraction and anchor-driven refinement each improve multimodal document understanding.
- 5.5 Local QA Performance: Hyper-M2RAG reaches 70.0% accuracy on TechReport and 78.0% on TechSlides in RealMMBench local QA.The reported results are presented as accuracy percentages.
- 5.5 Local QA Performance: Hyper-M2RAG achieves a 9 percentage point improvement over MegaRAG on the reported local QA comparison.The supplied passage truncates the dataset or metric context for this improvement.
- 5.5 Local QA Performance: The gain on TechSlides highlights the framework’s efficacy in visual-heavy contexts with fragmented information across visual blocks.Technical slides often distribute related content across figures and bullet points.
- 5.5 Local QA Performance: The high-order hypergraph clusters spatially disparate but semantically related elements, enabling more precise local retrieval than dense graph connectivity.The passage gives figures and corresponding bullet points as an example.
- 5.6 Ablation Study: Adding multimodal extraction consistently improves performance over the text-only HyperRAG baseline on both TechReport and TechSlides.The improvement is attributed to incorporating visual elements and layout information.
- 5.6 Ablation Study: Multimodal extraction helps preserve cross-modal semantic dependencies and enables more complete evidence retrieval from complex documents.This is the stated effect of adding visual and layout information.
- 5.6 Ablation Study: Anchor-driven refinement further improves performance over the multimodal hypergraph variant.The ablation isolates refinement after multimodal hypergraph construction.
- 5.6 Ablation Study: Selective reconstruction of local neighborhoods around cross-page anchors captures long-range dependencies while avoiding irrelevant-region processing.The ablation identifies both multimodal representation and anchor-driven refinement as essential components whose combination improves multimodal document understanding.
5.7 Structural Complexity Analysis · 5.8 Cross-Model Judge Audit · 5.9 Reduction in Token Cost and Latency
Hyper-M2RAG builds denser high-order multimodal hypergraph structures, achieves consistent cross-model judging gains, and reduces refinement cost through localized anchor-centered updates. Across the three analyses, its structural richness and efficiency improvements are supported by dataset-level counts, win rates, and token reductions.
- 5.7 Structural Complexity Analysis: Hyper-M2RAG builds 8,257 high-order hyperedges on TechReport versus HyperRAG’s 2,579, a 220% increase, and raises TechSlides counts from 1,716 to 6,571.These high-order hyperedges model dependencies among multiple entities.
- 5.7 Structural Complexity Analysis: HyperRAG+MM produces fewer high-order hyperedges than HyperRAG on TechReport, 2,078 vs. 2,579, whereas Hyper-M2RAG jointly models textual, visual, and layout elements.The comparison indicates that modality augmentation alone is insufficient, while hypergraph construction with anchor-driven refinement yields richer relations.
- 5.7 Structural Complexity Analysis: Hyper-M2RAG achieves the largest total hyperedge counts on both datasets: 22,478 on TechReport and 19,093 on TechSlides.The denser representation supports retrieval of evidence distributed across long, complex documents.
- 5.8 Cross-Model Judge Audit: The cross-model audit evaluates Hyper-M2RAG against HyperRAG on Sustainable Report and World History using Kimi-K2.6, GLM-5.1, and DeepSeek-V4 Pro.HyperRAG is selected because both methods use hypergraphs, while Hyper-M2RAG adds multimodal anchors and star-expansion refinement.
- 5.8 Cross-Model Judge Audit: Hyper-M2RAG consistently outperforms HyperRAG across both domains and all evaluators, with win rates ranging from 36.0% to 55.2%.The same ranking across model families indicates the improvement is not specific to a single judge.
- 5.9 Reduction in Token Cost and Latency: Hyper-M2RAG reduces build and refinement time to 12 min and lowers token consumption by 72% versus MegaRAG and 77% versus HyperRAG+Page.The efficiency comparison is conducted on World History per 100 pages.
- 5.9 Reduction in Token Cost and Latency: Anchor-centered star-neighborhood refinement recomputes only hyperedges incident to updated anchors, replacing global document- or page-level rescans.Because updated anchors grow more slowly than document length, refinement cost scales sub-linearly while preserving cross-modal linkages.
6 Conclusion
Hyper-M2RAG is presented as a multimodal retrieval-augmented generation framework for complex, high-density technical documents. It replaces binary graphs with high-order hypergraphs to capture semantic dependencies fragmented across document sections and modalities.
- 6 Conclusion: Hyper-M2RAG targets retrieval-augmented generation for complex, high-density technical documents.The paper presents it as a novel multi-modal framework.
- 6 Conclusion: Its high-order hypergraph structure replaces traditional binary graphs to capture multi-scale semantic dependencies fragmented across sections and modalities.The conclusion identifies this transition as enabling broader semantic dependency modeling.
- 6 Conclusion: The framework integrates layout-aware hyperedge construction with an anchor-driven mechanism.The supplied passage introduces this integration as the core contribution, but its description is truncated.