Source-linked AI summary
Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphs
Zongyu Wu, Yilong Wang, Xiaochen Wang, Minhua Lin, Zhichao Xu, Fenglong Ma, Xiang Zhang, Suhang Wang
TL;DR
Existing MMKG-based RAG methods underuse textual context because visual and textual modalities are largely processed separately, leaving visual elements insufficiently connected to distributed document semantics. CEMMKG designs multi-granular local and global context for visual elements and uses it across MMKG construction. Experiments report improved MMKG-based RAG performance and applicability across different methods.
Problem
Existing MMKG-based RAG methods use limited textual context during visual processing and modality fusion, although visual elements often depend on distributed prose for interpretation.
Method
CEMMKG constructs multi-granular local and global textual context for each visual element and utilizes it during vision-based graph construction and modality fusion.
Results
CEMMKG improves MMKG-based RAG performance across context configurations and is applicable to different MMKG-based RAG methods.
Takeaways & Limitations
Appropriately designed textual context can improve MMKG-based RAG and support its integration across different methods.
Takeaways & Limitations
The optimal local-context granularity is unresolved because broader context may add support while diluting information most relevant to the visual element.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models (LLMs) and multimodal large language models (MLLMs). In particular, knowledge graph (KG)-based RAG leverages structured knowledge to provide (M)LLMs with high-quality external information. Building on these works, recent studies have explored multimodal knowledge graphs (MMKGs) as knowledge bases for GraphRAG. This enables Graph RAG to integrate knowledge across multiple modalities, thereby further enhancing its performance. However, existing MMKG-based RAG methods generally follow a common pipeline in which different modalities are largely processed independently before being fusion. As a result, textual context is only used to a limited extent during visual information extraction and subsequent multimodal knowledge fusion. This brings a semantic gap between images and text which limits the multimodal GraphRAG performance. To address this issue, we propose a novel framework for constructing a Context-Enhanced MMKG (CEMMKG) to better support multimodal GraphRAG. The proposed CEMMKG enriches each image with complementary textual context at both local and global scopes. Local context goes beyond the surrounding text by incorporating sentences that are semantically related to the image, while global context provides a summary of the entire passage. We further introduce a multi-granularity design for the local context, allowing it to capture semantically relevant information at different levels of detail. Extensive experiments on the selected vision-centric dataset validate that CEMMKG is effective in leveraging contextual information to improve MMKG-based RAG performance. Moreover, its effectiveness across different MMKG-based RAG methods demonstrates its broad applicability.
1 Introduction
RAG and GraphRAG use external structured knowledge to support LLM and MLLM generation, but existing MMKG methods underuse distributed textual context during visual processing and fusion. CEMMKG addresses this with multi-granular local and global context, improving MMKG-based RAG performance and applicability.
- MLLMs remain prone to hallucinations, while RAG retrieves external information to support more reliable response generation.
- GraphRAG leverages entities and relations in knowledge graphs to provide structured external knowledge for retrieval and generation.
- Existing MMKG-based RAG methods largely process modalities separately, limiting textual context during visual information extraction and multimodal fusion.
- CEMMKG systematically designs textual context for visual elements at multiple granularities and uses it across MMKG construction stages.
- Experiments on a vision-centric MMLongBench-Doc subset show improved MMKG-based RAG performance across context configurations and compatibility with different methods.
2 Related Work
Multimodal GraphRAG extends structured retrieval to visual and textual evidence, but existing MMKG pipelines commonly select textual context by proximity and fuse independently constructed modality graphs. This leaves semantically relevant cross-modal information underused.
- MLLMs jointly reason over interleaved images and text, extending language-model capabilities to visual information.
- Multimodal RAG retrieves information such as images and charts to ground generation beyond text-only evidence.
- Multimodal GraphRAG builds MMKGs to support structure-aware retrieval across modalities.
- Existing methods often construct visual and textual graphs independently, then merge them at a later fusion stage.
- Textual context for visual elements is frequently restricted to adjacent chunks, selecting by proximity rather than semantic relevance.
3 Background and Preliminaries
MMKG-based RAG retrieves multimodal knowledge from a graph constructed from document text and visual elements, making graph quality fundamental to answer quality. The paper focuses on enriching visual graph construction and modality fusion with broader textual context.
- An MMKG-based RAG system extracts textual and visual content, constructs an MMKG, retrieves relevant knowledge, and supplies it as evidence to an LLM or MLLM.
- Because retrieval operates on represented graph knowledge, MMKG construction is a fundamental determinant of overall MMKG-based RAG performance.
- Existing pipelines independently build textual and image-based graphs before fusing them into a unified MMKG.
- The fusion module matches visual and textual entities using neighboring text chunks, clustering candidates and using an LLM to identify counterparts.
- Visual elements often require prose for interpretation, while semantically relevant textual context may occur elsewhere beyond nearby image chunks.
- The studied problem is how to establish comprehensive textual context for each visual element and use it to construct a higher-quality MMKG.
4 Method
CEMMKG constructs complementary local and global textual context for each visual element, then injects that context at both image-to-graph construction and modality fusion. Its local context uses multiple reference granularities, while global context represents document-level content.
- Framework overview: CEMMKG constructs textual context for each visual element and reuses it across multiple MMKG construction stages.The framework selects and organizes context, then applies it during image-to-graph construction and modality fusion.
- Local context: Local context combines surrounding text with reference-based information from sentences that mention the visual element.Reference text can occur far from the visual element because it is selected by identifier mentions rather than document position.
- Local context: The reference-based local context offers sentence, paragraph, and summary granularities, while these alternatives are selected rather than jointly used.Sentence context is fine-grained, paragraph context adds supporting information, and summary context provides a broader representation of the references.
- Local context: CEMMKG treats local-context granularity as an empirical design dimension because broader context may add support while diluting image-relevant information.The optimal granularity is evaluated experimentally rather than assumed in advance.
- Global context: Global context derives from the entire document as either its abstract or an LLM-generated document summary, selected according to availability.Global context is shared across visual elements, whereas local context is specific to each element.
- Multi-stage context utilization: Image-to-graph construction receives the full constructed context, whereas modality fusion receives only local context for focused entity alignment.The two stages use different context components according to their objectives: comprehensive visual understanding versus cross-modal correspondence.
5 Experiments
The experiments evaluate how local-context granularity affects MMKG-based RAG and whether CEMMKG transfers across different methods.
- RQ1 examines how different granularities of local textual context affect MMKG-based RAG performance.
- RQ2 examines whether CEMMKG can be applied effectively to different MMKG-based RAG methods.
5.1 Experimental Setup
The evaluation uses a vision-focused subset of MMLongBench-Doc, compares direct inference and two MMKG-based RAG baselines, and reports strict and soft accuracy.
- Dataset: VisionHeavy contains 106 questions, with 80.2% of answerable questions requiring visual information and 43.2% relying on multiple pages.
- Dataset: The subset spans six document types: academic paper, administration&industry files, brochure, guideline, research report, and tutorial/workshop.
- Baselines: Baselines include direct inference with an MLLM and MMKG-based RAG using RAG-Anything or MMGraphRAG.
- Evaluation: Evaluation extracts canonical answers with Llama-3.1-70B-Instruct and applies the released type-aware deterministic scorer.
- Evaluation: Results report strict full-credit accuracy alongside official MMLongBench-Doc soft accuracy, with fractional scores arising from ANLS-eligible answers.
- Implementation: The same retrieval procedure and hyperparameters are used across CEMMKG context configurations for each backbone framework.
5.2 Multimodal GraphRAG Performance
CEMMKG improves multimodal GraphRAG when textual context is appropriately granular, but adding complete paragraphs can introduce unhelpful context. Benefits also vary by visual-content type.
- Overall Performance: 34.91% hard accuracy and 36.84% soft accuracy are achieved by reference sentences plus document-level global context, versus 23.58% and 24.17% for MMGraphRAG.
- Overall Performance: 31.13% hard accuracy and 32.82% soft accuracy are achieved with reference paragraph summaries, also outperforming the MMGraphRAG baseline.
- Overall Performance: 22.64% soft accuracy from full reference paragraphs falls below MMGraphRAG’s 24.17%, indicating that more context is not necessarily better.
- Performance across Different Types: Reference-sentence context raises Figure-question soft accuracy from 6.25% to 24.70% and matches MMGraphRAG at 31.25% for Chart questions.
- Performance across Different Types: Reference paragraph summaries achieve the best Table-question soft accuracy at 26.46%, compared with 22.46% for MMGraphRAG.
- Performance across Different Types: For Misc questions, the proposed context design increases soft accuracy from 34.15% to 54.01% and hard accuracy from 34.15% to 51.22%.Misc questions lack Figure, Chart, or Table labels but may still involve multimodal information.
5.3 Applicability to Different Methods
CEMMKG also improves RAG-Anything on cross-page questions, showing applicability beyond the MMGraphRAG backbone. Reference-sentence context performs best among the compared configurations.
- On cross-page questions, RAG-Anything reaches 28.89% strict and 36.18% soft accuracy without CEMMKG.
- Adding reference sentences increases RAG-Anything to 35.56% strict and 41.12% soft accuracy.
- Performance improves progressively from reference paragraphs to LLM-generated summaries and then reference-sentence context.
6 Conclusion
CEMMKG systematically defines and uses textual context for visual processing and multimodal fusion in GraphRAG-oriented MMKG construction. Experiments show that appropriately designed context improves MMKG-based RAG across methods, while future work may extend the framework to video and audio.
- 6 Conclusion: CEMMKG defines textual context for visual elements using local context at different granularities and global, document-level representations.The framework also explores how this context enters different MMKG construction stages.
- 6 Conclusion: Extensive experiments show that appropriately designed textual context improves the performance of MMKG-based RAG.The experiments compare different context configurations.
- 6 Conclusion: CEMMKG applies effectively across different MMKG-based RAG methods, indicating broader applicability.
- 6 Conclusion: Future work could extend CEMMKG to modalities such as video and audio and investigate how contextual information transfers to them.
A GenAI Usage Disclosure
The paper discloses generative AI use for writing assistance, visualization icons, implementation support, and core methodological tasks, with generated writing reviewed by the authors.
- A GenAI Usage Disclosure: Generative AI assisted with writing, and the authors reviewed all generated content.
- A GenAI Usage Disclosure: GenAI tools generated several icons used for visualization in the figures.
- A GenAI Usage Disclosure: AI is integral to the proposed method, supporting textual graph generation, visual information processing, multimodal information fusion, and related tasks.The paper states that these uses are described in detail in the Method section.
Ethical Considerations
The work studies textual context for multimodal knowledge graph construction using publicly available models and datasets. The authors report no significant ethical concerns or negative societal impacts.
- Ethical Considerations: The study examines how to define and use textual context for multimodal knowledge graph construction.
- Ethical Considerations: All experiments use publicly available models and datasets.
- Ethical Considerations: The authors report no significant ethical concerns or negative societal impacts from the work.