PaperlayerView exact revision on arXiv

Source-linked AI summary

GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation

Linhao Luo, Zicheng Zhao, Gholamreza Haffari, Dinh Phung, Chen Gong, Shirui Pan

arXiv:2502.01113v3cs.IRcs.AIcs.CL

TL;DR

Conventional RAG struggles to represent cross-document relationships needed for multi-hop reasoning, and graph-based retrieval remains limited by graph quality and transfer. GFM-RAG uses a query-dependent GNN over a knowledge-graph index for single-step retrieval. It reports state-of-the-art performance across multi-hop QA and domain-specific RAG datasets, including generalization without additional training.

  • Problem

    Conventional RAG struggles to model cross-document relationships needed for multi-hop reasoning, while graph retrievers face noisy or incomplete graphs and limited transfer to unseen datasets.

    Source: §1 · p. 1, §3.2 · p. 5

  • Method

    GFM-RAG uses a query-dependent GNN over a knowledge-graph index to retrieve multi-hop evidence in one step.

    Source: §1 · p. 2

  • Results

    GFM-RAG achieves state-of-the-art performance across three multi-hop QA and seven domain-specific RAG datasets, generalizing without additional training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Its reported alignment with neural scaling laws suggests potential for further improvement through scaling.

    Source: §1 · p. 2

    The authors evaluate GFM-RAG only on multi-hop QA and KG-completion tasks.

    Source: G Limitations · p. 27

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) has proven effective in integrating knowledge into large language models (LLMs). However, conventional RAGs struggle to capture complex relationships between pieces of knowledge, limiting their performance in intricate reasoning that requires integrating knowledge from multiple sources. Recently, graph-enhanced retrieval augmented generation (GraphRAG) builds graph structure to explicitly model these relationships, enabling more effective and efficient retrievers. Nevertheless, its performance is still hindered by the noise and incompleteness within the graph structure. To address this, we introduce GFM-RAG, a novel graph foundation model (GFM) for retrieval augmented generation. GFM-RAG is powered by an innovative graph neural network that reasons over graph structure to capture complex query-knowledge relationships. The GFM with 8M parameters undergoes a two-stage training process on large-scale datasets, comprising 60 knowledge graphs with over 14M triples and 700k documents. This results in impressive performance and generalizability for GFM-RAG, making it the first graph foundation model applicable to unseen datasets for retrieval without any fine-tuning required. Extensive experiments on three multi-hop QA datasets and seven domain-specific RAG datasets demonstrate that GFM-RAG achieves state-of-the-art performance while maintaining efficiency and alignment with neural scaling laws, highlighting its potential for further improvement.

1 Introduction

Conventional RAG struggles with cross-document relationships, while graph-based retrieval faces noisy or incomplete structures and limited transfer. GFM-RAG combines a knowledge-graph index with a query-dependent GNN for single-step multi-hop retrieval, and reports strong results across QA and domain-specific RAG datasets.

1 Introduction

  • Conventional RAG methods retrieve documents independently, making cross-document integration difficult for multi-hop reasoning and applications such as legal judgment and medical diagnosis.Iterative retrieval and LLM reasoning can address multi-source reasoning but incur high computational costs.Source: §1 · pp. 1–2
  • GraphRAG models knowledge relationships explicitly, but graph noise or incompleteness limits performance, while GNN retrievers trained from scratch constrain generalization to new datasets.Existing graph foundation models largely focus on graph tasks such as node classification and link prediction, leaving GFM design for enhancing LLM reasoning unexplored.Source: §1 · p. 2
  • GFM-RAG uses a knowledge-graph index and query-dependent GNN to capture query–knowledge relationships and retrieve across multiple hops in a single step.The index connects factual triples to their source documents.Source: §1 · p. 2
  • Its 8M-parameter retriever is trained in two stages on 60 knowledge graphs, over 14M triples, and 700k documents, for retrieval on unseen datasets without further training.Training comprises self-supervised KG-completion pre-training and supervised document-retrieval fine-tuning.Source: §1 · p. 2
  • GFM-RAG achieves state-of-the-art performance across three multi-hop QA and seven domain-specific RAG datasets, with reported effectiveness, efficiency, and generalizability.The authors also report alignment with neural scaling laws and potential for further improvement.Source: §1 · p. 2

2 Related Work

Related work moves from document retrieval toward graph-based retrieval and graph foundation models. The paper positions GFM-RAG against limitations in modeling relationships, graph quality, transfer, and the task focus of existing GFMs.

2 Related Work

  • Dense-embedding methods retrieve documents by query similarity but struggle to capture complex document relationships, while iterative multi-step retrieval is computationally expensive.The passage cites dense retrieval works [30, 5, 34, 43] and multi-step retrieval works [64] [24].Source: §2 · p. 3
  • GraphRAG explicitly models knowledge relationships for retrieval and reasoning, but noisy or incomplete graphs can yield suboptimal performance, and GNN-based methods lack generalizability.HippoRAG uses personalized PageRank [16], while GNN-based approaches are discussed in [41] [18].Source: §2 · p. 3
  • Existing GFMs target tasks such as link prediction and node classification, leaving GNN-based foundation models for enhancing LLM reasoning an open question.Examples include ULTRA [11], OpenGraph [68], and GFT [66].Source: §2 · p. 3

3 Approach

GFM-RAG builds a knowledge-graph index from documents, then uses a pretrained graph foundation model retriever to select documents for an LLM. Its approach combines query-dependent graph reasoning and two-stage training with entity-based document ranking.

3 Approach

  • GFM-RAG constructs a KG-index from documents, retrieves relevant documents with a pretrained GFM retriever, then ranks them and generates an answer with an LLM.The index, retrieval, and answer-generation components form the overall pipeline.Source: §3 · p. 3, Figure 2 · p. 4

3.1 KG-index Construction

  • The KG-index combines extracted entity-relation triples with entity-resolution edges and an entity-to-document index.The index connects knowledge across documents and records which documents mention each entity.Source: §3.1 · p. 4

3.2 Graph Foundation Model (GFM) Retriever

  • The GFM retriever addresses KG-index noise and incompleteness by using a query-dependent GNN that transfers to new corpora and KGs without further training.It identifies relevant graph entities to help locate documents and represents query, document, and graph relationships in a unified transferable space.Source: §3.2 · pp. 4–5

3.2.1 Query-dependent GNN

  • Query-dependent message passing propagates information from query entities through the KG to estimate entity relevance, supporting multi-hop reasoning.Query-mentioned entities are initialized with the query embedding, while other entities start as zero vectors.Source: §3.2.1 · pp. 5–6

3.2.2 Training Process

  • Training combines binary cross-entropy and ranking losses, with KG-completion pre-training followed by supervised document-retrieval fine-tuning.Pre-training masks a triple's head or tail as a synthetic query target; fine-tuning uses entities from labeled supporting documents as targets.Source: §3.2.2 · p. 6

3.3 Documents Ranking and Answer Generation

  • The system selects top-T entities by relevance, weights them by inverse document frequency, and uses their scores to retrieve top-K documents for the LLM.The document ranker sums the weights of entities mentioned in each document.Source: §3.3 · p. 6

4 Experiment

The experiments assess multi-hop retrieval and QA, efficiency, generalization to unseen RAG datasets, scaling, and path interpretations. GFM-RAG is reported to outperform baselines, generalize without domain-specific tuning, and benefit from increased training data and model size.

4.1 Experimental Setup

  • Evaluation spans three multi-hop QA and seven domain-specific RAG datasets, comparing single-step, graph-enhanced, and multi-step retrieval baselines on retrieval and QA metrics.The GFM retriever has 8M parameters and uses six query-dependent message-passing layers.Source: §4.1 · p. 7

4.2 Retrieval Performance

  • 16.8%, 8.3%, and 19.8% higher R@2 than SOTA IRCoT + HippoRAG on HotpotQA, MuSiQue, and 2Wiki, respectively; GFM-RAG leads all three datasets.The authors report that this single-step retriever outperforms multi-step methods as well.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Question Answering Performance

  • 8.5%, 21.2%, and 3.9% EM gains on three datasets, respectively, when GFM-RAG is integrated with IRCoT; single-step GFM-RAG also achieves state-of-the-art QA performance.The evaluation uses GPT-4o-mini with the top five retrieved documents for answer generation.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Efficiency Analysis

  • GFM-RAG performs multi-hop reasoning with single-step GNN reasoning, more effectively than naive single-step methods and more efficiently than iterative multi-step methods.The authors attribute the costs of multi-step methods to iterative retrieval and LLM reasoning.Source: §4.4 · p. 8, Table 3 · p. 9

4.6 Model Generalizability

  • 18.9% higher average R@5 than HippoRAG across seven RAG datasets, with GFM-RAG requiring no domain-specific fine-tuning.The datasets are tested using a KG-index built from each dataset’s documents and the pretrained retriever.Source: Figure 3 · p. 9, §4.6 · p. 9

4.7 Model Neural Scaling Law

  • GFM-RAG’s MRR scales with training-data scale and model size, following a fitted power-law relationship.The authors interpret this pattern as evidence of scalability and potential for further improvement.Source: Figure 4 · p. 9, §4.7 · p. 10

4.5 Ablation Study

  • The model is not sensitive to sentence embedding choice, while its pre-training and loss-weighting strategies are crucial for performance.The ablations also examine ranking methods, training datasets, and KG-index construction.Source: §4.5 · p. 9

4.8 Path Interpretations

  • Path interpretations illustrate GFM-RAG’s multi-hop reasoning, including links from a singer to a football club and from a murder case to its presiding judge.The model ranks path importance by prediction-score derivatives and extracts top-k longest paths with beam search.Source: §4.8 · p. 10, Table 4 · p. 10

5 Conclusion

The paper presents GFM-RAG as a graph foundation model for retrieval, using a knowledge graph index and a query-dependent GNN for multi-hop reasoning. Its appendix explains how natural-language questions can act as soft queries over graph relations.

5 Conclusion

  • GFM-RAG combines a knowledge graph index and a pretrained query-dependent GNN to retrieve relevant knowledge through single-step multi-hop reasoning.The conclusion reports improved effectiveness, efficiency, and generalizability across three benchmark and seven domain-specific datasets.Source: §5 · p. 10

A Query-dependent GNNs for Multi-hop Reasoning and Retrieval

  • Prior studies [21] [52] theoretically support query-dependent GNNs for capturing multi-hop logical associations; the example question asks which country Barack Obama is a politician of.The appendix presents those graph associations as a way to answer the natural-language question.Source: A Query-dependent GNNs for Multi-hop… · p. 17
  • A natural-language question acts as a soft query that the query-dependent GNN uses to learn logical reasoning over knowledge graphs for retrieval.The conclusion describes the knowledge graph index as explicitly modeling relationships between knowledge and documents.Source: §5 · p. 10, A Query-dependent GNNs for Multi-hop… · p. 17

B.1 Multi-hop QA Datasets

The paper evaluates on multi-hop QA and domain-specific RAG datasets, compares retrieval baselines, and describes construction of its training graphs and model configuration.

B.1 Multi-hop QA Datasets

  • HotpotQA, MuSiQue, and 2WikiMultiHopQA are the three multi-hop QA datasets; the experiments use official splits and 1,000 validation samples per dataset to avoid data leakage.The datasets require reasoning across multiple documents or articles.Source: §B.1 · pp. 17–18

B.2 Domain-specific RAG Datasets

  • Seven test collections cover biomedical PubMedQA, customer support DelucionQA, TechQA, ExpertQA, and EManual, and general knowledge MS Marco and HAGRID.Test corpora merge candidate passages for KG-index construction.Source: §B.2 · pp. 18–19

C Baselines

  • Baselines comprise single-step naive, graph-enhanced, and multi-step methods, including BM25, HippoRAG, and IRCoT, respectively.IRCoT integrates retrieval with an LLM’s chain-of-thought reasoning and can work with arbitrary retrievers.Source: C Baselines · p. 19

D.1 Training Data Construction

  • The training pipeline extracts triples from merged candidate passages with GPT-4o-mini and OpenIE prompts, then constructs 60 KG-indexes from 60,000 training samples.Entity resolution uses ColBERTv2 similarity with a threshold of 0.8.Source: §D.1 · p. 20

D.2 Model Settings

  • The GFM is an 8M-parameter, six-layer query-dependent GNN with hidden dimension 512; retrieval selects the top 20 entities for document ranking.Its message function is DistMult with sum aggregation.Source: §D.2 · p. 20

D.3 Training Settings

Training pairs self-supervised knowledge-graph completion with supervised document-retrieval fine-tuning. Embedding comparisons find relatively small performance differences, while the GNN improves retrieval over sentence-embedding baselines.

D.3 Training Settings

  • KG completion pre-training masks a triple entity to create a query and target; supervised retrieval fine-tuning trains on question-supporting-document entities.For example, the query “Where was Barack Obama born in?” can target Honolulu and USA.Source: §D.3 · pp. 20–21
  • Training uses 60 KG-indexes for 30,000 pre-training steps, followed by five epochs of supervised document-retrieval fine-tuning.The reported training used AdamW and eight NVIDIA A100 GPUs.Source: §D.3 · p. 21

E.1 Effectiveness of Different Sentence Embeddings

  • Sentence-embedding performance varies relatively little; all-mpnet-v2 leads on three metrics and is the default for efficiency, despite its 512-token context limit.The authors leave exploration of larger-context embedding models for future work.Source: §E.1 · p. 21
  • GFM-RAG outperforms both pre-trained and QA-fine-tuned all-mpnet-v2, indicating that its GNN captures question-to-knowledge relationships for multi-hop reasoning.The comparison covers variants without the GNN and embedding-only systems.Source: §E.1 · pp. 21–22, Table 9 · p. 21

E.2 Effectiveness of Different Training Strategies

The ablations examine how supervised retrieval fine-tuning and KG completion pre-training contribute to retrieval and graph completion, then test the balance between BCE and ranking losses.

E.2 Effectiveness of Different Training Strategies

  • Removing supervised document-retrieval fine-tuning significantly decreases retrieval performance, underscoring its role in relating user queries to relevant knowledge.The table evaluates KG completion pre-training and supervised retrieval fine-tuning.Source: §E.2 · p. 22
  • Pre-training alone performs best on knowledge-graph completion, while the model trained with both stages performs best on multi-hop QA and second-best on graph completion.The authors describe pre-training as supporting general graph reasoning and supervised fine-tuning as learning the downstream task.Source: §E.2 · p. 22

E.3 Effectiveness of Loss Weights

  • Using only BCE or ranking loss is suboptimal; the best result uses α = 0.3, a smaller BCE weight consistent with the cited work’s setting of rare positive samples.The comparison reports MRR across loss weights.Source: §E.3 · pp. 22–23, Table 12 · p. 23

E.4 Effectiveness of Ranking Methods

The ranking ablation favors IDF-weighted scores from the top predicted entities, while dataset and domain experiments examine cross-dataset generalization and transfer after fine-tuning.

E.4 Effectiveness of Ranking Methods

  • IDF + Top-T Pred performs best among the four ranking variants, using IDF-weighted scores to map top predicted entities to documents.The inverted index bridges structured KG reasoning and the unstructured documents used by LLMs.Source: §E.4 · p. 23, Table 13 · p. 23
  • The authors identify end-to-end joint reasoning over structured and unstructured knowledge without an explicit inverted index as future work.Source: §E.4 · p. 23

E.5 Ablation Study of Training Datasets

  • A model trained on multi-domain datasets performs competitively across all three benchmarks, supporting generalization across domains from diverse KGs.The ablation also compares models trained separately on each dataset.Source: §E.5 · p. 23, Table 14 · p. 24

E.6 Model Transferability

  • GFM-RAG generalizes in zero-shot evaluation, with further improvements after domain-specific fine-tuning.The evaluation fine-tunes on each domain’s training split.Source: §E.6 · p. 23, Table 15 · p. 24

E.7 Details of Model Neural Scaling

Scaling experiments find higher performance with larger models and more training data, while deeper GNNs generally help but may offer limited benefit beyond the datasets’ reasoning-hop requirements. Predicted hop distributions are compared with ground truth and HippoRAG.

E.7 Details of Model Neural Scaling

  • Performance rises as model size (0.08M–8M parameters) and training data (3k–45k samples) increase; larger models require more data to reach their best performance.The fitted trends are shown for model size and training-data size.Source: §E.7 · p. 24, Figure 5 · p. 25
  • With hidden dimension fixed at 512, performance generally improves across 1–8 GNN layers, but peaks around four layers in some cases.The authors attribute limited benefit beyond four layers to datasets requiring at most four reasoning hops and lacking higher-hop training signals; they also identify other architectures as future work.Source: §E.7 · pp. 24–25, Table 17 · p. 25

E.8 Visualization of the Distribution of Multi-hop Prediction

  • GFM-RAG’s predicted hop distribution closely matches ground truth, whereas HippoRAG’s differs, especially on 2Wiki; the comparison supports single-step multi-hop reasoning.The distributions cover ground-truth paths and predictions from both systems.Source: §E.8 · p. 25, Figure 6 · p. 26

E.9 Cost and Impact of LLMs on KG-index Construction

The section weighs KG-index construction costs against indexing speed and retrieval performance under different graph builders. It also specifies the prompt source used for triple extraction.

E.9 Cost and Impact of LLMs on KG-index Construction

  • 48M tokens per 10k documents costs around $2.6 with GPT-4o-mini; LightRAG and GraphRAG use 57M and 76M tokens, respectively.GFM-RAG avoids community-level summaries, and its index construction time is compared in Table 20.Source: §E.9 · p. 26
  • GFM-RAG’s index process is quick because it avoids building a traditional vector database to store documents and entities.Table 20 compares graph indexing time.Source: §E.9 · p. 26, Table 20 · p. 26
  • The authors note that LLM-based KG construction incurs computational costs, while alternative KG construction methods can be integrated through their interface.They identify more efficient construction methods and process optimization as future work.Source: §E.9 · p. 26
  • GPT-4o-mini-built KG-indexes yield higher performance than GPT-3.5-turbo-built indexes for both GFM-RAG and HippoRAG.GFM-RAG performs significantly better than HippoRAG under both indexes, which the authors interpret as greater robustness to KG-index quality.Source: §E.9 · pp. 26–27, Table 21 · p. 27

F Prompts

  • Triple extraction follows the prompts used in HippoRAG, provided in Table 22.The prompt passage is listed under F Prompts.Source: F Prompts · p. 27

G Limitations

The authors identify construction cost, model size, and evaluation scope as limitations. They also report their claims, theory status, and reproducibility provisions in the accompanying checklist.

G Limitations

  • GFM-RAG’s KG-index construction can be costly and time-consuming, especially when LLMs perform OpenIE extraction.The authors plan to explore more efficient KG construction and optimize the process.Source: G Limitations · p. 27
  • The 8M-parameter GFM-RAG is relatively small compared with foundation models such as billion-parameter LLMs; scaling it remains future work.The authors caution that direct comparison between GNN-based and transformer-based models is not fair.Source: G Limitations · p. 27
  • Evaluation is limited to multi-hop QA and knowledge graph completion; other tasks, including KG question answering and reasoning, remain for future validation.Source: G Limitations · p. 27

1. Claims

  • The authors state that the introduction and abstract accurately reflect the paper’s contributions and scope.Source: §1 · p. 29

3. Theory assumptions and proofs

4. Experimental result reproducibility

  • The authors say Section D details data construction, model settings, and training for reproducibility, and that code is available through an anonymous link.They also direct readers to Section D for experimental settings.Source: §4 · p. 30, §5 · p. 31, §6 · p. 31

7. Experiment statistical significance

The checklist reports fixed-seed experiments without error bars and points to Section D.3 for compute resources. It also records the authors’ statements on ethics, impacts, safeguards, assets, human subjects, and LLM use.

7. Experiment statistical significance

  • Experiments use a fixed random seed, and no error bars are reported.Source: §7 · p. 31

8. Experiments compute resources

  • The authors state that experiment resources are detailed in Section D.3.Source: §8 · p. 32

9. Code of ethics

  • The authors report compliance with the NeurIPS Code of Ethics.Source: §9 · p. 32

10. Broader impacts

  • The authors describe the work as technical and state that it does not include societal impacts.Source: §10 · p. 32

11. Safeguards

  • The authors state that existing datasets and pretrained models have safeguards, and that they cited assets and complied with their licenses and terms.Source: §11 · p. 33, §12 · p. 33

13. New assets

16. Declaration of LLM usage

  • The authors state that LLM usage is described in Sections D and D.1.Source: §16 · p. 35
Loading 2502.01113v3…