Source-linked AI summary
Don't Forget to Connect! Improving RAG with Graph-based Reranking
Jialin Dong, Bahare Fatemi, Bryan Perozzi, Lin F. Yang, Anton Tsitsulin
TL;DR
RAG can overlook answer-bearing documents with weak links to the question, and existing rerankers may miss connections among documents. G-RAG reranks with document graphs and selected AMR information, and the paper reports that it outperforms state-of-the-art approaches with fewer computational resources.
Problem
RAG may disregard answer-bearing documents with less apparent connections to the question, and existing rerankers often fail to capture connections among retrieved documents.
Source: §1 · pp. 1–2Method
G-RAG builds a document graph whose edges capture shared concepts, incorporates selected AMR information, and trains with pairwise ranking loss.
Source: §1 · p. 2Results
G-RAG outperforms state-of-the-art approaches and requires fewer computational resources.
Source: §1 · p. 2Takeaways & Limitations
The findings emphasize the importance of reranker model design for improving RAG, including when considering large language models.
Source: §1 · p. 2The authors identify more sophisticated AMR processing and integration into node and edge features as directions for further study.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Retrieval Augmented Generation (RAG) has greatly improved the performance of Large Language Model (LLM) responses by grounding generation with context from existing documents. These systems work well when documents are clearly relevant to a question context. But what about when a document has partial information, or less obvious connections to the context? And how should we reason about connections between documents? In this work, we seek to answer these two core questions about RAG generation. We introduce G-RAG, a reranker based on graph neural networks (GNNs) between the retriever and reader in RAG. Our method combines both connections between documents and semantic information (via Abstract Meaning Representation graphs) to provide a context-informed ranker for RAG. G-RAG outperforms state-of-the-art approaches while having smaller computational footprint. Additionally, we assess the performance of PaLM 2 as a reranker and find it to significantly underperform G-RAG. This result emphasizes the importance of reranking for RAG even when using Large Language Models.
1 Introduction
RAG can overlook answer-bearing documents with weak links to the question, while existing rerankers often miss cross-document connections. G-RAG uses document graphs and selected AMR information to address these gaps and reports stronger performance with fewer computational resources.
1 Introduction
- RAG may disregard answer-bearing documents whose connections to the question context are less apparent, even when they connect to strongly relevant documents.The paper frames reranking as the bridge between retrieval and reading; identifying positive documents can improve exact-match performance when the reader returns a gold answer.Source: §1 · p. 1
- Existing approaches leave cross-document links out of reranking; external knowledge graphs can require substantial storage, and redundant AMR inputs can increase computation and GPU cost.The paper also notes interest in assessing untuned LLMs as rerankers.Source: §1 · p. 2
- G-RAG represents documents as nodes, shared concepts as edges, updates edge features through message passing, and uses selected AMR factors as node features.Figure 1 depicts AMR information feeding the document-level graph, which is then used for reranking.Source: §1 · p. 2, Figure 1 · p. 2
- G-RAG outperforms state-of-the-art approaches while requiring fewer computational resources.The contribution summary also introduces metrics intended to assess ranking scenarios with tied scores.Source: §1 · p. 2
- The paper introduces variants of MRR and MHits@10 for ranking scenarios with tied scores, which are common in LLM-generated relevance scores.It also evaluates PaLM 2 as a reranker and reports that excessive ties hinder pretrained LLM reranking effectiveness.Source: §1 · p. 2
2 Related Work
Related work spans pretrained-model rerankers, knowledge-graph and AMR approaches, and emerging LLM reranking. The paper distinguishes G-RAG through its use of cross-document connections in reranking.
RAG in ODQA
Graphs in ODQA
- Knowledge-graph methods use external knowledge bases and entity mapping [15] [1], while Wang et al. use AMR for text-level relations within individual documents.G-RAG instead uses document graphs to characterize cross-document connections and does not depend on external knowledge graphs.Source: §2 · p. 3
Abstract Meaning Representation (AMR)
- AMR represents textual semantics as a rooted, directed graph; Wang et al. improve reranking and reading but incur additional computation and GPU memory cost.The paper attributes the cost issue potentially to integrating all AMR node and edge tokens without selecting key factors.Source: §2 · p. 3
3 Proposed Method: G-RAG
G-RAG reranks retrieved documents with a graph neural network that combines connections among documents with AMR-derived semantic information. It builds document and feature representations from AMRs, updates them with a GNN, and uses the resulting node representations to score documents.
3.1 Establishing Document Graphs via AMR
- For each question, G-RAG connects the top 100 DPR-retrieved documents when their AMR graphs share nodes, then removes isolated document nodes.Each document is a graph node, and shared AMR nodes determine undirected edges.Source: §3.1 · p. 4
3.2 Graph Neural Networks for Reranking
- G-RAG uses document-graph structure and AMR semantics in a GNN reranker while avoiding redundant AMR information that can increase computation and GPU memory use.The authors contrast this design with integrating AMR information into language models.Source: §3.2 · p. 4
3.2.1 Generating Node Features
- Node inputs combine document text with AMR concepts extracted from nonredundant shortest paths starting at the AMR node labeled “question.”The path analysis notes that negative documents may lack adequate question-context connections or contain question-related paths without gold-answer information.Source: §3.2.1 · pp. 4–5
3.2.2 Edge Features
- Each document-graph edge records the number of common AMR nodes and edges between its two documents, and these features are normalized before GNN use.The edge-feature vector has two dimensions, one for common nodes and one for common edges.Source: §3.2.2 · p. 5
3.2.3 Representation Update
- At each GNN layer, edge features weight node features for message computation, a mean aggregator combines neighbor information, and a learnable nonlinear function updates node representations.The framework updates edge representations alongside node representations.Source: §3.2.3 · p. 5
3.2.4 Reranking Score and Training Loss
- Final GNN node representations are used to score documents, and training considers cross-entropy and pairwise ranking losses, with the authors emphasizing the pairwise loss.The paper notes that cross-entropy may struggle with ODQA’s imbalance between negative and positive documents.Source: §3.2.4 · p. 6
4 Experiments
The experiments evaluate G-RAG on ODQA reranking, including tied-score metrics, embedding-model variants, and zero-shot PaLM 2. Cross-document connections improve results, while zero-shot LLM rerankers perform poorly in this setting.
4.1 Setting
- Experiments use NQ and TQA, retrieve 100 DPR documents per question, and generate question-document AMR graphs with AMRBART.Source: §4.1 · p. 6
4.1.1 Metrics
- To handle common LLM score ties, the authors propose MTRR and TMHit@10; TMHit@10 reduces optimistic effects by dividing hit counts by the number of ties.Source: §4.1.1 · p. 7
4.2 Comparing Reranker Systems
- Graph-based GCN and G-RAG are competitive across MRR and MHits@10 and outperform no-reranker and MLP baselines; G-RAG-RL is a strong model.Source: §4.2 · p. 8, Table 1 · p. 8
4.3 Using different LLMs as Embedding Models
- Ember and Ember (HPs-T) are more likely to avoid overfitting and achieve the highest scores across all test sets.Source: §4.3 · pp. 8–9, Table 3 · p. 9
4.4 Investigating PaLM 2 Scores
- Zero-shot LLM rerankers perform poorly; this may reflect frequent tied scores, especially for smaller LLMs.Source: §4.4 · p. 9
- Cross-document connections bring improvements of up to 7 percentage points, while Table 2 reports G-RAG-RL considerably improving over both language-model types.Source: Table 2 · p. 8, §4.4 · p. 9
5 Conclusions
G-RAG combines implicit document connections with selectively integrated AMR information to improve ODQA reranking, including for answer context weakly connected to a question. The paper also identifies tied-score evaluation, AMR processing, and LLM fine-tuning as directions for further work.
- G-RAG leverages implicit document connections and strategically integrates AMR information to identify valuable documents even when their answer context is weakly connected to the question.Source: §5 · p. 9
- The paper proposes two metrics for fairly evaluating ranking scenarios with tied scores and notes that fine-tuning LLMs may improve their usefulness as rerankers.Source: §5 · p. 9
- Future work includes more sophisticated AMR processing and efficient ways to resolve ties in LLM-generated ranking scores.Source: §5 · p. 9
A Dataset Statistics
The AMR graph statistics cover NQ and TQA, with figures describing node and edge counts and question-rooted shortest paths. The analysis reports that some negative documents lack adequate connections to question context.
A Dataset Statistics
- Some negative documents cannot establish adequate connections to the question context within their text, according to the SSSP analysis.The analysis examines shortest single-source paths in document AMR graphs.Source: A Dataset Statistics · p. 14
- Figures 2 and 3 report AMR graph node and edge counts and SSSP counts, respectively, for NQ and TQA.Figure 2 covers train/dev/test sets; Figure 3 covers the training set.Source: Figure 2 · p. 14, Figure 3 · p. 15
- Figure 4 depicts the G-RAG pipeline.Source: Figure 4 · p. 15
B Simulation Results with Different GNN Models.
Across most cases in the reported comparison, GCN outperforms the other tested GNN models. The authors suggest small document graphs and existing edge features may help explain this result.
B Simulation Results with Different GNN Models.
- GCN outperforms the compared GNN models in most cases under the stated setting.The comparison includes GAT, GraphSage, GIN, and BART-GST; the table reports Mean Hits @ 10.Source: B Simulation Results with Different… · p. 15, Table 5 · p. 15
- The authors suggest that document graphs may be too small for advanced GNN models designed for graphs with thousands or millions of nodes.Source: B Simulation Results with Different… · p. 15
- The authors also suggest that adding more weight parameters could lead to overfitting because their model already considers edge features.Source: B Simulation Results with Different… · p. 15
C Qualitative Examples
The qualitative examples examine TQA, where the authors describe documents with weakly connected keywords or gold answers as a challenge for baseline reranking. G-RAG uses connections among documents and message passing to help such documents identify potential keywords and improve their ranking.
C Qualitative Examples
- The example is drawn from TQA, described as more complex than NQ and containing more positive documents.Source: C Qualitative Examples · p. 15
- The top-ten GNN reranking examples list each document’s AMR node, edge, and question-rooted SSSP counts alongside PaLM 2 scores and ranks and DPR ranks.Source: C Qualitative Examples · p. 16
- Message passing over document connections lets the 7th, 8th, and 9th documents use the well-connected first document to discern potential keywords and improve their ranking.The example documents share concepts such as “Sinatra” and “Ol’ Blue Eyes” with the first document.Source: C Qualitative Examples · pp. 18–19
D Examples of LLM-generate Relevant Score
Figure 5 illustrates examples of LLM-generated relevance scores.
D Examples of LLM-generate Relevant Score
- The illustrated examples concern LLM-generated relevance scoring.Source: D Examples of LLM-generate Relevant… · p. 19
- Figure 5 presents examples of relevant scores generated by an LLM.Source: Figure 5 · p. 19, D Examples of LLM-generate Relevant… · p. 19
- Figure 5 is identified as the visual for the score examples.Source: Figure 5 · p. 19