Source-linked AI summary
Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendation
Shijie Wang, Wenqi Fan, Yue Feng, Shanru Lin, Xinyu Ma, Shuaiqiang Wang, Dawei Yin
TL;DR
LLM-based recommenders face hallucinations and gaps in current, domain-specific knowledge, while vanilla RAG can add noise and overlook structure. K-RagRec retrieves, re-ranks, and graph-encodes KG sub-graphs to augment recommendations; experiments on three real-world datasets demonstrate effectiveness, with a reported average 41.6% improvement over the sub-optimal baseline across datasets for LLama-2-7B.
Problem
LLM-based recommender systems suffer from hallucinations, outdated knowledge, domain-specific knowledge gaps, and vanilla RAG’s noise and structural omissions.
Method
K-RagRec indexes item-KG sub-graphs, selectively retrieves and re-ranks them, then uses a GNN and projector to align their structure with the LLM.
Results
41.6% average improvement over the sub-optimal baseline across all datasets was achieved by K-RagRec with LLama-2-7B among prompt-tuning RAG methods.
Takeaways & Limitations
Experiments on three real-world datasets demonstrate the effectiveness of retrieving and encoding structured KG knowledge for LLM-based recommendation.
Takeaways & Limitations
Evaluation was limited by GPU resources to 7b and 8b models and used Freebase as the external knowledge graph.
Abstract
from arXiv · showhide
Recommender systems have become increasingly vital in our daily lives, helping to alleviate the problem of information overload across various user-oriented online services. The emergence of Large Language Models (LLMs) has yielded remarkable achievements, demonstrating their potential for the development of next-generation recommender systems. Despite these advancements, LLM-based recommender systems face inherent limitations stemming from their LLM backbones, particularly issues of hallucinations and the lack of up-to-date and domain-specific knowledge. Recently, Retrieval-Augmented Generation (RAG) has garnered significant attention for addressing these limitations by leveraging external knowledge sources to enhance the understanding and generation of LLMs. However, vanilla RAG methods often introduce noise and neglect structural relationships in knowledge, limiting their effectiveness in LLM-based recommendations. To address these limitations, we propose to retrieve high-quality and up-to-date structure information from the knowledge graph (KG) to augment recommendations. Specifically, our approach develops a retrieval-augmented framework, termed K-RagRec, that facilitates the recommendation generation process by incorporating structure information from the external KG. Extensive experiments have been conducted to demonstrate the effectiveness of our proposed method.
1 Introduction
LLM-based recommender systems offer broad language understanding but remain vulnerable to hallucinations and stale or domain-specific knowledge gaps. K-RagRec addresses these limitations by retrieving and encoding structured, up-to-date knowledge sub-graphs from item KGs.
- Motivation: LLM-based recommender systems can hallucinate fictional items and lack current or domain-specific recommendation knowledge.Examples include recommending the non-existent film “Godmother” and failing to recommend films released after December 2022.
- Motivation: Vanilla RAG can introduce noise while overlooking entity relationships, limiting recommendation accuracy, reliability, and reasoning.Knowledge graphs provide structured, factual, and editable knowledge for recommendations.
- Challenges: KG retrieval addresses higher-order neighborhood effects that triplet-only or first-order retrieval can neglect.The challenge arises because indiscriminate retrieval also retrieves information regardless of whether an item needs it.
- K-RagRec: K-RagRec indexes item-KG sub-graphs at coarse and fine granularities, retrieves selectively by popularity, and re-ranks relevant sub-graphs.The framework then uses a GNN and projector to align retrieved structures with the LLM’s semantic space.
- K-RagRec: The framework is designed to provide reliable, up-to-date knowledge for recommendation generation with LLMs.Its stated goal is to augment recommendation capability through relevant knowledge retrieved from item KGs.
2 Related Work
Prior RAG systems retrieve corpus fragments to guide language-model generation, while recommendation-specific RAG remains an emerging area. K-RagRec targets the under-explored use of faithful structured KG knowledge for LLM-based recommendations.
- Retrieval-Augmented Generation: Early RAG methods retrieve relevant fragments from large corpora to guide language-model generation.The related work discusses REALM, RETRO, and DPR as representative examples.
- RAG for Recommendations: Initial recommendation studies have used movie or book datasets as external knowledge for improving recommendations.These works represent early explorations of RAG for recommendation tasks.
- Research Gap: Faithful structured knowledge retrieval from knowledge graphs remains under-explored in recommendation research.K-RagRec is presented as a response to this gap by retrieving knowledge sub-graphs from KGs.
3 Methodology
K-RagRec retrieves structured knowledge sub-graphs from item knowledge graphs and integrates them into LLM recommendation generation. Its pipeline indexes multi-hop graph structure, selectively retrieves and re-ranks sub-graphs, then encodes them as soft prompts for a frozen LLM.
- K-RagRec comprises semantic indexing, popularity-selective retrieval, sub-graph retrieval, re-ranking, and knowledge-augmented recommendation.
- Hop-Field Knowledge Sub-graphs for Semantic Indexing: Hop-field indexing captures higher-order neighborhoods by representing each entity’s l-hop neighbors as a knowledge sub-graph stored in a vector database.The method addresses the limitations of retrieving only nodes, triplets, or first-order neighbors.
- Hop-Field Knowledge Sub-graphs for Semantic Indexing: The indexing GNN aggregates neighboring entity and relation information to produce semantic representations for coarse and fine graph structures.Node and relation text attributes provide the semantic inputs used in graph encoding.
- Popularity Selective Retrieval Policy: A popularity-selective policy determines which items require retrieval, reducing retrieval costs for recommendation histories with many items.The policy is motivated by the concentration of user behavior on a small proportion of popular items.
- Knowledge Sub-graphs Retrieval: The same PLM embeds an item query and knowledge sub-graphs into a shared space, enabling top-K similarity retrieval from the knowledge vector database.Retrieved sub-graphs for items are collected with knowledge associated with users’ historical interactions.
- Knowledge Sub-graphs Re-Ranking: Re-ranking uses the recommendation prompt embedding to select the Top-N most relevant retrieved sub-graphs before they are passed to the LLM.This step limits information overload from long user interaction histories and prioritizes relevant sub-graphs in the prompt.
- Knowledge-augmented Recommendation: A second GNN and an MLP projector encode retrieved sub-graphs into the LLM embedding space as soft graph prompts, while the LLM backbone remains frozen.Only the two GNNs and projector are learned using cross-entropy loss against the ground-truth output.
4 Experiment
The experiments evaluate K-RagRec across datasets, baselines, metrics, backbones, efficiency, ablations, and hyperparameters. Results show consistent performance gains, component contributions, and efficient inference.
- Experimental Setup: K-RagRec is evaluated on MovieLens-1M, MovieLens-20M, and Amazon Book using Accuracy and Recall@3/5 under leave-one-out evaluation.Experiments compare multiple KG-RAG-enhanced recommendation baselines and use Llama-2-7b, Llama-3-8b, and QWEN2-7b backbones.
- Overall Performance: With Llama-2-7B, K-RagRec improves by an average of 41.6% over the sub-optimal prompt-tuning baseline across all datasets.With Llama-3-8B and QWEN2-7B, it achieves average improvements ranging from 13% to 32%.
- Overall Performance: K-RagRec consistently outperforms all baseline methods on the MovieLens and Amazon Book datasets across the evaluated backbone models.Against LoRA fine-tuning with naive RAG, prompt-tuned K-RagRec performs close to or better in most settings, while LoRA-fine-tuned K-RagRec achieves the best performance.
- Ablation Study: Removing any framework component decreases recommendation performance, while removing the GNN Encoder reduces accuracy by 37% on MovieLens and 45.9% on Amazon Book.The ablation study uses Llama-2-7B on MovieLens and Amazon Book.
- Inference Efficiency: K-RagRec achieves the best computational efficiency among KG-RAG methods and is only about 0.1s slower than direct inference without retrieval.Inference efficiency is measured on MovieLens-1M with Llama-2-7b using two NVIDIA A6000-48G GPUs.
- Parameter Analysis: Recommendation performance first improves and then decreases as the popularity threshold increases; K=3 is selected, while N between 5 and 7 improves Amazon Book performance.The analyses indicate that retrieval and reranking counts should be chosen according to dataset and KG scale.
5 Conclusion
K-RagRec augments LLM recommendations by retrieving reliable, up-to-date KG knowledge through semantic indexing, selective retrieval, and graph encoding. Experiments on three real-world datasets demonstrate its effectiveness.
- K-RagRec retrieves reliable and up-to-date knowledge from KGs to augment LLM recommendation capabilities.
- The framework uses GNN and PLM semantic indexing for coarse- and fine-grained KG retrieval, a popularity-selective policy, and expressive graph encoding.Graph encoding helps LLMs leverage knowledge structure while avoiding long context inputs.
- Extensive experiments on three real-world datasets demonstrate the effectiveness of K-RagRec.
6 Limitations
The work identifies three limitations concerning model scale, external knowledge sources, and selective retrieval policy. It outlines corresponding directions for future research.
- Evaluation was limited to 7b and 8b models because of GPU resource constraints.The authors plan to extend evaluation to larger models to assess effectiveness and scalability.
- The framework used Freebase as its only external knowledge graph.Future work will examine YAGO, DBpedia, and Wikipedia to study how knowledge sources affect performance.
- Selective retrieval used popularity to determine which items to retrieve for efficiency.The authors identify more flexible policies, such as reinforcement learning, as a future direction.
A.1 Implementation Details
The implementation details describe K-RagRec’s training, graph, acceleration, and retrieval settings. Dataset and knowledge-graph statistics are summarized separately in Table 3.
- K-RagRec’s hyperparameters cover general training, GNN indexing and encoding, LoRA, acceleration, and retrieval.Retrieval settings include candidate item number M, popularity threshold p, retrieved subgraphs K, and reranked subgraphs N.
- The reported experiments average results from three runs with different random seeds unless otherwise specified.
- Table 3 reports basic statistics for the three datasets and the reconstructed knowledge graph.“Items in KG” counts items appearing in both a knowledge graph and its corresponding dataset.
A.2 Graph Neural Networks
Graph neural networks update node representations by aggregating information from neighboring nodes. This iterative operation captures graph topology and relational structure.
- GNNs capture graph topology and relational structure by iteratively updating node representations.
- A typical GNN operation aggregates neighboring nodes’ features before combining them with the node’s own information.
- N(x_j) denotes node j’s neighbors, while AGG aggregates their features and ⊕ combines them with the node itself.
A.3 More Related Work
Prior work applies LLMs to recommendation through unified training and prompting, recommendation-oriented fine-tuning, and broader modeling of collaborative knowledge. These approaches motivate continued use of LLMs for recommendation tasks.
- Recent studies harness LLM reasoning and generalization capabilities to enhance recommender systems.
- P5 unifies pre-training, prompting, and prediction across sequential recommendation and rating-prediction tasks.
- Tallrec fine-tunes LLaMA-7B on recommendation data for sequential recommendation.
A.4 Comparison with Existing Methods
The paper positions its approach as an effort to avoid costly, frequent fine-tuning by augmenting LLM recommendations with structured knowledge from knowledge graphs. It distinguishes K-RagRec through multi-granularity graph retrieval and popularity-aware selection.
- Existing LLM-based recommender systems often require frequent dataset-specific fine-tuning to address knowledge gaps and hallucinations, which is time-consuming and costly.
- K-RagRec augments LLM recommendation performance by retrieving structured data from knowledge graphs.
- Its indexing GNN uses representations from each layer to retrieve coarse- and fine-grained graph structures rather than only the final-layer representation.
- K-RagRec introduces popularity-selective retrieval to determine whether an item needs retrieval and reduce retrieval time in recommendation.
A.5 Comparison with 10 Candidate Items
The paper evaluates K-RagRec with ten candidate items and examines its comparative performance, generalization, hallucination reduction, cold-start behavior, and component choices. Across the reported studies, K-RagRec consistently outperforms baselines, generalizes across datasets, and benefits from its components and graph-encoder design.
- Comparison with 10 Candidate Items: K-RagRec consistently outperforms baseline KG RAG-enhanced recommendation methods on MovieLens and Amazon Book with M = 10.
- Ablation Study: Removing any framework component decreases overall recommendation performance, while removing the GNN encoder leads to a 37% decrease.
- Generalization Study: 21.6% improvement over SOTA baselines is achieved on MovieLens-20M in the zero-shot setting.
- Generalization Study: About 8.7% improvement over prompt-tuned baselines is achieved on Amazon Book when training on MovieLens-1M and evaluating zero-shot.
- Study of Hallucination: 93.1% reduction in hallucinations is reported for K-RagRec compared with direct inference on LLama-2.
- Encoder and Layer Studies: Four GNN encoder variants show close performance, and performance first improves then decreases as the number of GNN layers increases.