Personalized Graph-Based Retrieval for Large Language Models

1 Introduction

• Over 99.99% of Amazon Reviews users have fewer than three interactions, while LaMP and LongLaMP impose minimum profile sizes that exclude most such users.Existing benchmarks rely exclusively on user history to model personalization.Source: §1 · pp. 1–2, Figure 2 · p. 2
• PGraphRAG uses user-centric knowledge graphs to retrieve semantically relevant context from users' own and neighboring profiles and augment generation prompts.The graphs represent information such as interests, preferences, and prior interactions.Source: §1 · p. 2
• The Personalized Graph-based Benchmark for Text Generation supports evaluation of personalized generation in sparse-history settings across twelve tasks.The tasks span long-form generation, summarization, and classification.Source: §1 · p. 2
• PGraphRAG achieves state-of-the-art performance across all tasks in its benchmark and significantly outperforms LaMP in low-profile scenarios.The framework retrieves relevant context from neighboring profiles as well as the target user's own profile, including when the user has only one input.Source: §1 · p. 2

2 Personalized Graph-based Benchmark for LLMs

• The benchmark covers twelve real-world tasks across multiple domains, spanning long-form generation, short-form generation, and ordinal classification.Source: §2 · p. 3

2.1 Personalized Text Generation: Problem Definition

• Generation seeks an output aligned with the target’s semantics and style, conditioned on the input and a graph-derived user profile.The profile includes the user’s interactions and related signals from similar items or neighboring users; retrieval selects a subset for context.Source: §2.1 · p. 3

2.2 Task Definitions

• Task definitions cover product and hotel reviews, review titles and summaries, stylized feedback, and ratings, including multilingual product tasks.Source: §2.2 · pp. 3–5

• PGraphRAG uses graph-based retrieval to extend personalization beyond direct user history, including when histories are sparse or unavailable.Source: §3 · p. 5
• A user profile combines the target user’s interactions with reviews by other users for the same items.Source: §3 · p. 5
• The workflow transforms the input into a retrieval query, selects relevant profile entries, then combines them with the input to form the LLM prompt.Source: §3 · pp. 5–6
• Query formulation, retrieval logic, prompt construction, profile contents, and retriever choice are modular and can be adapted independently.The framework supports retrievers such as BM25 and Contriever.Source: §3 · p. 6

4 Experiments

• The experiments use LLaMA 3.1 8B Instruct and GPT-4o-mini; retrieval configurations compare user-only context with context that also includes neighboring users.Source: §4 · p. 6
• Baselines include no retrieval, random retrieval, and LaMP’s use of the target user’s history; generation is scored with ROUGE-1, ROUGE-L, and METEOR, and ratings with MAE and RMSE.Source: §4 · pp. 6–7

4.1 Baseline Comparison

• +32.1% ROUGE-1, +21.7% ROUGE-L, and +25.7% METEOR over LaMP on Hotel Experience Generation with LLaMA-3.1-8B-Instruct; PGraphRAG beats all baselines across long-text metrics.Source: §4.1 · p. 7
• PGraphRAG outperforms baselines in most short-text cases; on User Product Review Title Generation with LLaMA-3.1-8B-Instruct, gains over LaMP are +5.6% ROUGE-1, +5.9% ROUGE-L, and +6.8% METEOR.Source: §4.1 · p. 7
• Rating gains are modest: PGraphRAG beats LaMP in 1 of 4 LLaMA tasks and 2 of 4 GPT tasks; on Multilingual Product Ratings, LLaMA gains are +1.75% MAE and +1.12% RMSE.For GPT on that task, gains are +2.16% MAE and +3.17% RMSE; the passage notes limited additional signal in more homogeneous domains.Source: §4.1 · p. 7

4.2 Ablation Studies

• Ablations vary retrieval depth, retrieval scope (user-only versus user+neighbors), and retriever model (BM25 versus Contriever).Source: §4.2 · p. 7

5 Conclusion

• PGraphRAG integrates user-centric knowledge graphs into retrieval-augmented generation, enriching prompts with structured user profiles rather than relying solely on user history.The framework is intended to support adaptive personalization in sparse-data settings.Source: §5 · p. 7
• The experiments show that graph-based retrieval improves performance across diverse tasks and outperforms state-of-the-art baselines.The conclusion highlights robustness to data sparsity, cold starts, and context adaptation as relevant application settings.Source: §5 · p. 7
• The paper presents PGraphRAG as a scalable design that generalizes user preferences and adapts to new users through structural retrieval.Source: §5 · p. 7

Scalability considerations

• PGraphRAG constructs a unified, sparse user-item bipartite graph offline and limits inference retrieval to a localized subgraph centered on the input user.The authors identify compression and real-time profile updates as future work for dynamic environments.Source: §6 · pp. 7, 9

Graph completeness and data sparsity

• Retrieval quality can be influenced by graph completeness, although PGraphRAG is designed for sparse and noisy conditions and uses neighboring nodes to provide contextual signals.The authors identify implicit signals and more resilient retrieval for incomplete graphs as future directions.Source: §6 · p. 9

Generalization vs. user adaptation

• Balancing individual personalization with generalization across user populations remains a challenge; personalized fine-tuning or adapter layers may further address this tradeoff.Source: §6 · p. 9

Static user profiles

• Profiles are treated as static during evaluation, while preferences can evolve over time; modeling temporal dynamics and supporting updates are future directions.Source: §6 · p. 9

A.1 Data Construction and Splitting

• The user–item graph represents users and domain entities as nodes, with user-generated content linking them.For gold-label selection, each chosen interaction must involve an entity associated with another user; splits preserve global and local neighbors while stratifying profile sizes.Source: §A.1 · p. 9

A.2 Performance Gains

• PGraphRAG reports gains in long- and short-text generation, with reduced gains on Task 8.For long-text generation, GPT-4o-mini gains approximately 15% in ROUGE-1, 13% in ROUGE-L, and 15% in METEOR; LLaMA-3.1-8B gains 15%, 11%, and 13%, respectively. In short-text generation, GPT gains 5% across all metrics, while LLaMA gains range from 2–6%.Source: Table 6 · p. 10, §A.2 · p. 10

A.3 Prompt and Output Examples

• Task prompts use templates whose placeholders are filled with task-specific input and retrieved context at inference.The output example contrasts graph-derived hotel cues with LaMP’s generic user-history context, which can produce an irrelevant hotel location.Source: §A.3 · p. 10

Gold Review

• The gold review describes a clean, peaceful stay in downtown Seattle, with walkable access to shopping and restaurants.It says the train ride to the hotel was easy and that no rental car was needed.Source: Gold Review · p. 10

LaMP

• LaMP’s example review instead describes a stay at Newark Garden Hotel, including welcoming staff and hot breakfast.The passage identifies LaMP’s output as relying on generic context from the target user’s reviews, which can lead to irrelevant content.Source: LaMP · p. 10, §A.3 · p. 10

PGraphRAG

• PGraphRAG’s example describes a Grand Hyatt Seattle stay, including a soaking tub, city view, and proximity to Pike Place Market.The comparison passage says PGraphRAG captures specific contextual cues such as location and hotel amenities.Source: PGraphRAG · p. 10, §A.3 · p. 10

A.4 PGraphRAG Ablation Details

• The ablation assesses user-specific and neighbor-derived context by comparing full PGraphRAG retrieval from target-user profiles and neighboring users with neighbor-only PGraphRAG-N and user-only PGraphRAG-U variants.Table 9 reports ablation results for long-text generation using LLaMA-3.1-8B-Instruct and GPT-4o-mini.Source: §A.4 · pp. 10, 13, Table 9 · p. 11
• For long-text generation, PGraphRAG and PGraphRAG-N consistently outperform PGraphRAG-U across datasets.PGraphRAG-N performs on par with or slightly below the full method, suggesting neighbor context alone is often sufficient, especially for sparse-profile or cold-start users.Source: Table 9 · p. 11, §A.4 · p. 13
• For short-text generation, PGraphRAG and PGraphRAG-N outperform PGraphRAG-U across most tasks, except Hotel Experience Summary Generation.On that task, PGraphRAG-U slightly outperforms all graph-based variants; the passage suggests limited data variation or task-specific semantic mismatch as possible explanations.Source: Table 10 · p. 12, §A.4 · p. 13

A.5 Impact of the Retrieved Items k

• Increasing k generally improves generation across tasks and models, with larger gains from k = 1 to k = 2 than from k = 2 to k = 4 in some cases.The study varies k across 1, 2, and 4 retrieved entries.Source: §A.5 · p. 13, Table 11 · p. 13
• Many profiles, especially in cold-start settings, contain fewer than four qualifying interactions, so retrieval can return fewer entries than the specified k.Effective context size therefore varies across users, particularly in the low-profile regime.Source: §A.5 · p. 13

A.6 Impact of Retriever Method R

• The comparison tests BM25, a sparse keyword-based retriever, against Contriever, a dense unsupervised retriever based on sentence embeddings.The retrieval analysis varies the retriever model alongside retrieval depth and retrieval scope.Source: §A.6 · p. 13
• Across both evaluated models, differences between BM25 and Contriever are minor, and neither retriever dominates across datasets or metrics.The reported pattern supports robustness to retrieval choice rather than dependence on a fine-tuned or heavily engineered strategy.Source: §A.6 · pp. 13–14

A.7 Impact of Ranked Retrieval

• PGraphRAG** performs best with all available context, but its cost and context-length constraints limit scalability; fixed-k ranked retrieval balances performance and efficiency.PGraphRAG** includes all context within the model’s input limit, whereas PGraphRAG* samples k = 4 entries without ranking.Source: §A.7 · p. 14
• Removing ranking lowers ROUGE-1 by 2.29% for long-text and 3.18% for short-text tasks; user-only retrieval drops by 0.92% and 1.98%, respectively.The passage attributes these consistent declines to the importance of ranking for identifying relevant context.Source: §A.7 · p. 14

A.8 Evaluating Different GPT Variants

• GPT-4o-mini offers the best trade-off between accuracy, cost, and consistency on long-text generation tasks among the evaluated GPT variants.The comparison uses BM25 with k = 4.Source: §A.8 · pp. 14–15

A.9 Impact of Length Constraints in GPT Model

• A 5-word output constraint offers the best trade-off across short-text evaluation metrics, so the study adopts it as the default.The compared fixed constraints are 3, 5, and 10 words.Source: §A.9 · p. 15

A.10 Validation Results

• Best-performing configurations were selected per task using validation performance across language models, retrieval strategies, and top-k settings; validation trends remained consistent in test results.Validation experiments covered representative tasks, with tables reporting long-text, short-text, and ordinal-classification results.Source: §A.10 · p. 15

Personalization in NLP

• Prior personalized NLP includes user attributes, embeddings, and retrieval-augmented methods, yet most work assumes dense histories and few approaches use graphs to generalize beyond individual traces.Benchmarking frameworks such as LaMP and LongLaMP standardize evaluation of personalized tasks.Source: Personalization in NLP · pp. 15, 17

Knowledge Graphs and Retrieval-Augmented Generation (RAG)

• Knowledge graphs support relational reasoning and graph-integrated RAG, but real-world scaling, graph maintenance, and integration of symbolic signals with noisy or misaligned data remain challenges.The cited applications include question answering, entity linking, reasoning, multi-hop reasoning, and rare entity recognition.Source: Knowledge Graphs and… · pp. 17–18

Toward Structured Personalization via Graph-Augmented RAG

• PGraphRAG uses user-centric bipartite graphs to retrieve both user-authored content and interactions from similar users, supporting generalization, diversity, and data efficiency at inference time.The approach is presented as a way to personalize under sparse-history conditions.Source: Toward Structured Personalization via… · p. 18