Source-linked AI summary
Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Models
Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, Abhinav Sethy
TL;DR
Retrieval-based personalization for language models can lose user information, provide limited high-level understanding, and face cold-start challenges, while incorporating long histories can strain input limits and inference costs. The paper combines retrieval with offline LLM-generated user summaries; on most LaMP tasks, it reports comparable or better performance using 75% less retrieved user data.
Problem
Retrieval-based personalization can lose user information, offer limited deeper understanding of user data, and face cold-start challenges, while including long histories in prompts can exceed input limits and raise inference costs.
Source: §1 · pp. 1–2Method
The method augments retrieved profile items and the task input with an LLM-generated user summary that can be generated and stored offline.
Source: §1 · p. 2, §2.2 · p. 3Results
Takeaways & Limitations
Offline summaries combined with runtime retrieval are presented as suitable for personalization in runtime-constrained applications such as voice assistants.
Source: §1 · p. 2, §5 · p. 4The LaMP benchmark’s data and tasks are simplistic and narrow in scope, so the method’s potential on more realistic user data remains to be assessed.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Personalization, the ability to tailor a system to individual users, is an essential factor in user experience with natural language processing (NLP) systems. With the emergence of Large Language Models (LLMs), a key question is how to leverage these models to better personalize user experiences. To personalize a language model's output, a straightforward approach is to incorporate past user data into the language model prompt, but this approach can result in lengthy inputs exceeding limitations on input length and incurring latency and cost issues. Existing approaches tackle such challenges by selectively extracting relevant user data (i.e. selective retrieval) to construct a prompt for downstream tasks. However, retrieval-based methods are limited by potential information loss, lack of more profound user understanding, and cold-start challenges. To overcome these limitations, we propose a novel summary-augmented approach by extending retrieval-augmented personalization with task-aware user summaries generated by LLMs. The summaries can be generated and stored offline, enabling real-world systems with runtime constraints like voice assistants to leverage the power of LLMs. Experiments show our method with 75% less of retrieved user data is on-par or outperforms retrieval augmentation on most tasks in the LaMP personalization benchmark. We demonstrate that offline summarization via LLMs and runtime retrieval enables better performance for personalization on a range of tasks under practical constraints.
1 INTRODUCTION
The paper proposes combining retrieved user data with LLM-generated summaries to personalize language-model outputs. It presents this approach as a way to retain higher-level user context while supporting runtime-constrained applications.
1 INTRODUCTION
- Retrieval augmentation addresses long prompts by selecting relevant user data, but may lose information, miss deeper user understanding, and face cold-start challenges.Including comprehensive user histories directly in prompts can exceed input limits, increase inference cost, and degrade performance with lengthy contexts.Source: §1 · p. 2
- The approach combines retrieved profile items with task-aware LLM summaries that capture higher-level user context for downstream generation.For personalized paraphrasing, the summary prompt directs attention to writing style as well as semantic content.Source: §1 · p. 2, Figure 1 · p. 2
- Summaries can be generated and stored offline, with the authors proposing that available data from other applications or self-description could support personalization for new users.The paper identifies voice assistants as one example of a runtime-constrained application.Source: §1 · p. 2
- With 75% less retrieved user data, the method is comparable to or better than retrieval augmentation on most LaMP tasks and outperforms it on two.The reported reduction does not sacrifice performance on five of six tasks.Source: §1 · p. 2
2 METHODOLOGY
The methodology formulates personalization as generating an output from a task input and user profile, then augments retrieved profile items with an LLM-generated user summary. The combined context is used to train a downstream language model.
2.1 Problem Formulation
- Personalization is formulated as learning the output distribution given a natural-language task input and a user profile.The paper writes this distribution as p(y|x,u).Source: §2.1 · p. 2
2.1.1 Baseline.
- The retrieval-only baseline selects the top-k relevant profile items using a query derived from the task input, builds a prompt, and fine-tunes the downstream model.The downstream model is fine-tuned on input-output pairs.Source: §2.1.1 · p. 2
2.2 Integrating Summarization and Retrieval
- The proposed method adds an LLM-generated abstractive summary of salient user information to the retrieved items and task input in the downstream model’s context.Summaries can be generated and stored offline; retrieval and prompt construction occur at runtime.Source: §2.2 · pp. 2–3
3 EXPERIMENTS
The experiments use LaMP’s user-based split to study personalization for new users, with FlanT5-base and BM25 as the downstream model and retrieval method. The setup compares summary-augmented retrieval against retrieval-only baselines under a 512-token input limit.
3.1 Datasets and Evaluation Metrics
- LaMP includes seven personalization tasks—three classification and four generation tasks—but this study excludes LaMP-6 because its data are private.The experiments use the user-based split to address cold-start issues for new users.Source: §3.1 · p. 3
3.2 Experimental Setup
- The experiments use FlanT5-base and BM25; neural retrieval such as Contriever was considered too slow for voice-assistant scenarios.FlanT5-base ran at approximately 125 milliseconds per sample when user data filled the 512-token input limit; Contriever induced approximately 10–30 seconds of latency per sample.Source: §3.2 · p. 3
- Vicuna and ChatGPT generate summaries, with ChatGPT summaries omitted for tasks 2 and 3 because those tasks use a strict output template.Vicuna has a 2,048-token context length, while ChatGPT’s limit is 16,384 tokens.Source: §3.2 · p. 3
- The comparison uses retrieval-only baselines with k=0, 1, and 4, and reports means from three repeated runs.Because k=4 reached the input-length limit for some tasks, summary-augmented methods use k=1 for direct comparison and k=0 to test summaries alone.Source: §3.2 · p. 3
4 RESULTS AND DISCUSSION
On LaMP, summary-augmented retrieval with k=1 matches or outperforms retrieval-only with k=4 on most tasks, while GPT-3.5 summaries also outperform the k=4 baseline on LaMP-1 without retrieval. The results vary by summary model, and the authors note that the benchmark tasks are simplistic and narrow in scope.
4 RESULTS AND DISCUSSION
- With k=1, summary-augmented methods match or outperform the retrieval-only k=4 baseline on most tasks, using 75% less retrieved user data.They outperform baselines on tasks 1 and 2 at p-value < 0.05 and are comparable on tasks 3, 5, and 7, with no statistically significant difference.Source: §4 · p. 3, Table 3 · p. 4
- On LaMP-1, GPT-3.5 Summ. outperforms the retrieval-only k=4 baseline using offline summaries alone, and beats the k=1 baseline on all tasks.The authors report that ChatGPT summaries mostly outperform Vicuna summaries, likely due to the disparity in model size.Source: §4 · p. 3
- The authors attribute the gap between Vicuna Summ. and GPT-3.5 Summ. to differences in offline summary quality.They report that Vicuna did not perform as well as GPT-3.5 on LaMP-1 and LaMP-4, despite studies reporting Vicuna at up to 90% of ChatGPT performance.Source: §4 · pp. 3–4
- Table 2 uses the same prompt content for Vicuna and ChatGPT, with additional tokens for Vicuna to match its expected prompt format.Source: Table 2 · p. 4
- The authors identify the LaMP benchmark's data and tasks as simplistic and narrow in scope, and call for evaluation on more realistic user data.They also note that benefits may improve with fine-tuning a larger language model and end-to-end task training.Source: §4 · p. 4
5 CONCLUSION
The paper combines offline LLM-generated user summaries with retrieval to personalize NLP tasks. On most LaMP tasks, the approach achieves comparable or better performance while reducing retrieved user data by 75%, and the authors describe it as suitable for runtime-constrained applications such as voice assistants.
5 CONCLUSION
- The method augments retrieval-based personalization with offline-generated summaries of user data.Source: §5 · p. 4
- On most LaMP tasks, the method achieves comparable or better performance while reducing retrieved user data by 75%.In some cases, it achieves superior performance after retrieval is removed.Source: §5 · p. 4
- Offline summary generation and retrieval are described as suitable for runtime-constrained applications such as voice assistants.Source: §5 · p. 4
A SUMMARY EXAMPLES
The examples contrast errors by the LaMP baseline with stronger outputs from the paper's summary-based methods. In the second example, GPT-3.5 Summ. is the method reported to demonstrate superior performance.
A SUMMARY EXAMPLES
- In Example 1, the LaMP baseline model exhibits errors while the paper's methods demonstrate superior performance.Source: A SUMMARY EXAMPLES · p. 5
- In Example 2, both the LaMP baseline model and Vicuna Summ. exhibit errors, while GPT-3.5 Summ. demonstrates superior performance.Source: A SUMMARY EXAMPLES · p. 5
- The examples compare the LaMP baseline with the paper's summarization methods across two cases.Source: A SUMMARY EXAMPLES · p. 5
A.1 Example 1
In this example, the baseline predicts reference, while GPT-3.5 Summ. predicts [1]. The displayed Vicuna summary is the brief phrase “Network Traffic Analysis,” unlike the longer GPT-3.5 summary of the author's research interests.
A.1 Example 1
- For the reference-selection question, the baseline prediction is and GPT-3.5 Summ. predicts [1].Source: §A.1 · pp. 5–6
- The Vicuna user summary is shown as “Network Traffic Analysis.”Source: §A.1 · p. 5
- The GPT-3.5 user summary describes research interests across computer networking and wireless communications, including network security and traffic analysis.Source: §A.1 · pp. 5–6
A.2 Example 2
Example 2 asks which of two references is related to an author of a paper on visual-audio integration for partner-robot authentication. The supplied user-profile evidence spans machine-learning and data-analysis research, while the model predictions differ.
A.2 Example 2
- The profile summary describes work on entropy estimation, compressed counting, matrix completion, hashing, sketch-based sampling, optimization, classification, and dimensionality reduction.Listed publications include work on matrix completion, maximum inner product search, entropy estimation, and random projections.Source: §A.2 · p. 6
- GPT-3.5 predicts [1], while Vicuna and the baseline predict.The passages do not establish which prediction is correct.Source: §A.2 · p. 6