Source-linked AI summary
LaMP: When Large Language Models Meet Personalization
Alireza Salemi, Sheshera Mysore, Michael Bendersky, Hamed Zamani
TL;DR
Personalized LLM outputs are understudied despite the importance of adapting systems to users. The paper introduces LaMP and retrieval augmentation, finding average improvements of 12.2% zero-shot and 23.5% with fine-tuning across datasets.
Problem
Personalized response development and evaluation for LLMs remain relatively understudied despite personalization’s importance in real-world applications.
Method
The paper introduces the seven-dataset LaMP benchmark and two retrieval augmentation approaches, using multiple retrieval models to select profile items for personalized outputs.
Results
12.2% average performance improvements occur across datasets in zero-shot settings and 23.5% with fine-tuning.
Takeaways & Limitations
LaMP provides a diverse framework for studying personalized text classification and generation, while retrieval augmentation improves benchmark performance in both evaluated settings.
Takeaways & Limitations
The benchmark faces limitations from task definitions, possible leakage into LLM pretraining data, evaluation of personalized generations, and privacy concerns.
Abstract
from arXiv · showhide
This paper highlights the importance of personalization in large language models and introduces the LaMP benchmark -- a novel benchmark for training and evaluating language models for producing personalized outputs. LaMP offers a comprehensive evaluation framework with diverse language tasks and multiple entries for each user profile. It consists of seven personalized tasks, spanning three text classification and four text generation tasks. We additionally propose two retrieval augmentation approaches that retrieve personal items from each user profile for personalizing language model outputs. To this aim, we study various retrieval models, including term matching, semantic matching, and time-aware methods. Extensive experiments on LaMP for zero-shot and fine-tuned language models demonstrate the efficacy of the proposed retrieval augmentation approach and highlight the impact of personalization in various natural language tasks.
1 Introduction
Personalization is important for real-world LLM applications, yet personalized response development and evaluation remain understudied. LaMP addresses this gap with a diverse benchmark and retrieval-based personalization approaches.
- LaMP introduces a comprehensive benchmark for training and evaluating LLMs on personalized text classification and generation.
- Existing NLP benchmarks often follow a one-size-fits-all approach and do not support models that adapt to end-user needs.
- The benchmark spans three personalized classification tasks and four personalized generation datasets.The classification tasks cover citation identification, movie tagging, and product rating; generation covers headlines, scholarly titles, email subjects, and tweet paraphrasing.
- LaMP supports personalization for new users and future interactions of existing users through user-based and time-based data splits.
- The paper releases the benchmark, data construction, evaluation scripts, and leaderboard to foster personalized NLP research.
- Retrieval augmentation selects profile items for prompts because complete user profiles can exceed LLM context limits and impose substantial processing costs.
2 The LaMP Benchmark
LaMP formulates personalization as conditioning outputs on user profiles and evaluates this capability across seven classification and generation tasks. Its user-based and time-based splits measure personalization for new users and future interactions of existing users.
- Problem Formulation: A LaMP instance contains an input x, target output y, and user profile P_u consisting of the user’s historical data.
- Overview of LaMP: The benchmark evaluates personalized outputs across three classification tasks and four generation tasks.Tasks include citation identification, movie tagging, product rating, news headlines, scholarly titles, email subjects, and tweet paraphrasing.
- Evaluation Settings: User-based separation measures personalization for new users by ensuring that users do not overlap across train, validation, and test splits.
- Evaluation Settings: Time-based separation measures personalization for future interactions by using older profile items and more recent items for input-output pairs.
- Evaluation: The benchmark includes task-specific classification metrics and Rouge-1/Rouge-L for text generation tasks.
3 Retrieval Augmentation for Personalizing LLMs
The paper uses retrieval augmentation to select relevant entries from large user profiles before constructing personalized LLM inputs. It explores IPA and FiD alongside term-matching, dense, recency-based, and random retrieval models.
- Retrieval augmentation selects a subset of profile entries relevant to the current input, addressing context-length limits and the cost of processing long profiles.
- The framework uses query generation, profile retrieval, and prompt construction to build personalized model inputs.
- The two personalization strategies are In-Prompt Augmentation and Fusion-in-Decode.
- FiD requires language-model training and encoder-decoder architectures, whereas IPA can operate without training and across architectures.
- FiD can incorporate more profile items into the LLM input, while IPA and FiD therefore present different tradeoffs.
- The retrieval study compares BM25, Contriever, Recency, and Random document selection.
4 Experiments
Experiments on LaMP evaluate fine-tuned and zero-shot language models, retrieval strategies, and the number of profile entries used for personalization. Personalization generally improves performance, but retrieval choice and profile size affect results, with some tasks declining as more entries are added.
- Retrieval methods: The experiments establish fine-tuned personalization baselines and compare random selection, BM25, Contriever, and Recency retrieval methods.The study evaluates retrieval methods across user-based and time-based separation settings.
- Retrieval methods: Personalization improves performance for all LaMP tasks, often even when profile documents are selected randomly rather than retrieved selectively.Non-personalized comparisons include no retrieval augmentation and random items drawn from all user profiles.
- Retrieval methods: Contriever performs best for most classification tasks, while BM25 leads Email Generation and Scholarly Title Generation in the user-based setting.Contriever also leads News Headline Generation and Tweet Paraphrasing, whereas Recency surpasses it on time-based Product Rating.
- Profile size: Increasing the number of retrieved profile items improves performance for most datasets, although some tasks decline as more entries are added.The authors relate this trade-off to finite language-model context size and suggest unified prompts for multiple user entries as future work.
5 Research Problems Enabled by LaMP
LaMP opens research directions for personalizing language models, from profile-aware prompting and retrieval to user-sensitive evaluation and learned profile selection.
- Prompting Language Models for Personalization: Long user profiles make hard-prompt personalization difficult because language models have context-length and processing-cost constraints.The paper suggests generating personalization prompts or using soft prompts as alternatives to retrieved entries.
- Evaluation of Personalized Text Generation: Personalized text-generation metrics should account for user preferences rather than relying only on syntactic or semantic similarity.The paper identifies user-aware evaluation as a direction for improving personalized text-generation research.
- Learning to Retrieve from User Profiles: Learning to rank could optimize which profile entries are selected for personalized text classification and generation.This extends the benchmark’s retrieval component toward learned selection of user-specific evidence.
6 Related Work
Related personalization research spans information access, dialogue, language modeling, sentiment prediction, and annotation modeling, while LaMP focuses on personalized NLP datasets and tasks.
- Information Access: Personalization has been extensively studied in information access, including recommender systems, search, query auto-completion, and collaborative personalized search.The cited work covers applications ranging from recommendation challenges to search systems.
- Dialogue Agents: Dialogue-agent research personalizes generated utterances by conditioning them on users or specific personas.Some studies construct user-oriented dialogue data from persona-based authoring or extracted information.
- Language Modeling: Other work personalizes fundamental language modeling using publicly available user data from platforms such as Reddit, Facebook, Twitter, and blogs.This line includes personalized language modeling and related user-conditioned modeling settings.
- Sentiment and Rating Prediction: Personalized sentiment prediction and rating prediction connect personalization methods to recommendation tasks.The related work includes Yelp and IMDB sentiment data and parallels with LaMP’s rating-prediction task.
- Annotation Modeling: Personalization has also been applied to modeling annotators and accommodating human label variation in NLP classification.This connects user-sensitive modeling with emerging work on social norms and differing human labels.
7 Conclusion
The paper introduces LaMP as a benchmark for personalized text classification and generation and proposes retrieval-based augmentation for selecting user-profile information. Across datasets, personalization improves performance in both zero-shot and fine-tuned settings.
- Benchmark: LaMP contains seven datasets spanning three personalized classification tasks and four personalized generation tasks.The benchmark supports training and evaluating language models for personalized outputs.
- Methods: The paper proposes In-Prompt Augmentation and Fusion-in-Decode as retrieval augmentation approaches for personalizing LLM outputs.Experiments vary language models and retrieval techniques for selecting profile entries.
- Results: 12.2% average performance improvements occur across datasets in the zero-shot setting, compared with 23.5% with fine-tuning.These are the paper’s reported average improvements from its LLM personalization approaches.
- Conclusion: The paper concludes that personalization is important for developing user-centric natural language processing systems tailored to individual needs.This conclusion is stated within the scope of the benchmark and reported experiments.
Limitations
LaMP’s limitations concern task realism, possible overlap with LLM pretraining data, challenges in evaluating personalized generation, and privacy risks that the paper does not study.
- Overall limitations: The benchmark’s limitations span task definitions, data leakage, personalized-generation evaluation, and broader privacy concerns.These limitations arise from both benchmark design and the wider use of personalization.
- Task definitions: Some task definitions may not adequately reflect realistic scenarios or real-world applications.The paper specifically questions binary Personalized Citation Identification and notes limitations in how certain tasks model real use.
- Leakage to LLM pretraining data: Because LaMP relies largely on publicly available web data, some benchmark examples may have appeared in LLM pretraining data.Observed results may therefore differ from performance on unseen cases.
- Evaluating personalized generation: Most generation tasks use short texts because long-text personalization is difficult to evaluate reliably.Long-text evaluation involves subjectivity, no definitive reference, and challenges involving coherence and consistency.
- Privacy and personalization: Personalized fine-tuning with private data may expose users to privacy risks, including membership attacks.The paper highlights these concerns but does not study privacy issues, so further analysis is required.
A Data Creation Details for Tasks in the LaMP benchmark
LaMP constructs personalized task samples from user-grouped datasets, pairing each input with a target output and a profile of other user entries. The benchmark uses task-specific curation and splits for classification and generation settings.
- Benchmark construction: LaMP creates English-language samples across seven personalized tasks using task-specific datasets and templates.The supplied passages describe citation, movie, product, email, tweet, news, and scholarly-title data construction.
- Classification tasks: Movie-tagging samples retain the 15 most frequent tags and use chronologically earlier profile movies to predict tags for later movies.Descriptions are retrieved externally because MovieLense lacks film descriptions or summaries.
- Classification tasks: Product-rating samples filter users with fewer than 100 reviews and the top 1% prolific users, then use profile reviews to predict ratings for input reviews.The profile is intended to capture writing style, preferences, and tendencies; the input review’s score is the ground truth.
- Generation tasks: Email, tweet, news, and scholarly-title datasets group records by senders, users, authors, or paper authors and use remaining records as profiles for generation.The passages specify filtering or author/user thresholds for several generation tasks and author-based train, validation, and test splits.
- Classification tasks: Citation samples group papers by authors, retain authors with at least 50 papers, and use publication-year ordering to divide each author’s papers.The construction uses English papers with references, authors, and abstracts before author grouping and temporal division.
B Samples of the Tasks Introduced in the LaMP Benchmark
LaMP samples combine a task input and output with multiple user-profile entries, while task-specific templates convert those entries into prompts. Prompt aggregation trims profile material to fit the language model’s context limit.
- Sample structure: Each LaMP sample contains an input, an output, and a profile consisting of several user-related entries.The templates are designed to help the model produce personalized results for the user.
- Prompt construction: Task-specific prompts combine instructions, the task input, and retrieved profile entries to personalize the language model’s input.Table 5 documents the prompt templates and functions used for this augmentation.
- Prompt construction: Prompt creation has two stages: generating a Per Profile Entry Prompt for each entry and aggregating those prompts into one language-model prompt.The aggregated input prompt is formed by combining the individual PPEP prompts.
- Context management: Long PPEPs are trimmed by removing less important non-template parts, especially content unrelated to categories, scores, or titles.The paper sets the maximum task-input capacity to 256 tokens.
- Evaluation settings: The experiments evaluate fine-tuned and zero-shot language models under user-based and time-based separation settings.The validation results are reported in Tables 6–9 for these combinations of training and separation conditions.
E Performance of Some Other Non-Personalized Baselines on the LaMP Benchmark
The paper evaluates conventional and neural non-personalized baselines alongside LaMP tasks. SVM and BERT serve classification tasks, while BART serves generation tasks.
- Baseline models: SVM, BERT, and BART provide non-personalized baselines for LaMP’s classification and generation tasks.SVM and BERT are evaluated on classification, whereas BART is evaluated on generation.
F Dataset Licenses
The LaMP tasks retain the licenses and terms of use associated with their original datasets. The supplied passages also include unrelated experimental and writing-assistant material.
- Dataset licenses: The task licenses are inherited from the original datasets and include CC BY-NC-SA 4.0, academic non-commercial use, and the Avocado Collection agreement.The listed licenses cover the seven personalized tasks.