PaperlayerView exact revision on arXiv

Source-linked AI summary

Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning

Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, Meng Jiang

arXiv:2402.04401v3cs.CL

TL;DR

Personalized LLM research has used retrieval and profile prompts, but existing approaches face ownership, customization, privacy, and behavior-shift challenges. OPPU gives each user a personal PEFT module and combines its parametric knowledge with retrieval or profiles. It outperforms baseline personalization methods across seven LaMP tasks, including under behavior shifts.

  • Problem

    Prompt-based LLM personalization uses user history or profiles, while existing methods face model-ownership, customization, privacy, and behavior-pattern generalization challenges.

    Source: §1 · pp. 1–2

  • Method

    OPPU fine-tunes a personal PEFT module on each user’s behavior history and can combine its parametric knowledge with retrieval or profile augmentation.

    Source: §1 · p. 2, §3 · p. 3, §8 · p. 9

  • Results

    OPPU outperforms baseline personalization methods across seven LaMP tasks and performs strongly when behavior shifts make history less relevant.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support per-user PEFT as a way to provide model ownership while adapting to behavior shifts, with reported robustness across activity levels and history formats.

    Source: §8 · p. 9

    The evaluation mainly considers one task per user rather than behavior spanning multiple tasks and domains.

    Source: §9 · p. 9

Abstract

from arXiv · show

Personalization in large language models (LLMs) is increasingly important, aiming to align the LLMs' interactions, content, and recommendations with individual user preferences. Recent advances have highlighted effective prompt design by enriching user queries with non-parametric knowledge through behavior history retrieval and textual profiles. However, these methods faced limitations due to a lack of model ownership, resulting in constrained customization and privacy issues, and often failed to capture complex, dynamic user behavior patterns. To address these shortcomings, we introduce One PEFT Per User (OPPU), employing personalized parameter-efficient fine-tuning (PEFT) modules to store user-specific behavior patterns and preferences. By plugging in personal PEFT parameters, users can own and use their LLMs individually. OPPU integrates parametric user knowledge in the personal PEFT parameters with non-parametric knowledge from retrieval and profiles, adapting LLMs to user behavior shifts. Experimental results demonstrate that OPPU significantly outperforms existing prompt-based methods across seven diverse tasks in the LaMP benchmark. Further studies reveal OPPU's enhanced capabilities in handling user behavior shifts, modeling users at different activity levels, maintaining robustness across various user history formats, and displaying versatility with different PEFT methods.

1 Introduction

The introduction frames personalized LLMs as a response to one-size-fits-all models, while identifying ownership and behavior-pattern generalization as challenges for existing approaches. It proposes OPPU, which personalizes LLMs with per-user PEFT modules, and reports strong results across LaMP tasks.

1 Introduction

  • Existing personalization methods rely mainly on prompts enriched with user history or profiles, but centralized processing constrains customization and raises privacy concerns.Users may need to share personal data with the service provider.Source: §1 · pp. 1–2
  • Retrieval-based personalization may underperform when past behaviors do not match a query’s needs, since irrelevant context can distract LLMs (Shi et al., 2023).Source: §1 · p. 2
  • OPPU equips each user with a personal PEFT module that stores behavior patterns and preferences, supporting individual model ownership and customization.The introduction notes that updated PEFT parameters are typically less than 1% of the base LLM’s weight.Source: §1 · p. 2
  • OPPU achieves state-of-the-art performance across seven LaMP tasks and outperforms baseline personalization methods, including when user behavior shifts make history less relevant.The introduction also reports robustness to varied history formats and versatility across PEFT methods.Source: §1 · p. 2
  • The paper presents OPPU as a parametric personalization approach that opens new opportunities for democratizing personalized LLMs.Source: §1 · p. 2

2 Preliminaries

The preliminaries formulate personalization as predicting a user’s response from a query and prior behavior history. They also establish task-adapted base LLM variants for comparisons, including retrieval- and profile-augmented alternatives.

2.1 Research Problem Formulation

  • For user u at time t, personalization predicts response r_u from query q_u and prior behavior history H_u, which may be query-answer pairs or plain text.Source: §2.1 · p. 2

2.2 Base LLMs Task Adaption

  • For fair comparison, the paper task-adapts base LLMs using held-out users without target-user preferences and defines base, retrieval-augmented, and profile-augmented variants.Retrieval adds top-k relevant user history; profile augmentation adds a textual profile generated from history.Source: §2.2 · pp. 2–3
  • Task adaptation uses LoRA, updating about 0.5% of the LLM’s parameters before merging the trained LoRA parameters into the base model.Source: §2.2 · p. 3

3 One PEFT Per User (OPPU)

OPPU creates personalized LLMs by tuning a PEFT module with each user’s behavior history and plugging it into a base model. When history does not match the task’s query-answer format, the method uses history sequences for unsupervised next-token prediction.

3 One PEFT Per User (OPPU)

  • Each user’s history tunes a personal PEFT module, which is plugged into the corresponding base LLM to create a user-owned personalized model.The module’s parameters capture user behavior patterns and preferences.Source: §3 · pp. 3–4
  • Personal PEFT training can use base, retrieval-augmented, or profile-augmented objectives, with history restricted to behavior preceding the training example.Source: §3 · p. 3
  • When history consists of text sequences rather than task-matched query-answer pairs, OPPU replaces history outputs with right-shifted sequences for unsupervised next-token prediction.Source: §3 · pp. 3–4

4 Experimental Settings

Experiments use seven public LaMP tasks, comparing OPPU with non-personalized, retrieval-augmented, and profile-augmented baselines. Evaluation focuses on 100 highly active users and excludes LaMP-6 because its private data were inaccessible.

  • The LaMP evaluation uses seven public tasks: four classification tasks and three generation tasks.Source: §4 · p. 4
  • The test set comprises the 100 users with the longest histories in the time-based dataset; all other users train the base LLM.Source: §4 · p. 4
  • Baselines are non-personalized, retrieval-augmented, and profile-augmented models using Llama-2-7B and BM25 for retrieval.Source: §4 · p. 4
  • Classification is evaluated with accuracy and F1, LaMP-3 with MAE and RMSE, and generation with ROUGE-1 and ROUGE-L; lower is better for MAE and RMSE.Source: §4 · p. 4
  • LaMP-6 is excluded because its private data were inaccessible to the authors.Source: §4 · p. 4

5 Results

OPPU outperforms baseline personalization methods across the seven LaMP tasks, with further gains when parametric user knowledge is combined with retrieval or profiles. Results also show gains on tasks where user-history and task formats differ, while adding retrieved items is not uniformly beneficial.

  • Across seven tasks, adding retrieval gained 1.93% versus non-retrieval OPPU and 2.48% versus retrieval without OPPU; adding profiles gained 4.56% versus non-profile OPPU and 7.18% versus non-OPPU.Source: §5 · p. 4
  • For personalized citation identification, where publication-history format differs from the task, OPPU increased accuracy by 3.48% and F1-score by 3.52%.Source: §5 · pp. 4–5
  • OPPU outperformed all baseline personalization methods across seven LaMP tasks; scholarly title generation ROUGE-1 and ROUGE-L improved 3.42% and 3.87%, respectively.Source: §5 · p. 4, Table 1 · p. 5
  • Table 2 reports query counts, average input and output token lengths, and the number of history items adopted for each task.Source: Table 2 · p. 5
  • More retrieved items generally correlated with better performance, but exceptions were hypothesized to reflect noise and irrelevant behavior patterns complicating preference understanding.Source: §5 · p. 5

6 Analysis

The analysis finds OPPU more robust than retrieval-based personalization when history is irrelevant, and examines how activity level, retrieval, history format, and PEFT choice affect performance.

6 Analysis

  • With less relevant history, OPPU generalizes more robustly than retrieval-based methods; OPPU with k=1 helps generation, while k=0 performs better for classification.Limiting history to less relevant items significantly reduces retrieval-based performance, often bringing it close to non-personalized approaches.Source: §6 · pp. 5–6, Table 3 · p. 5

Modeling Users with Different Active Levels

  • Across activity levels, retrieval+OPPU performs strongest, with its advantage over baselines more pronounced for users with longer histories.The analysis randomly selected 20 users from each activity range.Source: Modeling Users with Different Active… · p. 6, Figure 3 · p. 6
  • As retrieved history grows, retrieval-only and retrieval+OPPU both improve, while their performance gap narrows.The authors suggest longer prompt histories reduce the gap between retrieved knowledge and behavior patterns encoded in PEFT parameters.Source: Modeling Users with Different Active… · p. 6, Figure 4 · p. 6
  • Combining parametric and retrieved knowledge keeps OPPU robust to incomplete or mismatched history formats.It performs close to full-history OPPU in text generation and outperforms models tuned on complete history in news categorization when retrieval is integrated.Source: Table 4 · p. 6, Modeling Users with Different Active… · p. 7
  • OPPU improves performance with LoRA, (IA)3, and prompt tuning; LoRA ranks highest, followed by (IA)3 and prompt tuning.Their trainable-parameter proportions are 0.01%, 0.06%, and 0.001%, respectively; the authors report that more trainable parameters generally correspond to better personalization.Source: Modeling Users with Different Active… · p. 7, Figure 6 · p. 7
  • In the movie-tagging case, OPPU recognizes a user’s frequent “based on a book” tag where query-only and retrieval-based methods answer incorrectly.The retrieval method fails to find history closely matched to the query, while OPPU uses the user’s broader behavior patterns.Source: Modeling Users with Different Active… · p. 7, Figure 5 · p. 7
  • Personal PEFT similarities range from 0.4 to 0.7, with the highest average in scholarly title generation; classification similarities vary more across users than generation similarities.The authors suggest that preferences in generation tasks are harder to categorize.Source: Modeling Users with Different Active… · pp. 7–8, Figure 7 · p. 8

7 Related Work

Prior LLM personalization mainly uses prompts containing user history, retrieved examples, or textual profiles. PEFT updates a small set of parameters, and OPPU applies it to store user-specific behavior in personal modules.

7.1 Personalization of LLMs

  • LLM personalization has used vanilla, retrieval-augmented, and profile-augmented prompts to incorporate user history and preferences.Vanilla prompts use in-context examples; retrieval selects history for limited context windows; profile methods add natural-language summaries.Source: §7.1 · p. 8
  • Personalization research also includes parameter merging and personalized reward models, beyond prompt-based approaches.These are described as another line of personalized alignment work.Source: §7.1 · p. 8

7.2 Parameter-Efficient Fine-tuning (PEFT)

  • PEFT updates a small number of extra parameters while keeping pretrained weights frozen, addressing the expense of full fine-tuning.Examples include adapters, prefix and prompt tuning, LoRA, and (IA)3.Source: §7.2 · p. 8
  • OPPU uses a personal PEFT module to store user history parametrically, providing each user a plug-in module for model ownership.The passage presents PEFT’s small parameter count and plug-and-play nature as suited to efficient personalization.Source: §7.2 · p. 8

8 Conclusion

OPPU equips each user with a personal PEFT module tuned on their history and integrates parametric knowledge with retrieval and profiles. It outperforms baselines across seven LaMP tasks, with further experiments showing versatility, robustness, and effectiveness across activity levels.

8 Conclusion

  • OPPU combines user-specific PEFT parameters with retrieval and profiles, outperforming baselines across all seven LaMP benchmark tasks.The personal module is tuned on a user’s history to capture behavioral patterns.Source: §8 · p. 9
  • Additional experiments show OPPU’s effectiveness for users with varying activity levels, as well as its versatility and robustness.The conclusion describes the framework as supporting PEFT-based personalization and LLM modularity.Source: §8 · p. 9

9 Limitations

The authors identify limits in task and domain coverage and in handling dynamic, potentially conflicting user histories.

9 Limitations

  • The study focuses on one task per user, leaving behavior across multiple tasks and domains for future investigation.The authors give movie tagging as an example where users’ histories consist solely of that activity.Source: §9 · p. 9
  • OPPU places the entirety of a user’s history in the PEFT module, while interests may shift or conflict over time.The authors identify history selection and management of discrepancies as future research directions.Source: §9 · p. 9

10 Ethical Considerations

The ethical discussion identifies privacy, bias, and accessibility concerns in personalized LLMs, while the hyperparameter appendix notes LoRA’s trainable low-rank matrices in Wq and Wv.

10 Ethical Considerations

  • Personalization may expose sensitive user details, so the passage calls for safeguards and respectful, secure data handling.The stated concern is unintended disclosure of personal information.Source: §10 · p. 9
  • Biased or unrepresentative personal data could lead personalized models to produce unfair or prejudiced responses.The passage calls for monitoring and mitigating bias in both the personal data and the resulting model.Source: §10 · p. 9
  • Resource-intensive LLMs may be difficult for smaller entities and researchers with limited compute or budgets to use.The passage warns this could widen gaps in AI research and application.Source: §10 · p. 9

A Hyperparameters

B Efficiency Analysis

This appendix evaluates per-user OPPU training time and documents its baselines, task formats, implementation context, and personalization prompts. It also specifies how PEFT-module similarities are calculated.

B Efficiency Analysis

  • Training time increases linearly with the number of history items, while growth with tokens per item is observed to be closer to linear than the theoretical quadratic trend.The analysis varies history-item counts from 10 to 100, then varies tokens per item from 10 to 100 with 10 items fixed; it excludes user count because private PEFT training can be simultaneous or distributed.Source: B Efficiency Analysis · p. 13

C Baseline Details

  • The non-personalized baselines use either no behavior history or random history, while personalized baselines add retrieved items, user profiles, or both.Retrieval uses k=1, 2, or 4; the profile-plus-retrieval setting uses k=1.Source: C Baseline Details · p. 13

E Computation Resources Details

  • The experiments use three NVIDIA A6000 GPUs and a 20-core Intel Xeon Silver 4210R CPU; training 100 personal PEFT modules sequentially takes about 12 minutes to 12 hours.Duration depends on behavior-history corpus size and sequence length.Source: E Computation Resources Details · p. 14

F PEFT Cosine Similarity Details

G Task Details

  • These tasks comprise binary citation choice based on a user’s publications, 15-way news categorization based on user history, 15-way movie tagging based on historical movie-tag pairs, and 5-way product rating based on historical review-rating pairs.Source: G Task Details · p. 14
  • The generation tasks test personalized tweet paraphrasing, news-headline generation, and scholarly-title generation from a user’s prior writing patterns.Source: G Task Details · p. 14

H Prompt for Personalization Tasks

I Prompt for User Profile Generation

  • Profile-generation prompts ask for user-history-based summaries or templates of preferences and writing styles across the task domains.The prompts include research interests, popular categories or tags, rating patterns, and headline, title, or tweet styles.Source: §I.1 · p. 15, §I.2 · p. 15, §I.3 · p. 15
Loading 2402.04401v3…