PaperlayerView exact revision on arXiv

Source-linked AI summary

User-LLM: Efficient LLM Contextualization with User Embeddings

Lin Ning, Luyang Liu, Jiaxing Wu, Neo Wu, Devora Berlowitz, Sushant Prakash, Bradley Green, Shawn O'Banion, Jun Xie

arXiv:2402.13598v2cs.CLcs.AIcs.LG

TL;DR

LLMs are difficult to contextualize with complex user timelines: text descriptions can require long context windows and miss subtle behavioral nuances. USER-LLM uses self-supervised user embeddings and cross-attention to condition LLMs on user history, outperforming text-prompt approaches across three datasets while reducing inference costs.

  • Problem

    User timelines are complex and potentially noisy, while text descriptions of them can require long context windows and may miss subtle user-behavior nuances.

    Source: §1 · p. 1

  • Method

    USER-LLM uses a self-supervised encoder to produce user embeddings from diverse interactions and integrates them with LLMs through cross-attention.

    Source: §1 · p. 2

  • Results

    Across MovieLens, Amazon Review, and Google Local Review, USER-LLM outperforms text-prompt approaches on tasks requiring deep user understanding, with up to 16.33% improvement.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    Its computational efficiency and ability to preserve LLM knowledge make USER-LLM well-suited to practical applications where real-time personalization is crucial.

    Source: §5 · p. 7

    Future work includes improving embedding pretraining and alignment and training on diverse tasks to enhance generalization across user scenarios.

    Source: §5 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable success across various domains, but effectively incorporating complex and potentially noisy user timeline data into LLMs remains a challenge. Current approaches often involve translating user timelines into text descriptions before feeding them to LLMs, which can be inefficient and may not fully capture the nuances of user behavior. Inspired by how LLMs are effectively integrated with images through direct embeddings, we propose User-LLM, a novel framework that leverages user embeddings to directly contextualize LLMs with user history interactions. These embeddings, generated by a user encoder pretrained using self-supervised learning on diverse user interactions, capture latent user behaviors and interests as well as their evolution over time. We integrate these user embeddings with LLMs through cross-attention, enabling LLMs to dynamically adapt their responses based on the context of a user's past actions and preferences. Our approach achieves significant efficiency gains by representing user timelines directly as embeddings, leading to substantial inference speedups of up to 78.1X. Comprehensive experiments on MovieLens, Amazon Review, and Google Local Review datasets demonstrate that User-LLM outperforms text-prompt-based contextualization on tasks requiring deep user understanding, with improvements of up to 16.33%, particularly excelling on long sequences that capture subtle shifts in user behavior. Furthermore, the incorporation of Perceiver layers streamlines the integration between user encoders and LLMs, yielding additional computational savings.

1 Introduction

User timelines contain rich behavioral information but can be complex and noisy; text-prompt approaches require long descriptions and may miss subtle preferences. USER-LLM instead contextualizes LLMs with user embeddings, improving efficiency and performance on tasks requiring deep user understanding.

1 Introduction

  • User timelines span diverse interactions and can be complex and noisy, while text descriptions require long context windows and may miss subtle behavioral nuances.The source highlights textual input, search queries, media consumption, social activity, and location visits as examples of timeline data.Source: §1 · p. 1
  • USER-LLM treats user timelines as a distinct modality and represents them directly as embeddings rather than relying on computationally expensive textual representations.The approach is inspired by integrating other modalities with LLMs through embeddings from modality-specific encoders.Source: §1 · pp. 1–2
  • Direct embeddings reduce the context window required for processing and can represent latent patterns and temporal dynamics in user interactions.The paper presents these properties as supporting more efficient processing and deeper understanding of users’ evolving needs.Source: §1 · p. 2
  • A self-supervised user encoder learns embeddings from diverse interactions that capture latent behaviors, interests, and their evolution over time.Cross-attention integrates these embeddings with LLMs so responses can adapt to users’ past actions and preferences.Source: §1 · p. 2
  • Across three public datasets, USER-LLM achieves up to 78.1X inference speedup and up to 16.33% improvement over text-prompt-based contextualization on tasks requiring deep user understanding.The approach particularly excels with long context inputs.Source: §1 · p. 2

2 Related Work

Related work spans multimodal LLM fusion, text-based personalization, long-context methods, and user modeling outside LLMs. USER-LLM draws on embedding-based contextualization while focusing its evaluation on LLM personalization.

2.1 Multimodal LLM

  • Multimodal LLM work progressed from aligning image and text representations, to cross-attention and soft-prompt fusion, and then to unified frameworks for diverse modalities.Examples include CLIP, Flamingo, PaLI (Chen et al. 2022a), and NextGPT (Wu et al. 2023b).Source: §2.1 · p. 2

2.2 Language Model based Personalization

  • LLM personalization has used user-history text prompts, while Doddapaneni et al. (2024) derived embeddings from text-based activity and integrated them via soft-prompt; USER-LLM uses activity tokens and fusion such as cross-attention.The paper evaluates its approach across datasets and tasks.Source: §2.2 · p. 2

2.3 Long Context in LLMs

  • Long-context methods extend context windows, retrieve or distill information, modify attention, or use linear attention; USER-LLM represents each user event with a single token.The passage states this representation is compatible with existing long-context techniques.Source: §2.3 · p. 2

2.4 User Modeling

  • Prior user-modeling methods generate user representations but primarily operate outside LLMs; comparisons with non-LLM user-modeling techniques are outside this work’s scope.The paper instead focuses on contextualizing LLMs with user representations.Source: §2.4 · pp. 2–3

3 Methodology

USER-LLM represents interaction histories as user embeddings and integrates them into a decode-only LLM through cross-attention. Its two-stage training and compact representations target personalized responses with less inference computation than text-based histories.

3.2 USER-LLM

  • USER-LLM encodes ID-based interaction histories as embeddings, then uses cross-attention to provide a decode-only LLM with context for personalized responses.The framework also supports alternative encoders and integration mechanisms such as soft prompts.Source: §3.2 · p. 3, User Embedding Generation · p. 3, LLM Contextualization with User… · p. 3
  • Each item’s feature IDs are embedded and concatenated into a fused representation, which the autoregressive Transformer converts into the user embeddings used for LLM contextualization.During pretraining, feature-specific projections support reconstruction of the original feature dimensions, and the overall loss averages cross-entropy losses across features.Source: User Embedding Generation · p. 3, User-encoder Pretraining · pp. 3–4
  • Cross-attention uses LLM text representations as queries and user embeddings as keys and values, integrating user-history information within the LLM.A trainable, zero-initialized scalar can gate each cross-attention layer’s contribution.Source: LLM Contextualization with User… · p. 3

3.3 Training Framework

3.4 Efficiency

  • User timelines become dense representations requiring one token per event, while smaller-dimensional user embeddings reduce computation during training and inference.Perceiver units further compress user embeddings with a shorter latent-query sequence, reducing the tokens needed to represent user history.Source: §3.4 · p. 4

4 Experiments

Experiments across three review and recommendation datasets evaluate USER-LLM on next-item prediction, preference prediction, and review generation against text-based baselines. USER-LLM generally improves performance while reducing computation, and ablations examine encoder pretraining, integration strategies, and model tuning.

4.1 Datasets and Tasks

  • The evaluation uses MovieLens20M, Amazon Review, and Google Review; sliding windows over timestamp-sorted user histories create next-item labels, with each user's latest interaction reserved for testing.Tasks include next-item prediction, favorite-genre or category prediction based on the most frequent historical choice, and review generation from item features.Source: §4.1 · pp. 4–5

4.2 Baselines and Experiment Setup

  • Text Prompt uses raw timeline text, while Text Summarization uses Gemini-M-generated summaries; resource limits restrict that comparison to Amazon Review.The described experiments use a pretrained Autoregressive Transformer and PaLM-2 XXS; reported USER-LLM results generally fine-tune the encoder, projection layers, and LLM together.Source: §4.2 · p. 5, §4.3 · p. 5

4.3 Performance and Efficiency Comparison

  • 12–21.9X fewer FLOPs than Text Prompt, with performance improvements up to 16.33% on most tasks across three datasets; Google Review next-item prediction is the exception.Figure 3 reports better performance with fewer tokens than Text Prompt and Text Summarization across three Amazon Review tasks.Source: Overall Performance · p. 5, Figure 3 · p. 5, Table 2 · p. 6
  • For sequence lengths 50, 100, and 200, USER-LLM uses a fixed 32-token LLM input versus Text Prompt’s 700, 1,350, and 2,500 tokens, respectively, reaching 78.1X FLOPs reduction at length 200.Source: Varying Context Length · p. 6
  • Freezing the LLM while tuning the encoder and projection layers achieves accuracy comparable to full fine-tuning with significantly fewer tuned parameters.The frozen-LLM strategy also surpasses text-prompt methods using full fine-tuning and LoRA tuning, while remaining comparable to USER-LLM with an unfrozen LLM.Source: Parameter efficiency · p. 6, Frozen LLM · p. 6, Table 3 · p. 6

4.4 Ablation Study

  • Pretrained encoders outperform randomly initialized encoders across all evaluated tasks.The study also reports that cross-attention generally outperforms soft-prompt integration, particularly on the two review-generation tasks.Source: Benefit of pretraining · p. 6, Soft Prompt v.s. Cross Attention · p. 7, Table 6 · p. 7
  • The Perceiver variant reduces the number of embedding tokens while maintaining competitive Recall@1 on MovieLens20M.Table 5 compares USER-LLM and its Perceiver variant on MovieLens20M using Recall@1.Source: Table 5 · p. 7
  • Gated cross-attention parameters indicate that upper LLM layers attend more to encoder output than lower layers on next-item prediction across three datasets.The authors suggest upper layers process semantic information such as user attributes and interests.Source: Gated Cross-attention · p. 7, Figure 4 · p. 7

5 Conclusion and Future Work

USER-LLM contextualizes LLMs with user embeddings that capture hidden preferences and their evolution, and evaluation reports strong performance on long histories. The authors identify embedding generation, embedding–language-model alignment, and broader task coverage as future research directions.

5 Conclusion and Future Work

  • USER-LLM integrates embeddings from self-supervised pretraining on diverse interactions with LLMs through cross-attention, enabling dynamic adjustment to user context.The embeddings capture hidden preferences and their evolution.Source: §5 · p. 7
  • Evaluation on MovieLens, Amazon Review, and Google Local Review finds consistent outperformance of text-prompt approaches on deep-understanding tasks, especially with long histories.The authors describe the framework as well-suited to real-time personalization because of its computational efficiency and ability to preserve LLM knowledge.Source: §5 · p. 7
  • Future work includes improving user-embedding pretraining, studying alignment with the language model, and training on diverse tasks to potentially improve generalization across user scenarios.Source: §5 · p. 7

A More details about Datasets

The evaluation uses MovieLens20M, Google Local Review, and Amazon Review data, with the latter two drawn from specified subsets.

A More details about Datasets

B Training Hyperparameters

The experiments use cosine learning-rate decay with linear warmup during the first 10% of training steps.

B Training Hyperparameters

  • Cosine learning-rate decay is used across experiments, with linear warmup over the first 10% of training steps.Table 7 contains the training hyperparameters, including learning rate, weight decay, batch size, and training steps.Source: B Training Hyperparameters · p. 11

C Encoder Architectures

The paper compares early-fusion autoregressive and late-fusion dual encoders for producing user embeddings, then considers cross-attention and soft-prompt integration with LLMs.

C Encoder Architectures

  • Early fusion combines each item's name, rating, and category features before encoding the sequence, while late fusion processes features separately and combines their embeddings later.Source: C Encoder Architectures · p. 11
  • The Autoregressive Transformer embeds each feature, concatenates them, and processes the sequence with a Transformer decoder to produce an embedding for each timeline item.Source: C Encoder Architectures · p. 11
  • The Dual Encoder uses separate user and label towers; LLM integration uses the user tower, while fusion layers can adjust the number of generated embeddings.The user and label embeddings are compared for loss computation via a softmax layer.Source: C Encoder Architectures · p. 11
  • Autoregressive Transformers capture long-range dependencies but output one embedding per input item, which can be inefficient for long sequences; Dual Encoders offer configurable embedding counts but may model sequential patterns less effectively.Source: C Encoder Architectures · p. 11
  • Without computation constraints, Autoregressive Transformers outperform Dual Encoders across three datasets and eight tasks, including across training strategies and encoder initialization choices.The comparison is reported in Table 9.Source: C Encoder Architectures · p. 11

D Integration Approaches

  • Cross-attention fuses encoder outputs with intermediate LLM representations, whereas soft-prompt prepends user embeddings to the text prompt; Table 11 compares these approaches and Table 8 covers short-term prompts alongside long-term embeddings.Table 8 distinguishes text-prompt baselines using 10 or 50 recent events from encoder inputs using 50 events, with one condition also adding 10 recent events as text.Source: D Integration Approaches · p. 11, Table 8 · p. 12

E More Results

Table 10 compares Autoregressive and Dual Encoder architectures on long-term and short-term user-context tasks.

E.1 Autoregressive Encoder v.s. Dual Encoder

  • Table 10 compares Autoregressive Encoder and Dual Encoder performance on long-term and short-term user-context tasks.The referenced Table 9 columns provide results for pretrained encoders using the Full training strategy.Source: §E.1 · p. 12
  • The comparison covers both long-term and short-term user-context tasks.Source: §E.1 · p. 12
  • For pretrained encoders with Full training, Table 9 columns 4 and 12 provide the corresponding performance results.Source: §E.1 · p. 12

E.2 Training Strategies and Encoder Pretraining

These results examine training and encoder choices, integration strategies, sequence length, and the combination of long- and short-term user context.

E.2 Training Strategies and Encoder Pretraining

  • Table 9 compares training strategies and pretrained versus randomly initialized user encoders during encoder–LLM integration.Source: §E.2 · p. 12

E.3 Cross-attention v.s. Soft-prompt

  • Table 11 compares cross-attention and soft-prompt integration strategies.Source: §E.3 · p. 12

E.4 Next Item Prediction

  • Table 12 compares text-prompt-based methods and USER-LLM across varying input activity sequence lengths on MovieLens20M next-item prediction.Source: §E.4 · p. 12

E.5 Long-term and short-term user context

  • USER-LLM generally performs better on next-item prediction when long-term user context is combined with short-term interactions.With 50 activities embedded and 10 recent activities used in the prompt, AR50TP10 outperforms TP10 and TP50; short-term context also improves next-item name prediction over AR50 while preserving computational efficiency.Source: §E.5 · p. 12

F Review Generation Examples

The examples illustrate personalized review generation on Amazon and Google review tasks, with responses that reflect user characteristics more than unpersonalized responses, though discrepancies from ground truth remain.

F Review Generation Examples

  • Personalized USER-LLM responses align more closely with user preferences and backgrounds than unpersonalized responses, although they can differ from ground truth.Source: F Review Generation Examples · p. 12
  • For the Firm - Aerobic Body Shaping prompt, the personalized response is “Terrible Waste of Time,” versus “Aerobic Workout” unpersonalized and “Save Your Money” as ground truth.Source: F Review Generation Examples · p. 12
  • For the A.I. - Artificial Intelligence prompt, the personalized response offers a science-fiction-oriented description, while the unpersonalized response repeats the title.Source: F Review Generation Examples · p. 13
Loading 2402.13598v2…