PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation

Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, Weinan Zhang

arXiv:2308.11131v6cs.IRcs.AI

TL;DR

The paper studies LLMs’ difficulty extracting useful information from long user behavior sequences in recommendation, even below the context limit. It proposes ReLLa, combining semantic behavior retrieval with retrieval-enhanced instruction tuning, and reports that few-shot ReLLa using less than 10% of training samples can outperform full-shot traditional CTR models. The comparison is made against baselines trained on the entire training set.

  • Problem

    LLMs may fail to extract useful information from long textual user behavior sequences for recommendation even when the context is far below their context limitation.

  • Method

    ReLLa uses semantic user behavior retrieval for zero-shot recommendation and retrieval-enhanced instruction tuning with mixed original and retrieved samples for few-shot recommendation.

  • Results

    Using less than 10% of training samples, few-shot ReLLa can outperform all full-shot traditional CTR models trained on the entire training set.

  • Takeaways & Limitations

    ReLLa demonstrates data efficiency and comprehension ability for long user behavior sequences within the reported recommendation experiments.

  • Takeaways & Limitations

    The comparison uses few-shot-tuned ReLLa against traditional recommendation baselines trained in full-shot settings.

Abstract

from arXiv · show

With large language models (LLMs) achieving remarkable breakthroughs in natural language processing (NLP) domains, LLM-enhanced recommender systems have received much attention and have been actively explored currently. In this paper, we focus on adapting and empowering a pure large language model for zero-shot and few-shot recommendation tasks. First and foremost, we identify and formulate the lifelong sequential behavior incomprehension problem for LLMs in recommendation domains, i.e., LLMs fail to extract useful information from a textual context of long user behavior sequence, even if the length of context is far from reaching the context limitation of LLMs. To address such an issue and improve the recommendation performance of LLMs, we propose a novel framework, namely Retrieval-enhanced Large Language models (ReLLa) for recommendation tasks in both zero-shot and few-shot settings. For zero-shot recommendation, we perform semantic user behavior retrieval (SUBR) to improve the data quality of testing samples, which greatly reduces the difficulty for LLMs to extract the essential knowledge from user behavior sequences. As for few-shot recommendation, we further design retrieval-enhanced instruction tuning (ReiT) by adopting SUBR as a data augmentation technique for training samples. Specifically, we develop a mixed training dataset consisting of both the original data samples and their retrieval-enhanced counterparts. We conduct extensive experiments on three real-world public datasets to demonstrate the superiority of ReLLa compared with existing baseline models, as well as its capability for lifelong sequential behavior comprehension. To be highlighted, with only less than 10% training samples, few-shot ReLLa can outperform traditional CTR models that are trained on the entire training set (e.g., DCNv2, DIN, SIM). The code is available \url{https://github.com/LaVieEnRose365/ReLLa}.

1 INTRODUCTION

The paper identifies a lifelong sequential behavior incomprehension problem in LLM-based recommendation and proposes ReLLa to address it in zero-shot and few-shot settings. ReLLa combines semantic retrieval with retrieval-enhanced instruction tuning, and experiments compare it with existing baselines.

  • Problem: LLMs may fail to extract useful information from long user behavior sequences even when the context remains below their context-window limit.On MovieLens-1M, Vicuna-13B peaks at K=15 and declines for K>15, unlike SIM’s steady gains.
  • Approach: ReLLa addresses this problem for both zero-shot and few-shot recommendation settings.The framework is designed to adapt a pure LLM for recommendation tasks.
  • Zero-shot recommendation: Semantic user behavior retrieval replaces simply truncated recent behaviors with the top-K behaviors semantically relevant to the target item.This improves sample quality and reduces the difficulty of extracting useful information from behavior sequences.
  • Few-shot recommendation: Retrieval-enhanced instruction tuning augments training with both original and retrieval-enhanced samples for few-shot recommendation.The mixed dataset is intended to improve robustness and generalization when training data are limited.
  • Evaluation: Experiments on three real-world public datasets evaluate ReLLa against existing baselines, including full-shot traditional recommendation models.The contribution statement emphasizes comparisons where ReLLa uses few-shot samples while baseline models use the entire training set.

2 PRELIMINARIES

The preliminaries define zero-shot and few-shot recommendation, formulate recommendation examples as textual input-output pairs, and describe how LLM outputs are converted into pointwise CTR scores. They also introduce semantic retrieval as an alternative to selecting only recent behaviors.

  • 2.1 Zero-shot and Few-shot Recommendations: Zero-shot recommendation uses an LLM directly on the target task without tuning on in-domain training data.Traditional recommendation models cannot perform this setting when randomly initialized, whereas LLMs can use open-world knowledge and reasoning abilities.
  • 2.1 Zero-shot and Few-shot Recommendations: Few-shot recommendation uses a relatively small number N of training samples and therefore requires data-efficient learning.Full-shot recommendation instead trains on the entire training set.
  • 2.2 Textual Input-Output Pair Formulation: Each recommendation sample is converted into textual sentences, while its binary label is represented by the answer words “Yes” and “No”.The textual input describes the user profile, behavior sequence, target item, and task description.
  • 2.2 Textual Input-Output Pair Formulation: Because user behavior sequences can contain tens to hundreds of behaviors, ReLLa selects the most relevant K behaviors toward the target item rather than simply the most recent K.Figure 2 illustrates the textual input-output format, while the method changes how the sequence portion is constructed.
  • 2.3 Pointwise Scoring with LLMs: For pointwise CTR prediction, the LLM’s scores for the binary answer words are converted into a floating-point click-through-rate estimate using a bidimensional softmax.The estimated score is used for evaluation on the testing set, while training retains instruction tuning and causal language modeling.
  • 2.3 Pointwise Scoring with LLMs: Semantic user behavior retrieval obtains item semantic vectors and retrieves behaviors relevant to the target item to improve zero-shot input quality.The retrieval process is intended to make useful information easier for the LLM to extract from long histories.

3 METHODOLOGY

ReLLa develops two retrieval-based techniques for zero-shot and few-shot recommendation: SUBR replaces recent behaviors with semantically relevant ones, while ReiT augments instruction-tuning data with retrieval-enhanced samples.

  • Zero-shot recommendation: SUBR replaces the truncated top-K recent behaviors with the top-K behaviors semantically relevant to the target item for zero-shot recommendation.The retrieved behaviors preserve roughly the original input-context length while improving sample quality.
  • Zero-shot recommendation: Semantic item encoding uses descriptive item text, LLM hidden-state average pooling, and PCA for dimension reduction and denoising.The resulting semantic vectors support behavior retrieval.
  • Few-shot recommendation: ReiT applies SUBR to training samples and combines original and retrieval-enhanced instances into a mixed dataset for few-shot instruction tuning.The mixed dataset contains 2N samples and enriches user-behavior patterns.
  • Few-shot recommendation: Pattern enrichment from SUBR is intended to regularize instruction tuning and improve robustness and generalization when extracting information from long behavior sequences.The paper contrasts this pattern enrichment with simply increasing the number of training samples and discusses both factors as contributors to improvement.
  • Few-shot recommendation: ReiT uses the causal language-modeling objective on the mixed dataset, while CTR scoring with BCE is reserved for testing rather than training.The method retains the original LLM structure without appending a randomly initialized prediction layer.

4 EXPERIMENT

Experiments on three real-world datasets evaluate ReLLa against recommendation baselines across zero-shot, full-shot, and few-shot settings, including sequence-length, data-efficiency, and ablation analyses. ReLLa improves long-sequence comprehension and generally outperforms baselines with limited training data, while retrieval and mixed-data training contribute to its performance.

  • Experimental setup: Experiments use BookCrossing, MovieLens-1M, and MovieLens-25M, evaluating AUC, Log Loss, and ACC across recommendation settings.The study compares zero-shot, full-shot, and few-shot performance and analyzes sequence length, shot number, and component effects.
  • Overall performance: ReLLa significantly outperforms Vicuna-13B on all three metrics for BookCrossing and MovieLens-1M, while on MovieLens-25M it improves Log Loss and ACC despite lower AUC.These results support retrieval-enhanced zero-shot recommendation but also reveal potential instability in zero-shot LLM AUC on MovieLens-25M.
  • Overall performance: Few-shot ReLLa generally outperforms baseline models while using less than 10% of their training samples, demonstrating data efficiency for sequential recommendation.For MovieLens-25M, ReLLa uses N = 65,536 samples versus SIM’s N = 19,349,912 full-shot samples.
  • Sequential behavior comprehension: ReLLa’s AUC continuously improves as behavior-sequence length grows, whereas Vicuna-13B peaks at K = 30/15/15 and then declines across the three datasets.The LLM’s decline occurs even when the input uses roughly 500/700/700 tokens, below its 2048-token context limit.
  • Data efficiency: ReLLa and SIM improve as shot number increases, but ReLLa consistently outperforms SIM and remains effective when training samples are extremely limited.On BookCrossing, SIM’s AUC is around 0.5 with 128 or 256 shots, while ReLLa retains few-shot inference ability.
  • Ablation study: Removing mixture training or retrieval reduces AUC by at least 1.12%, 0.99%, and 1.95% on BookCrossing, MovieLens-1M, and MovieLens-25M, respectively.Additional ablations attribute gains to doubled training samples, pattern enrichment, and semantic retrieval; pattern enrichment produces 0.21%, 0.16%, and 0.48% relative AUC improvements in the half-shot comparison.

5 RELATED WORK

Traditional CTR models capture feature interactions or sequential user behavior, while LLM-based recommenders can serve as feature engineers, encoders, or scoring/ranking functions. ReLLa focuses on LLMs as scoring functions for pointwise CTR prediction and addresses their difficulty comprehending lifelong behavior sequences.

  • Traditional CTR Models: Traditional CTR models include feature-interaction models and sequential recommendation models.Feature-interaction models capture higher-order feature patterns, while sequential models dynamically model user interests from behavior histories.
  • LLMs for Recommendation: LLMs in recommender systems serve as feature engineers, feature encoders, or scoring/ranking functions.These roles respectively support textual feature augmentation, semantic representation learning, and direct item scoring or generation.
  • ReLLa Positioning: ReLLa adopts LLMs as scoring/ranking functions for pointwise CTR prediction and targets their incomprehension of lifelong user behavior sequences.The paper identifies this problem and introduces retrieval techniques to improve LLM comprehension in recommendation.

6 CONCLUSION

The paper concludes that ReLLa combines semantic user behavior retrieval and retrieval-enhanced instruction tuning to address LLM incomprehension of long behavior sequences. Experiments on three datasets report that few-shot ReLLa can outperform full-shot traditional CTR models using less than 10% of their training samples.

  • 6 CONCLUSION: ReLLa uses semantic user behavior retrieval and retrieval-enhanced instruction tuning to improve LLM comprehension of long behavior sequences.SUBR addresses the retrieval stage, while ReiT incorporates retrieval-enhanced samples during instruction tuning.
  • 6 CONCLUSION: The paper evaluates ReLLa on three real-world public datasets and compares it with existing baselines.The experiments also include hard prompt templates for the datasets and compare behavior sequences with and without SUBR.
  • 6 CONCLUSION: Less than 10% training samples allow few-shot ReLLa to outperform all full-shot traditional CTR models.The reported result is presented as evidence of ReLLa’s data efficiency and behavior-sequence comprehension ability.

B DATA PREPROCESSING

The experiments use BookCrossing, MovieLens-1M, and MovieLens-25M with dataset-specific splits and rating thresholds for binary labels. Few-shot samples are uniformly drawn and nested across larger shot settings, while baseline prompts may use the most recent K items.

  • Dataset Splits: Experiments use BookCrossing, MovieLens-1M, and MovieLens-25M with dataset-specific train-test splits.MovieLens datasets use timestamp-based 8:1 splits, whereas BookCrossing uses a 9:1 random user split because it lacks timestamps.
  • Label Construction: Rating thresholds convert the three datasets’ scores into positive and negative samples.BookCrossing uses ratings above 5, MovieLens-1M uses ratings of 4 or 5, and MovieLens-25M uses ratings above 3.0.
  • Behavior Sequences: Without SUBR, baseline behavior sequences use the most recent K items.This contrasts with SUBR, which replaces them with semantically relevant historical items.
  • Few-shot Sampling: Few-shot training sets uniformly sample N instances, with smaller sampled sets included within larger-shot sets.This nesting keeps the few-shot comparisons consistent across different values of N.

C BASELINE IMPLEMENTATION

Baseline implementation covers traditional CTR models and language-model-based models with tuned architecture and optimization settings. ReLLa prompts represent users, histories, and target items textually, while SUBR uses item-description embeddings and semantically relevant behaviors.

  • Baseline Categories: Baseline implementation covers traditional CTR models and language-model-based models.The paper separates their hyperparameter configurations into these two categories.
  • Traditional CTR Models: Traditional CTR baselines tune embedding or hidden sizes, layer counts, dropout, learning rates, batch sizes, and optimizers.The specific search spaces vary across BookCrossing, MovieLens-1M, and MovieLens-25M.
  • Prompt Construction: ReLLa hard prompts encode user attributes, behavior histories, ratings, and target-item prediction instructions.The examples ask the model to output whether the user will like the target item.
  • Semantic Retrieval: SUBR obtains semantic item embeddings from item descriptions and retrieves relevant historical items for the user behavior sequence.The retrieval-enhanced prompts replace the most recent K behaviors with the most relevant K behaviors toward the target item.
  • Language-Model Baselines: Language-model baselines retain pretrained structures and use AdamW, with model-specific training settings for CTR-BERT, P5, and PTab.The implementations include two-tower encoding, sequence-to-sequence modeling, and pretraining followed by classification fine-tuning.

D ADDITIONAL EXPERIMENTS

Additional experiments test whether lifelong sequential behavior incomprehension generalizes across LLM backbones and whether ReLLa generalizes with different models. On MovieLens-1M, zero-shot AUC is evaluated across sequence lengths for five LLMs.

  • Universality and generalization: Five LLM backbones—Falcon-7B, Mistral-7B, Vicuna-7B, Vicuna-13B, and LLaMA-2-70B-Chat—are used to test generalization.The study includes models with different architectures and sizes.
  • Universality and generalization: Zero-shot AUC is measured on MovieLens-1M as user-behavior sequence length K varies from 5 to 30.The evaluated lengths are K ∈ {5, 10, 15, 20, 25, 30}.
  • Evaluation setting: LLaMA-2-70B-Chat is evaluated on a downsampled 10,000-example test set, whereas the other four LLMs use the full test set.The downsampling is attributed to the model’s time consumption.

D.1.2 Generalization of ReLLa.

ReLLa is evaluated across backbone LLMs and sequence lengths, with results indicating consistent gains over original LLMs and strong data efficiency. The analysis also identifies slower inference as a practical boundary and links recommendation performance to instruction-following capability.

  • Generalization of ReLLa: ReLLa consistently and significantly improves recommendation performance over the original LLMs in both zero-shot and few-shot settings.The comparison covers multiple backbone LLMs under the generalization experiment.
  • Generalization of ReLLa: Mistral-7B, Vicuna-7B, and Vicuna-13B with ReiT using fewer than 10% of samples significantly outperform full-shot SIM.The <10% setting uses 65,536 training samples, while full-shot SIM uses the whole training set.
  • Sequence-length analysis: ReLLa-equipped LLMs perform better with K=30, whereas zero-shot LLMs peak at K=5 or K=15.This pattern supports ReLLa’s ability to address user-sequence incomprehension at longer sequence lengths.
  • Complexity and deployment: ReLLa has slower inference than traditional recommendation models and is therefore currently suited to applications with high latency tolerance.The authors attribute this computational limitation to the large-scale property of LLMs.

D.3 Ablation on PCA & Distance Metric

The ablation study examines PCA dimensionality and distance metrics for SUBR using Vicuna-13B on MovieLens-1M. PCA dimensionality 512 is selected as a balance between performance and storage or computing cost, while cosine distance is used by default.

  • PCA dimensionality: PCA dimensionality is evaluated for ReLLa in zero-shot and few-shot settings on MovieLens-1M.The few-shot condition uses 8,192 samples, corresponding to <1% of training data.
  • PCA dimensionality: PCA dimensionality 512 generally achieves the best performance across the evaluated settings.The authors use Vicuna-13B as ReLLa’s backbone for this ablation.
  • PCA dimensionality: Smaller PCA dimensions perform worse because dimensionality reduction loses semantic information, while dimensions above 512 increase storage and computing costs.The authors therefore consider 512 a reasonable performance-cost balance.
  • Distance metric: SUBR compares cosine, L2, and L1 distance metrics in zero-shot and few-shot settings on MovieLens-1M.Cosine distance is selected as ReLLa’s default metric because it is widely used for textual similarity in NLP.

D.3.2 Impact of Distance Metric.

The analysis connects sequence heterogeneity with LLM comprehension and examines how SUBR changes that heterogeneity. Retrieval produces more homogeneous behavior sequences, which is associated with better handling of longer sequences.

  • Distance metric: Cosine similarity emphasizes angular difference and normalizes vector magnitude, while L1 and L2 distances can suffer from the curse of dimensionality.These properties motivate comparing the three metrics for semantic relevance.
  • Sequence heterogeneity: Sequence heterogeneity is defined as the diversity of user behaviors, such as the number of unique movie genres in a sequence.For example, [Fiction, Comedy, Comedy, Family] has heterogeneity score 3.
  • Sequence heterogeneity: Heterogeneity scores increase as sequence length K grows, and LLM performance may decline once heterogeneity exceeds a threshold.This is linked to the observed zero-shot performance peak around K=15.
  • Retrieval effect: SUBR substantially lowers heterogeneity scores compared with top-recent behavior sequences by retrieving top-relevant behaviors.The authors use this reduction to explain why retrieval can improve comprehension of longer behavior sequences.
Loading 2308.11131v6…