Source-linked AI summary
Leveraging Large Language Models for Sequential Recommendation
Jesse Harte, Wouter Zorgdrager, Panos Louridas, Asterios Katsifodimos, Dietmar Jannach, Marios Fragkoulis
TL;DR
Sequential recommendation seeks to predict users’ next interactions, but existing approaches have limited access to the semantic information encoded by LLMs. This paper evaluates three LLM-based strategies, finding that LLM-initialized BERT4Rec achieves the strongest results, while embedding similarity can also be competitive.
Problem
The paper addresses limited use of LLMs’ semantic information in sequential recommendation, where models predict a user’s next interaction from past interactions.
Method
The paper devises and evaluates three approaches: embedding-based similarity recommendations, fine-tuned prompt-based recommendations, and LLM-initialized sequential models.
Results
LLM2BERT4Rec obtains the highest NDCG@20 on both datasets, increasing NDCG by more than 20% on Beauty and over 15% on Delivery Hero.
Takeaways & Limitations
LLM-based item embeddings can substantially improve an existing sequential model, while embedding similarity alone can produce compelling recommendations depending on the dataset.
Takeaways & Limitations
Infrequent products in the Delivery Hero dataset may limit LLMSeqSim’s ability to find similar items.
Abstract
from arXiv · showhide
Sequential recommendation problems have received increasing attention in research during the past few years, leading to the inception of a large variety of algorithmic approaches. In this work, we explore how large language models (LLMs), which are nowadays introducing disruptive effects in many AI-based applications, can be used to build or improve sequential recommendation approaches. Specifically, we devise and evaluate three approaches to leverage the power of LLMs in different ways. Our results from experiments on two datasets show that initializing the state-of-the-art sequential recommendation model BERT4Rec with embeddings obtained from an LLM improves NDCG by 15-20% compared to the vanilla BERT4Rec model. Furthermore, we find that a simple approach that leverages LLM embeddings for producing recommendations, can provide competitive performance by highlighting semantically related items. We publicly share the code and data of our experiments to ensure reproducibility.
1 INTRODUCTION
Sequential recommendation predicts a user’s next interaction from an interaction sequence, and recent models increasingly draw on NLP architectures. This paper investigates three ways to leverage LLM semantics for the task, with experiments showing strong gains from LLM-initialized BERT4Rec.
- Sequential recommendation predicts the next user interaction from a sequence of past interactions.Applications include next-purchase, next-track, and next-Point-of-Interest prediction.
- Recent sequential recommendation models include GRU4Rec, SASRec, and BERT4Rec, which respectively adopt recurrent, transformer, and BERT-based architectures.
- LLMs provide semantically rich information, but limited research had leveraged that information for sequential recommendation.
- The paper devises three approaches: LLMSeqSim aggregates item embeddings for similarity-based recommendations, LLMSeqPrompt fine-tunes an LLM on prompt-completion pairs, and LLM2BERT4Rec initializes sequential models with LLM item embeddings.
- 15-20% NDCG improvement over vanilla BERT4Rec makes LLM-initialized BERT4Rec the best-performing model in the experiments.
- LLM embedding similarity alone can provide competitive recommendations by identifying semantically related catalog items.
2 BACKGROUND & RELATED WORK
LLMs have entered recommendation research through embeddings and direct recommendation generation, extending a longstanding connection between NLP architectures and sequential recommendation. This paper complements prior work by evaluating three alternative ways to use LLMs in sequential recommendation.
- LLMs are used in recommendation systems both to initialize existing models with embeddings and to generate recommendations from encoded knowledge.
- LLM-based recommendation can operate zero-shot, few-shot, or after fine-tuning on task examples.
- Prior sequential-recommendation work using LLM embeddings reports mixed results, including improvements for some datasets and models.
- VQ-Rec targets cross-domain recommendation with a new item representation scheme, whereas related work evaluates modality-based representations against item identifiers.
- This paper proposes and evaluates three alternative ways of leveraging LLMs for sequential recommendation.
- LLM-based initialization can outperform existing state-of-the-art sequential models, while embedding similarity can produce compelling recommendations depending on the dataset.
3 THREE LLM-BASED APPROACHES FOR SEQUENTIAL RECOMMENDATIONS
The paper develops three LLM-based recommendation strategies: embedding similarity, fine-tuned prompt completion, and LLM-initialized BERT4Rec. These approaches respectively exploit item semantics directly, generate next-item outputs, or transfer semantic representations into a sequential model.
- LLMSeqSim: LLMSeqSim aggregates embeddings of session items and recommends catalog products with the highest similarity to the resulting session embedding.The method tests average, position-weighted, and last-item aggregation, plus cosine, Euclidean, and dot-product similarity.
- LLMSeqPrompt: LLMSeqPrompt maps duplicate or hallucinated generated products to catalog products using embedding similarity to construct catalog recommendations.Duplicate frequency is used to rank recommendations, and hallucinated products are mapped using the same approach as duplicates.
- LLMSeqPrompt: LLMSeqPrompt fine-tunes an OpenAI ada model on sessions as prompts and the held-out final product as the completion.Test-session prompts are supplied to the fine-tuned model to obtain recommendations.
- LLM2BERT4Rec: LLM2BERT4Rec initializes BERT4Rec item embeddings with LLM embeddings, reducing their dimension with PCA before baseline-style training.PCA maps 1536-dimensional LLM embeddings to the configured BERT4Rec embedding dimension, such as 64.
- LLM2BERT4Rec: BERT4Rec uses item and positional embeddings, transformer encoders, and a masked-item prediction head.
4 EXPERIMENTAL EVALUATION
The evaluation compares sequential recommendation models on Amazon Beauty and Delivery Hero using accuracy and beyond-accuracy metrics. LLM-based initialization substantially improves BERT4Rec, while simpler LLM-based approaches vary by dataset and can favor coverage and novelty.
- Experimental setup: Experiments use Amazon Beauty and a real-world Delivery Hero dataset, with temporally ordered sessions and leave-one-out testing.The last interaction in each test session serves as ground truth.
- Experimental setup: Evaluation reports NDCG, MRR, and HitRate at 10 and 20, alongside catalog coverage, serendipity, and novelty.These metrics combine ranking accuracy with broader recommendation properties.
- Experimental setup: Models include GRU4Rec, SKNN, BERT4Rec, SASRec, three LLM-based approaches, and a popularity baseline.Hyperparameters are systematically tuned for most models using validation-fold NDCG@20.
- Accuracy results: LLM2BERT4Rec achieves the highest NDCG@20 on both datasets, with LLM-based embeddings increasing NDCG by more than 20% on Beauty and over 15% on Delivery Hero.A permutation experiment preserves embedding statistics while removing item semantics.
- Accuracy results: LLMSeqSim is highly competitive on Beauty, leads at NDCG@10, but performs poorly on Delivery Hero except against the popularity baseline.The authors relate this difference to infrequent items and the broader Delivery Hero catalog.
5 CONCLUSIONS
The paper evaluates three ways to leverage LLMs for sequential recommendation and finds LLM-initialized BERT4Rec performs best across two datasets. Several directions remain open, including generalization across domains, other LLMs, hybrid approaches, and richer item information.
- Three approaches leverage LLMs for sequential recommendation problems.
- LLM-initialized BERT4Rec achieves the best performance for both datasets.The evaluation also finds that this initialization substantially improves accuracy.
- Generalization to different recommendation domains and datasets with diverse characteristics remains to be investigated.
- The consistency of performance gains from other LLM architectures and training corpora remains open.
- Combining LLM2BERT4Rec with LLMSeqSim is proposed as a way to pursue accuracy and beyond-accuracy performance.
- The effect of providing category information and other non-name information to the LLM remains unresolved.