PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMEmb: Large Language Model Can Be a Good Embedding Generator for Sequential Recommendation

Qidong Liu, Xian Wu, Wanyu Wang, Yejing Wang, Yuanshao Zhu, Xiangyu Zhao, Feng Tian, Yefeng Zheng

arXiv:2409.19925v2cs.IRcs.CL

TL;DR

Sequential recommender systems struggle with low-popularity items, whose learned embeddings can be sparse and distant from those of popular items. LLMEmb adapts LLM-generated item embeddings for recommendation and integrates collaborative signals; across three datasets, it outperforms competitors, especially on long-tail items.

Abstract

from arXiv · show

Sequential Recommender Systems (SRS), which model a user's interaction history to predict the next item of interest, are widely used in various applications. However, existing SRS often struggle with low-popularity items, a challenge known as the long-tail problem. This issue leads to reduced serendipity for users and diminished profits for sellers, ultimately harming the overall system. Large Language Model (LLM) has the ability to capture semantic relationships between items, independent of their popularity, making it a promising solution to this problem. In this paper, we introduce LLMEmb, a novel method leveraging LLM to generate item embeddings that enhance SRS performance. To bridge the gap between general-purpose LLM and the recommendation domain, we propose a Supervised Contrastive Fine-Tuning (SCFT) approach. This approach includes attribute-level data augmentation and a tailored contrastive loss to make LLM more recommendation-friendly. Additionally, we emphasize the importance of integrating collaborative signals into LLM-generated embeddings, for which we propose Recommendation Adaptation Training (RAT). This further refines the embeddings for optimal use in SRS. The LLMEmb-derived embeddings can be seamlessly integrated with any SRS models, underscoring the practical value. Comprehensive experiments conducted on three real-world datasets demonstrate that LLMEmb significantly outperforms existing methods across multiple SRS models. The code for our method is released online https://github.com/Applied-Machine-Learning-Lab/LLMEmb.

Introduction

Sequential recommender systems struggle to recommend low-popularity items, while LLM embeddings offer a way to represent item semantics more uniformly. LLMEmb adapts LLMs for recommendation through contrastive fine-tuning and collaborative adaptation.

Introduction

  • Low-popularity items are hard to recommend: SASRec embeddings for these items are sparse and distant from popular-item embeddings, while LLaMA embeddings are more uniformly distributed.The paper links long-tail recommendation difficulty to reduced user serendipity and seller profits.Source: Introduction · pp. 1–2
  • Existing LLM-enhanced methods face a semantic gap and risk semantic loss through dimensionality reduction and continued training with SRS models.Some fine-tune for language modeling or category prediction while overlooking item attributes important for distinguishing items.Source: Introduction · p. 2
  • LLMEmb uses attribute-level data augmentation and supervised contrastive fine-tuning, then Recommendation Adaptation Training to inject collaborative signals while keeping LLM embeddings frozen.A trainable adapter transforms dimensions, and generated embeddings can be cached in SRS embedding layers without additional inference-time computational burden.Source: Introduction · p. 2
  • Experiments on three public datasets report superior performance for LLMEmb with three popular SRS models, with particular benefits for long-tail items.The paper presents this as a method to help alleviate the long-tail problem in sequential recommendation.Source: Introduction · p. 2

Preliminary

The preliminary section defines next-item recommendation from a user's time-ordered interaction sequence and describes the common embedding-sequence framework. The paper focuses on replacing the framework's usually scratch-trained item embedding function with LLM-generated embeddings.

Preliminary

  • A user's input sequence Q(u) consists of item interactions ordered by timeline, and the task is to recommend the next item.The sequence length is denoted Nu; the paper later omits the user-specific superscript for simplicity.Source: Preliminary · p. 2
  • Most SRS models first map item identities to dense embeddings, then use a sequence model to derive a user representation and calculate item recommendation probabilities.The user and item representations are combined by inner product, and the resulting probability vector is optimized with a loss such as binary cross-entropy.Source: Preliminary · p. 2
  • Because SRS embedding functions are often randomly initialized and trained from scratch, this paper focuses on using LLMs to generate embeddings compatible with most SRS models.The sequence-modeling backbone remains the component that models user interaction histories.Source: Preliminary · p. 2

Method

LLMEmb generates item embeddings for sequential recommendation through two training stages: SCFT makes LLM embeddings recommendation-friendly, and RAT adapts them for SRS models. At inference, the precomputed embeddings replace the model’s original embedding layer.

Overview

  • LLMEmb’s overview comprises an LLM and adapter: SCFT trains the LLM, then RAT trains the adapter to produce final item embeddings for the SRS backbone.During inference, the embeddings are precomputed and substitute for the SRS model’s original embedding layer.Source: Overview · p. 3

Supervised Contrastive Fine-Tuning (SCFT)

  • SCFT augments each item prompt into two copies by randomly dropping attributes, then fine-tunes the LLM with contrastive learning to distinguish items and produce recommendation-friendly semantic embeddings.The two copies form positive pairs; distinct items are treated as negative samples.Source: Supervised Contrastive Fine-Tuning… · pp. 3–4

Recommendation Adaptation Training (RAT)

  • RAT uses an adapter to reduce dimension mismatch, trains it with the SRS backbone to inject collaborative signals, and aligns final embeddings with pretrained collaborative embeddings.The LLM embeddings are frozen during adaptation to preserve their semantic relationships; alignment is intended to help optimization by learning coarse collaborative relationships.Source: Recommendation Adaptation Training (RAT) · pp. 4–5

Training and Inference

  • LoRA is used to reduce computational resources during SCFT, and cached final embeddings add no inference computational burden compared with traditional SRS models.The trained backbone handles sequence modeling while precomputed item embeddings serve as the embedding component.Source: Training and Inference · p. 5

Experiment

Experiments evaluate LLMEmb across three datasets and SRS backbones, finding it outperforms competing methods, particularly for long-tail items. Ablations and analyses examine its training components, hyperparameters, popularity-group performance, and embedding distributions.

Experimental Settings

  • Experiments use Yelp, Amazon Beauty, and Amazon Fashion, testing LLMEmb with GRU4Rec, Bert4Rec, and SASRec.The evaluation reports N@10 and H@10, averaging three runs with different random seeds.Source: Experimental Settings · p. 5

Overall Performance (RQ1)

  • LLMEmb outperforms all competitors across three datasets, with especially large gains for tail items, and consistently enhances three SRS models.The tail group comprises items in the last 80% by popularity rank.Source: Overall Performance (RQ1) · pp. 5–6, Table 1 · p. 6

Ablation Study (RQ2)

  • Removing SCFT lowers overall and tail performance; RAT variants show that the adapter and alignment matter, while not freezing LLM embeddings severely harms performance.The ablations use the Yelp dataset and SASRec backbone.Source: Ablation Study (RQ2) · p. 6, Table 2 · p. 6

Hyper-parameter Analysis (RQ3)

  • On Yelp with SASRec, overall HR@10 rises then falls as γ increases from 0.5 to 8; increasing α from 1e−3 to 1e−2 improves performance, but larger α lowers it.The authors attribute the latter decline to a convergence dilemma from stronger contrastive loss and also identify adapter dimension dm as important.Source: Hyper-parameter Analysis (RQ3) · pp. 6–7, Figure 4 · p. 6

Group Analysis (RQ4)

  • Across five popularity groups, LLMEmb improves long-tail items more than LLM2X and SAID, especially groups 1–4, but slightly underperforms LLM2X for popular items such as group 40+.The authors describe this as a seesaw between popular and long-tail items.Source: Group Analysis (RQ4) · p. 7, Figure 5 · p. 7

Visualization (RQ5)

  • t-SNE visualizations show LLMEmb embeddings distributed more uniformly, while SAID embeddings remain clustered by item popularity despite becoming more even.The authors associate SAID’s remaining popularity clustering with semantic loss.Source: Visualization (RQ5) · p. 7

Related Works

Prior sequential recommendation work emphasizes model architectures and training losses for capturing user preferences, while often overlooking skewed item embeddings. LLM-based recommendation includes both direct generation and embedding-based approaches, with resource and alignment challenges noted for existing methods.

Related Works

  • Prior SRS work improves preference and dynamics modeling through architecture and loss designs, while often overlooking skewed item-embedding distributions.Source: Related Works · p. 7
  • LLM recommendation includes direct item-name generation and adapter-based integration of collaborative signals, but direct LLM use is resource-consuming for real-time recommendation.Source: Related Works · p. 7
  • LLM2X [Harte et al. 2023], SAID, and TSLRec [Liu et al. 2024a] use LLM embeddings to enhance SRS but face semantic gap and loss, leading to suboptimal performance.Source: Related Works · p. 7

Conclusion

LLMEmb generates sequential-recommendation item embeddings using supervised contrastive fine-tuning and recommendation adaptation training. Experiments on three real-world datasets verify its effectiveness.

Conclusion

  • LLMEmb is an LLM-based generator that derives item embeddings for sequential recommendation.Source: Conclusion · p. 7
  • Supervised contrastive fine-tuning equips the LLM to identify items for recommendation tasks.Source: Conclusion · p. 7
  • Recommendation adaptation training updates a trainable adapter to avoid semantic loss and inject collaborative signals into LLM embeddings.Source: Conclusion · p. 7
  • Experiments on three real-world datasets verify LLMEmb's effectiveness.Source: Conclusion · p. 7

Related papers

Loading 2409.19925v2…