PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation

Qidong Liu, Xian Wu, Yejing Wang, Zijian Zhang, Feng Tian, Yefeng Zheng, Xiangyu Zhao

arXiv:2405.20646v2cs.IRcs.CL

TL;DR

Long-tail users and items challenge sequential recommendation, and existing approaches face seesaw, noise, or LLM-integration concerns. LLM-ESR enhances recommendation with cached LLM-derived embeddings, combining semantic and collaborative signals for items and retrieval-augmented self-distillation for users. Experiments across three real-world datasets and three backbone models support its effectiveness and flexibility.

  • Problem

    Long-tail users and items challenge sequential recommendation, while LLM-based approaches can incur inference costs or fail to preserve semantic information.

    Source: §1 · p. 2

  • Method

    LLM-ESR uses cached LLM-derived embeddings, dual-view modeling for items, and retrieval-augmented self-distillation for users.

    Source: §1 · p. 3, §3.2 · pp. 4–5

  • Results

    Experiments on three real-world datasets with three backbone SRS models validate LLM-ESR's effectiveness and flexibility.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The framework combines LLM semantics with collaborative signals to augment long-tail items and uses self-distillation to alleviate the long-tail user challenge.

    Source: §6 · p. 10

    Only OpenAI API embeddings are validated, and searching for the best self-distillation weight and number of retrieved users is time-consuming.

    Source: D Limitation · p. 20

Abstract

from arXiv · show

Sequential recommender systems (SRS) aim to predict users' subsequent choices based on their historical interactions and have found applications in diverse fields such as e-commerce and social media. However, in real-world systems, most users interact with only a handful of items, while the majority of items are seldom consumed. These two issues, known as the long-tail user and long-tail item challenges, often pose difficulties for existing SRS. These challenges can adversely affect user experience and seller benefits, making them crucial to address. Though a few works have addressed the challenges, they still struggle with the seesaw or noisy issues due to the intrinsic scarcity of interactions. The advancements in large language models (LLMs) present a promising solution to these problems from a semantic perspective. As one of the pioneers in this field, we propose the Large Language Models Enhancement framework for Sequential Recommendation (LLM-ESR). This framework utilizes semantic embeddings derived from LLMs to enhance SRS without adding extra inference load from LLMs. To address the long-tail item challenge, we design a dual-view modeling framework that combines semantics from LLMs and collaborative signals from conventional SRS. For the long-tail user challenge, we propose a retrieval augmented self-distillation method to enhance user preference representation using more informative interactions from similar users. To verify the effectiveness and versatility of our proposed enhancement framework, we conduct extensive experiments on three real-world datasets using three popular SRS models. The results show that our method surpasses existing baselines consistently, and benefits long-tail users and items especially. The implementation code is available at https://github.com/Applied-Machine-Learning-Lab/LLM-ESR.

1 Introduction

Long-tail users and items challenge sequential recommendation, while existing approaches can suffer from noisy or seesaw effects and costly or semantics-eroding LLM integration. LLM-ESR uses cached LLM-derived embeddings to enhance both sides of recommendation, and is evaluated across three datasets and three backbone models.

Long-tail challenges

  • Over 80% of Beauty users have fewer than 10 interactions, and 71.4% of items have no more than 30; SASRec performs worse for these long-tail groups.Figure 1 reports the preliminary SASRec experiments on Beauty.Source: §1 · p. 2, Figure 1 · p. 2

Challenges in existing approaches

  • Prior long-tail methods risk seesaw effects for items or noisy pseudo-items for users, while LLM integration can impose inference costs or erode semantic relations.Existing LLM-based embedding approaches also focus on items and may neglect user-side semantics.Source: §1 · p. 2

LLM-ESR

  • LLM-ESR derives user and item semantic embeddings that can be cached, avoiding additional LLM inference burden.The framework is presented as model-agnostic and adaptable to sequential recommendation models.Source: §1 · pp. 2–3
  • For long-tail items, LLM-ESR combines LLM-derived semantic information with collaborative signals in a dual-view modeling framework.The LLM-derived embeddings are frozen to retain their semantic information.Source: §1 · p. 2
  • For long-tail users, retrieval-augmented self-distillation uses similar users to enhance the SRS sequence encoder.User similarity is measured using LLM-derived user representations.Source: §1 · p. 2

Evaluation

  • The framework is evaluated on three real-world datasets with three backbone SRS models to assess its effectiveness and flexibility.The introduction reports experiments but gives no benchmark comparison or numerical result here.Source: §1 · p. 3

2 Problem Definition

Sequential recommendation predicts a user's next item from their chronologically ordered interaction history. This section defines head and tail users and items by ranking interaction counts and item popularity.

Problem definition

  • The task is to predict the next item a user may interact with from their historical interaction sequence.The sequence orders items by timeline.Source: §2 · p. 3

Head and tail groups

  • Users are ranked by sequence length and items by popularity; the top 20% of each are head, and the remainder are tail.The paper's objective is to improve recommendation performance for tail users and items.Source: §2 · p. 3

3 LLM-ESR

LLM-ESR combines cached LLM-derived semantics with collaborative recommendation signals, using distinct modules to address long-tail items and users. It fuses two preference views and distills information from similar users during training, while omitting the auxiliary distillation module at inference.

3.1 Overview

  • LLM-ESR encodes user histories and item attributes into semantic embeddings, then applies dual-view modeling and retrieval-augmented self-distillation to address item and user tails.Source: §3.1 · p. 4

3.2 Dual-view Modeling

  • Dual-view modeling combines frozen LLM item semantics with trainable collaborative embeddings, using cross-attention for sequence-level fusion and concatenated views for recommendation.The views share a sequence encoder, and PCA-reduced semantic embeddings initialize the collaborative layer.Source: §3.2 · pp. 4–5

3.3 Retrieval Augmented Self-Distillation

  • Retrieval-augmented self-distillation retrieves similar users from semantic user embeddings and mean-pools their dual-view representations to guide the target user's representation.The teacher representation supplies a guidance signal rather than being optimized.Source: §3.3 · pp. 5–6

3.4 Train and Inference

  • Training updates collaborative embeddings, the adapter, cross-attention, and the sequence encoder while freezing semantic bases; inference skips distillation and LLM calls using cached embeddings.Source: §3.4 · p. 6

4 Experiment

Across three datasets and three backbone models, LLM-ESR outperforms the competing baselines, with gains also reported for long-tail users and items. Ablations and group analyses examine its components and performance across user and item groups.

4.1 Experimental Settings

  • The experiments use three real-world datasets and compare LLM-ESR and competing methods with GRU4Rec, Bert4Rec, and SASRec.Evaluation uses H@10 and N@10; results are averages over three test runs with random seeds 42, 43, and 44.Source: §4.1 · pp. 6–7

4.2 Overall Performance

  • LLM-ESR outperforms all competing baselines across user and item groups and SRS models, with overall results reported across three datasets.Table 1 reports overall, tail, and head performance; the experiments use Yelp, Amazon Fashion, and Amazon Beauty with GRU4Rec, Bert4Rec, and SASRec.Source: §4.1 · p. 6, Table 1 · p. 7, §4.2 · p. 7
  • LLM-ESR ranks first for both tail and head items, and LLM-ESR, LLMInit, and MELT markedly augment the tail-user group.CITIES sometimes benefits tail items while harming popular items; LLMInit introduces semantics but does not use LLMs on the user side.Source: §4.2 · p. 8
  • LLM-ESR achieves the largest performance improvements across all three backbone SRS models, while other baselines vary in which backbone they benefit.CITIES and MELT tend to perform better with GRU4Rec; LLMInit is more beneficial with Bert4Rec and SASRec.Source: §4.2 · p. 8

4.3 Ablation Study

  • Removing the collaborative view harms head-group performance, removing the semantic view harms tail items, and removing self-distillation lowers long-tail-user performance.Removing sharing or cross-attention also decreases performance; one-layer adapters and random initialization underperform the full method.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Hyper-parameter Analysis

  • Performance rises and then falls as self-distillation weight α decreases from 1 to 0.01; the best number of retrieved users N is 10.Large α can interfere with ranking-loss convergence, while too-small α reduces performance; too-large N may reduce retrieved-user relatedness.Source: §4.4 · p. 9, Figure 3 · p. 9

4.5 Group Analysis

  • Across five user and item groups, LLM-based frameworks improve every group, and LLM-ESR gains more than LLMInit in long-tail groups.Examples include user groups 1–4 and item groups 1–9.Source: §4.5 · p. 9, Figure 4 · p. 9

5 Related Works

Prior sequential recommendation work developed RNN, CNN, self-attention, and MLP architectures, while long-tail approaches largely focused on items or users separately. LLM-based recommendation includes systems that perform recommendation directly and systems that use LLMs to enhance recommenders.

5.1 Sequential Recommendation

  • Sequential recommendation architectures progressed from RNNs and CNNs to self-attention and pure MLP models, while long-tail methods often address users or items separately.MELT addresses both long-tail challenges but remains collaborative; LLM-ESR addresses both from a semantic perspective.Source: §5.1 · pp. 9–10

5.2 LLMs for Recommendation

  • LLM recommendation research divides into direct recommendation by LLMs and LLM-based enhancement of recommenders, which avoids LLM use during recommendation.LLM-ESR belongs to the enhancement line and further addresses defects in semantic information.Source: §5.2 · p. 10

6 Conclusion

LLM-ESR targets long-tail user and item challenges through cached LLM semantics, dual-view modeling, and retrieval-augmented self-distillation; comprehensive experiments verify its effectiveness and flexibility.

6 Conclusion

  • LLM-ESR combines cached LLM-derived semantic embeddings with collaborative signals to address long-tail users and items.The framework is designed to enhance sequential recommendation.Source: §6 · p. 10
  • Dual-view modeling augments long-tail items, while retrieval-augmented self-distillation addresses the long-tail user challenge.Source: §6 · p. 10
  • Comprehensive experiments verify the effectiveness and flexibility of LLM-ESR.Source: §6 · p. 10

A Supplement to Method

The supplement details how LLM-ESR constructs prompts and trains and runs its recommendation framework.

A.1 Prompt Design

  • Item prompts organize dataset-specific attributes and descriptions, while the shared user prompt represents interaction history with item titles to limit input length.Source: §A.1 · p. 15
  • The user prompt asks the LLM to infer preferences from the titles of items the user has visited.Source: §A.1 · p. 15

A.2 Train and Inference Process

  • Training combines dual-view ranking loss with retrieval-augmented self-distillation loss, while inference uses dual-view modeling to produce recommendations.Semantic embeddings are obtained from prompts; inference loads these embeddings and trained parameters.Source: §A.2 · pp. 15–16

B Experimental Settings

This section introduces the paper’s experimental settings.

B Experimental Settings

B.1 Dataset and Preprocessing

The experiments use preprocessed Yelp, Fashion, and Beauty datasets, test popular sequential-recommendation backbones, and compare traditional and LLM-based baselines.

B.1 Dataset and Preprocessing

  • Experiments use Yelp check-in histories and Amazon Fashion and Beauty review data, with Yelp point-of-interest attributes also used.Source: §B.1 · p. 16
  • Preprocessing follows SASRec, treats reviews or ratings as implicit interactions, drops users with fewer than three items, and holds out the last two items for test and validation.Source: §B.1 · p. 16

B.2 Backbone and Baseline

  • Three popular sequential-recommendation models are tested; their main differences are the sequence encoder and ranking loss.Source: §B.2 · p. 17
  • The described backbones include GRU4Rec, Bert4Rec, and SASRec, which differ in their sequence encoders and training objectives.Source: §B.2 · p. 17
  • Traditional baselines enhance long-tail users or items through collaborative signals without introducing semantics; CITIES targets long-tail item embeddings and MELT targets both tails.Source: §B.2 · p. 17
  • LLM-based baselines combine LLM-derived semantics with recommendation models; comparisons include adapted RLMRec and LLMInit approaches that initialize item embeddings.Source: §B.2 · p. 17

B.3 Implementation Details

The implementation fixes shared training settings and selects the same best retrieval and distillation hyperparameters across all three datasets. The appendix also identifies Yelp ablation tables and introduces further experimental analyses.

B.3 Implementation Details

  • N=10 and α=0.1 are the best hyperparameter choices on all three datasets.The searched values were N ∈ {2, 6, 10, 14, 18} and α ∈ {1, 0.5, 0.1, 0.05, 0.01}.Source: §B.3 · p. 17
  • Experiments use an Intel Xeon Gold 6133 platform with Tesla V100 32G GPUs, batch size 128, learning rate 0.001, and Adam.The implementation uses Python 3.9.5 and PyTorch 1.12.0; early stopping uses 20-epoch patience.Source: §B.3 · p. 17
  • Tables 4 and 5 report Yelp ablations with Bert4Rec and GRU4Rec backbones, respectively.Boldface marks the highest score and underlining the next best result in each table.Source: Table 4 · p. 18, Table 5 · p. 18

C More Experimental Results

C.1 Ablation Study

Ablations support the dual-view design and retrieval-augmented self-distillation, while limited-text experiments favor freezing semantic embeddings. The visualization passage identifies a t-SNE comparison of item embeddings by popularity on Yelp.

C.1 Ablation Study

  • Removing either view hurts overall performance; collaborative information matters more for head items, while semantic information matters more for long-tail items.The relative view benefit differs by backbone: Bert4Rec benefits more from collaboration, whereas GRU4Rec gains more from semantics.Source: §C.1 · p. 18
  • Without self-distillation, tail-user performance consistently drops; separate encoders are slightly worse than a shared encoder, and removing cross-attention lowers overall performance.The shared encoder is also more parameter-efficient, while cross-attention provides sequence-level fusion.Source: §C.1 · p. 18
  • Freezing semantic embeddings improves performance with both full and cropped Yelp prompts and significantly benefits long-tail items.The prompts average 38.38 words when complete and 20.33 words when cropped; unfrozen embeddings show similar results across the two prompt settings.Source: §C.1 · pp. 18–19, Table 6 · p. 19

C.2 Visualization

  • Figure 5 visualizes Yelp item embeddings with t-SNE, coloring items by popularity group; CITIES, MELT, and LLM-ESR use SASRec.The caption also identifies LLM embeddings as derived from textual item prompts.Source: Figure 5 · p. 19

C.3 Group Analysis

The group analysis divides users and items into five groups and compares recommendation performance across those groups. LLM-ESR’s reported advantages are strongest among more long-tailed users and the 1–9 item group.

C.3 Group Analysis

  • LLM-ESR benefits all reported user groups and outperforms LLMInit, with a larger advantage for users in the 1–4 and 5–9 interaction groups.All methods perform worse for users with fewer interactions; MELT helps Bert4Rec across groups but harms several groups with GRU4Rec.Source: §C.3 · p. 19, Figure 6 · p. 20, Figure 7 · p. 20
  • Users and items are each divided into five groups for the analysis.The accompanying labels distinguish user groups from item groups.Source: §C.3 · p. 19
  • LLM-ESR leads in the 1–9 item group; methods that improve long-tail item accuracy show slight drops for popular items, yet larger tail gains accompany overall improvement.Source: §C.3 · p. 20, Figure 6 · p. 20, Figure 7 · p. 20

D Limitation

The authors identify two limitations: hyperparameter search takes time, and experiments validate only OpenAI API embeddings. They note that newer embedding models may perform better.

D Limitation

  • Searching for the self-distillation loss weight α and retrieved-user count N is time-consuming.Source: D Limitation · p. 20
  • Experiments validate only embeddings provided by the OpenAI API; newer models may yield better performance.The cited examples of more recent models are [3] [52].Source: D Limitation · p. 20
  • Despite these limitations, experiments across multiple datasets and backbone models consistently validate LLM-ESR’s effectiveness.Source: D Limitation · p. 20

Related papers

Loading 2405.20646v2…