Harnessing Large Language Models for Text-Rich Sequential Recommendation

1 INTRODUCTION

• Rich item text lengthens user histories, creating input-length, computational-cost, and preference-shift challenges for LLM recommenders.The introduction cites 1,024 tokens as an example input limit and notes Transformer complexity of O(n^2).Source: §1 · p. 1
• LLM-TRSR segments behavior histories, summarizes their blocks with a frozen LLM, and fine-tunes an LLM recommender on summaries, recent interactions, and candidate items.The recommender is trained with SFT, with LoRA used for parameter-efficient fine-tuning.Source: §1 · p. 2
• Hierarchical summarization progressively combines block summaries, while recurrent summarization updates a preference summary as each subsequent behavior block arrives.Both approaches are inspired respectively by CNN and RNN architectures.Source: §1 · p. 2
• LLM-TRSR was reported effective on Amazon-M2 and MIND, public datasets spanning e-commerce product and news recommendation.The supplied introduction reports effectiveness without giving comparative benchmark values.Source: §1 · p. 2
• The authors describe LLM-TRSR as the first LLM approach to text-rich sequential recommendation and validate it on two open-source datasets from distinct domains.The stated contribution includes the two preference-summary paradigms.Source: §1 · p. 2

2.1 Sequential Recommendation

• Prior sequential recommenders model behavior with sequence architectures, while text-rich approaches add item-text encoders; the authors say LLM use in this setting remained underexplored.Examples include GRU4Rec, NARM, BERT4Rec, LSTUR, and TempRec.Source: §2.1 · p. 2

2.2 Large Language Models

• LLMs are described as large neural language models trained on untagged text, with discriminative and generative types; the paper selects Llama models for summarization and recommendation.The related-work discussion notes that input-length extensions do not resolve computational overhead and performance degradation.Source: §2.2 · pp. 2–3

2.3 LLM for Recommendation

• LLM recommendation work includes discriminative and generative models and three paradigms; prior LLM sequential recommendation studies had not considered text-rich history challenges.The three paradigms are LLM Embeddings + RS, LLM Tokens + RS, and LLM as RS.Source: §2.3 · p. 3

3 PROBLEM FORMULATION

• A user's behavior history is an ordered sequence of interacted items, and each item has multiple attributes that can be represented as text.The sequence may include items the user clicked, bought, or read.Source: §3 · p. 3
• Text-Rich Sequential Recommendation estimates the click probability of a candidate item for a user given that user's historical behavior sequence.The formulation maps the candidate item to a real-valued prediction.Source: §3 · p. 3

4.1.1 History Sequence Construction and Segmentation.

• Extended user histories are segmented into manageable blocks and summarized individually to capture preferences within specific time frames.The segmentation addresses length and computational challenges posed by directly processing extensive text.Source: §4.1 · p. 3, §4.1.1 · p. 4

4.1.3 Hierarchical Summarization.

• Hierarchical summarization combines block summaries into increasingly abstract summaries of overall user preferences.The number of summarization layers can be extended as in a CNN; the example uses two layers for a ten-item sequence.Source: §4.1.2 · p. 4, §4.1.3 · p. 4

4.2 Recurrent LLM-based User Preference Summarization

• Recurrent summarization updates the preference summary by combining the preceding summary with each subsequent behavior block.The process begins with a summary of the first block and repeats until all blocks have been processed.Source: §4.2 · p. 4

4.2.1 First Block Summarization.

• The first block in recurrent summarization is summarized using the approach employed for hierarchical summarization.The paper illustrates this first-block step with a user from the MIND dataset.Source: §4.2 · p. 4
• The recurrent prompt is designed to capture long-term interests while updating short-term inclinations.The authors report that the generated summary captures both aspects.Source: §4.2.1 · p. 4

4.3 LLM-based Recommendation

• The recommender prompt combines a preference summary, recent behavior, candidate-item description, recommendation instruction, and final answer.The preference summary represents long-term interests, recent behavior indicates short-term interests, and the answer is whether the user interacted with the candidate.Source: §4.3 · pp. 4–5
• The recommendation model is trained with SFT, and LoRA reduces the number of trainable parameters during fine-tuning.The training objective uses next-token prediction over the prompt text.Source: §4.3 · p. 5

• The experiments section introduces the datasets, baseline methods, evaluation metrics, and experimental results.Source: §5 · p. 5

5.1 Dataset Description

• The experiments use Amazon-M2 for product recommendation and MIND for news recommendation.Source: §5.1 · p. 5
• Records are restricted to 10–25 interactions, with negative sampling at 1:1 for training and 1:20 for validation and test.Source: §5.1 · pp. 5–6

5.2.1 Baseline Methods and Evaluation.

• The baselines include non-sequential methods NCF and DIN, sequential methods, and the LLM-based instruction-tuned method TALLRec.For the listed baselines, the authors use pre-trained BERT for text embedding instead of a pure ID-based paradigm.Source: §5.2.1 · pp. 6–7
• Models are evaluated with Recall@K and MRR@K at K values of 3, 5, and 10.Source: §5.2.1 · p. 7

5.2 Experimental Settings

• The experiments use Llama-30b-instruct as summarizer and Llama-2-7b as recommender, with 8-bit and BF16 settings, respectively.The setup also specifies LoRA rank 8, AdamW learning rate 1e-4, batch size 1, 64 gradient-accumulation steps, 8 epochs, ZeRO stage 2, maximum length 2048, and 5 items per block.Source: §5.2.1 · p. 7

• LLM-TRSR surpasses all baseline methods across evaluation metrics and datasets.Source: §5.3 · p. 7
• LLM-based recommendation approaches consistently outperform traditional methods.Source: §5.3 · p. 7
• LLM-TRSR-Recurrent outperforms LLM-TRSR-Hierarchical on Amazon-M2, while the reverse holds on MIND.Source: §5.3 · p. 7

• Performance initially improves and then declines as historical items increase, with three items yielding the optimum.Source: §5.4 · p. 7
• Using zero historical items still achieves reasonably good performance based solely on user preference summaries.Source: §5.4 · p. 8

5.5 Discussion on Parameter Size

• Larger recommender models generally achieve better performance, but demand greater computational resources.The paper compares the Llama-2-7b backbone with Pythia-1.4b and Pythia-2.8b.Source: §5.5 · p. 8

5.5.1 Discussion on Parameter Size of Recommender.

• The recommender comparison uses Llama-2-7b as its backbone and evaluates smaller Pythia-1.4b and Pythia-2.8b models.Source: §5.5 · p. 8

5.5.2 Discussion on Parameter Size of Summarizer.

• Llama-30b-instruct significantly surpasses Llama-2-13b as a summarizer; the smaller model produces disorganized, harder-to-comprehend summaries.The comparison sets the historical-item count in the recommendation prompt to zero.Source: §5.5.2 · p. 8, §5.5.1 · p. 8

• LLM-TRSR segments user behavior sequences and uses an LLM-based summarizer to encapsulate user preferences.Source: §6 · p. 8
• The framework introduces hierarchical and recurrent preference-summarization paradigms, then fine-tunes an LLM recommender with SFT and LoRA.Source: §6 · p. 8
• Experiments on two public datasets are reported as evidence of the approach's efficacy.Source: §6 · p. 8