PaperlayerView exact revision on arXiv

Source-linked AI summary

A Bi-Step Grounding Paradigm for Large Language Models in Recommendation Systems

Keqin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang, Yancheng Luo, Chong Chen, Fuli Feng, Qi Tian

arXiv:2308.08434v2cs.IR

TL;DR

LLM4Rec methods often evaluate on restricted candidate sets, leaving comprehensive ranking capacity insufficiently examined. BIGRec addresses this gap with bi-step grounding from language space to recommendation space and then actual items. It shows strong few-shot and cross-domain performance, including performance comparable to full-data conventional models in most settings, while incorporating popularity and collaborative information.

  • Problem

    Existing LLM4Rec approaches often evaluate restricted candidate sets rather than the models’ comprehensive ability to rank all items.

  • Method

    BIGRec fine-tunes LLMs to generate meaningful recommendation tokens, then identifies matching actual items using latent representations while incorporating statistical information.

  • Results

    BIGRec trained on 1024 samples considerably outstrips traditional models trained on the same number of samples and performs on par with full-dataset conventional models in most settings.

  • Takeaways & Limitations

    Popularity and collaborative information can be infused into BIGRec, further enhancing its capability and indicating potential for future LLM4Rec research.

  • Takeaways & Limitations

    BIGRec relies heavily on semantic information different from collaborative information, and the paper identifies unresolved issues concerning its initial stage.

Abstract

from arXiv · show

As the focus on Large Language Models (LLMs) in the field of recommendation intensifies, the optimization of LLMs for recommendation purposes (referred to as LLM4Rec) assumes a crucial role in augmenting their effectiveness in providing recommendations. However, existing approaches for LLM4Rec often assess performance using restricted sets of candidates, which may not accurately reflect the models' overall ranking capabilities. In this paper, our objective is to investigate the comprehensive ranking capacity of LLMs and propose a two-step grounding framework known as BIGRec (Bi-step Grounding Paradigm for Recommendation). It initially grounds LLMs to the recommendation space by fine-tuning them to generate meaningful tokens for items and subsequently identifies appropriate actual items that correspond to the generated tokens. By conducting extensive experiments on two datasets, we substantiate the superior performance, capacity for handling few-shot scenarios, and versatility across multiple domains exhibited by BIGRec. Furthermore, we observe that the marginal benefits derived from increasing the quantity of training samples are modest for BIGRec, implying that LLMs possess the limited capability to assimilate statistical information, such as popularity and collaborative filtering, due to their robust semantic priors. These findings also underline the efficacy of integrating diverse statistical information into the LLM4Rec framework, thereby pointing towards a potential avenue for future research. Our code and data are available at https://github.com/SAI990323/Grounding4Rec.

1 INTRODUCTION

Existing LLM4Rec evaluations often use restricted candidate sets, motivating all-rank investigation. BIGRec addresses this with bi-step grounding, showing strong few-shot and multi-domain performance while supporting statistical information.

  • Motivation: Existing LLM4Rec evaluations often use limited candidate sets, leaving models’ overall global ranking ability insufficiently assessed.The paper motivates evaluating recommendation performance across all items rather than only sampled candidates.
  • BIGRec: BIGRec grounds LLMs through language space, recommendation space, and actual item space.The framework first generates meaningful tokens for item descriptions, then matches them to actual items using latent representations.
  • BIGRec: The second grounding step matches generated tokens to actual items while allowing popularity and collaborative information to influence matching.Statistical information can be incorporated by weighting representation distances, including according to item popularity.
  • Results: Experiments on two real-world datasets show that BIGRec largely outperforms traditional recommendation models and existing LLM4Rec methods.The results also demonstrate strong few-shot and multi-domain abilities.
  • Results: BIGRec can outperform most traditional models trained with 100 or even 1,000 times more samples.The paper reports that scaling training samples provides relatively modest benefits for BIGRec compared with traditional models.
  • Statistical information: The authors postulate that strong semantic priors limit LLMs’ assimilation of popularity and collaborative information from training samples.They further validate incorporating these statistical signals into BIGRec, indicating a potential direction for future LLM4Rec research.

2 RELATED WORK

Related work applies LLMs to recommendation through embeddings, instruction tuning, in-context learning, and semantic retrieval, but these approaches face limitations in data, adaptation, or all-item evaluation. BIGRec instead grounds generated outputs to real recommendation items.

  • Grounding in LLM: In-context recommendation can be limited by LLM input size and insufficient recommendation knowledge.The related work contrasts this with domain-specific instruction tuning and customized prompts.
  • LLM-based Recommendation: LLM-based recommendation studies use item embeddings with traditional recommenders or instruction tuning with recommendation data.These approaches seek to improve recommendation capability despite limited recommendation data during LLM pre-training.
  • LLM-based Recommendation: Existing instruction-tuned methods evaluate mainly in CTR or negative-sampling settings rather than ranking all items.This leaves their all-rank recommendation ability unexplored.
  • Sequential Recommendation: Sequential recommendation models include RNN-, CNN-, and attention-based approaches, alongside pretraining, augmentation, debiasing, and robust optimization.Conventional ID-based systems are described as having limited generalization and adaptation to novel scenarios.
  • Sequential Recommendation: Some language-model recommendation methods require substantial pretraining data or rely on word overlap, which does not exploit semantic information.The cited limitations distinguish feature extraction and lexical retrieval from semantic grounding.
  • Grounding in LLM: This paper uses contextual grounding and adds a step that grounds generated outputs to real-world items because recommendations must correspond to existing items.The paper does not prioritize modality grounding because of its large data and resource requirements.

3 BIGREC

BIGRec grounds LLM outputs through recommendation space into actual items using two stages. It matches generated-token representations to item embeddings and can incorporate popularity or collaborative information during grounding.

  • 3.1 Preliminary: The language space contains all sequences an LLM could generate, whereas recommendation space is a task-specific subset containing entities relevant to user preferences.Recommendation-space entities may be actual or imaginary items, so it is not itself sufficient for recommending available products.
  • 3 BIGREC: BIGRec uses two grounding steps: from language space to recommendation space, then from recommendation space to actual item space.The first step restricts LLM outputs to recommendation-related tokens; the second enables recommendations of existing items.
  • 3.2 BIGRec Implementation: Instruction tuning asks the LLM to generate a new item from a user’s past interactions, restricting outputs toward the designated recommendation space.This generative tuning procedure uses recommendation-specific instruction data.
  • 3.2.2 Step 2: Grounding Recommendation Space to Actual ItemsSpace.: BIGRec maps generated outputs to actual items by ranking item embeddings according to their L2 distance from the generated-token representation.The method then modifies this grounding with statistical information such as popularity and collaborative-model prediction scores.
  • 3.2.2 Step 2: Grounding Recommendation Space to Actual ItemsSpace.: Popularity reweighting assigns popular items smaller adjusted L2 distances and therefore higher ranks, with γ controlling popularity’s influence.Collaborative information is injected by substituting a collaborative-model prediction score for the popularity variable.

4 EXPERIMENTS

The experiments evaluate BIGRec against existing methods under limited-data conditions, scaling training data, and injecting popularity or collaborative information. They also examine whether BIGRec can ground recommendations across domains.

  • RQ1: The experiments ask how BIGRec compares with existing methods when trained on 1024 samples.This is the first limited-training-data research question.
  • RQ2: The study compares limited-data BIGRec with traditional recommendation models.This question targets the relative performance of the LLM-based method against conventional recommenders.
  • RQ3: The experiments test whether increasing training data yields substantial performance gains for BIGRec, as it does for traditional models.This evaluates the effect of training-sample scaling.
  • RQ4: The study measures how much popularity and collaborative information enhance BIGRec.These signals are incorporated during the grounding process.

4.1 Experimental Settings

Experiments use MovieLens10M and Amazon Games data with different popularity-bias profiles. The evaluation uses temporally ordered sequential splits, all-ranking metrics, and conventional and LLM-based baselines.

  • Datasets: The Movie dataset contains 10,682 items, 10,000,054 interactions, and 9,301,274 interaction sequences.It is the MovieLens10M benchmark dataset.
  • Datasets: The Game dataset contains 17,408 items, 496,315 interactions, and 149,796 interaction sequences.It is the 5-core subset of an Amazon video-games recommendation dataset.
  • Datasets: Movie has stronger popularity bias, while Game has a more equitable interaction distribution between popular and unpopular items.The datasets were selected to expose different popularity characteristics.
  • Data Splitting: Each dataset is divided into ten timestamp-based periods and split into training, validation, and testing sets at an 8:1:1 ratio.The temporal split ensures test interactions occur after training interactions and avoids testing-phase leakage.
  • Evaluation: HR and NDCG are computed with the all-ranking protocol, where every item not previously interacted with is a candidate.This evaluates ranking over the full set of unseen items rather than a restricted candidate sample.
  • Implementation Details: Because LLM inference is computationally expensive, validation and testing each use 5,000 randomly sampled interactions, and BIGRec uses beam size 4.The beam-size choice follows prior work and is motivated by GPU and inference costs.
  • Compared Methods: Baselines include GRU4Rec, Caser, SASRec, DROS, P5, and GPT4Rec-LLaMA.These cover conventional sequential recommenders and LLM-based recommendation methods.

4.2 Performance Comparison with Limited Training Data (RQ1)

Under limited training data, BIGRec outperforms conventional sequential baselines and most traditional models, while also supporting cross-domain grounding. Its advantage varies with dataset popularity bias, and BM25-based GPT4Rec-LLaMA performs poorly.

  • Limited Training Data: With limited training data, GRU4Rec, Caser, and SASRec perform significantly worse than LLM-based BIGRec.The conventional models depend on ID embeddings that are difficult to learn effectively from limited data.
  • Limited Training Data: BIGRec tends to rank items of interest to users in higher top-ranked positions.This finding is reported in the limited-data performance analysis.
  • Comparison with LLM-Based Methods: GPT4Rec-LLaMA performs poorly compared with BIGRec, which the authors attribute to BM25’s unsuitability for short item-title queries.BM25 is designed for document-level text and is susceptible to low-frequency-word noise in these queries.
  • Popularity Bias: BIGRec’s improvement over conventional models is significantly higher on Game than Movie, possibly because the datasets differ in popularity bias.Conventional methods capture popular-item bias more strongly, whereas BIGRec is less affected by it.
  • Multi-Domain Grounding: BIGRec trained on Movie and Game data performs comparably on each domain to BIGRec trained on that domain alone.This supports simultaneous grounding to different actual item spaces at least across domains.

4.3 Performance Comparison to Baselines Trained with More Data (RQ2)

BIGRec trained on only 1,024 samples generally matches or exceeds conventional recommendation models trained on substantially more data, especially on Game. On Movie, its advantage is concentrated at the top of the recommendation list, while performance is weaker at lower positions.

  • BIGRec performs better than conventional baselines at the top positions on Movie but worse at relative bottom positions.
  • BIGRec consistently outperforms traditional models excluding DROS on the Game dataset.
  • BIGRec trained with only 1,024 samples remains comparable to the optimized traditional method DROS in most cases.
  • BIGRec trained with 1,024 samples achieves comparable performance to conventional models trained with 100 or even 1,000 times more data.This supports BIGRec’s use when recommendation exposure resources are limited.
  • Increasing training-set size and expanding the range of grounded items may further improve BIGRec’s performance.

4.4 Performance of BIGRec with Increasing Training Samples (RQ3)

Increasing training data substantially improves traditional recommendation models but has limited impact on BIGRec. The results suggest that BIGRec relies more on LLM semantic information than on dataset-specific statistical information.

  • Increasing data volume has limited impact on BIGRec’s performance compared with traditional ID-based methods.
  • Increasing training data enhances traditional model performance, partly by providing more popularity and collaborative information.
  • Figure 4 compares SASRec, DROS, and BIGRec across Sample Num using NDCG@K performance and improvement relative to 1,024 samples.
  • BIGRec appears less aggressive in capturing dataset statistics and instead relies on the LLM’s semantic information.

4.5 Performance of Introducing Valuable Statistical Information (RQ4)

BIGRec benefits from incorporating popularity and collaborative information during grounding. These results indicate that adding statistical information can improve alignment between LLM outputs and actual recommendation items.

  • Popularity and collaborative information are introduced to help locate actual items from generated LLM tokens during grounding.
  • Introducing popularity information: Popularity injection improves BIGRec’s NDCG@K and Recall@K, particularly for larger values of K.Its performance is also substantially better than using popularity information alone.
  • Introducing collaborative information: Collaborative-information integration improves both BIGRec and conventional models, with larger gains for BIGRec.
  • Introducing collaborative information: BIGRec’s reliance on semantic information distinct from collaborative information may explain why collaborative integration yields additional gains.
  • More effective and efficient grounding techniques could further improve recommendation accuracy.

5 CONCLUSION AND FUTURE WORK

BIGRec studies all-rank LLM recommendation through a bi-step grounding paradigm and reports strong performance with limited data, while identifying unresolved efficiency and grounding-design questions. Its findings also indicate that statistical information can further enhance LLM-based recommendations.

  • Conclusion: BIGRec investigates LLM4Rec performance across all ranks using a bi-step grounding paradigm.The paradigm is formulated as language space → recommendation space → actual item space.
  • Conclusion: BIGRec trained on 1024 samples outstrips traditional recommender models trained on the same number and matches conventional models trained on the full dataset in most cases.
  • Conclusion: LLMs rely heavily on semantics for recommendations while ignoring some useful information, and increasing training data can enhance BIGRec performance.The conclusion discusses scaling training data and its relationship to model performance.
  • Conclusion: Infusing popularity and collaborative information into the LLM further enhances its capability and validates the authors' assumption.
  • Future Work: The initial grounding stage still raises questions about sample selection, recommendation-space membership, and the cost of training larger language models.The paper asks whether appropriate samples can reduce expenses and whether all items should belong to one recommendation space.
  • Future Work: The second grounding step remains crude because it extracts decoder embeddings and uses similarity calculations, while large-beam sampling is time-consuming.The authors identify more efficient connections to actual items and more effective use of conventional recommendation features as future directions.
Loading 2308.08434v2…