PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender System

Sein Kim, Hongseok Kang, Seungyoon Choi, Donghyun Kim, Minchul Yang, Chanyoung Park

arXiv:2404.11343v2cs.IRcs.AI

TL;DR

Collaborative filtering performs poorly with sparse cold-start interactions, while modality-aware and LLM-based recommenders can lack collaborative knowledge in warm settings. A-LLMRec aligns embeddings from a frozen pretrained CF recommender with a frozen LLM, and experiments report superiority across cold/warm, few-shot, cold-user, and cross-domain scenarios. The method is also model-agnostic and efficient because only its alignment network is trained.

  • Problem

    Collaborative filtering struggles with sparse cold-start interactions, while modality-aware and LLM-based recommenders can underperform traditional CF in warm scenarios because they lack collaborative knowledge.

  • Method

    A-LLMRec aligns collaborative knowledge from a frozen pretrained CF recommender with the token space of a frozen LLM using a trainable alignment network.

  • Results

    Experiments report that A-LLMRec outperforms CF recommenders and LLM-based recommenders across cold/warm, few-shot, cold-user, and cross-domain scenarios.

  • Takeaways & Limitations

    A-LLMRec combines collaborative knowledge with LLM capabilities while remaining model-agnostic and avoiding fine-tuning of the CF recommender and LLM.

  • Takeaways & Limitations

    The paper mainly focuses on the sequential recommendation task, although it states that A-LLMRec can also be applied elsewhere.

Abstract

from arXiv · show

Collaborative filtering recommender systems (CF-RecSys) have shown successive results in enhancing the user experience on social media and e-commerce platforms. However, as CF-RecSys struggles under cold scenarios with sparse user-item interactions, recent strategies have focused on leveraging modality information of user/items (e.g., text or images) based on pre-trained modality encoders and Large Language Models (LLMs). Despite their effectiveness under cold scenarios, we observe that they underperform simple traditional collaborative filtering models under warm scenarios due to the lack of collaborative knowledge. In this work, we propose an efficient All-round LLM-based Recommender system, called A-LLMRec, that excels not only in the cold scenario but also in the warm scenario. Our main idea is to enable an LLM to directly leverage the collaborative knowledge contained in a pre-trained state-of-the-art CF-RecSys so that the emergent ability of the LLM as well as the high-quality user/item embeddings that are already trained by the state-of-the-art CF-RecSys can be jointly exploited. This approach yields two advantages: (1) model-agnostic, allowing for integration with various existing CF-RecSys, and (2) efficiency, eliminating the extensive fine-tuning typically required for LLM-based recommenders. Our extensive experiments on various real-world datasets demonstrate the superiority of A-LLMRec in various scenarios, including cold/warm, few-shot, cold user, and cross-domain scenarios. Beyond the recommendation task, we also show the potential of A-LLMRec in generating natural language outputs based on the understanding of the collaborative knowledge by performing a favorite genre prediction task. Our code is available at https://github.com/ghdtjr/A-LLMRec .

1 INTRODUCTION

Traditional collaborative filtering struggles with sparse cold-start interactions, while modality-aware and LLM-based recommenders can lose collaborative knowledge in warm settings. A-LLMRec aligns pretrained CF embeddings with an LLM, aiming to combine collaborative knowledge with LLM capabilities efficiently across scenarios.

  • Motivation: In cold scenarios, sparse interactions make collaborative knowledge difficult to construct, motivating modality and LLM-based methods that use user or item information.The supplied passages define cold items as those in the bottom 35% of interactions and warm items as those in the top 35%.
  • Motivation: Modality-aware and LLM-based recommenders can outperform traditional CF in cold settings but fall behind it in warm settings because they lack collaborative knowledge.Figure 1 compares SASRec, MoRec, and TALLRec using Hit@1 on Amazon Movies and Video Games.
  • Approach: A-LLMRec aligns pretrained CF user/item embeddings with the LLM token space so the LLM can use collaborative knowledge for recommendation.Neither the CF recommender nor the LLM is fine-tuned; the alignment network transfers collaborative knowledge between them.
  • Advantages: A-LLMRec is model-agnostic because existing CF recommenders can be integrated and replaced without retraining the overall system.This supports practical use when recommender models are updated.
  • Efficiency: A-LLMRec trains approximately 2.53 times and infers 1.71 times faster than TALLRec while outperforming TALLRec and CF-RecSys in cold and warm scenarios.Its alignment network is the only trainable neural network, whereas TALLRec fine-tunes the LLM with LoRA.
  • Results: A-LLMRec outperforms CF recommenders and LLM-based recommenders across cold/warm, few-shot, cold-user, and cross-domain scenarios.The paper also evaluates natural-language favorite-genre prediction based on collaborative knowledge.

2 RELATED WORK

Collaborative filtering models infer preferences from similar users and items, while modality-aware and LLM-based recommenders add textual, visual, or language-based information. LLM recommenders can help with cold-start and cross-domain settings, but conventional models may remain stronger because of task and collaborative-knowledge gaps.

  • 2.1 Collaborative Filtering: Collaborative filtering recommends items by leveraging users’ historical preferences and similarities among users or items.Matrix factorization and later neural models refine the representation of latent user preferences and interaction patterns.
  • 2.2 Modality-aware Recommender Systems: Modality-aware recommenders use item titles, descriptions, images, and pretrained encoders to improve recommendation mainly in cold scenarios.Examples include BERT and ResNet or Vision-Transformer-based encoders.
  • 2.3 LLM-based Recommender Systems: LLM-based recommenders use prompting, item text, user descriptions, or parameter-efficient fine-tuning to support recommendation and cold-start tasks.TALLRec uses LoRA to bridge the gap between language-model training and recommendation.
  • 2.3 LLM-based Recommender Systems: LLM-based approaches can underperform traditional recommendation models because natural-language training tasks differ from recommendation tasks.TALLRec addresses this gap with LoRA and shows stronger results particularly for cold-start and cross-domain recommendation.

3 PROBLEM FORMULATION

The paper formulates sequential recommendation as predicting a user’s next item from historical interactions, using item embeddings and a collaborative filtering recommender. A-LLMRec is also applicable to non-sequential recommendation by replacing the CF backbone.

  • Task: Sequential Recommendation: Sequential recommendation predicts a user’s next item from the historical interaction sequence.Each user sequence contains ordered item interactions, and the task models the next interaction conditioned on the preceding sequence.
  • Task: Sequential Recommendation: Item embeddings for a user’s sequence are fed into a collaborative filtering recommender such as SASRec to predict subsequent behavior.The sequence embedding matrix is constructed from the item embedding matrix and passed to the CF-RecSys.
  • Task: Sequential Recommendation: A-LLMRec can support non-sequential recommendation by replacing the sequential CF backbone, such as substituting NCF for SASRec.The paper states that this model-agnostic extension is demonstrated experimentally.

4 PROPOSED METHOD: A-LLMREC

A-LLMRec aligns a frozen collaborative filtering recommender with a frozen LLM so the LLM can use collaborative knowledge alongside textual information. The framework uses two pre-training stages to bridge CF representations and the LLM token space.

  • 4 PROPOSED METHOD: A-LLMREC: A-LLMRec aligns a frozen pre-trained CF-RecSys with a frozen LLM to improve recommendation in both cold and warm scenarios.The framework bridges the modality gap by aligning collaborative knowledge with the LLM’s token space.
  • 4 PROPOSED METHOD: A-LLMREC: The approach uses two pre-training stages: aligning collaborative and textual knowledge, then aligning the resulting embeddings with the LLM.

4.1 Alignment between Collaborative and Textual Knowledge (Stage-1)

Stage 1 aligns frozen CF item embeddings with SBERT text embeddings to create joint collaborative-text representations. Reconstruction and recommendation objectives preserve information and explicitly incorporate collaborative knowledge, with text-based representations supporting unseen items.

  • 4.1 Alignment between Collaborative and Textual Knowledge (Stage-1): Stage 1 maps frozen CF item embeddings and SBERT text embeddings into a shared latent space using separate item and text encoders.The encoders transform the two modalities into latent embeddings and match their representations.
  • 4.1 Alignment between Collaborative and Textual Knowledge (Stage-1): Mean squared error matching aligns item embeddings with associated title-and-description embeddings for later use in the LLM.
  • 4.1.1 Avoiding Over-smoothed Representation.: Reconstruction losses are added because matching alone can over-smooth or collapse representations, while experiments show removing any Stage-1 loss reduces performance.The reconstruction terms preserve the original item and text information.
  • 4.1.2 Recommendation Loss.: A recommendation loss explicitly incorporates collaborative knowledge while informing the model about the recommendation task.
  • 4.1 Alignment between Collaborative and Textual Knowledge (Stage-1): For unseen or interaction-sparse items, the text encoder supplies joint embeddings expected to retain textual and implicit collaborative knowledge.The default item embedding is used when interactions exist; text-derived embeddings are used for cold, few-shot, and cross-domain cases.

4.2 Alignment between Joint Collaborative-Text Embedding and LLM (Stage-2)

Stage 2 projects user representations and joint item embeddings into the frozen LLM’s token space, then inserts them into prompts for recommendation. The prompt combines collaborative representations with recommendation instructions and adapts wording across datasets.

  • 4.2 Alignment between Joint Collaborative-Text Embedding and LLM (Stage-2): Stage 2 aligns Stage-1 joint embeddings with the LLM token space and designs prompts that use the learned collaborative knowledge.The Stage-1 item encoder is frozen during Stage 2.
  • 4.2.1 Projecting collaborative knowledge onto the token space of LLM: Two 2-layer MLPs project user representations and joint item embeddings into the LLM token space.The projected outputs can be supplied as LLM prompt inputs without fine-tuning the LLM.
  • 4.2.1 Projecting collaborative knowledge onto the token space of LLM: The example prompt preserves a common format across datasets while adapting verbs and nouns to each domain.
  • 4.2.1 Projecting collaborative knowledge onto the token space of LLM: The prompt directly inserts projected user and item representations as token embeddings alongside recommendation instructions.These projected representations can be treated as ordinary tokens by the LLM.

5 EXPERIMENTS

A-LLMRec is evaluated across datasets and recommendation scenarios, including cold/warm items, cold users, few-shot training, and cross-domain transfer. Results indicate that incorporating collaborative knowledge improves performance across these settings.

  • 5.1 Experimental Setup: Experiments use four Amazon datasets, compare collaborative filtering, modality-aware, and LLM-based baselines, and evaluate with Hit@1.The datasets include Movies and TV, Video Games, Beauty, and Toys.
  • 5.2.1 Overall Performance: A-LLMRec outperforms LLM-based recommenders that do not use collaborative knowledge, while MLP-LLM also underperforms A-LLMRec.The comparison covers four datasets and uses Hit@1 as the evaluation metric.
  • 5.2.2 Cold/Warm Item Scenarios: A-LLMRec outperforms all other baselines across both cold and warm item scenarios.Items are labeled cold or warm using the bottom and top 35% of interaction counts, respectively.
  • 5.2.3 Cold User Scenario: A-LLMRec consistently outperforms other models in the cold user scenario, where users have exactly three interactions.Models are trained without the sampled users and then evaluated on their final interaction.
  • 5.2.4 Few-shot Training Scenario: A-LLMRec outperforms all other baselines in the few-shot scenario despite training with extremely few users.The setting limits the training set to K users and creates many unseen or new items at inference.
  • 5.2.5 Cross-domain Scenario: A-LLMRec outperforms all baselines in cross-domain evaluation from Movies and TV to Video Games.The A-LLMRec (SBERT) variant performs particularly well when collaborative information is lacking.

5.3 Ablation Studies

Ablation and analysis experiments examine A-LLMRec’s alignment components, prompt inputs, efficiency, generalization, backbone choice, and language-generation capability. The findings support the value of alignment while showing a trade-off between broader training and efficiency.

  • Stage-1 Ablation Studies: Removing any Stage-1 loss decreases performance across all datasets, with Lmatching removal causing a significant decline.The result supports alignment between item and textual information as a contributor to recommendation performance.
  • Stage-2 Ablation Studies: Excluding either the user representation or joint embedding from the prompt reduces performance, with joint-embedding removal causing the larger decrease.The joint embedding transfers collaborative knowledge and also captures textual information about items.
  • Efficiency Analysis: A-LLMRec has faster training and inference than TALLRec because it trains only the alignment network rather than fine-tuning the LLM.The faster inference is presented as practical for real-time recommendation services.
  • Training with All Items in Each Sequence: Using all items in each user sequence improves recommendation performance but increases training time approximately threefold, making the improvement marginal.Vanilla A-LLMRec uses only the last item in each sequence and retains comparable performance.
  • Model-Agnostic Analysis: A-LLMRec remains model-agnostic by supporting sequential and non-sequential CF-RecSys backbones, although SASRec performs best among the tested choices.The tested alternatives include NextItNet, GRU4Rec, and NCF.
  • Favorite Genre Prediction: A-LLMRec is evaluated for favorite genre prediction using the same OPT-6.7B backbone as LLM-Only.The task tests whether aligned collaborative knowledge can support natural-language outputs about users’ preferred movie genres.

6 CONCLUSION

A-LLMRec uses collaborative knowledge from pre-trained CF-RecSys to outperform existing recommender systems across cold/warm, cold-user, few-shot, and cross-domain settings, while remaining model-agnostic and efficient.

  • 6 CONCLUSION: A-LLMRec outperforms existing CF-RecSys, modality-aware recommenders, and LLM-based recommenders across cold/warm, cold-user, few-shot, and cross-domain scenarios.The method also demonstrates potential for natural-language generation from collaborative knowledge.
  • 6 CONCLUSION: A-LLMRec avoids fine-tuning both the pre-trained CF-RecSys and LLM, supporting its model-agnostic and efficiency advantages.
  • 6 CONCLUSION: A-LLMRec can generate natural-language outputs based on its understanding of collaborative knowledge from CF-RecSys.

A BASELINES

The baselines span traditional collaborative filtering, modality-aware and LLM-based recommenders, and additional architectures used to compare recommendation strategies.

  • A BASELINES: The baseline set includes NCF, NextItNet, and GRU4Rec, covering neural collaborative filtering, temporal convolution, and recurrent session-based recommendation.
  • A BASELINES: MoRec and CTRL use item text with SASRec as backbone, representing modality-aware collaborative filtering approaches.
  • A BASELINES: LLM-Only uses 6.7B OPT with recommendation prompts, while TALLRec fine-tunes an LLM with LoRA from text prompts and interaction histories.
  • A BASELINES: Figure 6 shows the Amazon Movies prompt used by LLM-Only and TALLRec, and Table 12 lists source-code links for baseline methods.
  • A BASELINES: MLP-LLM connects frozen CF-RecSys and LLM embeddings through MLP layers, whereas A-LLMRec uses auto-encoders.

B LANGUAGE GENERATION TASK

The favorite genre prediction results show that A-LLMRec can generate valid natural-language outputs from aligned collaborative representations, whereas the compared systems cannot do so reliably.

  • B LANGUAGE GENERATION TASK: A-LLMRec generates favorite genres for users using aligned user representations and item embeddings, while LLM-only fails to do so.
  • B LANGUAGE GENERATION TASK: TALLRec cannot generate valid natural-language outputs because instruction tuning restricts its responses to particular prompts used during tuning.
  • B LANGUAGE GENERATION TASK: Figure 5 presents additional favorite genre prediction results for the experiment in Section 5.4.4.

C REPRODUCIBILITY

The reproducibility materials identify the baseline implementations and direct readers to the source code and execution instructions for reproducing the reported experiments.

  • C REPRODUCIBILITY: Baseline implementations follow the official code released by the respective authors.
  • C REPRODUCIBILITY: Table 12 provides source-code links for the baseline methods.
  • C REPRODUCIBILITY: The paper directs readers to the source code and instructions for reproducing the reported experimental results.
Loading 2404.11343v2…