PaperlayerView exact revision on arXiv

Source-linked AI summary

CoLLM: Integrating Collaborative Embeddings into Large Language Models for Recommendation

Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, Xiangnan He

arXiv:2310.19488v3cs.IR

TL;DR

Existing LLM recommenders often emphasize text semantics while neglecting collaborative information, leaving a gap in warm-start recommendation. CoLLM maps collaborative representations from an external conventional model into the LLM embedding space and reports improved recommendation performance across two real-world datasets. Its main scope limitation is that experiments were conducted only with Vicuna-7B, while inference cost remains a practical challenge.

  • Problem

    Existing LLMRec approaches often rely on text semantics and inadequately model collaborative information from user-item interactions, especially for warm users and items.

  • Method

    CoLLM maps collaborative information from a well-trained external conventional model into the LLM through an MLP and tunes the mapping module after language-only LoRA fine-tuning.

  • Results

    CoLLM's best version outperforms baselines on both metrics across two datasets, with experiments demonstrating effectiveness and adaptability in warm and cold recommendation scenarios.

  • Takeaways & Limitations

    Externalizing collaborative models lets CoLLM incorporate collaborative information while retaining flexibility to adjust the modeling mechanism.

  • Takeaways & Limitations

    Experiments were conducted exclusively on Vicuna-7B, and the authors identify relatively high inference cost as a practical challenge.

Abstract

from arXiv · show

Leveraging Large Language Models as Recommenders (LLMRec) has gained significant attention and introduced fresh perspectives in user preference modeling. Existing LLMRec approaches prioritize text semantics, usually neglecting the valuable collaborative information from user-item interactions in recommendations. While these text-emphasizing approaches excel in cold-start scenarios, they may yield sub-optimal performance in warm-start situations. In pursuit of superior recommendations for both cold and warm start scenarios, we introduce CoLLM, an innovative LLMRec methodology that seamlessly incorporates collaborative information into LLMs for recommendation. CoLLM captures collaborative information through an external traditional model and maps it to the input token embedding space of LLM, forming collaborative embeddings for LLM usage. Through this external integration of collaborative information, CoLLM ensures effective modeling of collaborative information without modifying the LLM itself, providing the flexibility to employ various collaborative information modeling techniques. Extensive experiments validate that CoLLM adeptly integrates collaborative information into LLMs, resulting in enhanced recommendation performance. We release the code and data at https://github.com/zyang1580/CoLLM.

1 INTRODUCTION

LLMRec methods leverage language models for recommendation but often underuse collaborative information from user-item interactions, limiting performance especially for warm users and items. CoLLM addresses this gap by mapping collaborative information from external conventional models into LLM embeddings without modifying the LLM itself.

  • LLMRec methods use LLM capabilities for recommendation, particularly benefiting cold users and items through language understanding and world knowledge.
  • Existing LLMRec approaches can struggle to surpass conventional recommender models for warm users and items because they primarily rely on text semantics.
  • Collaborative information from user-item co-occurrence patterns is especially useful for warm users and items, motivating integration into LLMs for both warm and cold scenarios.
  • CoLLM treats collaborative information as a separate modality and maps it from a well-trained conventional collaborative model into the LLM using an MLP.
  • CoLLM uses two-step tuning: language-only LoRA fine-tuning followed by tuning the mapping module so LLMs can use collaborative information for recommendation.
  • Extensive experiments on two real-world datasets demonstrate CoLLM's effectiveness, while its external-model design supports flexible collaborative modeling.

2 RELATED WORK

Related work explores LLM-based recommendation, combinations of language and collaborative models, and multimodal LLMs. CoLLM differs by mapping collaborative embeddings into the LLM space rather than merely ensembling models or directly learning ID embeddings there.

  • 2.1 LLMRec: LLMRec research has used in-context learning and recommendation-data tuning to provide language models with recommendation capabilities.
  • 2.1 LLMRec: Tuned LLMRec methods can still underperform traditional recommender models in certain settings because they often rely on semantic priors while overlooking collaborative information.
  • 2.1 LLMRec: BIGRec combines LLMs with collaborative models through ensembling, whereas CoLLM integrates collaborative information into the LLM generation process.
  • 2.1 LLMRec: Concurrent collaborative-embedding methods directly learn ID embeddings in the LLM space, while CoLLM maps embeddings produced by collaborative models into that space.
  • 2.2 Multimodal LLM: Multimodal LLM studies align frozen visual or sensor encoders with language models to extend language models beyond text inputs.

3 PRELIMINARIES

The paper formulates recommendation from interaction and item-text data, aiming for an LLM to use collaborative information beyond text. It introduces LLMs and conventional latent-factor collaborative recommenders as framework components.

  • Problem Definition: The recommendation dataset contains user-item-label tuples, with item textual information such as titles available for LLM fine-tuning.
  • Problem Definition: The objective is to enable an LLM to leverage both interaction data and textual information for superior warm- and cold-start recommendation.
  • Large Language Model: LLMs are language models with at least several billion parameters trained on massive text datasets and capable of natural-language understanding and generation.
  • Conventional Collaborative Recommender: Conventional collaborative recommenders such as MF and LightGCN encode users and items with latent factors or embeddings derived from interaction data.
  • Conventional Collaborative Recommender: Latent user and item representations are processed by an interaction module, with prediction-error minimization teaching them to encode collaborative information.

4 METHODOLOGY

CoLLM treats collaborative information as a separate modality and integrates it into LLM recommendation through prompt fields, hybrid encoding, and parameter-efficient prediction. It uses conventional collaborative models to create embeddings in the LLM token space without modifying the LLM itself.

  • Architecture Overview: CoLLM’s architecture comprises prompt construction, hybrid encoding, and LLM prediction.These components jointly represent textual and collaborative information within the LLM token embedding space.
  • Prompt Construction: CoLLM adds user and item ID fields to recommendation prompts as placeholders for collaborative information.The prompt also contains interaction-history titles and the target item title as textual inputs.
  • Hybrid Encoding: Hybrid encoding uses the LLM embedding lookup for text and a collaborative model for user and item ID fields.The resulting sequence combines textual token embeddings with collaborative embeddings suitable for LLM processing.
  • Hybrid Encoding: A conventional collaborative model generates user and item representations, while an MLP maps them into the LLM token embedding space.The collaborative model may be a latent-factor recommender, with mapping dimension d1 to the LLM embedding size d2.
  • LLM Prediction: The LLM uses the hybrid embedding sequence to generate recommendation predictions through an additional LoRA module.LoRA adds rank-decomposition weight matrices while keeping the pretrained LLM parameters fixed, enabling parameter-efficient recommendation learning.
  • Discussion: Inference remains computationally costly, although CoLLM adds only a smaller CIE module and does not update the LLM during its training.Acceleration techniques are deferred to future research, while the paper reports that CoLLM does not introduce excessive additional overhead compared with existing LLMRec methods.

5 EXPERIMENTS

The experiments address whether CoLLM improves recommendation by augmenting LLMs with collaborative information and examine how its design choices affect performance and efficiency. They also consider performance across datasets and LLM backbones.

  • Research Questions: The experiments compare whether CoLLM’s collaborative-information augmentation improves recommendation against existing methods.This is posed as research question RQ1.
  • Research Questions: The study examines how CoLLM’s design choices affect performance and efficiency.This forms part of research question RQ2.
  • Research Questions: The experiments evaluate CoLLM on other datasets and LLM backbones.These settings are also included in RQ2.

5.1 Experimental Settings

Experiments use two recommendation datasets, timestamp-based splits, standard ranking metrics, and diverse collaborative, hybrid, and LLMRec baselines to evaluate CoLLM.

  • Datasets: Experiments use ML-1M and Amazon-Book, with ratings or review scores transformed into binary interaction labels.ML-1M labels ratings greater than 3 as positive; Amazon-Book uses a threshold of 4.
  • Datasets: Timestamp-based splits allocate temporally ordered interactions to training, validation, and testing to simulate real-world recommendation and prevent leakage.ML-1M uses 10 months for training, 5 for validation, and 5 for testing within the most recent 20 months; Amazon-Book uses 2017 interactions.
  • Evaluation Metrics: Performance is assessed with AUC, UAUC, and NDCG, covering overall prediction accuracy and user-level or ranking quality.UAUC averages per-user AUC across exposed items, while NDCG measures normalized discounted cumulative gain.
  • Compared Methods: Baselines span conventional collaborative methods, language-collaborative models, and LLMRec methods.The collaborative group includes MF, LightGCN, SASRec, and DIN; CTRL represents the language-collaborative group, while ICL, Prompt4NR, and TALLRec represent LLMRec.
  • Compared Methods: CoLLM is implemented with four collaborative backends: MF, LightGCN, SASRec, and DIN.For SASRec and DIN, sequence representations are incorporated into the user representation within the CIE module.
  • Implementation: All methods use PyTorch 2.0, with BCE generally serving as the optimization loss and AdamW or Adam selected by model type.Learning rates of [1e-2, 1e-3, 1e-4] and embedding sizes of [64, 128, 256] are explored.

5.2 Performance Comparison (RQ1)

CoLLM outperforms the baselines across both metrics and datasets, while its collaborative-information integration improves LLMRec performance in warm scenarios without sacrificing cold-start proficiency.

  • The best CoLLM version outperforms baselines on both metrics across ML-1M and Amazon-Book.The overall comparison evaluates AUC, UAUC, and NDCG, with the cited observation referring to both reported metrics.
  • CoLLM’s gains are attributed to collaborative-information modeling rather than solely to adaptively updatable prompts.Prompt4NR falls short of CoLLM and even trails TALLRec despite fine-tuning the LLM and using adaptable prompts.
  • CoLLM improves LLMRec performance over corresponding collaborative baselines in almost all tested configurations.This pattern holds when the CIE module uses different collaborative models, demonstrating integration flexibility.
  • CoLLM’s integration mechanism outperforms simple ensemble averaging of MF and TALLRec, while ensembling CoLLM-MF yields only marginal further improvements.The ensemble comparison is reported on the AUC metric.
  • Warm and Cold Scenarios: In warm scenarios, CoLLM outperforms both MF and TALLRec across the two datasets on the reported AUC and UAUC comparisons.For AUC, the ordering is TALLRec below MF below CoLLM; for UAUC, MF is below TALLRec below CoLLM.
  • Warm and Cold Scenarios: In cold scenarios, TALLRec and CoLLM outperform MF, while CoLLM remains broadly comparable to TALLRec.CoLLM performs better in AUC and slightly worse in UAUC than TALLRec in the reported cold-scenario comparison.

5.3 In-depth Analysis (RQ2)

CoLLM’s CIE module is central to modeling collaborative information, while tuning choices trade recommendation quality against efficiency. Experiments also indicate strong generalization across scenarios, datasets, and LLM backbones.

  • The Effect of CIE Module: Removing CIE reduces performance, while directly adding user/item tokens performs even worse than omitting collaborative-information modeling.The authors attribute this to tokenization redundancy, whereas CIE maintains collaborative information in a low-rank form.
  • The Influence of Tuning Choices: Tuning the collaborative model within the two-step framework improves performance in most cases.T1 initializes the collaborative model from its pretrained state, whereas T2 tunes the entire CIE model from scratch.
  • The Influence of Tuning Choices: At least five times the training effort is required in ML-1M when additional collaborative-model tuning is used.This overhead motivates choosing more efficient tuning approaches despite their potential performance gains.
  • The Influence of Tuning Choices: One-step tuning usually underperforms the two-step alternatives, particularly in cold scenarios.The authors link this result to the first text-only tuning step’s role in learning recommendation tasks for cold-start settings.
  • Efficiency: CIE adds limited inference cost because its module is much smaller than the LLM and its extra prompt tokens are a small proportion of the prompt.Pretraining the collaborative module also reduces the need for extensive additional training.
  • Method Generalization: CoLLM consistently outperforms TALLRec and MF on the Qwen2-1.5B backbone and remains effective on additional datasets.The additional datasets are Video Games and CDs & Vinyl from Amazon.

6 CONCLUSION

The paper presents CoLLM as an approach for incorporating collaborative information into LLM-based recommendation, especially for warm scenarios. Its experiments support effectiveness across warm and cold settings, while broader backbones and incremental learning remain future work.

  • Conclusion: CoLLM incorporates collaborative information into LLM-based recommendation by externalizing traditional collaborative models.The approach is designed to provide flexibility in adjusting the collaborative-information modeling mechanism.
  • Conclusion: Collaborative-information modeling is particularly significant for improving LLMRec recommendation performance in warm scenarios.The conclusion also reports that CoLLM enables strong recommendation performance in both warm and cold scenarios.
  • Conclusion: The experiments were conducted exclusively on Vicuna-7B, leaving other LLM backbones and incremental learning for future investigation.The authors identify the evolving nature of real-world collaborative information as motivation for studying incremental learning.
Loading 2310.19488v3…