Text-like Encoding of Collaborative Information in Large Language Models for Recommendation

1 Introduction

• Collaborative information captures user-item co-occurrence, is pivotal for modeling interests—especially for active users and items—and differs in modality from text.Source: §1 · p. 1
• From-scratch user/item tokens risk low learning efficacy and tokenization redundancy (Deletang et al., 2024; Zhang et al., 2023b), while external embeddings add mapping-model training overhead (Zhang et al., 2023b; Li et al., 2023c; Liao et al., 2024).Source: §1 · p. 1
• Text-like encodings are promising because LLMs excel at processing text, but sequences need only be LLM-interpretable, not human-readable or conventional natural language.Source: §1 · pp. 1–2
• BinLLM converts external collaborative embeddings into binary text-like features, optionally compresses them with dot-decimal notation, and is evaluated extensively on two datasets.Source: §1 · p. 2

2.1 Model Architecture

• BinLLM converts collaborative embeddings into binary sequences that can be included in prompts for LLM-based recommendation.Prompts combine textual fields, such as item titles, with ID fields that carry collaborative information.Source: §2.1 · p. 2, §2.1.1 · pp. 2–3

2.1.1 Prompt Construction

• The prompt template has text fields for textual data and ID fields for collaborative information, and follows the format used by CoLLM.Table 1 provides an example of the template.Source: §2.1.1 · p. 2, Table 1 · p. 2
• A fully connected layer followed by a sign function converts user and item collaborative embeddings into binary representations.The binary sequences can be compressed optionally by converting each eight bits to a decimal number and separating the numbers with dots.Source: §2.1.1 · pp. 3–4

2.1.2 LLM Prediction

• The generated prompt is fed to an LLM with an additional LoRA module for recommendation prediction.Depending on the task, the prediction can be a next item or the likelihood of liking a candidate item.Source: §2.1.2 · p. 4

2.2 Training

• Training has two modules: the text-like encoding module learns binary sequences, while LoRA tuning instructs the LLM to make recommendations using collaborative information.The encoding module is tuned independently of the LLM.Source: §2.2 · p. 4

2.2.1 Pre-training for Text-like Encoding

• The text-like encoding module is trained by fitting its binarized representations to interaction data using a recommendation loss.The work uses binary cross-entropy loss and approximates the sign function’s gradient with a straight-through estimator for end-to-end training.Source: §2.2.1 · p. 4

2.2.2 LoRA Tuning

• Intuitive tuning trains LoRA from scratch with collaborative prompts, while two-step tuning first trains without collaborative information and then fine-tunes with it.The two-step strategy addresses the potential for intuitive tuning to over-rely on binary representations and neglect other attributes.Source: §2.2.2 · p. 4

3.1 Experimental Settings

• Experiments focus on click/rating prediction using Vicuna-7B, two processed datasets, and AUC and UAUC evaluation metrics.Dataset processing follows CoLLM; BinLLM uses 32-dimensional collaborative embeddings without compression by default.Source: §3.1 · pp. 5–6

3.2.1 Overall Performance (RQ1)

• BinLLM achieves the best overall performance against baselines except against CoLLM-DIN on UAUC, and consistently outperforms CoLLM-MF and PersonPrompt.Table 3 compares overall performance on ML-1M and Amazon-Book; BinLLM also generally outperforms CoLLM-DIN except on Amazon-Book UAUC, despite using MF rather than its more advanced collaborative model.Source: Table 3 · p. 6, §3.2.1 · pp. 6–7

3.2.2 Warm and Cold Performance

• In warm scenarios, BinLLM is best among the compared methods; in cold scenarios, all LLMRec methods outperform MF, and BinLLM beats CoLLM in most cases.Figure 2 compares warm and cold performance on ML-1M and Amazon-Book, with AUC on the left axis and UAUC on the right.Source: Figure 2 · p. 6, §3.2.2 · p. 7

3.2.1 Overall Performance (RQ1)

• Across the comparisons, LLMRec methods incorporating collaborative information generally improve performance and can surpass traditional collaborative and language-model-based methods.Among LLMRec methods using collaborative information, PersonPrompt performs worst; the authors attribute this to low learning efficacy from adding tokens and embeddings.Source: §3.2.1 · p. 7

3.3.1 Ablation Study

• Ablations find that using the LLM, collaborative information, both text and collaborative fields, and the default two-step tuning each matter to performance.BinMF underperforms BinLLM variants; text-only underperforms other variants; ID-only generally underperforms intuitive-tuning and default BinLLM; intuitive-tuning consistently trails the default. Directly tuning prompts with collaborative information from scratch may underutilize both information types.Source: §3.3.1 · pp. 7–8
• On ML-1M, BinLLM-emb-str scores AUC 0.7343, versus 0.7425 for BinLLM and 0.7243 for CoLLM.BinLLM-emb-str reduces embedding dimensions with UMAP and converts them to strings; it trails BinLLM but exceeds CoLLM.Source: §3.3.1 · p. 8

3.3.2 The Influence of Compression

• Compression generally maintains comparable recommendation performance while reducing collaborative-representation length.Dot-decimal notation shortens representations by approximately 2.5 times, but inference-time gains were smaller because these representations comprise a smaller part of the prompt.Source: §3.3.2 · p. 8

Collaborative Information Modeling

• Collaborative modeling progressed from statistical methods to latent-factor and neural models, motivating exploration of collaborative information modeling for LLMRec.Source: §4 · p. 8

LLMRec

• LLMRec research includes direct prompting and tuning methods, with tuning motivated by recommendation data and LLMs’ insufficient attention to collaborative information.Source: §4 · pp. 8–9
• Prior methods add ID tokens trained from scratch or map latent factors into LLM semantic space; these approaches face low learning efficacy, extra mapping training, or alignment challenges.Source: §4 · p. 9

• Extensive results demonstrate BinLLM’s superiority in LLMRec.Source: §5 · p. 9
• BinLLM binarizes collaborative embeddings to incorporate collaborative information in a text-like format.Source: §5 · p. 9
• The encoding is intended to align collaborative information with how LLMs process information and enhance recommendation performance.Source: §5 · p. 9

• Experiments rely solely on Vicuna-7B, limiting validation to one LLM.Source: Limitations · p. 9
• Evaluation covers rating/click prediction but omits other recommendation tasks, including next-item prediction.Source: Limitations · p. 9
• Low inference efficiency poses a challenge for real-world recommendation, particularly in the all-ranking setting.The authors suggest exploring pruning and recommendation generation that avoids multiple inferences per user.Source: Limitations · p. 9

• The authors state that binarizing numerical embeddings does not raise ethical concerns and that the data are publicly available.Source: Ethical Considerations · p. 9
• The datasets do not include sensitive details like gender, but recommendation can still raise privacy concerns involving user behavioral data.Source: Ethical Considerations · p. 9
• The authors flag potential hidden societal biases from LLM use and advise thorough risk assessment and caution.Source: Ethical Considerations · p. 9