PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Representations Can be What Recommenders Need: Findings and Potentials

Leheng Sheng, An Zhang, Yi Zhang, Yuxin Chen, Xiang Wang, Tat-Seng Chua

arXiv:2407.05441v4cs.IRcs.AI

TL;DR

The paper asks whether language-model representations encode user preferences, a connection that remains unclear in recommendation. It tests linear mapping into behavior space and builds AlphaRec using language representations alone. The resulting model outperforms leading ID-based collaborative-filtering baselines, while the analysis identifies initialization, zero-shot, and intention-aware potentials.

  • Problem

    It remains unclear whether language models inherently encode user-preference and behavior information in their representations.

    Source: §1 · pp. 1–2

  • Method

    The paper linearly maps frozen language representations of item titles into behavior space, then develops AlphaRec with a trainable projector and contrastive loss using language representations rather than item IDs.

    Source: §1 · pp. 2–3, §3.1 · p. 4, §4 · p. 6

  • Results

    AlphaRec consistently outperforms leading CF baselines across all metrics and datasets, improving Recall@20 by 6.79% to 9.75% over the best baseline.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The analysis identifies language representations as potential item-representation initialization, zero-shot recommendation on new datasets, and a basis for text-based user-intention awareness.

    Source: §1 · p. 3, §7 · p. 10

    AlphaRec lacks a personalized design because it uses one MLP for all users, and its fixed user intention is less practical without exploration of personalized intentions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent studies empirically indicate that language models (LMs) encode rich world knowledge beyond mere semantics, attracting significant attention across various fields. However, in the recommendation domain, it remains uncertain whether LMs implicitly encode user preference information. Contrary to prevailing understanding that LMs and traditional recommenders learn two distinct representation spaces due to the huge gap in language and behavior modeling objectives, this work re-examines such understanding and explores extracting a recommendation space directly from the language representation space. Surprisingly, our findings demonstrate that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance. This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation, implying that collaborative signals may be implicitly encoded within LMs. Motivated by these findings, we explore the possibility of designing advanced collaborative filtering (CF) models purely based on language representations without ID-based embeddings. To be specific, we incorporate several crucial components to build a simple yet effective model, with item titles as the input. Empirical results show that such a simple model can outperform leading ID-based CF models, which sheds light on using language representations for better recommendation. Moreover, we systematically analyze this simple model and find several key features for using advanced language representations: a good initialization for item representations, zero-shot recommendation abilities, and being aware of user intention. Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.

1 INTRODUCTION

Although language models encode knowledge beyond semantics, it is unclear whether their representations capture user preferences. The paper tests whether recommendation representations can be derived from language representations and finds that language-only approaches can support competitive recommendation.

1 INTRODUCTION

  • Prior work leaves the relationship between language and behavior spaces largely unexplored, despite using language models to align or enrich recommendation systems.The paper frames this gap against a prevailing view that general language models and traditional recommenders encode distinct representation spaces.Source: §1 · pp. 1–2
  • Advanced language representations, linearly mapped into behavior space, achieve superior recommendation performance and may encode user-preference similarities beyond textual similarity.The mapped representations cluster some movies with low textual similarity but high user-preference similarity.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • AlphaRec uses item titles with a frozen language model, a trainable projector, and a contrastive loss, and can outperform leading ID-based collaborative-filtering models.The projector is a two-layer MLP with graph convolution.Source: §1 · p. 3
  • Language representations may provide good item-representation initialization, zero-shot recommendation on new datasets after multi-dataset co-training, and text-based user-intention awareness.The reported zero-shot performance can match or surpass fully trained LightGCN without additional training on the new datasets.Source: §1 · p. 3

2 PRELIMINARY

The preliminary section defines collaborative filtering as matching users and items from interaction behavior, then contrasts item representations built from IDs with those derived from text using language models.

2.1 TASK FORMULATION

  • Collaborative filtering selects items matching user preferences from binary interaction histories, using user and item representations followed by a relevance score.The score estimates how likely a user is to interact with an item.Source: §2.1 · p. 3

2.2 ITEM REPRESENTATION GENERATION

  • ID-based generators transform item identifiers into dense representations with trainable embeddings, sometimes augmented by graph convolution.The passage notes poor domain transferability and lack of user-intention awareness as problems associated with ID-only features.Source: §2.2 · p. 4
  • LM-based generators encode item text with a frozen language model, then use a trainable projector to produce recommendation representations.The passage notes that few studies had shown such generators could outperform ID-based generators alone.Source: §2.2 · p. 4

3 UNCOVERING COLLABORATIVE SIGNALS IN LMS VIA LINEAR MAPPING

The section tests whether language representations contain collaborative signals and whether those signals scale or withstand prompt noise. It uses linear mapping to project item-title representations into a recommendation behavior space and evaluates the resulting representations.

3.1 LINEAR MAPPING

  • Linear mapping projects frozen-LM item-title representations into behavior space, averages a user's historical-item representations, and trains the mapping matrix with InfoNCE.The authors use test-set recommendation performance as an indication of a possible homomorphism between language and behavior spaces.Source: §3.1 · p. 4

3.2 EMPIRICAL FINDINGS

  • Advanced LM representations mapped into behavior space outperform leading CF models in most cases, while earlier BERT-style models perform similarly to or worse than MF.The comparison covers three Amazon datasets and includes MF, MultVAE, and LightGCN; the reported pattern suggests the capability emerged in more recent, advanced LMs.Source: §3.2 · p. 5, Table 1 · p. 5
  • Mapped language representations capture user-preference similarities beyond semantic textual similarity, including movies that cluster together despite differing textual meaning.The paper uses this pattern to suggest that preferences not immediately apparent from text can be uncovered through mapping.Source: §3.2 · p. 5
  • Linear-mapping performance improves consistently as Llama2 model size increases from 7B to 70B.Figure 2 presents performance for different language-model sizes.Source: §3.2 · p. 5, Figure 2 · p. 5
  • Adding 5–10 random letters to item titles has minimal impact on linear-mapping performance, with greater prompt-noise impact on Movies & TV.The paper attributes the greater impact to comparatively shorter item titles in that dataset.Source: §3.2 · p. 6, Table 2 · p. 6

4 LEVERAGING LANGUAGE REPRESENTATIONS FOR BETTER RECOMMENDATION

This section introduces AlphaRec, a language-representation-based collaborative filtering model, and evaluates whether its design can make advanced language representations useful for recommendation. AlphaRec outperforms leading ID-based CF baselines, while its components contribute positively and combining language with ID embeddings does not surpass it.

4 LEVERAGING LANGUAGE REPRESENTATIONS FOR BETTER RECOMMENDATION

  • The section asks whether language representations can support advanced CF models that outperform prevailing ID-based methods, following evidence of possible space homomorphism and encoded collaborative signals.It introduces AlphaRec as a simple model based solely on language representations.Source: §4 · p. 6

4.1 ALPHAREC

  • AlphaRec combines a two-layer MLP, graph convolution, cosine similarity, and InfoNCE optimization to build a CF model from language representations.The model is designed to explore language representations rather than introduce new CF mechanisms.Source: §4.1 · pp. 6–7

4.2 EMPIRICAL FINDINGS

  • AlphaRec consistently outperforms leading CF baselines by a large margin across all metrics on all datasets, with a 6.79% to 9.75% Recall@20 improvement over the best baseline.Each component contributes positively. Replacing XSimGCL’s ID embeddings with language representations does not bring a stable and remarkable improvement.Source: §4.2 · p. 7
  • Figure 3 compares component effects on Books and epochs to convergence; its caption describes AlphaRec’s convergence as breakneck.The component ablation results are also summarized as positive in the main comparison.Source: §4.2 · p. 7, Figure 3 · p. 8
  • KAR and RLMRec improve over the most advanced CF methods, but their combination of ID embeddings and language representations does not outperform AlphaRec.The authors attribute this to naive combination designs.Source: §4.2 · p. 8

5 EXPLORING POTENTIALS OF LANGUAGE REPRESENTATIONS FOR RECOMMENDATION

The section examines three potentials of advanced language representations: initializing item representations, supporting zero-shot recommendation, and incorporating user intentions. AlphaRec demonstrates these potentials across initialization, new-dataset recommendation, and intention-aware recommendation experiments.

5.1 GOOD INITIALIZATION FOR ITEM REPRESENTATIONS (POTENTIAL 1)

  • Advanced language representations provide good item initialization: AlphaRec converges rapidly, and only minor adjustments are needed to produce effective behavior representations.Its convergence speed is comparable to or faster than SGL and XSimGCL.Source: §5.1 · p. 8

5.2 ZERO-SHOT ABILITY (POTENTIAL 2)

  • The zero-shot evaluation jointly trains on Books, Movies & TV, and Video Games, then tests on MovieLens-1M, Book Crossing, and Amazon Industrial & Scientific without further training.The test sets have no user or item overlap with training data.Source: §5.2 · p. 8
  • AlphaRec shows strong zero-shot recommendation, comparable to or better than fully trained LightGCN, and outperforms LLMRank; its improvement on entirely new datasets is significant (p-value << 0.05).The test uses three target datasets with no item or user overlap and no further target-dataset training.Source: §5.2 · pp. 8–9, Table 4 · p. 9
  • Training AlphaRec jointly on multiple datasets generally improves zero-shot performance over single-dataset training, with little performance decline on source datasets.The authors suggest that more training data spanning more themes could support general recommendation capabilities.Source: §5.2 · p. 9

5.3 INTENTION-AWARE ABILITY (POTENTIAL 3)

  • Adding user intention significantly refines pretrained AlphaRec recommendations, and AlphaRec outperforms TEM without additional search-task training.The reported α curve suggests that both historical interests and user intention matter; α controls their relative strength.Source: §5.3 · p. 10, Table 5 · p. 10, Figure 4 · p. 10

6 LIMITATIONS

The paper identifies missing theoretical guarantees and personalized design as limitations. Its intention setup also lacks exploration of personalized user intentions.

6 LIMITATIONS

  • The work provides no theoretical guarantee and introduces no new CF-model components.Source: §6 · p. 10
  • AlphaRec uses a single MLP for all users rather than a personalized design.Source: §6 · p. 10
  • The fixed user intention in Section 5.3 is less practical, and personalized user intentions remain unexplored.Source: §6 · p. 10

7 CONCLUSION

The conclusion presents AlphaRec as a language-representation-based CF model motivated by evidence of a possible relationship between language and behavior spaces. The accompanying related-work and implementation material describes representation methods, model choices, and complementary research on that relationship.

7 CONCLUSION

  • Empirical findings suggest a possible homomorphism between advanced LM representations and effective item representations, motivating AlphaRec and its three analyzed potentials.Source: §7 · p. 10

A RELATED WORKS

  • The paper describes ID-based CF as difficult to transfer because one-hot identifiers contain no feature information beyond item identity.Source: A RELATED WORKS · p. 18
  • Related work groups LM-based recommendation into LM-enhanced recommenders, LM text encoders, and LMs that directly recommend items.Source: A RELATED WORKS · p. 18

B.1 BRIEF OF USED LMS

  • The evaluated language models include BERT, RoBERTa, Llama2-7B, Mistral-7B, OpenAI embedding models, and SFR-Embedding-Mistral.The passage distinguishes encoder-only, decoder-only, and text-embedding models.Source: §B.1 · p. 19

B.2 GENERATING ITEM REPRESENTATIONS FROM LMS

  • Item representations are extracted from encoder [CLS] states or decoder last-token states, while closed-source embedding models are accessed through APIs.Source: §B.2 · p. 19

B.3 CONNECTION WITH FINE-TUNED LLM-BASED APPROACHES FOR EXPLORING THE RELATIONSHIP BETWEEN LANGUAGE SPACE AND BEHAVIOR SPACE.

  • Prior work on interpreting behavior representations with fine-tuned LLMs complements this paper’s exploration of the language–behavior space relationship.Those studies project behavior representations into LLM token space and fine-tune LLMs to generate explanations.Source: §B.3 · p. 19

C.1 DATASETS

The appendices describe the datasets and baseline families used in the study. They also state the interaction-splitting and item-title choices and explain the baseline comparison setup.

C.1 DATASETS

  • The study uses six datasets: four from Amazon and two from other platforms.The Amazon datasets are Books, Movies & TV, Video Games, and Industrial & Scientific; the others are MovieLens-1M and Book Crossing.Source: §C.1 · p. 19, Table 6 · p. 20
  • Dataset preparation splits each user’s interactions 4:3:3, removes users with fewer than 20 interactions, and filters validation and test items absent from training.The paper uses item titles as its text description.Source: §C.1 · p. 20

C.2 BASELINES

  • General-recommendation baselines cover classical CF, contrastive-learning CF, and LM-enhanced CF; the LM-enhanced comparisons use XSimGCL as backbone.The authors omit LM-as-recommender baselines due to inference cost at millions of interactions and candidate-selection or next-item-prediction task limitations.Source: §C.2 · p. 20
  • RLMRec aligns semantic and collaborative representations with an additional contrastive loss, while the appendix also describes MF, MultVAE, LightGCN, SGL, BC Loss, XSimGCL, and KAR.The authors use the same item and user representations for KAR and XSimGCL to make comparisons relatively fair.Source: §C.2 · pp. 20–21

C.3 ABLATION STUDY

The ablations show that AlphaRec’s nonlinear mapping, contrastive loss, and graph convolution each contribute positively. Advanced language representations also outperform earlier BERT-style representations, with AlphaRec narrowing the performance gap between them.

C.3 ABLATION STUDY

  • Each AlphaRec component contributes positively: nonlinear mapping captures implicit preference similarities more effectively, while contrastive loss and graph convolution further enhance recommendation.Replacing the MLP with a linear matrix, InfoNCE with BPR, or removing graph convolution each degrades performance.Source: §C.3 · p. 21

C.4 EFFECT OF VARYING LANGUAGE REPRESENTATIONS IN ALPHAREC

  • Advanced language representations consistently outperform early BERT-style representations in AlphaRec.The model’s nonlinear transformations and neighborhood aggregation narrow the performance gap between language representations.Source: §C.4 · p. 21, Table 7 · p. 22

C.5 ABLATION STUDY ON THE MODIFICATION OF RLMREC

These appendices compare AlphaRec with alternative profile generation and recommendation approaches, examine its representations and training cost, and describe zero-shot evaluation settings. The reported results include strong zero-shot performance against LLMRank and continued performance on a newer dataset.

C.5 ABLATION STUDY ON THE MODIFICATION OF RLMREC

  • The modified RLMRec using AlphaRec’s item and user representations performs better than RLMRec with LLM-generated profiles, which can be unstable or degrade performance.The authors attribute the gap to possible noise and hallucination in generated profiles.Source: §C.5 · p. 22

C.6 PERFORMANCE ON AMAZON ELECTRONIC 2023

  • On Amazon 2023, AlphaRec consistently outperforms baselines using interactions after 2022 to reduce possible pretraining-data leakage.The authors attribute the outcome to language models understanding preferences behind item-title descriptions rather than simply remembering training data.Source: §C.6 · p. 22, Table 9 · p. 23

C.7 THE T-SNE VISUALIZATION COMPARISON

  • AlphaRec’s nonlinear mapping gathers obscure preference-similar movies that remain dispersed with a linear mapping, while preserving obvious clusters.The comparison visualizes language-space representations, linear-mapped representations, and AlphaRec representations.Source: §C.7 · p. 22, Figure 7 · p. 23

D.1 FAST CONVERGENCE SPEED

  • AlphaRec converges quickly and requires only a small amount of training time.Table 10 reports seconds per epoch and total training cost until convergence; Amazon-Mix combines Books, Movies & TV, and Video Games.Source: §D.1 · p. 23, Table 10 · p. 24

D.2.1 CO-TRAINING ON MULTIPLE DATASETS

  • For co-training, negative items are sampled from the same dataset as each positive item; other training procedures match single-dataset training.Source: §D.2.1 · p. 23

D.2.2 BASELINES

  • Zero-shot baselines adapt three sequential-recommendation methods—ZESRec, UniSRec, and VQ-Rec—to use their original model structures with the CF training paradigm.The appendix also describes Random and Pop baselines; Pop samples from the most popular items, where popularity is interaction count.Source: §D.2.2 · pp. 23–24

D.2.3 COMPARISON WITH LLMRANK

  • AlphaRec significantly surpasses LLMRank in zero-shot performance, with its improvement rising as the NDCG K decreases.Evaluation uses 19 negative items per positive item and NDCG on the candidate set.Source: §D.2.3 · p. 24, Table 11 · p. 24

D.2.4 THE EFFECT OF TRAINING DATASETS

The appendix examines how training data, intention queries, and intention strength relate to AlphaRec’s recommendation behavior. More training data and thematically similar source datasets generally support zero-shot performance, while co-training is generally comparable to single-dataset training.

D.2.4 THE EFFECT OF TRAINING DATASETS

  • Training on more datasets generally improves zero-shot performance, and source datasets with similar themes contribute more on untrained targets.Movies & TV and MovieLens-1M are given as an example of thematic similarity.Source: §D.2.4 · p. 24, Table 12 · p. 25
  • Co-training on three Amazon datasets generally yields performance similar to single-dataset training, except for some Video Games degradation attributed to the τ choice.The mixed-dataset setting uses τ = 0.15, while Video Games’ optimal τ is around 0.2; the authors say one AlphaRec can capture preferences across datasets.Source: §D.2.4 · p. 24, Table 13 · p. 25

D.3.1 INTENTION QUERY GENERATION

  • Each item receives a fixed natural-language intention query generated with ChatGPT using a Chain-of-Thought prompt and a rule-based quality check.Incorrect queries are regenerated; ChatGPT3.5 API is used for budget reasons.Source: §D.3.1 · p. 25

D.3.2 BASELINE

  • AlphaRec is compared with TEM, a personalized-search method, to assess its user-intention capture despite not being designed specifically for search tasks.TEM encodes the intention query with user history behaviors.Source: §D.3.2 · p. 25

D.3.3 CASE STUDY

  • In two case studies, AlphaRec places the target item at the top while maintaining general user preferences.Source: §D.3.3 · p. 25

D.3.4 EFFECT OF THE INTENTION STRENGTH ALPHA

  • A larger α gives more weight to the intention query and less to historical interests; Video Games shows a trend similar to MovieLens-1M.Source: §D.3.4 · p. 25

E HYPERPARAMETER SETTINGS AND IMPLEMENTATION DETAILS

The implementation appendix records experimental settings and provides tables for hyperparameter search, AlphaRec hyperparameters, and language-representation extraction costs. It also includes an example figure for item-query generation.

E HYPERPARAMETER SETTINGS AND IMPLEMENTATION DETAILS

  • Experiments use PyTorch, one NVIDIA RTX A5000 GPU, an AMD EPYC 7543 CPU, Adam, and early stopping after 20 evaluations without Recall@20 improvement.ID-based CF methods use two graph-propagation layers, embedding size 64, and 256 sampled negatives.Source: E HYPERPARAMETER SETTINGS AND… · p. 25

Intention Query Generation

  • Figure 8 illustrates item-query generation, while Table 16 reports the cost of extracting language representations on Amazon Movie & TV.Source: Figure 8 · p. 26, Table 16 · p. 27

E HYPERPARAMETER SETTINGS AND IMPLEMENTATION DETAILS

Related papers

Loading 2407.05441v4…