Source-linked AI summary
Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences
Scott Sanner, Krisztian Balog, Filip Radlinski, Ben Wedin, Lucas Dixon
TL;DR
The paper asks whether language-based preferences can support recommendation as an alternative to item-history methods, particularly when user data are scarce. It collects a dataset pairing language and item preferences with recommendation feedback, then compares prompted LLMs with item-based collaborative filtering. Zero-shot and few-shot LLMs perform competitively for pure language-based preferences in near cold-start settings, while the resulting representations are explainable and scrutable.
Problem
Controlled evidence comparing natural-language preference descriptions with traditional item-based approaches is limited, motivating evaluation of language-based recommendation against item-based collaborative filtering.
Method
The authors collect a dataset containing natural-language preference descriptions, item preferences, and ratings, then compare prompted LLM recommendation with collaborative filtering methods.
Results
Zero-shot and few-shot LLMs provide competitive recommendation performance for pure language-based preferences in near cold-start settings compared with item-based collaborative filtering methods.
Takeaways & Limitations
Language-based preference representations are explainable and scrutable, making competitive near-cold-start recommendation possible without task-specific supervised training.
Takeaways & Limitations
The conclusions are based on preferences from a relatively small pool of 153 raters.
Abstract
from arXiv · showhide
Traditional recommender systems leverage users' item preference history to recommend novel content that users may like. However, modern dialog interfaces that allow users to express language-based preferences offer a fundamentally different modality for preference input. Inspired by recent successes of prompting paradigms for large language models (LLMs), we study their use for making recommendations from both item-based and language-based preferences in comparison to state-of-the-art item-based collaborative filtering (CF) methods. To support this investigation, we collect a new dataset consisting of both item-based and language-based preferences elicited from users along with their ratings on a variety of (biased) recommended items and (unbiased) random items. Among numerous experimental results, we find that LLMs provide competitive recommendation performance for pure language-based preferences (no item preferences) in the near cold-start case in comparison to item-based CF methods, despite having no supervised training for this specific task (zero-shot) or only a few labels (few-shot). This is particularly promising as language-based preference representations are more explainable and scrutable than item-based or vector-based representations.
1 INTRODUCTION
This study examines whether LLM prompting can recommend items from natural-language preferences, especially near cold-start, compared with item-based collaborative filtering. It introduces a dataset and experimental framework for evaluating language- and item-based preferences, finding competitive performance from pure language-based descriptions.
- Controlled quantitative comparisons between natural-language preference descriptions and traditional item-based approaches have been limited.
- The study asks how effective LLM prompting is for recommendations from natural-language preferences compared with collaborative filtering based solely on item ratings.
- The experiments compare completion, instruction, and few-shot LLM prompts with text-based and item-based baselines, including collaborative filtering methods.
- The authors build a test collection containing natural-language preference descriptions and item ratings to study language-based recommendation.
- The study evaluates whether language can replace items in near-cold-start recommendation, whether combining both improves performance, and whether dispreferences help.
- Pure language-based LLM recommendation provides competitive near-cold-start performance with an explainable and scrutable preference representation.
2 RELATED WORK
Prior work spans item-based collaborative filtering and language-based recommendation, but had not directly compared prompting-based LLM recommendations from natural-language preferences with item-based approaches. Existing datasets and studies also lacked detailed parallel language preferences paired with ratings on unseen items.
- Item-based recommendation: Traditional recommendation methods use item ratings, ranging from nearest-neighbor and matrix-factorization models to jointly learned user–item embeddings.These approaches exploit user–item interaction histories to represent preferences and generate recommendations.
- Natural language in recommendation: Recent work explores natural-language preferences, tags, narratives, item descriptions, and reviews as inputs for recommendation.Approaches differ in whether language remains scrutable or is converted into an inscrutable user representation.
- Language models for recommendation: Prior language-model recommendation studies include theoretical proposals, generated preference narratives, pretrained BERT, custom-trained models, and item-rating prompting approaches.BERT was shown to outperform information-retrieval baselines for language-based recommendation from descriptions.
- Research gap: Earlier work did not directly assess the relative performance of language-based versus item-based recommendation from actual natural-language preferences.The present study addresses this gap with a unified prompting framework for large language models.
- Research gap: Existing datasets generally lack detailed natural-language preference descriptions paired with ratings of unseen items.The authors describe their dataset as filling this gap for comparative recommendation experiments.
- Prompting in large language models: Prompting research covers few-shot methods using example input-output pairs and zero-shot methods in which models receive task instructions without examples.These prompting paradigms motivate testing LLM recommendation without task-specific supervised training or with limited labels.
3 EXPERIMENTAL SETUP
The experiment collects consistent item-based and language-based preferences from the same raters, then evaluates recommendations using both biased algorithmic pools and unbiased random items. A two-phase movie study prioritizes fair comparison and near-cold-start conditions over dataset scale.
- Study design: The study collects both item-based and language-based preferences from the same raters to compare their recommendation utility consistently.The design responds to the lack of parallel corpora containing both preference types.
- Phase 1: Preference Elicitation: Raters first described liked and disliked movies in free-form text, selected five liked and five disliked examples, and then wrote final descriptions based on those selections.Initial descriptions required at least 150 characters; final descriptions were collected after item ratings.
- Recommendation pools: SP-EASE uses top-10 EASE collaborative-filtering recommendations with λ=5000.0, while SP-BM25-Fusion ranks items by the highest BM25 score among their reviews.BM25-Fusion matches rater natural-language preferences against item reviews.
- Evaluation: Random popular and mid-popular pools are unbiased because they use no user information, whereas personalized recommendation pools can contain preference-related bias.Performance on the merged Unbiased Set is central to the study’s conclusions.
- Study design: The controlled elicitation protocol limits data per rater and prioritizes a realistic near-cold-start amount of preference information over data scale.Manual collection effort also constrained the number of recruited raters, making the protocol a primary contribution.
4 METHODS
The paper compares traditional item- and language-based recommenders with LLM methods using items, language, or both. LLM prompts rank each candidate item by the likelihood assigned to its target-item suffix.
- Baselines and evaluated inputs: The evaluation compares collaborative-filtering baselines, a language-based BM25-Fusion baseline, random ordering, and LLM methods using items, language, or both.Item-based baselines include MostPopular, Item-kNN, WR-MF, BPR-SLIM, and EASE.
- LLM prompting: The LLM experiments use a 62-billion-parameter PaLM variant trained over 1.4 trillion tokens.
- LLM prompting: Prompts combine a prefix with the target movie name as a suffix, and the model’s suffix log likelihood ranks the 40 candidate items.
- Prompting strategies: Completion, zero-shot, and few-shot templates are defined for item-only, language-only, and combined Language+Items preferences.
- Prompting strategies: Zero-shot variants additionally test negative item and language preferences alongside positive preferences.
5 RESULTS
Language-based preferences perform competitively with item-based collaborative filtering in the near cold-start setting, while combining language and item preferences provides little apparent benefit. Few-shot prompting generally performs best among the tested LLM prompting strategies.
- Data collection: 153 raters supplied preference descriptions, item preferences, and ratings for personalized recommendation pools.
- Rating analysis: Seen items received an average rating of 4.29, compared with a neutral 3.00 for unseen items, reflecting positive bias toward previously seen movies.
- Research questions: LLM Language Few-shot (3) is competitive with most traditional item-based CF methods, while Item+Language LLM methods show no performance boost.
- Research questions: Pretrained LLM methods can perform comparatively well against CF methods using item-only preferences, despite lacking the large supervised datasets used to train CF.
- Prompting strategies: Few-shot prompting generally outperforms zero-shot and completion prompting, but adding more few-shot examples does not improve performance.
- Preference polarity: Combining positive and negative preferences yields no meaningful improvement, while negative-only preferences perform at or below the popularity baseline.
6 ETHICAL CONSIDERATIONS
The paper identifies ethical and scope limitations involving recommendation bias, study scale, and the English-only preference data. It also notes that possible bias cannot be ruled out and requires larger studies to bound.
- Future ethical analysis: The authors identify measuring unintended bias in language-driven versus classic recommenders as an important direction for future study.
- Bias and evaluation scope: The fixed-corpus ranking design scores all items, but possible recommendation bias cannot be ruled out and would require larger-scale studies to bound.
- Study population: The study’s conclusions rely on a relatively small pool of 153 raters, limiting the evidence base.
- Study population: Because preferences were collected only in English from a restricted sample, the results may not generalize to other languages or cultures.
7 CONCLUSION
The paper introduces a dataset for comparing recommendation from item-based and language-based preferences, using LLM prompting strategies and ratings of independent recommendations. LLMs achieve competitive performance for pure language-based preferences in near cold-start settings, while preserving explainable and scrutable preference representations.
- The authors collected item-based and language-based preferences from raters, together with ratings of an independent set of item recommendations.The dataset supports comparisons across pure item-based, pure language-based, and combined preferences.
- Prompting strategies in large language models enabled quantitative comparisons of recommendation efficacy across preference modalities.The comparisons included pure item-based, pure language-based, and combined preferences.
- Zero-shot and few-shot LLM strategies were remarkably competitive for pure language-based preferences in the near cold-start case against item-based collaborative filtering methods.These results concern language-based preferences without item preferences.
- General-purpose LLMs performed competitively with fully supervised item-based CF methods when using either item-based or language-based preferences.
- The approach provides a competitive near cold-start recommender based on an explainable and scrutable language-based preference representation.The paper presents this representation as a path toward effective and novel LLM-based recommenders using language-based preferences.