Source-linked AI summary
Uncovering ChatGPT's Capabilities in Recommender Systems
Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, Jun Xu
TL;DR
Recommendation capabilities and limitations of ChatGPT remain insufficiently understood. The paper reformulates point-wise, pair-wise, and list-wise ranking as domain-specific prompts and evaluates them across four datasets. ChatGPT outperforms other LLMs across the ranking policies, while list-wise ranking offers the best cost-performance trade-off and shows potential for cold-start and explainable recommendation.
Problem
Existing evidence leaves ChatGPT’s recommendation capabilities and limitations unclear, including whether LLMs can function as few-shot or zero-shot recommenders.
Method
The paper reformulates point-wise, pair-wise, and list-wise ranking into domain-specific prompts and evaluates ChatGPT and other LLMs on four recommendation benchmarks.
Results
ChatGPT outperforms other LLMs across all three ranking policies and shows consistent advantages in 22 of 24 comparisons.
Takeaways & Limitations
List-wise ranking provides the best cost-performance trade-off, while ChatGPT shows potential for mitigating cold start and supporting explainable recommendation.
Takeaways & Limitations
The evaluation treats some semantically meaningful ChatGPT answers as non-compliant because existing methods require valid answers to match the candidate set.
Abstract
from arXiv · showhide
The debut of ChatGPT has recently attracted the attention of the natural language processing (NLP) community and beyond. Existing studies have demonstrated that ChatGPT shows significant improvement in a range of downstream NLP tasks, but the capabilities and limitations of ChatGPT in terms of recommendations remain unclear. In this study, we aim to conduct an empirical analysis of ChatGPT's recommendation ability from an Information Retrieval (IR) perspective, including point-wise, pair-wise, and list-wise ranking. To achieve this goal, we re-formulate the above three recommendation policies into a domain-specific prompt format. Through extensive experiments on four datasets from different domains, we demonstrate that ChatGPT outperforms other large language models across all three ranking policies. Based on the analysis of unit cost improvements, we identify that ChatGPT with list-wise ranking achieves the best trade-off between cost and performance compared to point-wise and pair-wise ranking. Moreover, ChatGPT shows the potential for mitigating the cold start problem and explainable recommendation. To facilitate further explorations in this area, the full code and detailed original results are open-sourced at https://github.com/rainym00d/LLM4RS.
1 INTRODUCTION
The paper investigates whether LLMs can support recommendation by aligning them with point-wise, pair-wise, and list-wise ranking. It reformulates these capabilities as domain-specific prompts and evaluates ChatGPT and other LLMs across four recommendation benchmarks.
- LLMs can potentially serve as few-shot or zero-shot recommenders, but their recommendation capabilities require further investigation.
- The study examines point-wise, pair-wise, and list-wise ranking perspectives for constructing personalized top-K recommendation lists.
- The authors reformulate the three ranking capabilities into domain-specific prompts and evaluate ChatGPT and other OpenAI LLMs on four recommendation benchmarks.
- List-wise ranking is recommended for practice because its performance improvement relative to cost offers the best trade-off among the three ranking strategies.
- ChatGPT shows potential for mitigating cold start, explainable recommendation, and understanding item similarity.
2 BACKGROUND
Prior work established that language models can support recommendation through task reformulation and prompting. This paper therefore evaluates ChatGPT’s potential and limitations as an off-the-shelf recommender.
- Large language model research has shown that models can perform diverse tasks without gradient updates, using textual instructions or a few examples.
- Recommendation research has used language-model training strategies to reformulate tasks and learn representations for recommendation.
- The paper conducts a preliminary evaluation of ChatGPT’s potential and limitations in recommender systems.
3 PROBING CHATGPT FOR RECOMMENDATION CAPABILITIES
The paper adapts point-wise, pair-wise, and list-wise recommendation into domain-specific prompts for off-the-shelf LLMs. Prompts combine task descriptions, demonstration examples, and a new user query, after which valid outputs are evaluated.
- Point-wise ranking predicts an individual preference score for each candidate item and then ranks items by those scores.
- Pair-wise ranking compares two candidate items for a user, predicts the preferred item, and ranks items using relative preference scores.
- List-wise ranking predicts scores for a complete candidate list and sorts the items according to those scores.
- Prompt adaptation: The method expresses the three ranking capabilities as tasks with domain-specific prompts for off-the-shelf LLMs.
- Prompt adaptation: Each prompt contains a domain-aware task description, few-shot demonstration examples, and a new user query with candidate items to rank.
- Prompt adaptation: Point-wise, pair-wise, and list-wise prompts use one item, an item pair, and a candidate-item list, respectively.
- Evaluation procedure: LLMs generate predictions from the elicited ranking capabilities, and manually checked valid answers are retained for evaluation while invalid answers are excluded.
4 EXPERIMENTS
The experiments evaluate ChatGPT and GPT-3.5 models across ranking capabilities and recommendation domains, compare them with collaborative filtering methods, measure cost, and test prompt-shot effects.
- Research questions: The experiments evaluate ChatGPT and GPT-3.5 models on different ranking capabilities across various recommendation domains.
- Research questions: The study compares LLM-based recommenders with traditional collaborative filtering methods.
- Research questions: The experiments measure the cost of LLM-based recommenders under different ranking capabilities.
- Research questions: The study examines how the number of prompt shots affects LLM-based recommender performance.
4.1 Experimental Settings
The study evaluates LLM-based recommendation across four domains using datasets, binary feedback processing, and standardized candidate lists and ranking metrics.
- Datasets: Evaluations cover movie, book, music, and news recommendation using four established datasets from different domains.The movie benchmark is MovieLens-1M2; books and music use Amazon subsets, while news uses MIND-small.
- Data processing: Ratings above 3 are labeled positive for movies, books, and music, while the News dataset retains its original binary feedback labels.Item titles serve as descriptions in the prompts.
- Evaluation protocol: Each dataset contributes 500 randomly sampled evaluation records because LLM evaluation is expensive.The evaluation uses one positive item and four randomly sampled negative items as candidates.
- Prompt settings: The experiments use one shot for pair-wise and list-wise ranking and two shots for point-wise ranking.These settings follow existing practice for in-context recommendation evaluation.
- Metrics: Performance is reported with NDCG@3, MRR@3, and Compliance Rate, which measures the proportion of valid answers among test samples.Compliance Rate addresses outputs that are not included in the candidate set.
4.2 RQ1: Overall Performance
Across the evaluated domains and ranking settings, ChatGPT generally outperforms comparison LLMs and conventional baselines, while performance varies by ranking policy and domain.
- Baseline comparison: 24.71% average improvement over random recommendation was observed for the point-wise task on the Movie dataset in NDCG@3.The study also reports that LLMs performed better than random recommendation in almost all cases and generally produced compliant answers.
- LLM comparisons: ChatGPT outperformed text-davinci-002 and text-davinci-003 in 22 out of 24 comparisons across two ranking metrics, three ranking capabilities, and four domains.The two exceptions occurred for point-wise ranking in the news domain against text-davinci-003.
- Ranking capabilities: ChatGPT performed better with list-wise ranking except in the movie domain, while the other compared LLMs generally performed better with pair-wise ranking.Pair-wise ranking exceeded point-wise ranking in 11 out of 12 cases but required more inference cost.
- Domain effects: LLM-based recommenders outperformed popularity-based recommendation for movies, books, and music but underperformed it in the news domain.The paper relates this domain difference to news popularity, rapid delivery, and limited interaction data in LLM training corpora.
- Zero-shot evaluation: In zero-shot evaluation, more than 50% of original-prompt cases were invalid, so logit_bias was used to control output tokens.Detailed zero-shot results are provided externally because of page limitations.
- Few-shot evaluation: Few-shot LLM settings generally outperformed zero-shot settings, and LLMs exceeded random and popularity-based policies in the zero-shot setting.These results support the potential of LLMs as recommendation systems and the usefulness of few-shot prompting.
4.3 RQ2: Comparison with Collaborative Filtering Models
The study compares LLM-based recommenders with traditional collaborative-filtering models under limited training data and concludes that LLM-based models can help mitigate cold start.
- The comparison evaluates Matrix Factorization and Neural Collaborative Filtering after training on varying proportions of data.The models were parameter-tuned for a fair comparison with LLMs.
- Figure 3 reports NDCG@3 improvement per unit cost across four datasets and compares list-wise, point-wise, and pair-wise ranking costs.The figure labels the costs as 1x, 5x, and 10x for list-wise, point-wise, and pair-wise ranking, respectively.
- LLM-based recommendation models use a few prompt samples rather than training data to understand the recommendation task.
- The authors conclude that LLM-based recommendation models can be applied in practice to mitigate the cold start problem.
4.4 RQ3: Performance Scaling by Cost
The paper evaluates recommendation gains relative to prompt cost across ranking strategies. List-wise ranking generally provides the best improvement per unit cost and is recommended for practice because it combines decent performance with lower cost.
- The analysis calculates each LLM’s improvement per unit cost to account for performance gains and computational expense.
- List-wise ranking needs one prompt for all candidate items, whereas point-wise needs N prompts and pair-wise needs N(N−1)/2 prompts.
- With N=5 candidates, point-wise, pair-wise, and list-wise ranking cost 5x, 10x, and 1x, respectively.
- Almost all three LLMs achieve their best improvement per unit cost with list-wise ranking, except text-davinci-002 on the Book dataset.
- Although pair-wise ranking may have better absolute metrics than point-wise ranking, its multiple prompts create additional cost.
- The authors recommend list-wise ranking in practice because it offers decent performance at lower cost.
4.5 RQ4: Performance Under Different Shots Examples
The paper studies how varying the number of in-context examples affects recommendation performance on the Movie dataset. More examples do not consistently produce the best results.
- Experiments vary the number of examples M from 1 to 5 and evaluate ChatGPT and GPT3.5s using NDCG@3.
- The best recommendation results do not always correspond to the maximum number of example shots.
- Additional examples can provide more context for understanding the recommendation task but may also introduce drawbacks.
4.6 Case Study of Exceptions
The case study examines exceptional ChatGPT outputs under pair-wise ranking. Some answers judged non-compliant nevertheless reflect item relevance and similarity judgments.
- LLMs can generate invalid answers under few-shot prompting, causing compliance rates below 100%.
- Two ChatGPT pair-wise cases lack a correct answer because both options are negative samples.
- In one example, science-fiction viewing histories are paired with cartoon answer choices, which ChatGPT identifies as unrelated to the history.
- The authors interpret these responses as evidence that ChatGPT can use interaction histories and item similarity when recommending.
- Because the evaluation marks such responses non-compliant, the paper suggests evaluating LLM recommenders beyond learning to rank.
5 CONCLUSION
The paper evaluates off-the-shelf LLMs for recommendation using point-wise, pair-wise, and list-wise ranking across four datasets. ChatGPT performs best overall, shows promise for cold-start and explainable recommendation, and is generally stronger at list-wise and pair-wise than point-wise ranking.
- ChatGPT outperforms other LLMs in recommendation across all three ranking capabilities and four datasets.
- LLMs excel at list-wise and pair-wise ranking but are not proficient at point-wise ranking in most cases.
- ChatGPT shows potential for mitigating the cold start problem and supporting explainable recommendation.