Source-linked AI summary
Large Language Models are Zero-Shot Rankers for Recommender Systems
Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, Wayne Xin Zhao
TL;DR
This paper investigates whether LLMs can rank recommendation candidates without task-specific training, using sequential user histories as conditions. It formalizes recommendation as conditional ranking and evaluates prompting and bootstrapping strategies, finding promising zero-shot ranking abilities alongside order-perception and prompt-bias challenges. These issues can be alleviated by the proposed strategies, particularly when candidates come from multiple candidate generators.
Problem
Existing recommender models are often domain-specific and may not capture explicit interests or broader knowledge, motivating study of LLMs as zero-shot rankers.
Method
The paper prompts LLMs to rank retrieved candidate items conditional on sequential interaction histories, using designed prompts and bootstrapping strategies.
Results
LLMs show promising zero-shot ranking abilities, but struggle with interaction order and suffer from position and popularity biases; these issues can be alleviated with designed prompting and bootstrapping.
Takeaways & Limitations
Zero-shot LLMs can challenge existing zero-shot recommendation methods, especially when ranking candidates retrieved by multiple candidate generators.
Takeaways & Limitations
Most experiments use closed-source ChatGPT, and observations may be biased by specific prompts and datasets.
Abstract
from arXiv · showhide
Recently, large language models (LLMs) (e.g., GPT-4) have demonstrated impressive general-purpose task-solving abilities, including the potential to approach recommendation tasks. Along this line of research, this work aims to investigate the capacity of LLMs that act as the ranking model for recommender systems. We first formalize the recommendation problem as a conditional ranking task, considering sequential interaction histories as conditions and the items retrieved by other candidate generation models as candidates. To solve the ranking task by LLMs, we carefully design the prompting template and conduct extensive experiments on two widely-used datasets. We show that LLMs have promising zero-shot ranking abilities but (1) struggle to perceive the order of historical interactions, and (2) can be biased by popularity or item positions in the prompts. We demonstrate that these issues can be alleviated using specially designed prompting and bootstrapping strategies. Equipped with these insights, zero-shot LLMs can even challenge conventional recommendation models when ranking candidates are retrieved by multiple candidate generators. The code and processed datasets are available at https://github.com/RUCAIBox/LLMRank.
1 Introduction
This work studies LLMs as zero-shot ranking models within recommender-system pipelines, where sequential histories condition ranking over retrieved candidates. Experiments identify order-perception, position, and popularity biases, while prompting and bootstrapping strategies improve these weaknesses.
- Motivation: Existing recommender models often rely on domain-specific behavior data, limiting their ability to capture explicit interests and broader knowledge.The introduction highlights clicked-item histories and “narrow experts” as limitations of conventional models.
- Research focus: The paper formalizes recommendation as conditional ranking: sequential interaction histories provide conditions, while retrieved items form the candidate set.The study focuses on ranking because LLMs are expensive to run over large candidate sets and ranking performance depends on retrieved items.
- Key findings: LLMs struggle to perceive the order of sequential interaction histories, but designed prompting can trigger order perception and improve ranking performance.The paper investigates prompting strategies that emphasize sequential structure and recent interactions.
- Key findings: LLM rankers suffer from position bias and popularity bias, which can be alleviated through bootstrapping or specially designed prompting strategies.These biases arise while ranking candidates arranged in prompts and are addressed by the proposed strategies.
- Key findings: LLMs outperform existing zero-shot recommendation methods, especially when candidates are retrieved by multiple candidate-generation models with different practical strategies.The findings indicate promising zero-shot ranking ability in a more challenging multi-generator candidate setting.
2 General Framework for LLMs as Rankers
The framework adapts LLMs to recommendation by prompting them to rank retrieved items conditional on a user’s historical interactions. It represents histories and candidates as natural-language patterns, explores order-aware prompting, and parses the resulting rankings back onto the candidate set.
- Conditional ranking formulation: The framework gives an LLM a chronological interaction history and retrieved candidate items, asking it to rank items of interest higher.Candidate sets are much smaller than the full item set, and each item is associated with descriptive text.
- Prompt construction: Two natural-language patterns for histories and candidates are filled into an instruction template that requests a ranked recommendation list.The final prompt combines sequential history, retrieved candidates, and an instruction describing the ranking objective.
- History prompting: The framework tests sequential, recency-focused, and in-context-learning prompts to help LLMs represent the order of historical interactions.Recency-focused prompts explicitly emphasize the most recently interacted item, while in-context examples augment the input sequence.
- History prompting: Demonstration examples are created from prefixes of the input interaction sequence and their successors rather than from other users.This design avoids introducing noise from users with different preferences.
- Candidate handling: The method ranks 20 candidates and varies their prompt order to examine position bias and apply bootstrapping for mitigation.Candidate items have no specific order under the classic candidate-generation setup, so alternative arrangements enable controlled testing.
- Output processing: Heuristic substring matching grounds natural-language LLM outputs on the specified candidate set, while out-of-candidate items are treated as errors or reprocessed.For GPT-3.5, such deviations occur in 3% of cases.
3 Empirical Studies
The empirical studies evaluate LLM ranking on two recommendation datasets and find strong zero-shot performance alongside weaknesses in sequential-order perception, position bias, and popularity bias. Designed prompting and bootstrapping strategies improve these behaviors, while performance depends on candidate-set composition.
- Experimental Setup: Experiments use MovieLens-1M and Games, filtering users and items with fewer than five interactions and representing items by titles.Leave-one-out evaluation uses NDCG@K over candidate sets, with the last interaction as the test ground truth.
- Sequential Historical Behaviors: LLMs perform similarly with chronological and randomly shuffled histories, indicating limited sensitivity to the order of historical interactions.Compared variants use one ground-truth item with 19 randomly sampled negatives; fake histories perform worse than the original histories.
- Sequential Historical Behaviors: Increasing history length can hurt ranking because LLMs may treat behaviors equally and become overwhelmed; shorter histories help them focus on recent interactions.The study varies the number of latest behaviors from 5 to 50.
- Sequential Historical Behaviors: Recency-focused prompting and in-context learning can trigger order perception and improve ranking performance.These strategies are introduced specifically to address the weakness of default prompting.
- Biases and Debiasing: LLMs exhibit position and popularity biases: candidate placement affects rankings, popular items receive higher positions, and bootstrapping or history-focused prompting alleviates these biases.Bootstrapping repeatedly ranks randomly shuffled candidates, while reducing history length lowers the popularity score of top-ranked items.
- Zero-Shot Ranking Performance: LLM ranking performance varies across datasets and candidate sets, reflecting use of popularity, text features, and user behaviors.The LLM-based method is affected by interaction-based hard negatives, but less severely than SASRec.
- Zero-Shot Ranking Performance: The LLM-based ranker achieves promising zero-shot results, including second-best performance on most metrics in Table 4 and better performance than Pop and BPRMF there.The strongest results occur especially when candidates are retrieved by multiple candidate-generation models.
4 Related Work
Related work covers transfer learning and language-model-based recommender systems. It highlights that prior methods use sparse cross-scenario anchors or fine-tuning, while few studies examine what determines LLM recommendation performance.
- Transfer Learning for Recommender Systems: Transfer-learning methods for recommender systems draw on other domains, markets, or platforms, typically using shared users, shared items, or shared representation spaces.These anchors are often sparse across scenarios.
- Large Language Models for Recommender Systems: Language-model-inspired recommender systems evolved from word2vec and neural networks toward PLMs and LLMs with stronger language understanding and generation abilities.Researchers have integrated LLMs with conventional recommenders and fine-tuned them with specialized instructions.
- Large Language Models for Recommender Systems: Although early studies report some zero-shot recommendation ability, few works investigate what determines LLM recommendation performance.This gap motivates empirical analysis of LLMs as ranking models.
5 Conclusion
The conclusion presents prompting, bootstrapping, and bias-focused strategies for using LLMs as zero-shot recommender rankers. It reports promising ranking ability while identifying order perception and bias reduction as continuing challenges.
- Contributions: The study constructs prompts containing historical interactions, candidates, and instruction templates for LLM-based ranking.It also proposes prompting strategies to trigger perception of sequential behavior order.
- Contributions: Bootstrapping and prompting strategies are introduced to alleviate position bias and popularity bias in LLM-based ranking.These methods target biases arising during candidate ranking.
- Conclusion and Future Work: Empirical studies indicate that LLMs have promising zero-shot ranking abilities and motivate better order perception, bias alleviation, and efficient downstream tuning.The authors propose future work on technical solutions and efficiently tuned personalized recommendation models.
6 Limitations
The evaluation relies primarily on ChatGPT, whose closed-source nature and performance gap over open-source alternatives complicate interpretation. Findings may also depend on the chosen prompts and datasets.
- ChatGPT is the primary target LLM in most experiments, but its closed-source commercial service may include additional performance-enhancing techniques.
- Open-source models such as LLaMA 2 and Mistral show a notable performance disparity compared with ChatGPT, complicating evaluation using only open-source models.
- The reported observations might be biased by the specific prompts and datasets used.