PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models as Zero-Shot Conversational Recommenders

Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, Julian McAuley

arXiv:2308.10053v1cs.IRcs.AI

TL;DR

Conversational recommendation has been less studied than action-based recommendation, motivating empirical evaluation of LLMs in this setting. The paper constructs a real-world dataset, revises evaluation to remove repeated-item shortcuts, and analyzes why zero-shot LLMs work. Across three datasets, zero-shot LLMs surpass fine-tuned CRS models, while content/context knowledge is especially important and popularity bias and geographic sensitivity remain limitations.

  • Problem

    Conversational recommendation remains comparatively immature, while existing LLM recommendation studies largely focus on traditional settings using users’ past actions.

  • Method

    The paper evaluates representative LLMs as zero-shot conversational recommenders across three datasets, including Reddit-Movie, and probes model knowledge and dataset information.

  • Results

    Zero-shot LLMs surpass fine-tuned existing CRS models after repeated-item shortcuts are addressed, with analyses highlighting content/context knowledge in their recommendations.

  • Takeaways & Limitations

    CRS evaluation should be re-benchmarked, datasets should diversify their sources and settings, and CRS should be distinguished from traditional recommendation through its content/context information.

  • Takeaways & Limitations

    LLM recommendations show popularity bias and sensitivity to geographical regions.

Abstract

from arXiv · show

In this paper, we present empirical studies on conversational recommendation tasks using representative large language models in a zero-shot setting with three primary contributions. (1) Data: To gain insights into model behavior in "in-the-wild" conversational recommendation scenarios, we construct a new dataset of recommendation-related conversations by scraping a popular discussion website. This is the largest public real-world conversational recommendation dataset to date. (2) Evaluation: On the new dataset and two existing conversational recommendation datasets, we observe that even without fine-tuning, large language models can outperform existing fine-tuned conversational recommendation models. (3) Analysis: We propose various probing tasks to investigate the mechanisms behind the remarkable performance of large language models in conversational recommendation. We analyze both the large language models' behaviors and the characteristics of the datasets, providing a holistic understanding of the models' effectiveness, limitations and suggesting directions for the design of future conversational recommenders

1 INTRODUCTION

This paper studies large language models as zero-shot conversational recommenders, addressing the limited attention to natural-language conversational recommendation. It contributes a real-world dataset, revised evaluation, and analyses of model knowledge and limitations.

  • Conversational recommender systems use multi-turn natural-language inputs to elicit preferences and provide personalized items and human-like responses.
  • Current LLM recommendation research largely evaluates traditional settings based on past user actions, while conversational recommendation remains comparatively immature.
  • Data: Reddit-Movie contains over 634k naturally occurring recommendation-seeking dialogs and is reported as the largest public conversational recommendation dataset.
  • Evaluation: Repeated ground-truth items can enable trivial copying baselines and produce spurious conclusions about conversational recommendation ability.
  • Evaluation: After removing repeated items, zero-shot LLMs outperform existing fine-tuned conversational recommendation models across ReDIAL, INSPIRED, and Reddit-Movie.
  • Analysis: The analyses find that LLMs rely primarily on content/context knowledge, while CRS datasets contain substantial such information; popularity bias and geographic sensitivity remain limitations.

2 LLMS AS ZERO-SHOT CRS

The framework prompts an LLM with a task description, output format, and conversation context, then post-processes generated item titles into ranked recommendations.

  • 2.1 Task Formation: A conversation consists of speakers, utterances, and mentioned-item sets over multiple turns.
  • 2.1 Task Formation: At the recommender’s kth turn, the conversational context from preceding turns is used to generate a ranked item list matching ground-truth items.
  • 2.2 Framework: Without fine-tuning, prompting combines a task description template, format requirement, and pre-kth-turn conversational context for the LLM.
  • 2.2 Framework: The experiments include GPT-3.5-turbo, GPT-4, BAIZE, and Vicuna, with decoding temperature set to 0 to encourage deterministic results.
  • 2.2 Framework: A post-processor such as fuzzy matching converts natural-language recommendation lists into ranked item lists, implementing generative retrieval.

3 DATASET

The paper introduces Reddit-Movie, a large-scale dataset of naturally occurring movie-recommendation conversations, complementing smaller crowd-sourced CRS datasets with richer real-world preference content.

  • Dataset Motivation: Reddit-Movie complements ReDIAL and INSPIRED by capturing real-world conversations rather than crowd workers role-playing users and recommenders.The authors use all three datasets for evaluation and analysis.
  • Dataset Construction: Reddit-Movie contains over 634k naturally occurring recommendation-seeking dialogs collected from Reddit.The dataset was constructed from five movie-related subreddits using conversation identification, movie mention recognition, and entity linking.
  • Dataset Construction: The dataset pipeline processes Reddit posts from January 2012 through December 2022 and uses the most recent 9k Reddit-Moviebase conversations for testing.The source posts come from r/movies, r/bestofnetflix, r/moviesuggestions, r/netflixbestof, and r/truefilm.
  • Dataset Statistics: Reddit-Movie covers 51,203 movies and has fewer multi-turn conversations than ReDIAL and INSPIRED.The shorter conversations reflect the inherent characteristics of Reddit posts.
  • Dataset Characteristics: Compared with existing datasets, Reddit-Movie conversations tend to contain more complex and detailed user preferences in their textual content.Existing datasets more often include users explicitly specifying preferences, whereas Reddit inputs provide richer textual information.

4 EVALUATION

The evaluation separates repeated from new recommendation items to expose a shortcut in existing CRS benchmarks, then compares representative CRS models with zero-shot LLMs across three datasets. After removing repeated items, LLMs achieve the strongest new-item performance, while GPT-based models generally lead open-source LLMs and produce mostly recognizable movie titles.

  • Evaluation Setup: The evaluation reports Recall@K with K={1,5} for representative CRS models and LLMs across the three datasets.Baseline results are re-run from author-provided training code, with means and standard errors reported.
  • Repeated Items Can Be Shortcuts: Current CRS evaluation can favor copying items already mentioned in the conversation, allowing a trivial baseline to outperform most previous models.This shortcut risks measuring repeated-item matching rather than genuine recommendation ability.
  • Repeated Items Can Be Shortcuts: Repeated items appeared in more than 15% of INSPIRED ground-truth items, motivating their removal from training and testing for re-evaluation.The revised evaluation focuses on new-item recommendation and re-trains the baselines.
  • LLMs Performance: Large language models achieve the best performance on new-item recommendation across ReDIAL, INSPIRED, and Reddit without fine-tuning.All models perform lower on Reddit, potentially because it contains more items and fewer conversation turns.
  • LLMs Performance: GPT-4 generally outperforms GPT-3.5-t, while Vicuna and BAIZE perform significantly below GPT-3.5-t despite being comparable to prior models on most datasets.The authors hypothesize that GPT-4 retains more correlations between movie names and user preferences from pre-training.
  • LLMs Performance: Around 95% of GPT-based generated recommendations can be string matched in IMDB, compared with around 81% for BAIZE and 87% for Vicuna.These matching rates are lower bounds indicating that only a few LLM recommendations are hallucinated movie titles.

5 DETAILED ANALYSIS

The analysis finds that LLMs rely primarily on content/context knowledge, while collaborative knowledge is weaker and dataset-dependent. It also identifies information-rich datasets, popularity bias, and geographical sensitivity as important factors shaping zero-shot CRS performance.

  • 5.1 Knowledge in LLMs: Content/context knowledge is primary: ItemOnly causes an average Recall@5 drop of more than 60%, while ItemRemoved or ItemRandom reduce GPT-based performance by less than 10% on average.The ablations indicate that GPT-based models retain much of their performance without item mentions, whereas removing conversational content is more damaging.
  • 5.1 Knowledge in LLMs: LLMs generally have weaker collaborative knowledge than existing CRS models, underperforming them by 30% on INSPIRED and ReDIAL with item-only context.Possible explanations include insufficient item-similarity information in pretraining and dataset- or platform-dependent interaction patterns.
  • 5.1 Knowledge in LLMs: On Reddit, LLMs outperform baselines in Recall@1 and Recall@5, plausibly because many items are rarely interacted with and title semantics help with cold-start items.Reddit contains 12,982 items mentioned no more than three times as responses, making collaborative ranking difficult.
  • 5.2 Information from CRS Data: Reddit contains more content/context information than ReDIAL and INSPIRED, aligning with the larger relative improvements achieved by LLMs on that dataset.Its information level is close to that of conversational search and question-answering datasets.
  • 5.2 Information from CRS Data: Collaborative information is insufficient for satisfactory recommendations, and its usefulness can depend on the dataset or platform.Pretraining on ML-25M beats random on linked Reddit data, but a gap remains versus Reddit fine-tuning and combined pretraining plus fine-tuning adds no further improvement.
  • 5.3 Limitations of LLMs as Zero-shot CRS: LLM recommendations show popularity bias and geographical sensitivity, concentrating more on popular items and performing better for movies from English-speaking regions.For example, The Shawshank Redemption appears around 5% of generated recommendations on ReDIAL and around 1.5% on Reddit.

6 RELATED WORK

Prior conversational recommender research includes traditional template- and critiquing-based systems, followed by deep CRS supporting natural-language interaction. Related LLM work emphasizes generalization, scaling, instruction tuning, and prompting for unseen tasks.

  • Conversational Recommendation: Traditional CRS includes template-based and critiquing-based systems, while later deep CRS supports interactions in natural language.
  • Large Language Models: LLMs exhibit strong generalization toward unseen tasks and domains, with larger parameter sizes improving downstream performance and instruction tuning enabling broader task generalization.

7 CONCLUSION AND DISCUSSION

The study finds that zero-shot LLMs are highly effective conversational recommenders, while also exposing evaluation shortcuts and limitations that motivate broader re-benchmarking and more diverse datasets.

  • Zero-shot LLMs surpass all fine-tuned existing CRS models in the reported experiments.
  • Previous standard CRS evaluations may produce unreliable conclusions because of a repeated-item shortcut.The study argues that more CRS models should be systematically re-benchmarked to clarify recommendation abilities and task characteristics.
  • The newly created discussion-website dataset provides more diverse and realistic conversations for CRS research.The authors describe it as the largest public CRS dataset.
  • LLMs show promise for CRS because they offer strong content and contextual knowledge without fine-tuning.The authors also identify efficient tuning and collaborative-filtering ensembling as opportunities for further improvement.
  • The analysis supports creating CRS datasets from diverse sources, domains, languages, and cultures.Suggested sources include crowd-sourcing platforms, discussion forums, and realistic CRS applications.
Loading 2308.10053v1…