PaperlayerView exact revision on arXiv

Source-linked AI summary

Is ChatGPT a Good Recommender? A Preliminary Study

Junling Liu, Chao Liu, Peilin Zhou, Renjie Lv, Kang Zhou, Yan Zhang

arXiv:2304.10149v3cs.IR

TL;DR

Traditional recommendation methods are often task-specific, while ChatGPT’s effectiveness in recommendation remains under-explored. The paper evaluates ChatGPT as a prompt-driven, general-purpose recommender across five tasks without fine-tuning, finding uneven accuracy but strong human-rated explainability. It also identifies positional bias as a practical limitation for direct recommendation.

  • Problem

    Most traditional recommendation methods are task-specific, and ChatGPT’s ability to perform classical recommendation tasks has not been thoroughly investigated.

  • Method

    The paper uses ChatGPT as a general-purpose recommendation model, applying prompts and few-shot interaction information across five recommendation tasks without fine-tuning.

  • Results

    ChatGPT performs well in rating prediction, poorly in sequential and direct recommendation, and outperforms state-of-the-art methods in human evaluations of explainable recommendation tasks.

  • Takeaways & Limitations

    The benchmark provides insights into ChatGPT’s strengths and limitations and a starting point for exploring large language models in recommendation systems.

  • Takeaways & Limitations

    Direct recommendation is constrained by positional bias in the item pool, with item ordering substantially affecting evaluation results.

Abstract

from arXiv · show

Recommendation systems have witnessed significant advancements and have been widely used over the past decades. However, most traditional recommendation methods are task-specific and therefore lack efficient generalization ability. Recently, the emergence of ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. Nonetheless, the application of ChatGPT in the recommendation domain has not been thoroughly investigated. In this paper, we employ ChatGPT as a general-purpose recommendation model to explore its potential for transferring extensive linguistic and world knowledge acquired from large-scale corpora to recommendation scenarios. Specifically, we design a set of prompts and evaluate ChatGPT's performance on five recommendation scenarios. Unlike traditional recommendation methods, we do not fine-tune ChatGPT during the entire evaluation process, relying only on the prompts themselves to convert recommendation tasks into natural language tasks. Further, we explore the use of few-shot prompting to inject interaction information that contains user potential interest to help ChatGPT better understand user needs and interests. Comprehensive experimental results on Amazon Beauty dataset show that ChatGPT has achieved promising results in certain tasks and is capable of reaching the baseline level in others. We conduct human evaluations on two explainability-oriented tasks to more accurately evaluate the quality of contents generated by different models. And the human evaluations show ChatGPT can truly understand the provided information and generate clearer and more reasonable results. We hope that our study can inspire researchers to further explore the potential of language models like ChatGPT to improve recommendation performance and contribute to the advancement of the recommendation systems field.

1 INTRODUCTION

The paper addresses the limited generalization of task-specific recommendation methods by evaluating ChatGPT as a general-purpose recommender. Across five tasks, ChatGPT performs unevenly: strong in rating prediction, weak in sequential and direct recommendation, but superior in human evaluation for explainability-oriented tasks.

  • Traditional recommendation methods are typically task-specific, requiring separate data and models for different tasks or application scenarios.
  • The paper benchmarks ChatGPT as a general-purpose recommendation model across rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization.The evaluation uses prompts without fine-tuning and also investigates few-shot prompting to inject user-interaction information.
  • ChatGPT performs well in rating prediction but poorly in sequential and direct recommendation, reaching early-baseline levels on some metrics.
  • Human evaluations show ChatGPT outperforms state-of-the-art methods on explanation generation and review summarization despite poor objective metrics.The result suggests objective metrics may not accurately reflect its explainable recommendation capabilities.

2 RELATED WORK

The related work establishes language models as adaptable tools for recommendation and positions this study as a self-contained evaluation of ChatGPT on classic recommendation tasks. Unlike conversational interfaces that augment existing recommenders, the paper treats ChatGPT itself as the recommendation system.

  • Large language models use extensive linguistic and world knowledge from large-scale corpora to improve downstream NLP tasks and motivate their use in recommendation.
  • Prior recommendation research includes prompt-based task reformulation, shared language-generation frameworks such as P5, and foundation models such as M6-Rec.
  • Chat-REC uses ChatGPT as a conversational interface that augments existing recommender models and makes recommendation more interactive and explainable.
  • This work instead evaluates ChatGPT as a self-contained recommendation system without external systems across classic tasks including sequential recommendation and rating prediction.

3 RECOMMENDATION WITH CHATGPT

ChatGPT recommendation uses task-specific prompts to translate recommendation problems into natural-language inputs, optionally injecting user behavior through few-shot examples. Its outputs are then checked and refined to satisfy task formats.

  • Workflow: The workflow constructs task-specific prompts, sends them to ChatGPT, and checks and refines its generated recommendation outputs.The refinement module returns compliant outputs directly, corrects them with predefined rules, or requests re-recommendation when correction fails.
  • Prompt construction: Each prompt combines a task description, behavior injection for user-item interaction information, and a format indicator constraining output structure.Behavior injection supports few-shot prompting, while format indicators make results more comprehensible and assessable.
  • Rating prediction: Rating prediction prompts ask ChatGPT to estimate a user’s rating for a target product, with examples including exact numeric or formatted outputs.The paper describes two prompt types and provides examples for the Beauty dataset.
  • Sequential recommendation: Sequential recommendation prompts cover direct next-item prediction, candidate selection, and judging whether a specified item is next.These formats use interaction history to formulate alternative next-item recommendation tasks.
  • Direct recommendation: Direct recommendation uses an item-selection prompt to choose the most appropriate product from candidate items based on explicit feedback.The task is framed around ratings or reviews rather than only implicit behavior.
  • Explainability-oriented tasks: Explainability-oriented prompts ask ChatGPT to generate textual explanations or summaries, with auxiliary information such as hint words and star ratings optionally included.The paper provides zero-shot and few-shot prompt examples for these tasks.
  • Output refinement: ChatGPT’s response randomness can produce different outputs for identical inputs, complicating evaluation despite format indicators and refinement procedures.The paper identifies randomness as a practical issue in recommendation use.
  • Output refinement: Similarity-based text matching maps ChatGPT predictions back to the original dataset during correction, but may not perfectly reflect ChatGPT’s ability.The authors describe this mapping as an indirect demonstration of potential in sequential recommendation.

4 EVALUATION

The evaluation compares ChatGPT with representative state-of-the-art baselines and uses ablation studies across tasks. It also examines how few-shot prompting and human evaluation affect assessment of recommendation outputs.

  • Research questions: The experiments compare ChatGPT with state-of-the-art baseline models to assess its performance across recommendation tasks.This comparison is posed as the first research question.
  • Research questions: Ablation studies examine the impact of few-shot prompting on recommendation performance.The study explicitly asks how few-shot prompting changes results.
  • Research questions: Human evaluation is designed to assess explanation generation and review summarization beyond numerical task metrics.This is the third stated research question.

4.1 Experimental Setup

The study evaluates ChatGPT on the Beauty category of a real-world Amazon review dataset using numerical metrics for rating and recommendation tasks. It conducts zero-shot and few-shot experiments using item titles and interaction histories.

  • Dataset: The Amazon dataset contains customer review text and metadata for 29 product categories, while this paper focuses on Beauty.The evaluation uses the Beauty category as its experimental setting.
  • Metrics: Rating prediction is evaluated with Root Mean Square Error (RMSE) and Mean Absolute Error (MAE).These metrics measure rating-prediction performance.
  • Metrics: Sequential and direct recommendation are evaluated with top-k Hit Ratio and top-k Normalized Discounted Cumulative Gain.The reported settings include HR@{1,5,10} and NCGG@{5,10} as written in the passage.
  • Experimental protocol: Using gpt-3.5-turbo, the study runs zero-shot and few-shot experiments across five tasks without task-specific training data.Few-shot settings provide k historical-record examples, while experiments collect n interacted items and use item titles as metadata.

4.2 Baselines for multiple tasks

The evaluation uses task-matched baselines spanning rating prediction, direct and sequential recommendation, explanation generation, and review summarization. Rating prediction is separately identified in a performance-comparison table.

  • Rating prediction: Rating prediction uses MF and MLP as baselines, both evaluated with mean square root loss.The corresponding results are presented in a performance-comparison table for rating prediction.
  • Direct recommendation: Direct recommendation uses BPR-MF, BPR-MLP, and SimpleX as comparison methods.These baselines represent direct recommendation approaches in the evaluation.
  • Sequential recommendation: Sequential recommendation is compared against Caser, HGN, GRU4Rec, BERT4Rec, FDSA, SASRec, and S3-Rec.The listed methods cover recurrent, attention-based, and other sequential recommendation approaches.
  • Explainability-oriented tasks: Explanation generation uses Att and PETER as baselines, while review summarization uses pretrained T0 and GPT-2.These baselines are matched to the two explainability-oriented tasks.

4.3 Performance Comparison on 5 Tasks (RQ1&2)

Across five recommendation tasks, ChatGPT’s performance varies substantially: it is strongest for rating prediction and human-rated explainability, but generally trails classical or supervised baselines on direct and sequential recommendation.

  • Rating prediction: Few-shot prompts outperform MF and MLP on both MAE and RMSE for seen-category rating prediction.These results support using conditional text generation for rating prediction.
  • Sequential recommendation: ChatGPT performs considerably worse than baselines in zero-shot sequential recommendation, while few-shot prompting improves performance but remains inferior to classical methods in most cases.NDCG@5 surpasses GRU4Rec under few-shot prompting, but the broader comparison still favors classical sequential recommenders.
  • Direct recommendation: Few-shot prompting improves direct recommendation by adding historical preferences, but ChatGPT still fails to surpass supervised recommendation baselines.Zero-shot prompting performs significantly worse because insufficient information limits the model’s ability to capture user interests.
  • Direct recommendation: Item-pool construction strongly affects direct recommendation, with placing the ground-truth item first producing metrics approximately ten times higher than shuffling the pool.This result indicates a positional bias toward items near the beginning of the prompt’s item pool.
  • Explanation generation: P5 achieves better objective metrics than ChatGPT for explanation generation, whereas ChatGPT’s interaction-oriented design produces more diverse and creative text.Human evaluation is introduced to measure generated-content quality beyond objective metrics.
  • Review summarization: ChatGPT lacks an advantage in review-summarization metrics but generates more effective and meaningful summaries by understanding information across the entire review.This motivates human evaluation for the review-summarization task.

4.4 Human Evaluation (RQ3)

Human evaluations compare ChatGPT, baseline, and ground-truth outputs for explanation generation and review summarization. Despite weaker objective-metric performance, ChatGPT receives clearer, more reasonable, and more accurate human assessments.

  • Evaluation Setup: Human evaluations rank outputs from ChatGPT, a baseline, and ground truth using sampled explanation and summarization prompts.Evaluators assess generated outputs, with avg_top1_ration measuring first-place rankings and avg_position measuring average rank.
  • Explanation Generation: Annotators show some subjectivity but reach relatively consistent judgments about explanation quality.The explanation-generation evaluation uses four manual annotators.
  • Explanation Generation: ChatGPT produces clearer and more reasonable explanations than the ground truth, while P5 generates generic and less fluent explanations.The comparison is also illustrated through example explanation results.
  • Review Summarization: ChatGPT accurately summarizes reviews and extracts useful information, whereas P5 produces overly general summaries based on limited keywords.The review-summarization comparison is presented in example results and human evaluation results.
  • Review Summarization: ChatGPT achieves the best review-summarization performance, surpassing both ground truth and P5 by a large margin.All annotators unanimously agree on this result.

5 CONCLUSION AND FUTURE WORK

The paper benchmarks ChatGPT as a self-contained recommender and finds uneven performance across recommendation tasks. It performs well for rating prediction and human-rated explainability, but poorly for sequential and direct recommendation, motivating further work on interaction data and semantic alignment.

  • Conclusion: The benchmark compares ChatGPT with traditional recommendation models across recommendation tasks.The paper presents this benchmark as a source of insights into ChatGPT’s strengths and limitations.
  • Conclusion: ChatGPT performs well in rating prediction but poorly in sequential and direct recommendation tasks.The conclusion states that these results indicate a need for further exploration and improvement.
  • Conclusion: ChatGPT outperforms state-of-the-art methods in human evaluations of explainable recommendation tasks despite its limitations.The reported potential concerns generating explanations and summaries.
  • Future Work: Future work will investigate incorporating user interaction data into language models and bridging the semantic gap between language and user interests.These directions are presented as the paper’s next steps.
Loading 2304.10149v3…