PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction

Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, Derek Zhiyuan Cheng

arXiv:2305.06474v1cs.IRcs.LG

TL;DR

The paper addresses the absence of comprehensive evaluations comparing LLMs of varying sizes with strong recommendation baselines. It evaluates LLMs for user rating prediction and finds that zero- and few-shot models lag behind supervised methods, while fine-tuning can largely close the gap and offer data-efficiency benefits.

  • Problem

    Comprehensive evidence comparing LLMs of varying sizes with carefully optimized, strong recommendation baselines is lacking.

  • Method

    The paper evaluates LLMs as recommendation systems for user rating prediction and fine-tunes them using human interaction data.

  • Results

    Zero-shot and few-shot LLMs fall behind fully supervised methods, whereas fine-tuned LLMs can largely close the gap with carefully designed baselines in key metrics.

  • Takeaways & Limitations

    LLM-based recommenders may provide better data efficiency and simpler feature processing by converting information into prompts.

  • Takeaways & Limitations

    Certain LLMs may not always follow the provided instruction, sometimes offering additional reasoning or other output.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated exceptional capabilities in generalizing to new tasks in a zero-shot or few-shot manner. However, the extent to which LLMs can comprehend user preferences based on their previous behavior remains an emerging and still unclear research question. Traditionally, Collaborative Filtering (CF) has been the most effective method for these tasks, predominantly relying on the extensive volume of rating data. In contrast, LLMs typically demand considerably less data while maintaining an exhaustive world knowledge about each item, such as movies or products. In this paper, we conduct a thorough examination of both CF and LLMs within the classic task of user rating prediction, which involves predicting a user's rating for a candidate item based on their past ratings. We investigate various LLMs in different sizes, ranging from 250M to 540B parameters and evaluate their performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct comprehensive analysis to compare between LLMs and strong CF methods, and find that zero-shot LLMs lag behind traditional recommender models that have the access to user interaction data, indicating the importance of user interaction data. However, through fine-tuning, LLMs achieve comparable or even better performance with only a small fraction of the training data, demonstrating their potential through data efficiency.

1 INTRODUCTION

The paper examines whether LLMs can predict user ratings from prior behavior and compares them with strong recommender baselines across model sizes and training regimes. Zero-shot models trail interaction-based recommenders, while fine-tuning reaches comparable or better performance with a small fraction of the training data.

  • Research gap: The paper addresses the absence of a comprehensive evaluation of LLMs with varying sizes against carefully optimized recommendation baselines.
  • Study scope: The study evaluates LLMs from 250M to 540B parameters for user rating prediction under zero-shot, few-shot, and fine-tuning regimes.The models are compared with state-of-the-art recommendation models on two widely adopted recommendation benchmark datasets.
  • Zero-shot results: Models over 100B parameters provide reasonable zero-shot recommendations in the cold-start scenario, with performance comparable to heuristic baselines.The cited contribution describes this result for larger off-the-shelf models.
  • Zero-shot results: Zero-shot LLMs fall behind traditional recommender models that use human interaction data and are only comparable to trivial average-rating baselines.The comparison indicates the importance of user interaction data for rating prediction.
  • Fine-tuning results: Fine-tuned LLMs achieve comparable or better performance than traditional models using only a small fraction of the training data.The paper presents this result as evidence of data efficiency.

2 RELATED WORK

Related work applies language models to recommendation, ranking, retrieval, click-through-rate prediction, and zero-shot item prediction. These studies establish several formulations and capabilities but do not replace the paper’s broader comparison of LLM sizes and recommendation settings.

  • Language-model recommendation: Earlier work formulates recommendation as a natural-language task with BERT and GPT-2, but reports lower performance than well-tuned GRU4Rec baselines.
  • LLMs for recommendation: P5 uses T5 to unify ranking, retrieval, and summary explanation, while M6-Rec fine-tunes an LLM for click-through-rate prediction.
  • Zero-shot recommendation: ChatRec and another prompting approach investigate zero-shot recommendation, with ChatRec also supporting interaction and explanations.The other approach generates next-item recommendations on MovieLens and reports competitive metrics without beating strong baselines.
  • LLM capabilities: LLMs’ instruction following enables generalization to unseen tasks from natural-language descriptions, and instruction fine-tuning and RLHF improve this capability.

3 METHOD

The method formulates rating prediction from a user’s time-ordered rated-item history and candidate-item information, and evaluates prompting and fine-tuning approaches for LLMs. Fine-tuning uses either five-way classification or regression, while prompt-based methods require engineering to obtain numeric outputs.

  • 3.1 Problem formulation: Rating prediction uses a user’s time-ordered interaction history and candidate item to predict the user’s rating.Each historical interaction includes item information and the rating the user gave it.
  • 3.2 Zero-shot and Few-shot LLMs for Rating Prediction: Zero- and few-shot prompts represent past ratings and candidate title and genre as text, specify the rating scale, and parse the model’s response.The prompts are demonstrated on MovieLens and are designed to elicit a numeric rating.
  • 3.2 Zero-shot and Few-shot LLMs for Rating Prediction: LLMs can be highly sensitive to prompt wording and may add reasoning or fail to provide a numerical rating.The authors address this with instructions such as giving a single number without explanation.
  • 3.3 Fine-tuning LLMs for Rating Prediction: The method emphasizes training LLMs with human interaction data to improve recommender performance and understanding of user preference.The study focuses on publicly available Flan-T5 models with competitive benchmark performance.
  • 3.3 Fine-tuning LLMs for Rating Prediction: Classification projects decoder outputs to vocabulary logits, trains with cross-entropy, and predicts the class with the largest output probability.The projection matrix has shape (d, |V|), where d is the decoder hidden dimension and |V| is vocabulary size.

4 EXPERIMENTS

The experiments examine whether off-the-shelf LLMs recommend effectively, how they compare with traditional recommenders, and how model size and convergence affect their use for recommendation. These questions cover zero-shot and few-shot performance, fair comparisons, scaling, and training efficiency.

  • Research questions: The experiments ask whether off-the-shelf LLMs perform well for zero-shot and few-shot recommendation.
  • Research questions: The study compares LLMs with traditional recommender models in a fair evaluation setting.
  • Research questions: The experiments investigate how model size affects LLM performance when LLMs are used as recommenders.
  • Research questions: The study tests whether LLMs converge faster than traditional recommender models.

4.1 Datasets and Evaluation Setup

The evaluation uses MovieLens-1M and 5-core Amazon-Books ratings, with chronological single-time-point splits and rating and ranking metrics.

  • Datasets: Both datasets contain user review ratings on a 1-to-5 scale.
  • Datasets: The study evaluates user rating prediction on MovieLens-1M and 5-core Amazon-Books datasets.MovieLens-1M contains 1 million movie ratings, while Amazon-Books filters users and items with fewer than 5 interactions.
  • Training / Test Split: Ratings are chronologically ordered after removing items without metadata, then split into the first 90% for training and the remainder for testing.
  • Training / Test Split: Each training example includes a user ID, item ID, item metadata, and a 5-point Likert rating label.
  • Evaluation Protocol: Experiments use 2,000 randomly sampled test tuples and truncate user histories to the 10 most recent interactions.
  • Evaluation Metrics: RMSE and MAE measure rating prediction, while ROC-AUC evaluates ranking with ratings ≥4 treated as positives.

4.2 Baselines and LLMs

The comparison includes traditional recommenders, heuristics, a Transformer-MLP matched to the LLM inputs, and LLMs spanning multiple sizes and training settings.

  • Baselines: Traditional baselines include Matrix Factorization and Multi-layer Perceptrons using user and item IDs as input features.
  • Baselines: The authors construct a Transformer-MLP because no existing method was designed for historical metadata and rating inputs in this setting.The model processes the same input information supplied to the LLM.
  • Transformer-MLP: The Transformer-MLP embeds sparse features, aggregates user behaviors with Add or Concat operations, models sequences with bidirectional self-attention, and predicts ratings through an MLP.
  • Baselines: Heuristic baselines predict using global, candidate-item, or user-past average ratings.
  • LLMs: The study evaluates zero-shot and few-shot LLMs including GPT-3 models, ChatGPT, and 540B-parameter Flan-U-PaLM.
  • LLMs: Fine-tuning experiments use Flan-T5-Base (250M) and Flan-T5-XXL (11B) models with a fixed 50k-step training schedule.

4.3 Zero-Shot and Few-shot LLMs (RQ1)

Zero-shot and few-shot LLMs can produce reasonable rating predictions, but they generally underperform supervised recommenders trained with user-item interaction data.

  • Zero-Shot Results: Zero-shot LLMs produce reasonable ratings and outperform global-average predictions in most cases.
  • Zero-Shot Results: Zero-shot LLMs are comparable to item- or user-average baselines, with text-davinci-003 varying across MovieLens and Amazon-Books.It performs slightly worse than the candidate-item average on MovieLens but better on Amazon-Books.
  • Few-Shot Results: Few-shot prompting improves AUC over zero-shot prompting, while RMSE and MAE show no clear improvement pattern.The few-shot setting provides three examples in the prompt.
  • Comparison: Both zero-shot and few-shot LLMs underperform traditional recommendation models trained with interaction data.
  • Interpretation: The authors attribute inferior LLM performance partly to missing user-item interaction data in pre-training and dataset-specific rating statistics.

4.4 LLMs vs. Traditional Recommender Models (RQ2)

Fine-tuning substantially improves LLM performance, with regression objectives outperforming classification and Flan-T5-XXL exceeding the strongest Transformer-MLP baseline on MAE and AUC.

  • Fine-Tuning Results: Fine-tuned LLMs perform much better than zero-shot and few-shot LLMs by incorporating dataset statistics.
  • Objective Choice: Fine-tuning Flan-T5-Base with classification loss performs worse than regression loss on RMSE, MAE, and AUC.
  • Comparison: Flan-T5-XXL achieves better MAE and AUC than the strongest Transformer-MLP baseline.The result suggests fine-tuned LLMs may be more suitable for ranking tasks.

4.5 Effect of Model Size (RQ3)

Model size affects zero-shot rating prediction, with only models above 100B parameters performing reasonably well. Fine-tuning also favors the larger Flan-T5-XXL model over Flan-T5-Base on both datasets.

  • 250M–500B-parameter LLMs generated rating predictions from zero-shot or few-shot prompts on a 1–5 scale.The experiments covered LLMs across different model sizes and used prompting to produce rating predictions.
  • Only LLMs larger than 100B parameters performed reasonably well for zero-shot rating prediction.The paper reports this pattern in Figure 1.
  • Flan-T5-XXL outperformed Flan-T5-Base on both datasets in the fine-tuning experiments.The comparison appears in the last two rows of Table 2.

4.6 Data Efficiency of LLMs (RQ4)

The paper compares LLM and traditional recommender convergence to assess data efficiency. LLMs require only a small fraction of the data for good AUC performance, while RMSE can look reasonable for both methods with limited data because average-rating predictions are strong baselines.

  • For RMSE, both LLMs and traditional methods reached reasonable performance with a small fraction of the data.The paper attributes this partly to the relatively low RMSE obtained by predicting average item ratings.
  • AUC makes the data-efficiency difference clearer because predicting the average rating produces an AUC of 0.5.The paper uses this baseline to distinguish the convergence trends.
  • LLMs required a small fraction of the data for good performance, whereas Transformer+MLP needed at least 1 more training data.

5 CONCLUSION

The paper evaluates LLMs for user rating prediction across zero-shot, few-shot, and fine-tuning settings against traditional recommenders. Zero-shot and few-shot LLMs fall behind supervised methods, while fine-tuning largely closes the gap and offers data-efficiency and modeling advantages.

  • The study evaluates LLM rating prediction in zero-shot, few-shot, and fine-tuning settings against traditional recommender methods.
  • Zero-shot and few-shot LLMs fall behind fully supervised methods, indicating the importance of incorporating the target dataset distribution.
  • Fine-tuned LLMs can largely close the gap with carefully designed baselines on key metrics.
  • LLM-based recommenders offer better data efficiency and simpler feature processing and modeling through prompt conversion.The approach avoids manually designing feature-processing strategies, embedding methods, and network architectures for different information types.
  • The paper identifies conversational recommendation as a potential application of LLM-based recommenders.
  • The authors plan to further improve LLM-based recommender performance.
Loading 2305.06474v1…