PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Rec: Personalized Recommendation via Prompting Large Language Models

Hanjia Lyu, Song Jiang, Hanqing Zeng, Yinglong Xia, Qifan Wang, Si Zhang, Ren Chen, Christopher Leung, Jiajie Tang, Jiebo Luo

arXiv:2307.15780v3cs.CLcs.AIcs.IR

TL;DR

Text-based recommendation is challenged by incomplete item descriptions that limit alignment with user preferences. LLM-REC uses four prompting strategies to augment descriptions and combine the resulting text with original inputs. Experiments show improved recommendation quality, with simple MLP models reaching comparable or superior performance to more complex content-based methods.

  • Problem

    Incomplete item descriptions may omit characteristics needed to align recommendations with user preferences.

  • Method

    LLM-REC uses four prompting strategies to generate augmented item text for a recommendation module.

  • Results

    LLM-REC improves recommendation performance and enables simple MLP models to achieve comparable or superior results to more complex content-based methods.

  • Takeaways & Limitations

    Combining augmented and original text can provide additional context for capturing nuanced item information and aligning recommendations with user preferences.

  • Takeaways & Limitations

    LLM-REC incurs extra computational cost during augmentation, including cost from the length of generated output text.

Abstract

from arXiv · show

Text-based recommendation holds a wide range of practical applications due to its versatility, as textual descriptions can represent nearly any type of item. However, directly employing the original item descriptions may not yield optimal recommendation performance due to the lack of comprehensive information to align with user preferences. Recent advances in large language models (LLMs) have showcased their remarkable ability to harness commonsense knowledge and reasoning. In this study, we introduce a novel approach, coined LLM-Rec, which incorporates four distinct prompting strategies of text enrichment for improving personalized text-based recommendations. Our empirical experiments reveal that using LLM-augmented text significantly enhances recommendation quality. Even basic MLP (Multi-Layer Perceptron) models achieve comparable or even better results than complex content-based methods. Notably, the success of LLM-Rec lies in its prompting strategies, which effectively tap into the language model's comprehension of both general and specific item characteristics. This highlights the importance of employing diverse prompts and input augmentation techniques to boost the recommendation effectiveness of LLMs.

1 Introduction

Text-based recommendation applies across domains but is limited by incomplete item descriptions that may not capture characteristics relevant to user preferences. LLM-REC addresses this gap by enriching descriptions with LLM-generated information.

  • Text-based recommendation spans products, movies, books, news, short videos, and social media posts.
  • Incomplete item descriptions hinder alignment between item characteristics and user preferences.
  • Movie descriptions may omit genre, tone, cinematography style, and thematic depth, reducing recommendation effectiveness.
  • User-generated recipe descriptions may omit dietary information and terms such as “vegan” or “plant-based.”
  • LLM-REC uses prompting strategies to enrich input text by leveraging LLM knowledge, reasoning, and task-solving capabilities.
  • LLM-REC achieves comparable or superior performance to advanced content-based approaches using augmented text.

2 Related Work

Prior work enriches text-based recommendation through contextual cues, inferred attributes, and external knowledge, while LLM-REC uses LLM knowledge and reasoning to generate augmented item text.

  • LLM-REC aligns with both text augmentation for recommendation and LLM-based recommendation research.
  • Text-based recommendation research addresses incomplete information through reviews, linguistic attributes, and external knowledge bases.
  • Unlike approaches that predict interactions from final prompts, LLM-REC generates augmented input text capturing item characteristics and nuances.

3 LLM-Rec

LLM-REC enriches item descriptions through four prompting strategies and feeds concatenated augmented text into a recommendation module. The prompts incorporate description content, recommendation goals, and user engagement.

  • LLM-Rec: LLM-REC investigates basic, recommendation-driven, engagement-guided, and combined recommendation-driven plus engagement-guided prompting.
  • LLM-Rec: The enriched texts are concatenated and supplied to the final recommendation module.
  • Basic Prompting: Basic prompting paraphrases descriptions, summarizes them with tags, or infers broader emotional characteristics.
  • Recommendation-driven Prompting: Recommendation-driven prompting adds instructions to generate content or tags useful for recommending an item.
  • Engagement-guided Prompting: Engagement-guided prompting combines a target item description with T important neighbor descriptions selected by user engagement.
  • Recommendation-driven + Engagement-guided Prompting: The combined strategy incorporates recommendation-driven and engagement-guided instructions in one prompt.
  • How does LLM-REC affect personalized recommendation?: LLM-REC is described as versatile across categorically structured and novel, unclassified items, while also improving transparency through inspectable augmented text.

4 Experiments

Experiments evaluate LLM-REC on movie and recipe recommendation using multiple language models, baselines, and top-K metrics. Augmented text improves performance, and analyses examine which generated information contributes most.

  • Experiment Setup: Experiments use Movielens-1M and Recipe datasets, content-based baselines, text-augmentation baselines, GPT-3, and LLAMA-2.
  • Experiment Setup: Precision@10, Recall@10, and NDCG@10 are averaged across five test-set splits using an MLP and dot-product recommendation module.
  • Main Results: 6.24% to 8.54% NDCG@10 relative gains occur on Movielens-1M, while Recipe gains range from 8.97% to 21.72% over MLP.
  • Main Results: LLM-REC enables simple MLP models to achieve comparable or superior performance to more complex content-based methods.
  • Understanding Augmented Information: Augmented content provides more detailed and expressive descriptions of item characteristics than original descriptions.
  • Understanding Augmented Information: Recommendation-driven augmentation adds genre, descriptive, viewer-oriented, and recipe-relevant characteristics such as “easy to make.”
  • Understanding Augmented Information: Masking words unique to recommendation-driven responses reduces performance, whereas selecting vital keywords can outperform retaining all response words.

LLM-REC augmentation outperforms other text augmentation methods for recommenda-

LLM-REC improves recommendation by augmenting item descriptions with strategically prompted LLM text and integrating multiple enriched representations. Across analyses, performance depends on prompt objectives and how augmented information is combined.

  • Comparisons: Knowledge-based augmentation improves recommendations for well-classified items but is limited for novel, dynamic user-generated content.The reported limitation is linked to potential mismatch between generated knowledge and the target item.
  • Comparisons: LLM-REC's prompting strategies outperform TagGPT while improving recommendation performance over baseline methods.The study reports gains for tag generation over baseline methods, but finds LLM-REC prompts more effective than TagGPT's prompts.
  • Prompting strategies: Recommendation performance consistently improves when original descriptions are combined with augmented text from multiple prompting strategies.The ablation compares original descriptions alone with combinations of augmented text across two benchmark datasets.
  • Prompting strategies: In Movielens-1M, combined recommendation-driven and engagement-guided prompting performs best, whereas recommendation-driven prompting alone is strongest for Recipe.These domain differences show that combining multiple objectives does not always yield the best result.
  • Information integration: Concatenating embeddings from more augmented information consistently improves performance, with Concat-All combining outputs from all four prompting strategies.The compared variants include Basic, Concat-Basic, Concat-Rec, and Concat-All.
  • Information integration: With an unchanged text encoder, converting each response into an embedding before concatenation outperforms concatenating the responses before encoding.The comparison indicates that the improvement is not attributable merely to increased embedding size.
  • Response consistency: Prompt wording changes produce largely similar responses, while lower cosine similarity mainly reflects differences in response format.The paper notes that fine-tuning or prompt instructions may mitigate these formatting differences.

5 Discussions and Conclusions

The paper presents LLM-REC as a flexible framework that enriches incomplete item descriptions with LLM-generated text for personalized recommendation. Experiments indicate that combining augmented and original text improves recommendation quality and supports simpler training with content-based backbones.

  • Contributions: LLM-REC is designed as input-text augmentation suitable for content-based backbone recommendation models.This distinguishes it from approaches designed around a specific recommendation model.
  • Contributions: LLM-REC uses LLMs to augment incomplete item descriptions and improve personalized recommendation.The framework is presented as a simple yet impactful mechanism for recommendation through LLMs.
  • Findings: Combining augmented input text with original item descriptions yields notable improvements in recommendation quality.The conclusion attributes the improvement to incorporating additional context into the recommendation input.

Limitations

LLM-REC has additional computational cost during augmentation and faces difficulty incorporating the latest knowledge. Future work targets selecting useful generated words and retrieving current external knowledge.

  • Limitations: LLM-REC incurs extra computational cost from the augmentation phase and the length of augmented output text.The paper identifies this as its first limitation.
  • Limitations: Selecting important words instead of all response words can improve recommendation performance, but the trade-off remains for future study.The authors plan to examine the balance between generated-word count and performance gains.
  • Limitations: LLM-REC, like many LLM-based studies, has difficulty promptly incorporating the latest knowledge.Future work will investigate gathering and summarizing current knowledge from external sources.

A.1 Datasets

The study evaluates LLM-Rec across movie and recipe recommendation settings, using augmented text, comparison baselines, frozen text embeddings, and user-item interaction training. It also uses graph-based neighbor selection for engagement-guided prompting.

  • Datasets: MovieLens-1M contains 1,000,209 ratings from 6,040 users covering 3,900 movies, while Recipe contains recipe details and reviews from Food.com.MovieLens ratings are converted into implicit feedback; Recipe provides descriptions, ingredients, directions, ratings, and reviews.
  • Datasets: GPT-3 generates MovieLens item descriptions with a one-sentence prompt that excludes the movie title and uses temperature 0.The generated response becomes the item description for evaluation.
  • Datasets: The benchmarks contrast categorically organized movies with diverse, user-generated recipes that are less strictly structured and more novel.Their descriptions also differ substantially in content and form.
  • Baselines: LLM-Rec is compared with original-description baselines ranging from MLP to AutoInt, DCN-V2, and EDCN to measure the value of augmented text.The comparison focuses on whether augmented input improves recommendation outcomes.
  • Baselines: The study also compares LLM-Rec with KAR and TagGPT, while excluding collaborative-filtering experiments because the focus is incomplete item descriptions.LLM-Rec emphasizes LLM common-sense reasoning and does not require domain expertise.
  • Model training: Sentence-BERT converts original and augmented text into frozen embeddings, while user IDs are mapped to latent representations and trained with binary cross-entropy.Negative samples pair users with items lacking recorded interactions, and early stopping uses a five-step window and evaluation frequency.
  • Engagement-guided prompting: Engagement-guided prompting selects the top 3 item neighbors by Personalized PageRank scores from a user-item bipartite graph.For MovieLens-1M, selected neighbors share the target item's genre.

B.1 Additional Discussions on Augmented Text

The additional analysis shows that prompting strategies enrich descriptions with general and specific item characteristics, but their outputs depend on neighbor composition and model choice. The combined strategy exhibits characteristics of both component strategies, while 7B LLAMA-2-CHAT performs poorly relative to GPT-3.

  • Recommendation-driven prompting: Recommendation-driven prompting adds themes and contextual descriptors absent from basic prompting, making item descriptions more detailed and expressive.Examples include “Supernatural,” “Paranormal,” “Psychological Thriller,” and “Troubled Child.”
  • Engagement-guided prompting: Engagement-guided prompting can resemble recommendation-driven prompting when neighbor and target descriptions are similar, or derive additional information from neighbor descriptions.Which behavior occurs depends on the composition of important neighbor items.
  • Limitations: The neighbor composition depends on the sampling method, whose deeper investigation is outside the study's scope.The authors leave more detailed exploration for future research.
  • Combined prompting: The recommendation-driven plus engagement-guided strategy generates text with characteristics of both sub-strategies, but their quantitative combination remains unresolved.The paper presents examples of this combined behavior without determining how the two components contribute numerically.
  • Language-model comparison: 7B LLAMA-2-CHAT performs worse than GPT-3 in the analyzed examples and sometimes generates no content.The paper also reports this issue in Appendix D.

B.2 Additional Discussions on Applicable Item Domains and Available Textual Information

The study tests whether LLM-Rec can enrich item descriptions across ten Amazon domains, including domains with sparse textual information. It finds broader, more expressive descriptions and reports improved adjective use as a proxy for added information.

  • Cross-domain evaluation: The domain analysis uses item descriptions from ten Amazon review domains, sampling 50 items per domain and prompting GPT-3.The analysis targets applicability across domains and text lengths.
  • Evaluation measures: The study uses adjective-count increases as a proxy for added expressive information and total word count to approximate augmentation volume.No single metric directly quantifies the added information.
  • Results: LLM-Rec enriches descriptions across multiple domains, including domains without rich textual content.The analysis specifically includes sparse-text settings.
  • Results: Movie and digital music descriptions average 20.34 and 30.18 words, respectively, and LLM-Rec increases their expressiveness through greater adjective use.These averages illustrate the sparse textual inputs examined in the cross-domain analysis.

B.3 Additional Experiments on Applying LLM-Rec to Other Baselines

Additional experiments show that LLM-Rec generally improves recommendation performance across other text-based recommendation systems. The analysis also finds that embedding separate text segments can outperform concatenating them before encoding, while prompt wording produces comparable outputs and dynamic prompting introduces computational costs.

  • Other recommendation baselines: LLM-Rec can be adapted to various text-based recommendation systems and generally improves performance over using original text.The results are reported in Table 10.
  • Embedding variants: Concat-All encodes individual text segments before concatenating embeddings, whereas Text Concatenation encodes the combined text as one embedding.Both methods use a frozen text encoder with fixed output dimensions.
  • Embedding variants: Concat-All performs better because concatenating long text before encoding increases information loss for a frozen encoder.The encoder may not retain crucial information from longer inputs when it is not fine-tuned for the data's nuances.
  • Prompt wording: The study tests alternative wording for each LLM-Rec prompt and compares generated responses using cosine similarity on 50 sampled MovieLens-1M items.The variants preserve the original prompts' intended meaning while changing word choice.
  • Dynamic prompting: Dynamic prompts incorporate recently interacted items to generate descriptions aligned with users' current interests, but each interaction requires real-time processing.This processing can be resource-intensive in large-scale systems with many users and items.
  • Dynamic prompting: Caching, efficient generation, and incremental prompt updates are proposed to reduce the computational cost of dynamic prompting.Incremental updates avoid regenerating prompts completely after every interaction.
  • Dynamic prompting: Dynamic prompting remains a promising personalization approach despite implementation challenges.The authors frame it as a possible way to better accommodate individual user needs.

C Extended Related Work

Prior text-based recommendation work enriches item descriptions with external knowledge, while LLM-based recommendation studies often use language models directly as recommenders. LLM-Rec instead uses diverse prompting strategies to augment item text, with experiments showing benefits that vary by dataset, model, and embedding design.

  • Augmentation in Text-based Recommendation: Prior approaches enriched item descriptions with external databases containing movie information such as actors, directors, genres, and categories.
  • LLM for Recommendation: LLM-based recommendation studies construct task-specific prompts containing recommendation tasks, user profiles, item attributes, and user-item interactions.
  • LLM for Recommendation: LLM-Rec differs by using LLMs to generate augmented item text rather than using them directly as recommender models.The paper identifies memorization of training samples as a potential evaluation challenge for direct LLM recommenders.
  • Empirical Examples: GPT-3 and LLAMA-2 can enrich item descriptions with supplementary information, although 7B LLAMA-2-CHAT performs comparatively worse, possibly because of limited parameter scale.
  • Embedding Design: Concatenating embeddings of LLM-augmented responses outperforms duplicate original-description embeddings and raw-text concatenation, with gains also shown across text-based recommendation modules.
  • Prompting Strategies: Ablations indicate that LLM-generated text augmentation improves recommendation performance, while improvement magnitude depends on dataset characteristics and prompting strategy.The basic strategies include paraphrase, tag, and infer variants, alongside recommendation-driven and engagement-guided prompting.
Loading 2307.15780v3…