PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)

Yashar Deldjoo, Zhankui He, Julian McAuley, Anton Korikov, Scott Sanner, Arnau Ramisa, René Vidal, Maheswaran Sathiamoorthy, Atoosa Kasirzadeh, Silvia Milano

arXiv:2404.00579v2cs.IRcs.AI

TL;DR

Generative recommender systems expand recommendation beyond traditional user-item histories to textual, visual, and multimodal data, but their evaluation remains complex because open-ended tasks and societal impacts are difficult to assess. This survey synthesizes interaction-driven, language-based, and multimodal approaches while identifying evaluation challenges and open research directions.

  • Problem

    Traditional recommender systems mainly rely on user-item histories, while Gen-RecSys introduces complex multimodal tasks whose performance, impact, and harm require broader evaluation.

  • Method

    The survey organizes Gen-RecSys research across interaction data, textual recommendation with LLMs, multimodal recommendation, and applications including prompting, fine-tuning, retrieval, and feature extraction.

  • Results

    The survey connects advances showing that pretrained LLMs support recommendation and explanation, and establishes a broad foundation spanning user-item interactions, text, and multimodal contexts.

  • Takeaways & Limitations

    Gen-RecSys research should evaluate not only system performance and capabilities but also fairness, privacy, and societal impact.

  • Takeaways & Limitations

    Multimodal recommendation remains constrained by the difficulty of combining modalities so shared information aligns while complementary information is retained.

Abstract

from arXiv · show

Traditional recommender systems (RS) typically use user-item rating histories as their main data source. However, deep generative models now have the capability to model and sample from complex data distributions, including user-item interactions, text, images, and videos, enabling novel recommendation tasks. This comprehensive, multidisciplinary survey connects key advancements in RS using Generative Models (Gen-RecSys), covering: interaction-driven generative models; the use of large language models (LLM) and textual data for natural language recommendation; and the integration of multimodal models for generating and processing images/videos in RS. Our work highlights necessary paradigms for evaluating the impact and harm of Gen-RecSys and identifies open challenges. This survey accompanies a tutorial presented at ACM KDD'24, with supporting materials provided at: https://encr.pw/vDhLq.

1 INTRODUCTION

Generative models broaden recommender systems beyond narrow, domain-specific preference modeling by handling interactions, text, and other modalities. This survey organizes Gen-RecSys methods by data modality, model paradigms, applications, and evaluation challenges.

  • Large language models support recommendation applications including personalization, conversational interfaces, and explanation generation through emergent reasoning and few-shot abilities.
  • Gen-RecSys models are applied either through direct training on user-item interactions or through pretrained models adapted with prompting, fine-tuning, retrieval, or feature extraction.
  • The survey covers a broader range of generative models and modalities than prior surveys focused mainly on LLMs, foundation models, or GANs.
  • Its framework classifies models by data and modality, including user-item data, text-driven recommendation, and multimodal recommendation.
  • The survey evaluates Gen-RecSys across benchmarks, performance, fairness, privacy, societal impact, harm, and conversational settings while identifying open research challenges.

2 GENERATIVE MODELS FOR INTERACTION-DRIVEN RECOMMENDATION

Interaction-driven recommendation uses only user-item interactions, yet supports diverse generative paradigms for producing recommendations and modeling user behavior. The survey reviews auto-encoding, autoregressive, adversarial, diffusion, and other generative models in this setting.

  • Interaction-driven recommendation uses user-item interactions as input and produces recommended item lists or grids without textual or visual information.
  • Auto-Encoding Models: Denoising auto-encoding models recover corrupted inputs and can support next-item prediction, as illustrated by BERT4Rec’s masked-item training.
  • Auto-Encoding Models: Variational autoencoders learn stochastic mappings to simpler distributions, from which decoders sample outputs for collaborative filtering, sequential recommendation, and slate generation.
  • Autoregressive Models: Autoregressive models learn conditional probabilities for sequences and are used in session-based, sequential, bundle, and model-attacking applications.
  • Self-Attentive Models: Self-attentive models improve long-term dependency handling and enable parallel training, supporting sequential, session-based, bundle, and next-basket recommendation.
  • Other Generative Models: GANs sample informative or negative items and synthesize user preferences or interactions, while diffusion models learn from corrupted interactions for prediction and sequence augmentation.

3 LARGE LANGUAGE MODELS IN RECOMMENDATION

Natural language provides a unified and interpretable medium for representing preferences, items, interactions, and recommendation tasks. The survey reviews how pretrained LLM abilities are adapted for recommendation and explanation.

  • Natural language can represent item features, user preferences, user-system interactions, recommendation tasks, and external knowledge in one expressive medium.
  • Pretrained LLMs have demonstrated reasoning abilities for making and explaining recommendations and facilitating conversational recommendation dialogues.
  • LLM-based recommendation methods use prompting, fine-tuning, retrieval, and external tools to augment pretrained abilities.
  • The survey covers encoder-only retrieval and cross-encoding, generative sequence-to-sequence recommendation, retrieval-augmented generation, feature extraction, and conversational recommendation.

3.1 Encoder-only LLM Recommendation

Encoder-only LLMs support recommendation through scalable dense retrieval and more interactive cross-encoder scoring. Cross-encoders model user-item interactions more richly but require substantially more computation.

  • Dense Retrieval: Dense retrieval ranks items by comparing separately encoded natural-language preference queries and item documents using embedding similarity.
  • Dense Retrieval: Separate query and document encoding enables scalable indexed search, particularly with approximate-search libraries such as FAISS.
  • Cross-Encoding: Cross-encoder architectures jointly embed item and preference descriptions and use an MLP head for rating prediction.
  • Cross-Encoding: Cross-encoders often perform strongly because user and item representations interact, but their computational cost makes them better suited to small item sets or reranking.

3.2 LLM-based Generative Recommendation

LLM-based generative recommendation expresses recommendation tasks as prompts and generates outputs such as item lists, ratings, or explanations. Research explores off-the-shelf, tuned, and prompt-tuned LLMs across recommendation and explanation tasks.

  • Task formulation: LLM-based recommendation converts tasks into prompts and generates item lists, ratings, or explanations as token sequences.These methods use seq2seq LLMs pretrained on large text corpora.
  • Off-the-shelf LLMs: Off-the-shelf LLM studies commonly prompt models with natural-language user preferences to recommend the next k items or predict ratings.Evaluations often target movies and books, which are prevalent in pretraining corpora.
  • Model adaptation: Fine-tuning uses natural-language interaction examples and task descriptions for rating prediction and sequential recommendation.Fine-tuning adds task-specific knowledge to the model’s internal parameters.
  • Model adaptation: P5 extends fine-tuned LLM recommendation to top-k recommendation, explanation generation, and review summarization.Prompt tuning instead adjusts continuous or soft prompt vectors without tuning internal LLM weights.
  • Generative explanation: Explanation-generation methods train on review-derived explanations using fine-tuning, prompt tuning, chain-of-thought prompting, or controllable decoding.Controllable decoding can use predicted parameters such as ratings to steer generation.

3.3 Retrieval Augmented Recommendation

Retrieval-augmented generation externalizes knowledge and conditions LLM outputs on retrieved information. In recommendation, retrieval commonly supplies candidate items for LLM reranking or review context for explanation generation.

  • RAG principles: RAG conditions LLM outputs on external information, enabling online updates, reducing hallucinations, and generally requiring fewer LLM parameters.This avoids retuning internal knowledge for every system update.
  • RAG for recommendation: Recommendation RAG commonly retrieves or constructs a candidate item set, then prompts an encoder-decoder LLM to rerank it.The candidate set can be based on a user query or interaction history.
  • RAG for recommendation: For RAG-based explanation generation, interaction history is used to generate queries that retrieve item reviews as context for explanations.The retrieved reviews provide explanatory evidence for the recommendation.

3.4 LLM-based Feature Extraction

LLMs can generate feature representations that serve as inputs to conventional or neural recommender systems. Examples use LLM-derived item or query embeddings and semantic IDs within downstream recommendation models.

  • LLM-derived features: LLM2-BERT4Rec initializes BERT4Rec item embeddings from item texts.The LLM supplies text-derived representations to the recommender.
  • LLM-derived features: Query-SeqRec feeds LLM query embeddings into a transformer-based recommender.The LLM-generated query representation becomes an input feature for recommendation.
  • LLM-derived features: TIGER embeds item text with an LLM, quantizes the embedding into a semantic ID, and trains a T5-based recommender.This pipeline converts textual semantics into discrete identifiers for recommendation.

3.5 Conversational Recommendation

LLMs enable fully natural-language, multi-turn recommendation dialogues and introduce dialogue history as interaction data. Conversational recommendation integrates recommendation with explanation, question answering, critiquing, preference elicitation, and dialogue management.

  • Conversational recommendation: LLM advances make fully natural-language system-user dialogues a feasible recommendation interface.Conversational recommendation studies multi-turn, multi-task, and mixed-initiative interactions.
  • Conversational recommendation: Dialogue history provides a rich new form of interaction data for conversational recommendation.This expands the information available beyond conventional user-item interactions.
  • Conversational elements: Conversational recommendation integrates dialogue management, recommendation, explanation, question answering, critiquing, and preference elicitation.These elements support diverse conversational recommendation functions.
  • System architectures: Some ConvRec systems use a monolithic LLM, whereas others combine an LLM dialogue layer with a recommender module.The latter architecture uses the LLM to facilitate natural-language dialogue while the recommender generates item recommendations.

4 GENERATIVE MULTIMODAL RECOMMENDATION SYSTEMS

Generative multimodal recommender systems address requests and outputs that require integrating text, images, and other modalities. The survey covers alignment methods, multimodal generative architectures, and challenges including data collection, representation fusion, and computational demands.

  • Motivation: Multimodal recommendation supports requests combining images, text, or audio and outputs such as virtual try-on or multimodal shopping assistance.
  • Motivation: Late fusion can be insufficient for cold-start recommendation because diverse item information may transfer preferences to new products or customers.
  • Challenges: Multimodal systems face incomplete annotations, difficulty preserving both shared and complementary information, and data requirements orders of magnitude above unimodal models.
  • Open directions: The survey identifies synthetic-data generation, stronger unimodal components, latent-space alignment, efficient training, and structured representations as advances toward multimodal generative recommendation.
  • Alignment: Contrastive approaches align modalities in shared embedding spaces, with CLIP using parallel encoders and ALBEF adding multimodal fusion and multiple pretraining objectives.
  • Generative approaches: Generative approaches include VAEs, diffusion models, and multimodal LLMs, which provide mechanisms for structured latent representations, image synthesis, and multimodal natural-language interaction.

5 EVALUATING FOR IMPACT AND HARM

Evaluating generative recommender systems requires assessing system performance, real-world effects, and societal harm rather than relying on a few model metrics. The survey reviews offline, online, conversational, efficiency, benchmark, and harm evaluation while calling for comprehensive frameworks.

  • Evaluation scope: Gen-RecSys evaluation is complicated by multi-component systems, open-ended tasks, limited benchmarks, and effects on users’ experiences, opinions, and actions.
  • Efficiency: Computational efficiency during training and inference is important because generative recommender models impose substantial computational burdens.
  • Benchmarks: Existing recommender benchmarks remain useful only narrowly for generative systems, motivating new benchmarks for tasks enabled by Gen-RecSys.
  • Performance evaluation: Offline accuracy metrics can be followed by live experiments, while A/B tests and simulations help assess performance in real-world and longer-term settings.
  • Conversational evaluation: Generative tasks can use NLP metrics such as BLEU and ROUGE, but conversational evaluation should supplement them with task- or objective-specific metrics and human assessment.
  • Societal impact: Potential harms include bias, environmental impact, worker displacement, manipulation, unintended consequences, and hyper-personalization risks.
  • Holistic evaluation: The survey proposes comprehensive Gen-RecSys evaluation spanning offline metrics, online performance, harm, and differing stakeholder perspectives.

6 CONCLUSIONS AND FUTURE DIRECTIONS

The survey identifies major future challenges for Gen-RecSys, including tool-augmented and retrieval-based systems, personalized content generation, and adversarial testing. It also broadens evaluation beyond performance to fairness, privacy, and societal impact.

  • Future challenges and opportunities: Important future directions include RAG, tool-augmented LLMs for conversational recommendation, personalized content generation, and red-teaming before deployment.The survey highlights data fusion, retriever-generator training, proactive dialogue, virtual try-on, and adversarial stress testing as specific opportunities.
  • Future challenges and opportunities: Tool-augmented conversational recommenders should coordinate dialogue, recommender modules, external reasoners, retrievers, and other tools.The survey emphasizes architecture design for LLM-driven control and proactive conversational recommendation.
  • Evaluation and research scope: The survey addresses evaluation challenges spanning performance, fairness, privacy, and societal impact while establishing a foundation for future Gen-RecSys research.Its scope extends across interaction data, textual data, and multimodal contexts.
Loading 2404.00579v2…