PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation

Yuyang Ye, Zhi Zheng, Yishan Shen, Tianshu Wang, Hengruo Zhang, Peijun Zhu, Runlong Yu, Kai Zhang, Hui Xiong

arXiv:2408.09698v5cs.IRcs.AI

TL;DR

Multimodal sequential recommendation remains challenging for MLLMs because ordered image inputs and evolving user preferences are difficult to process. MLLM-MSR summarizes item modalities, recurrently infers preferences, and fine-tunes an MLLM recommender; the paper reports improved performance across datasets compared with existing methods.

  • Problem

    MLLMs face challenges processing sequential multimodal inputs and capturing evolving user preferences, while fine-tuning for recommendation also raises overfitting and generalizability concerns.

    Source: Introduction · p. 1, Recurrent User Preference Inference · p. 3

  • Method

    MLLM-MSR summarizes item visual and textual data, recurrently infers user preferences from enriched item information, and fine-tunes an MLLM recommender with supervised learning.

    Source: Introduction · p. 2, Tuning MLLM based Recommender · p. 5

  • Results

    MLLM-MSR consistently outperforms compared methods in classification and ranking, and the authors report validation across three public datasets.

    Source: Introduction · p. 2, Performance Analysis · p. 6

  • Takeaways & Limitations

    The experiments support MLLM-MSR's ability to adapt to dynamic preferences across datasets while integrating diverse modalities into recommendation.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) have demonstrated significant potential in the field of Recommendation Systems (RSs). Most existing studies have focused on converting user behavior logs into textual prompts and leveraging techniques such as prompt tuning to enable LLMs for recommendation tasks. Meanwhile, research interest has recently grown in multimodal recommendation systems that integrate data from images, text, and other sources using modality fusion techniques. This introduces new challenges to the existing LLM-based recommendation paradigm which relies solely on text modality information. Moreover, although Multimodal Large Language Models (MLLMs) capable of processing multi-modal inputs have emerged, how to equip MLLMs with multi-modal recommendation capabilities remains largely unexplored. To this end, in this paper, we propose the Multimodal Large Language Model-enhanced Multimodaln Sequential Recommendation (MLLM-MSR) model. To capture the dynamic user preference, we design a two-stage user preference summarization method. Specifically, we first utilize an MLLM-based item-summarizer to extract image feature given an item and convert the image into text. Then, we employ a recurrent user preference summarization generation paradigm to capture the dynamic changes in user preferences based on an LLM-based user-summarizer. Finally, to enable the MLLM for multi-modal recommendation task, we propose to fine-tune a MLLM-based recommender using Supervised Fine-Tuning (SFT) techniques. Extensive evaluations across various datasets validate the effectiveness of MLLM-MSR, showcasing its superior ability to capture and adapt to the evolving dynamics of user preferences.

Introduction

The paper frames multimodal sequential recommendation as a challenge for MLLMs: ordered image inputs, evolving preferences, and fine-tuning constraints complicate recommendation. MLLM-MSR combines item summarization, recurrent preference inference, and supervised fine-tuning, and is reported to improve recommendation performance across datasets.

Introduction

  • The paper situates MLLMs within recommendation research that has expanded from text-focused LLM methods toward multimodal inputs and fusion.Prior work also explores sequential recommendation and multimodal information, motivating the paper's focus on combining these settings.Source: Introduction · p. 1
  • Sequential multimodal recommendation poses challenges in processing ordered images, capturing temporal preference dynamics, and fine-tuning without overfitting or losing pretraining generalizability.The paper identifies computational demands and scalability as additional constraints for sequential multimodal inputs.Source: Introduction · p. 1
  • MLLM-MSR combines MLLM-based item summarization, recurrent user-preference inference, and supervised fine-tuning of an MLLM recommender.The approach converts item visual and textual data into cohesive descriptions, uses enriched item information to infer evolving preferences, and integrates these with interaction ground truth for fine-tuning.Source: Introduction · p. 2
  • The authors report that MLLM-MSR outperforms other methods in recommendation performance and is validated across multiple datasets.They describe the approach as improving accuracy and interpretability through detailed user preferences.Source: Introduction · p. 2

Related Work

Related work traces sequential recommenders from matrix and neural models to multimodal fusion methods. It also surveys LLM recommendation approaches, including embedding-based, token-based, direct-model, and MLLM-based systems.

Multimodal Sequential Recommendation

  • Sequential recommendation developed from matrix-based and neural architectures, while multimodal sequential models use early, late, and hybrid fusion strategies.The surveyed neural models include GRU4Rec, SASRec, and BERT4Rec; fusion methods integrate modalities at different stages or combine fusion with sequential modeling.Source: Multimodal Sequential Recommendation · p. 2

LLM for Recommendation

  • LLM recommendation research builds on language models used to analyze user-item interactions and generate personalized, context-aware recommendations.The passage names BERT, GPT-3, BERT4Rec, and RLMRec in this research lineage.Source: LLM for Recommendation · p. 2
  • LLM recommendation approaches are categorized as embeddings-based, token-based, or direct model applications, with MLLM frameworks addressing diverse data types.The passage describes these approaches as extracting representations, generating semantic tokens, or directly generating recommendations.Source: LLM for Recommendation · p. 2

Preliminary

This section defines sequential multimodal recommendation as predicting a user's next interaction from historical text and image data, and preliminarily evaluates image summarization. The reported GRU4Rec comparison finds image summaries comparable to direct image processing.

Preliminary

  • The preliminary section introduces a problem formulation and a study of the image-summarizing approach's effectiveness.The study uses LLaVA to summarize images, embeds summaries with BERT, and feeds them to GRU4Rec for comparison with direct VGG19 image representations using AUC.Source: Preliminary · p. 3, Effectiveness of Multiple Images Summary · p. 3

Problem Definition

  • The user behavior sequence contains interacted items, with each item associated with a textual description and an image.The paper gives clicking, purchasing, and watching as examples of interactions.Source: Problem Definition · p. 3

Definition 1 (Multimodal Sequential Recommendation)

  • Multimodal sequential recommendation predicts the probability of a user's next interacted item from historical behavior containing textual and visual data and a candidate item.The task is formulated as predicting the next interaction, such as a click, for the user and candidate item.Source: Problem Definition · p. 3

Effectiveness of Multiple Images Summary

  • GRU4Rec using image summaries performs comparably to direct VGG19 image representations, supporting the summaries' preservation of semantic information for sequential modeling.The passage presents this as preliminary validation of the image-summary approach to handling multiple ordered images.Source: Table 1 · p. 3, Effectiveness of Multiple Images Summary · p. 3

Technical Details

MLLM-MSR combines multimodal item summarization, recurrent user-preference inference, and supervised fine-tuning of an MLLM-based recommender. It converts item images and text into summaries, updates preference narratives across interaction sessions, and uses them with item data to predict interactions.

Technical Details

  • The framework combines multimodal user-preference inference with supervised fine-tuning of an MLLM-based recommender.These are the two main components of MLLM-MSR.Source: Technical Details · p. 3

Multimodal Item Summarization

  • Item summarization converts image and text information into unified textual descriptions for processing multimodal item sequences.Separate prompts summarize text and describe images; a fusion prompt integrates both outputs, which are calibrated to similar lengths.Source: Multimodal Item Summarization · p. 3

Recurrent User Preference Inference

  • Long historical multimodal sequences can produce unstable outputs and excessively long prompts, so the method updates user-preference summaries across interaction sessions.Interactions are segmented into session blocks and represented as concise narratives; each session's prompt incorporates previous interactions to refine current preferences.Source: Recurrent User Preference Inference · p. 4

Tuning MLLM based Recommender

  • The recommender uses user preferences, an item's textual description and image, and a system prompt to predict the probability of interaction.Its prompt restricts outputs to “yes” or “no.”Source: Tuning MLLM based Recommender · p. 4
  • Supervised fine-tuning trains the MLLM on interaction sequences containing user preferences, item descriptions, and images using next-token prediction.LoRA is used for parameter-efficient fine-tuning and to accelerate training.Source: Tuning MLLM based Recommender · p. 5

Experiments

Experiments evaluate MLLM-MSR on three recommendation datasets against sequential, multimodal, and LLM-based baselines. The model outperforms the compared methods, while ablations and parameter analyses examine its components and settings.

Dataset Description

  • The evaluation uses Microlens, Amazon-Baby, and Amazon-Game datasets with interaction histories and item images, plus video introductions or product descriptions.Infrequent users and items were removed to meet minimum history-length criteria.Source: Dataset Description · p. 5

Baseline Methods

  • Baselines include basic sequential recommenders, multimodal recommenders, multimodal-feature-enhanced sequential models, and LLM-based sequential recommenders.The LLM group includes TALLREC, which uses textual inputs, and LLaVA without recommendation-specific fine-tuning.Source: Baseline Methods · pp. 5–6

Performance Analysis

  • MLLM-MSR consistently outperformed the compared methods on classification and ranking metrics across the reported evaluation.Results use 5-fold cross-validation, various random seeds, and a 95% confidence level.Source: Table 3 · p. 6, Performance Analysis · p. 6

Metrics

  • Performance is measured with AUC, HR@5, and MRR@5, using standardized candidate-set sizes across methods.Source: Metrics · p. 6

Ablation Study

  • Across three datasets, MLLM-MSR outperformed variants that replaced recurrent inference with direct inference or omitted image summaries from preference inference.The comparison supports the roles of recurrent preference inference and image-summary input.Source: Figure 4 · p. 7, Ablation Study · p. 7

Parameter Analysis

  • Block size requires balancing limited context, which can miss preference dynamics, against long prompts, which increase computation and reduce available blocks.The analysis concerns recurrent user-preference inference.Source: Figure 5 · p. 7, Parameter Analysis · p. 7
  • Short preference-summary context loses information, while performance stabilizes after context length reaches a threshold.The study fixes output length while varying context length; the authors report that a suitable range avoids unnecessary computational complexity.Source: Figure 6 · p. 7, Parameter Analysis · p. 7

Conclusion

MLLM-MSR uses two-stage preference summarization and supervised fine-tuning to apply MLLMs to multimodal sequential recommendation. The authors report adaptation to dynamic preferences across datasets and performance improvements over existing methods.

  • MLLM-MSR combines two-stage user-preference summarization with supervised fine-tuning and is reported to adapt to dynamic preferences across datasets.The authors report improved performance over existing methods and added interpretability through detailed user-preference analysis.Source: Conclusion · p. 7

Related papers

Loading 2408.09698v5…