Source-linked AI summary
Multimodal Recommender Systems: A Survey
Qidong Liu, Jiaxi Hu, Yutian Xiao, Xiangyu Zhao, Jingtong Gao, Wanyu Wang, Qing Li, Jiliang Tang
TL;DR
Multimedia services create a need for recommender systems that understand multimodal item content while addressing data sparsity. This survey synthesizes MRS procedures, challenges, models, datasets, code, and future directions through a four-part technical taxonomy. It organizes the field around Modality Encoder, Feature Interaction, Feature Enhancement, and Model Optimization techniques, while identifying computational complexity, interpretability, and the lack of a universal solution as ongoing challenges.
Problem
MRS must represent complex multimodal inputs, connect heterogeneous modalities, learn under sparse interactions, and optimize parameterized encoders with recommendation models.
Method
The survey organizes MRS research by four technical categories and summarizes general procedures, datasets, open-source code, challenges, and future directions.
Results
The survey provides taxonomies of MRS techniques, summarizes dataset and code resources, and proposes challenges and future directions.
Takeaways & Limitations
The technical taxonomy is intended to help readers understand general MRS architectures and recent advancements in the field.
Takeaways & Limitations
MRS implementation remains difficult because multimodal architectures can contain parameter-intensive modality encoders and multiple branches requiring different training pipelines.
Abstract
from arXiv · showhide
The recommender system (RS) has been an integral toolkit of online services. They are equipped with various deep learning techniques to model user preference based on identifier and attribute information. With the emergence of multimedia services, such as short videos, news and etc., understanding these contents while recommending becomes critical. Besides, multimodal features are also helpful in alleviating the problem of data sparsity in RS. Thus, Multimodal Recommender System (MRS) has attracted much attention from both academia and industry recently. In this paper, we will give a comprehensive survey of the MRS models, mainly from technical views. First, we conclude the general procedures and major challenges for MRS. Then, we introduce the existing MRS models according to four categories, i.e., Modality Encoder, Feature Interaction, Feature Enhancement and Model Optimization. Besides, to make it convenient for those who want to research this field, we also summarize the dataset and code resources. Finally, we discuss some promising future directions of MRS and conclude this paper. To access more details of the surveyed papers, such as implementation code, we open source a repository.
1 Introduction
Multimodal recommender systems use item modalities such as images, audio, and text to represent items and user preferences, while addressing challenges across the recommendation pipeline. The survey organizes these challenges and techniques into a unified procedure and four technical categories.
- 1 Introduction: Multimodal information can alleviate data sparsity and reveal user preferences at a deeper semantic level.
- 1 Introduction: MRS uses multimodal item features, including image, audio, and text, alongside identifier and tabular information.
- General Procedures: The unified MRS procedure comprises Raw Feature Representation, Feature Interaction, and Recommendation.
- 1.2 Taxonomy: MRS research addresses challenges in representing complex modalities, interacting heterogeneous features, enhancing sparse representations, and optimizing encoders with recommendation models.
- 1.2 Taxonomy: The survey organizes techniques into Modality Encoder, Feature Interaction, Feature Enhancement, and Model Optimization categories.
2 Modality Encoder
Modality encoders transform multimodal item inputs into dense representations for recommendation. The survey groups visual encoders by their underlying architectures and notes the progression from CNN-based models toward transformer-based models.
- Modality Encoder: Modality encoders obtain dense representations from multimodal features such as images, text, and audio.
- Visual Encoder: Visual encoders extract and compress information from raw image pixels for recommendation models.
- Visual Encoder: Visual encoders are categorized as CNN-based, ResNet-based, or Transformer-based.
- Visual Encoder: Recent MRS models use ViT to extract visual features, following advances in transformer-based visual pre-training.
3 Feature Interaction
Feature interaction connects heterogeneous multimodal spaces with recommendation signals through Bridge, Fusion, and Filtration techniques. These approaches model user–item relations, combine modality information at different granularities, and remove noise.
- Feature Interaction: Feature interaction transforms heterogeneous modality spaces into a common space to improve recommendation performance and generalization.
- Bridge: Bridge methods construct multimodal information channels that capture relationships between users and items.
- Bridge: User-item, item-item, and knowledge graphs provide distinct structures for modeling multimodal preferences and item representations.
- Fusion: Fusion methods combine modalities within items, with attention mechanisms assigning different weights and focus to multimodal information.
- Fusion: Fine-grained fusion selectively integrates local features such as clothing patterns, image patches, or news regions across modalities.
- Fusion: Combined attention aims to preserve global aggregation while retaining fine-grained multimodal information.
- Filtration: Filtration removes irrelevant or noisy information from multimodal features and interaction graphs to improve recommendation performance.
4 Feature Enhancement
Feature enhancement improves multimodal recommendation by separating shared and modality-specific information and strengthening representations through disentanglement, contrastive learning, and data augmentation.
- 4 Feature Enhancement: Disentangled and contrastive learning distinguish unique and common characteristics across modality representations to improve recommendation performance and generalization.These methods target entangled factors and complementary information in multimodal data.
- 4 Feature Enhancement: Contrastive learning aligns representations across modalities by increasing semantic similarity for corresponding item features.Examples include text-to-image and image-to-text objectives, graph-view alignment, and visual-textual pair matching.
- 4 Feature Enhancement: Data augmentation constructs positive or negative recommendation samples through graph enhancement, modal edge dropping, masking, semantic transformations, and fashion matching schemes.These approaches exploit relationships among multimodal or graph-based samples for contrastive objectives.
5 Model Optimization
Model optimization in multimodal recommendation addresses the high computational cost of multimodal encoders through end-to-end or two-step training and efficiency-oriented techniques.
- 5 Model Optimization: MRS training uses either end-to-end optimization, which updates all layers jointly, or two-step training, which pretrains multimodal encoders before recommendation training.The two schemes differ in how encoder and recommender parameters are trained.
- 5 Model Optimization: Pretrained encoders such as ViT, ResNet, and BERT provide modality representations, but their large parameter counts challenge available computing resources.ViT-Base is cited as having 86M parameters.
- 5 Model Optimization: Parameter sharing and freezing reduce training computation and memory costs while supporting recommendation performance and denoising.MKGformer shares attention parameters, whereas FREEDOM freezes graph-structure parameters.
- 5 Model Optimization: Two-step training can target downstream tasks better but requires substantially more computing resources, limiting its adoption.Pretraining objectives include graph reconstruction, masked node-feature reconstruction, and fashion-matching knowledge acquisition.
- 5 Model Optimization: Knowledge distillation and prompt tuning are used to improve training efficiency and simplify finetuning for multimodal recommendation.SGFD distills a lighter modality encoder, while PromptMM uses a pretrain-prompt scheme.
6 Applications and Resources
MRS applications benefit from domain-specific modeling, while datasets, standard evaluation metrics, and open-source frameworks support practical research and reproducibility.
- 6 Applications and Resources: Typical applications may require unique models because user motivations and item attributes differ across domains such as fashion recommendation.The paper contrasts domain-specific design with general MRS models.
- 6 Applications and Resources: The survey organizes popular MRS datasets by application and directs researchers to corresponding citations and websites.These resources are intended to ease access to datasets for typical applications.
- 6 Applications and Resources: MRS commonly evaluates recommendation models with general RS metrics, including hit rate and normalized discounted cumulative gain.The survey notes that multimodal recommendation often uses the same evaluation metrics as general recommendation.
- 6 Applications and Resources: Multimodal architectures are technically difficult to implement because they combine extensive modality encoders with multiple interaction branches.Large encoders are difficult to deploy, and varied branches complicate a unified training pipeline.
- 6 Applications and Resources: MMRec and Cornac provide open-source support for multimodal recommendation models, data, metrics, and experimental procedures.MMRec integrates more than ten models, while Cornac supports TensorFlow and PyTorch.
7 Challenges and Future Directions
The survey identifies unresolved challenges in MRS, including the absence of a universal solution, interpretability, computational demands, overfitting, and privacy, and highlights several future research directions.
- 7 Challenges and Future Directions: MRS lacks an up-to-date universal solution that combines techniques across different modeling stages.The survey presents this as an open challenge despite methods addressing individual stages.
- 7 Challenges and Future Directions: Model complexity can limit interpretability, trust, and transparency, while parameter-intensive encoders increase training and inference costs.These constraints are especially challenging for large datasets, populations, and real-time recommendation.
- 7 Challenges and Future Directions: Sparse data and informative modality-encoder representations leave MRS models susceptible to overfitting, while multimodal information also raises privacy concerns.The survey lists both risks among the field’s unresolved challenges.
- 7 Challenges and Future Directions: Future work includes stronger cross-modal representation learning through transfer learning and large-scale pretrained encoders.The survey expects these approaches to improve MRS effectiveness and efficiency.
- 7 Challenges and Future Directions: Adapting multimodal large language models to MRS is identified as a promising research direction because of their understanding and reasoning abilities.The survey specifically highlights MLLM adaptation for multimodal recommendation.
8 Conclusion
The survey organizes multimodal recommender systems around four technical taxonomies and supplements them with research resources and future directions.
- The survey proposes four MRS categories: modality encoder, feature interaction, feature enhancement, and model optimization.
- It summarizes datasets and open-source codes to support research in multimodal recommender systems.
- The paper identifies challenges and future directions intended to inspire further MRS research.