PaperlayerView exact revision on arXiv

Source-linked AI summary

Bootstrap Latent Representations for Multi-modal Recommendation

Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, Feijun Jiang

arXiv:2207.05969v3cs.IR

TL;DR

Multi-modal recommendation must exploit item content while avoiding the computational and memory costs of auxiliary graphs and noisy negative sampling. BM3 uses dropout-generated latent views and three jointly optimized objectives, outperforming prior methods on three datasets while training 2-9× faster.

  • Problem

    Multi-modal recommendation seeks to use item content, but auxiliary graphs and BPR negative sampling can be costly and noisy on large graphs.

  • Method

    BM3 uses latent representation dropout instead of graph augmentation to generate contrastive views, then jointly reconstructs interactions and aligns modality features.

  • Results

    BM3 significantly improves over state-of-the-art methods across three datasets and trains 2-9× faster than baseline methods.

  • Takeaways & Limitations

    BM3 demonstrates that self-supervised multi-modal recommendation can combine interaction reconstruction with inter- and intra-modality representation alignment.

  • Takeaways & Limitations

    BM3 models observed user-item interactions in a graph whose nodes are users and items.

Abstract

from arXiv · show

This paper studies the multi-modal recommendation problem, where the item multi-modality information (e.g., images and textual descriptions) is exploited to improve the recommendation accuracy. Besides the user-item interaction graph, existing state-of-the-art methods usually use auxiliary graphs (e.g., user-user or item-item relation graph) to augment the learned representations of users and/or items. These representations are often propagated and aggregated on auxiliary graphs using graph convolutional networks, which can be prohibitively expensive in computation and memory, especially for large graphs. Moreover, existing multi-modal recommendation methods usually leverage randomly sampled negative examples in Bayesian Personalized Ranking (BPR) loss to guide the learning of user/item representations, which increases the computational cost on large graphs and may also bring noisy supervision signals into the training process. To tackle the above issues, we propose a novel self-supervised multi-modal recommendation model, dubbed BM3, which requires neither augmentations from auxiliary graphs nor negative samples. Specifically, BM3 first bootstraps latent contrastive views from the representations of users and items with a simple dropout augmentation. It then jointly optimizes three multi-modal objectives to learn the representations of users and items by reconstructing the user-item interaction graph and aligning modality features under both inter- and intra-modality perspectives. BM3 alleviates both the need for contrasting with negative examples and the complex graph augmentation from an additional target network for contrastive view generation. We show BM3 outperforms prior recommendation models on three datasets with number of nodes ranging from 20K to 200K, while achieving a 2-9X reduction in training time. Our code is available at https://github.com/enoche/BM3.

1 INTRODUCTION

Multi-modal recommendation seeks to exploit item content beyond user-item interactions, but graph augmentation and negative sampling can make existing methods costly and noisy. BM3 addresses these issues with dropout-based self-supervised views and a joint multi-modal objective.

  • Item texts, images, and videos remain underused despite their potential to improve recommendations beyond historical user-item interactions.
  • Existing methods integrate modality features through concatenation, attention, or graph neural networks, sometimes adding user-user and item-item auxiliary graphs.
  • Random negative sampling in BPR can be expensive on large graphs and introduce noisy supervision; uniform sampling exceeds 25% of LightGCN’s training time per epoch.
  • BM3 removes the target network and uses latent embedding dropout to generate contrastive views without auxiliary graph augmentation.
  • BM3’s Multi-Modal Contrastive Loss jointly reconstructs interactions and aligns features across modalities and augmented views without negative samples.

2 RELATED WORK

Prior work combines collaborative filtering, multi-modal features, graph neural networks, and self-supervised learning, but large-scale multi-modal recommendation remains constrained by auxiliary graphs and negative sampling. BM3 is positioned as a simpler SSL alternative.

  • Early multi-modal recommenders extend collaborative filtering by incorporating visual or other item features into item representations.
  • Graph neural networks enhance user and item representations using structural information from user-item and auxiliary graphs.
  • Graph-based multi-modal models may require auxiliary graphs and many BPR negatives, increasing computational complexity and memory costs on large graphs.
  • Self-supervised learning can learn representations without negative sampling, using distorted views and asymmetric online and target networks.
  • BUIR learns user and item representations from positive interactions using distinct views and a slow-moving target encoder.
  • BM3 applies a simplified SSL framework to multi-modal recommendation to address the open question of using multimodal features efficiently.

3 BOOTSTRAPPED MULTI-MODAL MODEL

BM3 consists of a multi-modal latent space converter, a contrastive view generator, and a multi-modal contrastive loss.

  • BM3 is organized around three components: a multi-modal latent space converter, contrastive view generator, and multi-modal contrastive loss.

3.1 Multi-modal Latent Space Convertor

BM3 maps ID and modality-specific features into a shared latent space, then uses LightGCN-based interaction propagation and residual representations for recommendation.

  • Multi-modal Latent Space Convertor: User and item ID embeddings have dimension d, while modality-specific features have modality-dependent dimensions d_m.
  • Multi-modal Latent Space Convertor: BM3 converts vision and text features into a common latent space so their representations can be compared with ID embeddings.
  • Multi-modal Latent Space Convertor: Each modality-specific feature is projected into the latent space with an MLP, producing representations that share the ID-embedding space.
  • ID Embeddings: The LightGCN backbone encodes user-item graph structure because directly initialized ID embeddings contain no interaction-graph information.
  • ID Embeddings: A GCN recursively propagates stacked user and item embeddings through normalized adjacency, with nonlinear transformation in its typical formulation.
  • ID Embeddings: LightGCN removes feature transformations and nonlinear activations, leaving linear aggregation through a normalized adjacency transition matrix.
  • ID Embeddings: BM3 aggregates hidden-layer representations with a readout and adds a residual connection from initial item embeddings to mitigate over-smoothing.
  • ID Embeddings: The model produces user ID, item ID, and uni-modal item latent embeddings for subsequent loss optimization without negative samples.

3.2 Multi-modal Contrastive Loss

BM3 generates contrastive views by applying dropout to latent embeddings after online encoding, then trains with positive user-item pairs and multi-modal alignment objectives.

  • Contrastive View Generator: BM3 replaces graph augmentation with latent embedding dropout to generate contrastive views while stopping gradients on the contrastive branch.The original embedding is passed through an MLP predictor, while the contrastive view receives stop-gradient treatment.
  • Graph Reconstruction Loss: The reconstruction loss uses positive user-item pairs and symmetrized negative cosine similarity between online embeddings and detached contrastive views.The loss jointly predicts the perturbed item from the user and the perturbed user from the item.
  • Inter-modality Feature Alignment Loss: BM3 aligns each item’s multi-modal feature embedding with its target ID embedding using negative cosine similarity.This alignment encourages ID embeddings to be close for items with similar multi-modal features.
  • Intra-modality Feature Masked Loss: An intra-modality feature masked loss applies dropout masking to latent modality embeddings and encourages predictors to learn from sparse representations.The method randomly masks a subset of each modality’s latent embedding.
  • Overall Objective: The final objective adds regularization to online user and item embeddings alongside reconstruction and multi-modal losses.The multi-modal contrastive formulation jointly combines interaction reconstruction, inter-modality alignment, and intra-modality feature masking.

3.3 Top-𝐾Recommendation

BM3 ranks candidate items by predicted user-item interaction scores and recommends the top K items, using predictor-transformed embeddings for scoring.

  • Top-K Ranking: BM3 predicts interaction scores for candidate items, ranks them in descending order, and selects the top K recommendations.This procedure converts predicted user-item scores into the final recommendation list.
  • Interaction Scoring: The interaction score uses the inner product between predictor-transformed user and candidate-item embeddings.Classical collaborative filtering instead ranks items using the inner product of the original user and item embeddings.
  • Interaction Scoring: A higher interaction score indicates that the user prefers the item.The score is used to order candidate items before selecting recommendations.

3.4 Computational Complexity

BM3 has graph-convolution cost comparable in order to LightGCN, with additional costs for multi-modal projection and contrastive prediction; constructing auxiliary item graphs can be more expensive.

  • Graph Propagation: O(2L|E|d/B) is the graph-convolution complexity of both LightGCN and BM3.Here, L is the number of LightGCN layers and B is the training batch size.
  • BM3 Overhead: BM3’s total cost is O(2L|E|d/B + Σ_m∈M |I|d_md + (2 + 2|M|)dB).The terms represent graph convolution, multi-modal feature projection, and contrastive loss computation.
  • Auxiliary Graph Construction: LATTICE additionally incurs O(|I|^2d_m) to build item similarity matrices, O(|I|^3) to normalize them, and O(k|I| log(|I|)) to retrieve top-k neighbors.These costs arise from constructing an item-item graph from multi-modal features.

4 EXPERIMENTS

BM3 is evaluated against general and multi-modal recommendation baselines on three Amazon datasets using standard top-K metrics and controlled settings. The experiments assess effectiveness, efficiency, modality contributions, and loss components.

  • Datasets and evaluation: The experiments use Baby, Sports, and Electronics Amazon datasets containing product descriptions and images, with an 8:1:1 user-history split.Recall@K and NDCG@K are computed under the all-ranking protocol rather than negative sampling.
  • Baselines: BM3 is compared with general collaborative-filtering and multi-modal recommendation methods, including models using negative sampling, self-supervision, GNNs, and auxiliary graphs.The evaluation includes BPR, LightGCN, BUIR, VBPR, MMGCN, GRCN, DualGNN, and LATTICE.
  • Effectiveness: BM3 significantly outperforms general and state-of-the-art multi-modal methods on all three datasets.The reported gains over the best baselines in Recall@10 are 3.68% on Baby, 6.15% on Sports, and 20.39% on Electronics.
  • Self-supervised comparison: BM3 achieves higher recommendation accuracy than BUIR, whose performance varies across datasets because it depends on perturbed graph views.BUIR is the self-supervised comparison model, whereas BM3 uses latent representation dropout for view generation.

4.6 Efficiency of BM3 (RQ2)

The efficiency and ablation studies examine BM3’s computational profile and the contributions of multi-modal features and loss components. Results show dataset-dependent effects, especially on the large Electronics dataset.

  • Efficiency: BM3 reduces training time by 2–9× per epoch while using less or comparable memory to other graph-based multi-modal baselines.Compared with LATTICE, BM3 requires half the training time and half the consumed memory.
  • Efficiency: BM3’s efficiency comes from removing negative-sampling time and using fewer GCN layers, without much additional cost beyond LightGCN’s multi-modal features.Graph-based multi-modal models generally retain the interaction graph and store multi-modal features, increasing memory requirements.
  • Multi-modal features: Combining textual and visual features gives BM3 the best recommendation accuracy on Baby, Sports, and Electronics.The importance of the two modalities varies by dataset; single-modality effects are negligible on some datasets.
  • Contrastive loss: Multi-modal losses improve accuracy on Baby and Sports, but either inter-modality or intra-modality loss can reduce performance on Electronics.The relative importance of inter- and intra-modality losses varies across datasets.
  • Dataset dependence: On Electronics, uni-modal features and uni-loss variants do not improve accuracy, suggesting that the existing learning signals already produce strong representations there.Adding coarse multi-modal signals does not help improve the interaction-only variant on the large dataset.

5 CONCLUSION

BM3 is a self-supervised multi-modal recommendation framework that removes randomly sampled negatives and uses latent embedding dropout to generate contrastive views. Its multi-modal contrastive loss combines interaction reconstruction with inter- and intra-modality objectives, yielding accuracy improvements while training 2–9× faster than baselines.

  • Conclusion: BM3 removes randomly sampled negative examples from user-item interaction modeling.It instead uses self-supervised learning with latent embedding dropout to perturb user and item representations.
  • Conclusion: BM3 jointly optimizes interaction-graph reconstruction, alignment between item ID embeddings and multi-modal features, and masked within-modality learning.These objectives form the model’s multi-modal contrastive loss.
  • Conclusion: BM3 improves accuracy over state-of-the-art multi-modal recommendation methods while training 2–9× faster than baseline methods.The evaluation covers three real-world datasets, including one large-scale dataset.

6 APPENDICES

The appendix evaluates BM3’s recommendation accuracy under different GCN-layer, embedding-dropout, and regularization settings. Results show dataset-dependent sensitivity to layer depth and dropout, while Figure 3 visualizes accuracy across hyper-parameter combinations.

  • The Number of GCN Layers: BM3 is tested with GCN-layer counts L in {1, 2, 3, 4}.
  • The Number of GCN Layers: On Baby and Sports, recommendation accuracy degrades relatively slowly as the number of GCN layers increases.
  • The Number of GCN Layers: On Electronics, recommendation accuracy can improve with more than one GCN layer.
  • Embedding Dropout and Regularization: The sensitivity study varies embedding dropout from 0.1 to 0.5 and the regularization coefficient λ across {0.0001, 0.001, 0.01, 0.1}.
  • Embedding Dropout and Regularization: On Sports, a larger embedding-dropout ratio usually helps BM3 achieve better recommendation accuracy.
  • Embedding Dropout and Regularization: Figure 3 compares recommendation accuracy across dropout-ratio and regularization-coefficient combinations, with darker backgrounds indicating better accuracy.
Loading 2207.05969v3…