PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Self-Supervised Learning for Recommendation

Wei Wei, Chao Huang, Lianghao Xia, Chuxu Zhang

arXiv:2302.10632v5cs.IR

TL;DR

Multimedia recommenders need to learn fine-grained modality-aware preferences despite reliance on interaction labels and sparse behavior data. MMSSL combines adversarial modality-aware self-augmentation with cross-modal contrastive learning, and experiments report significant gains over baselines, especially for sparse interactions.

  • Problem

    Existing multimedia recommenders rely heavily on observed interaction labels and have limited robustness when user behavior is sparse.

  • Method

    MMSSL unifies modality-aware adversarial self-augmentation with cross-modal contrastive learning to model collaborative dependencies and modality-specific user preferences.

  • Results

    MMSSL achieves significant performance gains over various baselines across public datasets and shows strong gains with highly sparse user interactions.

  • Takeaways & Limitations

    The results support MMSSL as an effective and robust self-supervised approach for multimedia recommendation under sparse interaction labels.

  • Takeaways & Limitations

    The theoretical knowledge-transfer analysis assumes that the target relation function is L-Lipschitz continuous.

Abstract

from arXiv · show

The online emergence of multi-modal sharing platforms (eg, TikTok, Youtube) is powering personalized recommender systems to incorporate various modalities (eg, visual, textual and acoustic) into the latent user representations. While existing works on multi-modal recommendation exploit multimedia content features in enhancing item embeddings, their model representation capability is limited by heavy label reliance and weak robustness on sparse user behavior data. Inspired by the recent progress of self-supervised learning in alleviating label scarcity issue, we explore deriving self-supervision signals with effectively learning of modality-aware user preference and cross-modal dependencies. To this end, we propose a new Multi-Modal Self-Supervised Learning (MMSSL) method which tackles two key challenges. Specifically, to characterize the inter-dependency between the user-item collaborative view and item multi-modal semantic view, we design a modality-aware interactive structure learning paradigm via adversarial perturbations for data augmentation. In addition, to capture the effects that user's modality-aware interaction pattern would interweave with each other, a cross-modal contrastive learning approach is introduced to jointly preserve the inter-modal semantic commonality and user preference diversity. Experiments on real-world datasets verify the superiority of our method in offering great potential for multimedia recommendation over various state-of-the-art baselines. The implementation is released at: https://github.com/HKUDS/MMSSL.

1 INTRODUCTION

Multimedia recommendation benefits from modality-rich item content but remains constrained by label scarcity and sparse user behavior. MMSSL addresses these issues through modality-aware generative and contrastive self-supervision, with strong performance under sparse interactions.

  • Motivation: Existing multimedia recommenders incorporate visual, acoustic, and textual item features, but their performance usually requires sufficient observed user-interaction labels.These modalities can reflect fine-grained user preferences.
  • Motivation: Recent self-supervised methods alleviate sparsity in collaborative filtering but do not adapt their augmentation schemes to multimedia recommendation.MMSSL instead derives modality-aware supervisory signals from unlabeled data.
  • Empirical evidence: For Amazon-baby, MMSSL achieves significant gains when user interaction frequency is highly sparse, including groups with fewer than 4 interactions.Its item-specific prediction distribution also contains more long-tail items in the highlighted high-performance area.
  • MMSSL: MMSSL unifies generative modality-aware collaborative self-augmentation with contrastive cross-modality dependency encoding.Its first stage models modality-specific collaborative effects, while its second stage captures interactions among modality-wise preferences.
  • Empirical evidence: The method is supported by extensive evaluation, theoretical analysis, and visual studies examining effectiveness, robustness, knowledge transfer, and cross-modal representation learning.The contribution summary explicitly reports significant gains and in-depth analyses.

2 PRELIMINARY

MMSSL formulates recommendation over a user-item interaction graph enriched with multimodal item information. The task is to learn item-adoption likelihood from these modality-aware relations.

  • Interaction Graph with Multi-Modality: MMSSL constructs a user-item graph whose edges represent observed interactions and incorporates textual, visual, and acoustic information into the graph.Users and items form the graph nodes, while observed user-item relations form its edges.
  • Task Formulation: The multimodal recommendation task learns a function that forecasts how likely an item is to be adopted by a user.The formulation uses a generated multimodal interaction graph and item modality feature embeddings.

3 METHODOLOGY

MMSSL uses modality-aware adversarial self-augmentation to learn collaborative relations from multimodal context, then aligns generated and observed relations for robust self-supervision. Gumbel transformation and Wasserstein gradient penalty address the distribution gap caused by sparse interactions.

  • 3.1 Multi-Modal Self-Augmentation: MMSSL supplements interaction modeling with a modality-aware adversarial self-augmentation task and a collaborative relation generator.The generator produces self-supervision signals from multimodal context to capture modality-aware user preference under sparse labels.
  • 3.1.1 Modality-aware Collaborative Relation Learning: The generator learns modality-specific user-item relation matrices from multimodal representations and collaborative effects.Each learned matrix captures the dependency between the collaborative view and the multimodal context view.
  • 3.1.2 Discriminator with Adversarial Generation: The discriminator distinguishes generated modality-aware relations from observed user-item interactions, while the generator refines relations to confuse it.This adversarial process enhances dependency modeling between collaborative and multimodal views.
  • 3.1.3 Adversarial SSL against Distribution Gap: Sparse observed interaction matrices create a distribution mismatch with dense generated relations, risking mode collapse and convergence difficulty.MMSSL uses Gumbel-Softmax to transform observed interactions into a dense proxy matrix that bridges this gap.
  • 3.1.4 Adversarial SSL Loss: The adversarial SSL objective aligns generated modality-aware relations with the transformed interaction proxy using minimax optimization and Wasserstein gradient penalty.The penalty improves robustness against distribution gap and data sparsity.

3.2 Cross-Modal Contrastive Learning

MMSSL models dependencies among modality-specific user preferences through modality-aware representations and cross-modal contrastive learning. The contrastive objective aligns each modality-specific user embedding with the user’s overall representation while separating different users.

  • Modality-aware Contrastive View: MMSSL uses modality-aware semantic neighbors to aggregate information for contrastive learning.
  • Modality-wise Dependency Modeling: A multi-head self-attention encoder captures correlations between pairs of modality-specific user preferences.Query and key transformations calculate relations between modality pairs, and softmax weights these relations across attention heads.
  • Multi-Modal High-Order Connectivity: The modality-aware encoder uses graph neural networks to recursively propagate high-order collaborative effects and produces multi-modal user and item representations.The interaction graph uses degree-normalized message passing, with layer-specific embeddings aggregated through mean-pooling.
  • Cross-Modal Contrastive Augmentation: The cross-modal contrastive objective maximizes mutual information between each user’s modality-specific embedding and overall embedding while treating other users as negatives.This self-discrimination strategy supplies supervision for dependency modeling among modality-specific preferences.
  • Cross-Modal Contrastive Augmentation: Contrastive learning separates different user representations, preserving modality-specific preference diversity with uniformly distributed embeddings.The similarity function and temperature coefficient define the contrastive scoring process.

3.3 Multi-Task Model Training

MMSSL generates final user and item representations by combining collaborative and multi-modal embeddings, then jointly optimizes recommendation and both self-supervised learning tasks. The objective includes regularization to reduce overfitting.

  • Multi-Task Model Training: MMSSL aggregates collaborative-view and multi-modal embeddings to form final user and item representations for prediction.Normalization reduces scale differences, while the aggregation weight controls the combination.
  • Multi-Task Model Training: The model predicts unobserved user-item interactions using the inner product of the final user and item embeddings.
  • Multi-Task Model Training: MMSSL jointly optimizes interaction prediction, adversarial modality-aware relation learning, and cross-modal contrastive learning.The three tasks are combined in a multi-task learning objective with separate loss-weighting hyperparameters.
  • Multi-Task Model Training: The training objective adds L2 weight-decay regularization against overfitting.

3.4 Theoretical Discussion of MMSSL

Theoretical analyses explain how MMSSL transfers collaborative knowledge through adversarial augmentation and improves user-specific discrimination through contrastive learning. The analysis also addresses distribution gaps and oversmoothing in representation learning.

  • Theoretical Discussion of MMSSL: MMSSL’s theoretical discussion covers collaborative knowledge transfer and cross-modality commonality.
  • Theoretical Analysis of Knowledge Transfer: The adversarial SSL task injects collaborative signals into modality-specific distributions by relating adversarial transferability to knowledge transferability.The analysis defines both transferability concepts and studies their correlation.
  • Theoretical Analysis of Knowledge Transfer: The transfer analysis assumes the target relation is L-Lipschitz continuous and uses a composite function to characterize knowledge transferability.
  • Theoretical Analysis of Knowledge Transfer: High adversarial transferability implies an affine transformation with bounded norm, providing theoretical support for self-augmentation and helping prevent mode collapse.
  • User-specific Patterns Modeling Through Gradient: Contrastive learning assigns larger gradients to hard negative samples, enhancing representation discrimination and modeling user-specific preferences.This mechanism is presented as a response to the oversmoothing risk of stacked GNN architectures.

4 EVALUATION

MMSSL is evaluated against multimodal, GNN-based, and SSL-based recommender baselines across public datasets using standard top-K metrics. Results examine overall accuracy, component contributions, sparsity robustness, convergence, and hyperparameter sensitivity.

  • Experimental setup: Experiments use TikTok, Amazon-Baby, Amazon-Sports, and Allrecipes, with multimodal item features varying across visual, acoustic, and textual content.TikTok includes visual, acoustic, and title features; Amazon product features include text and images; Allrecipes uses recipe images and ingredients.
  • Experimental setup: Recommendation accuracy is measured with Recall@20, Precision@20, and NDCG@20 using all-item ranking and averages over test users.
  • Overall comparison: MMSSL consistently outperforms all baselines across different datasets, with gains attributed to integrated generative and contrastive self-supervision for modality-aware augmentation.Multimodal methods generally outperform graph-based collaborative filtering, while existing SSL baselines obtain only marginal gains over NGCF and LightGCN.
  • Ablation study: Removing adversarial self-augmentation, cross-modal contrastive learning, Gumbel transformation, or the adversarial component reduces performance, supporting the contribution of each design choice.The comparison with r/p-GAE also favors MMSSL for capturing implicit modality-aware user-item relations.
  • Sparsity evaluation: MMSSL consistently outperforms compared approaches across interaction sparsity degrees, with results measured by NDCG@20 on Amazon-Baby and Allrecipes.The authors associate this robustness with self-augmented multimodal signals that transfer multimodal knowledge into user representations.
  • Efficiency and sensitivity: MMSSL converges faster than compared methods on Amazon-Baby while maintaining superior recommendation accuracy.The study also reports best performance with 2 or 3 GNN layers, ζ=100, and λ_1=1; deeper networks suffer from oversmoothing.

5 RELATED WORK

Related work spans GNN-based recommendation, self-supervised learning for recommendation, and multimedia recommendation. These lines model collaborative relationships, augment supervision, or incorporate multimedia item content.

  • GNN-based Recommender Systems: GNN-based recommender systems model user-item, social, or sequential relationships, and MMSSL uses GNNs to capture high-order collaborative relations with multimodal context.
  • Self-Supervised Learning for Recommendation: Self-supervised recommendation methods augment original supervision with auxiliary tasks, often contrasting positive node pairs generated through graph-based operators.Examples include random-walk sampling and other augmentation strategies in contrastive learning.
  • Multimedia Recommendation: Multimedia recommendation methods incorporate item content through matrix factorization, attention mechanisms, or graph neural networks to model complex user preferences.Representative approaches include VBPR, ACF, VECF, and later graph-based multimodal systems.

6 CONCLUSION

The paper proposes MMSSL for multimedia recommendation, combining modality-aware adversarial perturbation with cross-modal contrastive learning. Experiments report significant gains over baselines, while future work targets richer user interests and explainability.

  • Conclusion: MMSSL uses modality-aware adversarial perturbation to capture multimodal user preference under sparse interaction labels.
  • Conclusion: Cross-modal contrastive learning models dependencies among modality-specific user interaction patterns.
  • Conclusion: Extensive experiments on public datasets report significant performance gains over various baselines.
  • Conclusion: Future work includes multi-interest user embeddings and GNN-based explainers for causal effects on modality-aware user-item graphs.

A APPENDIX

The appendix provides deeper analyses of MMSSL's user-preference diversity, adversarial knowledge transfer, computational complexity, and additional experimental evidence.

  • Appendix analyses: The appendix analyzes user-preference diversity from contrastive-learning gradients and discusses the theoretical basis of self-augmented adversarial collaborative knowledge transfer.
  • Appendix analyses: It also analyzes MMSSL's time complexity and presents additional experiments further justifying the proposed method.

A.1 Derivation of Negative Sample Gradient

The derivation analyzes how contrastive-learning gradients depend on positive and negative pairs, focusing on how negative-sample similarity determines gradient magnitude. It shows that harder negative samples receive larger gradients, enhancing user-representation discrimination.

  • Motivation: The hardness-aware gradient property is used to address over-smoothing when encoding high-order collaborative signals and learning distinguishable embeddings.The motivation is that greater gradients push hard negative instances away from the anchor.
  • Gradient derivation: Contrastive loss gradients for the anchor user are determined by both positive and negative pairs after applying the derivation and chain rule.The normalized anchor and instance definitions support the gradient calculation in the shared hyperspace.
  • Negative-sample gradient: The negative-sample gradient is analyzed by expressing its similarity with the anchor as x = q_u · q_N ∈ [−1, 1].Because q_u and q_N are unit vectors, x abbreviates their similarity for the gradient-function analysis.
  • Negative-sample gradient: As negative-sample similarity x increases, the gradient of negative samples rises, assigning larger gradients to hard negatives.The paper interprets these hard negatives as other users and links the larger gradients to improved discrimination of user representations.
  • Temperature analysis: Figure 5 plots the gradient function 𝜙(x) for temperature coefficients τ = 0.02, τ = 0.1, and τ = 0.5.The figure states that the gradient increases as the temperature coefficient decreases.
Loading 2302.10632v5…