DiffMM: Multi-Modal Diffusion Model for Recommendation

1 INTRODUCTION

• Sparse observed interactions limit supervised recommenders’ ability to generate accurate embeddings of complex user preferences.The paper identifies interaction sparsity relative to the full interaction space as the central challenge.Source: §1 · p. 1
• Existing collaborative-filtering SSL methods use augmentations that overlook multimedia characteristics, limiting their capture of modality-aware preferences.Examples include stochastic node and edge dropout and discovery of implicit semantic node correlations.Source: §1 · p. 1
• Multimodal SSL methods may use simplistic random augmentation or intuitive cross-view alignment, introducing irrelevant noise and motivating adaptive modality-aware augmentation.The paper identifies misclick behavior and popularity bias among possible sources of noisy self-supervisory signals.Source: §1 · p. 1
• DiffMM generates a modality-aware user-item graph through diffusion, then uses modality-aware graph aggregation and cross-modal contrastive learning to model preferences.The reverse diffusion process is guided by modality-aware signal injection; contrastive learning examines interaction-pattern consistency across modalities.Source: §1 · p. 2
• DiffMM’s experiments on multiple benchmark datasets show significant performance improvements over competitive baselines.The contribution summary reports effectiveness across multiple benchmark datasets.Source: §1 · p. 2

2.1 Preliminaries

• The task is to predict whether a user will adopt an item from a user-item interaction graph enriched with item features across modalities.The graph represents user-item interactions, and the item modalities include textual, visual, and acoustic features.Source: §2.1 · p. 2

2.2 Multi-Modal Graph Diffusion Model

• DiffMM progressively corrupts user-item interactions and learns reverse denoising to restore them while incorporating modality information and mitigating noisy modality features.Its modality-aware signal injection guides interaction restoration; an MSE loss aligns modality-feature aggregation using predicted probabilities with item-ID aggregation using observed interactions.Source: §2.2 · p. 2, §2.2.2 · p. 4
• For inference, DiffMM forward-corrupts the interaction graph, then deterministically reverse-denoises it and uses the resulting predictions to rebuild the user-item graph.The reverse process uses the predicted mean and ignores variance.Source: §2.2.2 · p. 4, §2.2.3 · p. 4

2.3 Cross-Modal Contrastive Augmentation

• Cross-modal contrastive learning uses modality-specific graph views and InfoNCE objectives to align different modalities and the main-task embedding with modality views.The two paradigms use modality views and the main-task embedding as alternative anchors; embeddings from different users serve as negatives for the modality-view anchor.Source: §2.3.1 · p. 4, §2.3.2 · p. 4, §2.3 · p. 4

2.4 Multi-Modal Graph Aggregation

• Final user and item representations combine modality-aware graph representations with high-order collaborative signals from the original interaction graph, weighting modalities with learnable parameters.Layer-specific embeddings are sum-pooled, and predictions use the dot product of final user and item embeddings.Source: §2.4 · p. 5

2.5 Multi-Task Model Training

• Joint training optimizes the diffusion module with ELBO and MSI losses, and the recommendation task with BPR and cross-modal contrastive losses.The recommendation objective also includes L2 regularization; its training data pairs observed interactions with unobserved interactions.Source: §2.5 · p. 5

3.1 Experimental Settings

• Experiments use TikTok, Amazon-Baby, and Amazon-Sports, with all-rank top-K evaluation using Recall@K, Precision@K, and NDCG@K.The baselines span collaborative filtering, graph-based, self-supervised, diffusion, and multimodal recommenders.Source: §3.1 · p. 5, §3.1.1 · p. 5, §3.1.2 · p. 5

3.2 Performance Comparison (RQ1)

• DiffMM consistently outperforms all baselines across the evaluated datasets, with cross-modal contrastive learning and modality-aware diffusion-based augmentation among the reported advantages.The comparison covers TikTok and Amazon datasets and Recall@20, Precision@20, and NDCG@20.Source: §3.2 · p. 6, Table 1 · p. 6

3.3 Model Ablation Test (RQ2)

• Removing cross-modal contrastive learning or modality-aware signal injection lowers performance, while the ablation comparison also examines replacing diffusion with VGAE.The authors report that the full diffusion approach is superior to other generative models and that signal injection helps create the modality-aware user-item graph.Source: §3.3 · p. 6, Table 2 · p. 6

3.4 Handling Sparse Interaction Data (RQ3)

• DiffMM consistently outperforms VBPR, LATTICE, SLMRec, and BM3 across user groups with different interaction sparsity levels.The authors report larger improvement under Recall for sparser user groups and attribute this advantage to modality-aware self-supervised signals.Source: §3.4 · pp. 6–7, Figure 2 · p. 7

3.5 In-Depth Model Analysis (RQ4)

• Dataset-specific λ0 values yield optimal performance, while ω must balance high-order aggregation against over-smoothing and loss of high-order information.The analysis also reports that different τ and λ1 values per dataset lead to best performance.Source: §3.5 · p. 7, Table 3 · p. 7
• Modality-view anchors perform better on Amazon-Baby and Amazon-Sports, whereas main-view anchors perform better on TikTok; linear alignment also wins on TikTok.Alignment methods have minimal impact on Amazon-Baby and Amazon-Sports; the authors attribute the TikTok linear method's advantage to lower dimensionality and potential overfitting with a parametric matrix.Source: §3.5 · pp. 7–8, Table 4 · p. 7

3.6 Effectiveness of Diffusion-enhanced Data Augmentation Paradigm (RQ5)

• On Amazon-Baby and Amazon-Sports, performance declines as the fusion ratio shifts from modality-aware graph views toward random augmentation.A ratio of 0 uses only the modality-aware graph, while 1 uses only random augmentation.Source: §3.6 · p. 8, Figure 4 · p. 8

4.1 SSL-Augmented Recommender Systems

• Graph SSL contrasts positive node pairs, sequence SSL uses cropping, masking, and reordering, and MHCN captures high-order connectivity.These methods use graph augmentation, sequence augmentation, and mutual-information maximization, respectively.Source: §4.1 · p. 8

4.2 Multi-Modal Recommendation Methods

• Multimodal recommendation progressed from feature-augmented matrix factorization (VBPR) and attention (ACF, VECF) to graph models of high-order dependencies (MMGCN, GRCN).Source: §4.2 · p. 8

4.3 Generative Models for Recommendation

• Recommendation generative work spans GANs, VAEs, interaction-probability diffusion [22], embedding diffusion, conditional sequence and item denoising [24], and knowledge-graph diffusion [11].Source: §4.3 · p. 8

• DiffMM reconstructs a comprehensive user-item graph with multimodal graph diffusion and uses cross-modal data augmentation to provide self-supervision.Source: §5 · p. 8
• Experiments comparing DiffMM with competitive baselines establish its reported superiority in recommendation performance.Source: §5 · p. 8
• Future work will integrate large language models to guide diffusion using their semantic understanding.Source: §5 · p. 8

• The appendix describes experimental datasets by size, composition, and key characteristics.Source: A APPENDIX · p. 10
• It analyzes the time complexity of key model components and documents implementation details for the model and baselines.Source: A APPENDIX · p. 10
• An interpretability study includes a case study of how the model makes predictions and which factors contribute to them.Source: A APPENDIX · p. 10

• The three datasets are TikTok short-form videos, Amazon-Baby products, and Amazon-Sports products.Source: §A.1 · p. 10
• Sentence-BERT generates text embeddings from TikTok captions and comments and from Amazon product titles and descriptions.Source: §A.1 · p. 10
• Amazon product images are represented by 4096-dimensional visual feature embeddings.Source: §A.1 · p. 10

A.2 Time Complexity Analysis

• Graph aggregation has complexity O((L + 2|M|) × |G| × d + Σ_m |G_m| × d), combining interaction-graph and modality-aware graph edges.L is the number of graph neural layers, |G| and |G_m| are edge counts, |M| is the modality set, and d is embedding dimensionality.Source: §A.2 · p. 10
• Cross-modal contrastive learning has complexity O(L × B × (I + J) × d), where B is batch size and I and J are item and user counts.The analysis defines L as graph neural layers and d as embedding dimensionality.Source: §A.2 · p. 10
• Graph diffusion costs O(B × I × d_diff) during training and O(t × B × I × d_diff) during inference, with t denoting inference steps.The authors describe its practical complexity as acceptable and similar to the contrastive-learning module.Source: §A.2 · p. 10

A.3 Implementation Details

• DiffMM uses PyTorch, Adam, Xavier initialization, batch size 1024, embedding dimension 64, learning rate 1e−3, and one GCN layer.Adam and Xavier use default parameters.Source: §A.3 · p. 10
• DiffMM tunes L2 regularization, loss weights, ω, and temperature τ over the specified candidate values.The searched ranges are λ2: 1e−1 through 1e−6; λ0: 1, 0.5, 0.1, 0.01, 0.001; λ1: 1, 0.1, 0.01; ω: 0.10–1.00; and τ: 0.1, 0.5, 1.0.Source: §A.3 · p. 10
• Baselines share DiffMM’s optimizer, initializer, batch size, and embedding dimension, with shared hyperparameters tuned over the same ranges.Self-supervised methods tune contrastive temperature over {0.1, 0.5, 1.0}; random-message-dropout methods tune dropout from 0.1 to 0.9.Source: §A.3 · p. 10

A.4 Details of Compared Baselines

• The conventional and GNN-based CF baselines include factorization-based MF-BPR, NGCF, and LightGCN.MF-BPR uses pairwise BPR loss; NGCF propagates through the interaction graph, while LightGCN removes redundant transformations and nonlinear activations.Source: §A.4 · p. 10
• Self-supervised baselines include SGL, NCL, and HCCF, using augmentation, clustered-neighbor positive pairs, or local/global cross-view contrast.SGL uses random augmentation; NCL uses EM-based clustering to identify neighboring nodes; HCCF uses a hypergraph neural network.Source: §A.4 · p. 10
• Multi-modal baselines cover feature-enhanced factorization and message passing, refined interaction graphs, item-similarity graphs, and contrastive or augmentation methods.Examples include VBPR, MMGCN, GRCN, LATTICE, CLCRec, MMGCL, SLMRec, and BM3.Source: §A.4 · pp. 10–11

A.5 Model Interpretability Study (RQ6)

• Figure 5 presents a randomly sampled Amazon-Baby visual-modality subgraph and a heat map of item-wise modality-feature similarity.The reported examples connect the graph’s shared-user item pairs with high similarity scores.Source: §A.5 · p. 11
• In Amazon-Baby’s visual-modality graph, items 1334 and 2108 share user 1108 and have similarity 0.97, while items 1131 and 337 share user 1171 and score 0.85.These pairs are not directly connected in the original interaction graph; their links arise through modality features.Source: §A.5 · p. 11
• The case study attributes modality-specific graph generation to step-wise forward and reverse denoising and modality-signal injection.The authors state that these design elements capture latent interaction patterns and incorporate modality-specific information into generated graphs.Source: §A.5 · p. 11