PaperlayerView exact revision on arXiv

Source-linked AI summary

LGMRec: Local and Global Graph Learning for Multimodal Recommendation

Zhiqiang Guo, Jianjun Li, Guohui Li, Chaoyang Wang, Si Shi, Bin Ruan

arXiv:2312.16400v2cs.IR

TL;DR

Existing multimodal recommenders couple collaborative and multimodal signals while underexploring robust global interests. LGMRec decouples local graph embeddings and adds global hypergraph embeddings, substantially outperforming baselines across three datasets.

  • Problem

    Existing multimodal recommenders often share user ID embeddings across collaborative and multimodal modules, while robust global user interests remain underexplored.

  • Method

    LGMRec independently learns collaborative and modality-related local embeddings with graph propagation, captures global dependencies through hypergraph embeddings, and fuses them for prediction.

  • Results

    LGMRec substantially outperforms various baselines across three benchmark datasets, with superior performance consistently observed across user groups of different sparsity levels.

  • Takeaways & Limitations

    Modeling decoupled local interests together with global hypergraph dependencies supports accurate multimodal recommendation and improves performance when user interactions are sparse.

Abstract

from arXiv · show

The multimodal recommendation has gradually become the infrastructure of online media platforms, enabling them to provide personalized service to users through a joint modeling of user historical behaviors (e.g., purchases, clicks) and item various modalities (e.g., visual and textual). The majority of existing studies typically focus on utilizing modal features or modal-related graph structure to learn user local interests. Nevertheless, these approaches encounter two limitations: (1) Shared updates of user ID embeddings result in the consequential coupling between collaboration and multimodal signals; (2) Lack of exploration into robust global user interests to alleviate the sparse interaction problems faced by local interest modeling. To address these issues, we propose a novel Local and Global Graph Learning-guided Multimodal Recommender (LGMRec), which jointly models local and global user interests. Specifically, we present a local graph embedding module to independently learn collaborative-related and modality-related embeddings of users and items with local topological relations. Moreover, a global hypergraph embedding module is designed to capture global user and item embeddings by modeling insightful global dependency relations. The global embeddings acquired within the hypergraph embedding space can then be combined with two decoupled local embeddings to improve the accuracy and robustness of recommendations. Extensive experiments conducted on three benchmark datasets demonstrate the superiority of our LGMRec over various state-of-the-art recommendation baselines, showcasing its effectiveness in modeling both local and global user interests.

Introduction

LGMRec addresses multimodal recommendation’s limitations in modeling user interests by decoupling collaborative and modality-related local representations and capturing global user–item dependencies. This jointly models local and global interests to improve recommendation robustness, particularly under sparse interactions.

  • Background: Multimodal recommender systems integrate item images, texts, and videos with user–item interactions to capture comprehensive user interests.The approach supports personalized services on online media platforms such as YouTube and TikTok.
  • Limitations: Existing multimodal recommenders are limited by low-order user–item interaction modeling and by coupling collaborative and multimodal signals through shared user ID embeddings.Collaborative signals reflect similar behavior patterns, whereas modal knowledge reflects content similarity.
  • LGMRec: LGMRec introduces a local graph embedding module that independently captures collaborative-related and modality-related local user interests through message propagation on user–item interactions.The module is designed to address the coupling limitation.
  • LGMRec: LGMRec jointly models local and global user and item representations to facilitate multimodal recommendation.The proposed framework is explicitly named the Local and Global Graph Learning-guided Multimodal Recommender.

Related Work

Related work develops graph-based, hypergraph-based, and multimodal recommendation methods to model user–item connectivity, complex dependencies, and modality-informed preferences. Existing multimodal approaches extend collaborative filtering, use modality-specific interaction graphs, or learn auxiliary semantic graphs from multimodal features.

  • Graph-based Recommendation: Graph-based recommendation methods convert user histories into user–item bipartite graphs and learn representations by aggregating high-order connectivity.These methods build on graph neural networks’ message-propagation mechanisms.
  • Hypergraph learning for Recommendation: Hypergraph recommendation methods construct hyperedges and node–hyperedge connections to capture complex, high-order interaction patterns.Hyperedges contain more than two nodes and can improve generalization by modeling complex node dependencies.
  • Multi-modal Recommendation: Multimodal recommendation analyzes item images, textual descriptions, and user behaviors to provide personalized services, initially extending vanilla collaborative filtering with multimodal side information.Reported behaviors include reviews and clicks.
  • Multi-modal Recommendation: Later multimodal methods model user high-order interests on modality-specific interaction graphs or learn auxiliary semantic graphs from multimodal features.MMGCN incorporates modality information into graph message passing to infer modality-related user preferences.

Methodology · Problem Statement and Overview · Local Graph Embedding (LGE) Module

LGMRec formulates multimodal recommendation as predicting user–item preference scores from interactions and item modalities, then combines decoupled local graph learning with global dependency modeling. Its LGE module separately learns collaborative-related and modality-related representations to avoid coupling signals and capture local user interests.

  • Methodology: LGMRec’s methodology first formulates multimodal recommendation and presents the framework before detailing its components.
  • Problem Statement and Overview: The interaction matrix R defines a user–item graph whose edges correspond to observed interactions, while users and items have d-dimensional ID embeddings.
  • Problem Statement and Overview: Multimodal recommendation predicts the score of item i for user u by jointly modeling user behaviors and item multimodal contents.The considered modalities are vision and text.
  • Problem Statement and Overview: LGMRec comprises local graph embedding, global hypergraph embedding, and a framework for learning local and global user and item representations.The LGE module uses GNNs for collaborative-related and modality-related local interests, whereas GHE captures global hypergraph dependencies.
  • Local Graph Embedding (LGE) Module: The LGE module independently learns collaborative-related and modality-related user and item representations from local topology to promote decoupled interest learning.This design avoids unstable updates of user embeddings.
  • Local Graph Embedding (LGE) Module: Collaborative graph propagation applies a lightweight graph convolution to the interaction adjacency matrix and combines layers to obtain local collaborative embeddings.The initial embedding matrix is E0 = Eid, and layer integration uses the mean function.
  • Local Graph Embedding (LGE) Module: For modality graph embedding, item features from different pretrained spaces are projected into the unified space R^d using modality-specific transformation matrices.ResNet and BERT are cited as examples of pretrained models producing the original visual and textual features.
  • Local Graph Embedding (LGE) Module: User modal features are initialized by aggregating neighboring item modal features, enabling separate ID and modality updates before light graph propagation.The final K-th layer embeddings are selected as modality-related embeddings with local modal information.

Global Hypergraph Embedding (GHE) Module · Fusion and Prediction

LGMRec’s GHE module learns modality-aware global user and item representations through hypergraph dependencies and message passing, then fuses them with decoupled local embeddings for prediction. Cross-modal contrastive learning and a joint BPR objective support robust multimodal representation learning and recommendation.

  • Hypergraph Dependency Constructing: Learnable implicit attribute vectors serve as modality-specific hyperedge embeddings, adaptively modeling dependencies between attributes, items, and users.This addresses unavailable explicit item-modality attributes, especially for visual modalities.
  • Hypergraph Dependency Constructing: Items with similar modal features tend to share hyperedges, while user-hyperedge dependencies derive from interactions and reflect users’ attribute preferences.Frequent interactions with items under an attribute indicate stronger preference for that attribute.
  • Hypergraph Message Passing: Hypergraph message passing uses attribute hyperedges as intermediate hubs to transfer global information to users and items without hop-distance limitations.The process takes item collaborative embeddings and modality-aware hypergraph dependencies as inputs.
  • Hypergraph Message Passing: Global user and item embeddings are computed across modality-specific hypergraph layers and aggregated into a unified global embedding matrix.Collaborative item embeddings initialize the global embedding process at h = 0.
  • Hypergraph Message Passing: Cross-modal hypergraph contrastive learning treats same-user embeddings across modalities as positive pairs and different users as negative pairs using InfoNCE.The user-side contrastive loss considers visual and textual modalities, with temperature factor τ generally set to 0.2.
  • Fusion and Prediction: Final user and item representations fuse two local embedding types with global embeddings after normalization, with α controlling the global-embedding contribution.Normalization alleviates scale differences among the embeddings.
  • Fusion and Prediction: Preference scores are computed by the inner product of final user and item representations, and model parameters are optimized with Bayesian personalized ranking loss.The training triples contain observed user-item pairs and unobserved negative items.
  • Fusion and Prediction: The hypergraph contrastive loss and BPR loss form a unified objective optimized with Adam, while weight decay regularizes model parameters.The hyperparameter λ2 weights the contrastive-loss term.

Experiment

Experiments on three multimodal Amazon datasets evaluate LGMRec against diverse recommendation baselines using standard top-n metrics. Results show consistent gains from combining decoupled local interests with global hypergraph dependencies, including under sparse interactions and varying hyperparameters.

  • Experimental Setup: LGMRec is evaluated on Baby, Sports, and Clothing using 5-core multimodal Amazon datasets with original visual and textual features.The visual features are 4096-dimensional and the textual features are 384-dimensional.
  • Experimental Setup: Interactions are randomly split into training, validation, and testing sets at an 8:1:1 ratio, with performance reported using Recall and NDCG at n ∈ {10, 20}.Results are averaged over all users in the testing set.
  • Overall Comparison: LGMRec substantially outperforms all listed baselines across the three datasets, attributed to separated local embeddings and modality-related global hypergraph dependencies.Baselines span general collaborative filtering, graph-based, hypergraph-based, and multimodal recommendation models.
  • Ablation Study: Removing multimodal contents reduces LGMRec to LightGCN and yields the worst performance, while removing either local or global graph embedding causes further performance drops.The variant without local graph embedding performs worse than the variant without global hypergraph embedding.
  • Sparsity Analysis: LGMRec maintains superior performance across user groups with different interaction sparsity levels on Baby and Sports compared with five multimodal baselines.The compared baselines are MMGCN, LATTICE, MMGCL, SLMRec, and BM3.
  • Hyperparameter Analysis: On Clothing, increasing the hyperedge number improves performance, while the fusion weight peaks at α = 0.2 before declining.The results indicate that excessive weighting of global embeddings can negatively affect performance.

Conclusion

LGMRec models multimodal recommendation through local graph embeddings and global hypergraph embeddings. Experiments on three datasets demonstrated the model’s superiority.

  • Conclusion: LGMRec captures local embeddings with local topological information and global embeddings with hypergraph dependencies.The model combines local graph structure with global dependency modeling.
  • Conclusion: Its local graph embedding module independently learns collaborative-related and modality-related local user interests.The two local interest types are learned separately within the local graph embedding module.
  • Conclusion: Its global hypergraph embedding module mines global user interests.The global module uses hypergraph dependencies to capture global user interests.
  • Conclusion: Extensive experiments on three datasets demonstrated LGMRec’s superiority.The supplied conclusion reports experiments across three datasets but does not provide quantitative results.

Appendix · Complexity Analysis of LGMRec

LGMRec’s complexity is dominated by local graph embedding and global hypergraph embedding. The analysis specifies costs for collaborative propagation, modal initialization, hypergraph construction and propagation, and contrastive learning.

  • Complexity Analysis of LGMRec: O(L × |E| × d) is the collaborative graph embedding complexity in the local graph embedding module.L denotes graph message-passing layers, |E| the interaction-graph edges, and d the embedding dimension.
  • Complexity Analysis of LGMRec: O(|M|×dm ×d) is the computational cost of modal feature initialization in the local graph embedding module.|M| is the number of modalities, while dm and d denote the corresponding dimensional quantities specified in the analysis.
  • Complexity Analysis of LGMRec: The local graph embedding module separately accounts for collaborative graph embedding and modality graph embedding costs.The supplied analysis identifies these as the two local graph components, although the modality propagation expression is truncated.
  • Complexity Analysis of LGMRec: O(|M| × A × |I| × (|U| + dm)) is the time complexity of hypergraph dependency construction.A is the number of hyperedges; |I| and |U| denote item- and user-related quantities in the stated expression.
  • Complexity Analysis of LGMRec: O(|M| × (|I| × H + |U|) × A × d) is the complexity of hypergraph message passing with global information propagation.H denotes the number of hypergraph layers.
  • Complexity Analysis of LGMRec: O(B × (|U| + |I|) × d) is the cost of hypergraph contrastive learning with only two modalities, v and t.B is the batch size.

Baselines

The paper compares LGMRec with Bayesian, graph-based, hypergraph-enhanced, and multimodal recommendation baselines. These methods represent progressively richer approaches to collaborative, global, and modality-specific preference modeling.

  • Collaborative Filtering: BPR learns low-dimensional user and item embeddings and optimizes them with Bayesian pairwise ranking loss.It represents the traditional latent-factor collaborative-filtering paradigm.
  • Graph-based Recommendations: LightGCN learns high-order user–item connectivity, while SGL and NCL enhance graph collaborative filtering through contrastive learning.SGL uses random edge dropout for data augmentation; NCL treats structural and semantic neighboring nodes as positive samples.
  • Graph-based Recommendations: HCCF injects global collaborative relations through hypergraph neural networks, whereas SHT combines hypergraph encoding and multi-head attention for contrastive global embeddings.Both baselines model global collaborative information beyond ordinary graph connectivity.
  • Multi-Modal Recommendations: VBPR integrates modal features with ID embeddings, while MMGCN, GRCN, and LATTICE model modality-specific preferences or modality-informed graph structure.MMGCN performs modality-specific message passing, and GRCN identifies noisy interaction edges to refine graph structure.

Parameter Setting

The experiments use a common optimization setup across models, with dataset-specific optimal parameters additionally reported in Table 4 and a shared hardware environment.

  • Parameter Setting: All models use batch size 2048, learning rate 0.001, and embedding size d = 64 for fair comparison.Dataset-specific optimal parameter settings are additionally presented in Table 4.
  • Parameter Setting: Experiments run in the same environment with an Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz and GeForce RTX 3090.

Hyperparameter Analysis

The hyperparameter analysis identifies dataset-dependent settings for local graph depth and hyperedge count, while favoring shallow modality and hypergraph propagation. Performance also improves with a moderate global-embedding weight before declining.

  • Collaborative graph layers L: Multi-layer collaborative graphs outperform single-layer models, with deeper local receptive fields preferred for sparse datasets: L = 4 on Sports and L = 3 on Clothing.Combining ID embeddings with sufficient multi-layer local structure improves user and item representations.
  • Modality graph layers K: Two modality graph layers achieve the best performance, while additional layers provide no improvement and may introduce redundant modality knowledge.The passage attributes the decline to decreasing node discrimination as deeper neighbors are aggregated.
  • Hypergraph layers H: Shallow hypergraph propagation performs better than multi-layer propagation, so H = 1 is used for all datasets.Deeper propagation may excessively smooth node representations and reduce performance.
  • Hyperedge number A: Hyperedge settings are dataset-dependent: performance increases with hyperedge count on sparse Clothing, while Baby and Sports typically peak at A = 4.The results support capturing multi-hyperedge global structures, especially for sparser datasets.
  • Adjustable factor α: Performance consistently rises to an optimum and then falls as α increases, with α = 0.3, 0.6, and 0.2 selected for Baby, Sports, and Clothing, respectively.The trend indicates that properly supplementing global embeddings supports robust user-interest modeling.
Loading 2312.16400v2…