PaperlayerView exact revision on arXiv

Source-linked AI summary

Mining Latent Structures for Multimedia Recommendation

Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, Liang Wang

arXiv:2104.09036v2cs.IRcs.LGcs.MM

TL;DR

Multimedia recommendation needs to capture semantic item-item relationships that traditional methods leave implicit when using multimodal features as side information. LATTICE learns modality-aware item graphs and applies graph convolutions to ID embeddings before integrating them with collaborative filtering; experiments across three public datasets support its effectiveness.

  • Problem

    Existing multimedia recommendation methods mainly use multimodal features as side information while modeling user-item interactions, leaving genuine semantic item-item relationships insufficiently explicit.

  • Method

    LATTICE learns modality-aware item graphs from multimodal features, fuses them, applies graph convolutions to item ID embeddings, and integrates the resulting representations with downstream CF methods.

  • Results

    LATTICE significantly and consistently outperforms original CF methods and CF+feats variants across three CF methods, with 17.6% average improvements over CF+feats variants.

  • Takeaways & Limitations

    Mining latent item relationships from multimodal features supplements collaborative signals and supports more comprehensive candidate discovery in multimedia recommendation.

  • Takeaways & Limitations

    The method uses item ID embeddings as graph-convolution inputs rather than multimodal features, because the convolutions are intended to capture item-item affinities directly.

Abstract

from arXiv · show

Multimedia content is of predominance in the modern Web era. Investigating how users interact with multimodal items is a continuing concern within the rapid development of recommender systems. The majority of previous work focuses on modeling user-item interactions with multimodal features included as side information. However, this scheme is not well-designed for multimedia recommendation. Specifically, only collaborative item-item relationships are implicitly modeled through high-order item-user-item relations. Considering that items are associated with rich contents in multiple modalities, we argue that the latent semantic item-item structures underlying these multimodal contents could be beneficial for learning better item representations and further boosting recommendation. To this end, we propose a LATent sTructure mining method for multImodal reCommEndation, which we term LATTICE for brevity. To be specific, in the proposed LATTICE model, we devise a novel modality-aware structure learning layer, which learns item-item structures for each modality and aggregates multiple modalities to obtain latent item graphs. Based on the learned latent graphs, we perform graph convolutions to explicitly inject high-order item affinities into item representations. These enriched item representations can then be plugged into existing collaborative filtering methods to make more accurate recommendations. Extensive experiments on three real-world datasets demonstrate the superiority of our method over state-of-the-art multimedia recommendation methods and validate the efficacy of mining latent item-item relationships from multimodal features.

1 INTRODUCTION

Multimedia recommendation methods commonly extend collaborative filtering with multimodal side information but do not explicitly model semantic item-item relationships. LATTICE mines modality-aware latent item graphs, uses graph convolutions to inject item affinities, and can plug into existing collaborative filtering models.

  • Multimedia recommendation predicts user interactions with items containing modalities such as images, text, and videos.
  • Traditional methods incorporate multimodal contents as item side information but primarily model direct user-item interactions, limiting their expressiveness.
  • Prior graph-based methods model high-order user-item relationships, but item relationships are only implicitly discovered through item-user-item co-occurrences.
  • Semantic item relationships from multimodal features can supplement collaborative signals and help recommend visually or semantically related items that interaction patterns miss.
  • LATTICE learns modality-aware item structures, aggregates modality-specific graphs, and applies graph convolutions to inject high-order item relationships into item representations.
  • LATTICE mines item graphs from multimodal features while applying graph convolutions to ID embeddings, then adds the resulting representations to downstream CF item embeddings.
  • LATTICE addresses limited-interaction settings through neighbor feedback and serves as a plug-and-play module for existing recommender models.

2 THE PROPOSED METHOD

LATTICE mines modality-aware latent item graphs from multimodal features, then uses graph convolutions to inject high-order item relationships into representations for downstream collaborative filtering.

  • Modality-aware latent structure learning: LATTICE learns item graph structures separately from multimodal features, fuses them, and uses the resulting latent graph to model item relationships.Its framework combines modality-aware structure learning, graph convolutional embedding, and downstream CF methods.
  • Modality-aware latent structure learning: For each modality, LATTICE constructs a cosine-similarity kNN graph, suppressing negative similarities and retaining only top-k edges.The resulting adjacency matrices are sparsified and normalized for computational efficiency and gradient stability.
  • Learning latent structures: The model dynamically learns graph structures from transformed high-level features and combines them with initial raw-feature graphs through a skip connection.The coefficient λ controls how much information is retained from the initial structure.
  • Aggregating multimodal latent graphs: Learned modality-specific graphs are adaptively fused with learnable modality weights into a normalized graph representing multimodal item relationships.The weights assign different importance scores to modality-specific graphs.
  • Graph convolutions: Graph convolutions propagate item representations across neighbors, and stacking layers captures high-order item-item relationships constructed from multimodal information.LATTICE uses simple message passing without feature transformations or nonlinear activations, then enhances downstream CF item embeddings with the learned representations.
  • Combining with collaborative filtering: LATTICE combines its item representations with downstream CF methods and is designed as a plug-and-play module for different collaborative filtering methods.The framework enhances item embeddings before computing user-item preference scores.

3 EXPERIMENTS

The experiments evaluate LATTICE against state-of-the-art multimedia recommendation and collaborative filtering methods across warm-start and cold-start settings, while also testing learned item graphs and hyperparameter sensitivity.

  • The experiments compare LATTICE with state-of-the-art multimedia recommendation and collaborative filtering methods.
  • Evaluation covers both warm-start and cold-start recommendation settings.
  • The study examines the effectiveness of item graphs learned from multimodal features.
  • The experiments assess sensitivity to several key hyperparameters.

3.1 Experiments Settings

The evaluation uses three multimodal Amazon datasets, compares LATTICE with collaborative filtering and content-aware baselines, and reports ranking metrics under warm-start and cold-start protocols.

  • Datasets: Experiments use Amazon Clothing, Sports, and Baby datasets containing visual and textual modalities.Visual features have 4,096 dimensions, while textual embeddings have 1,024 dimensions.
  • Baselines: Baselines comprise CF methods MF, NGCF, and LightGCN, plus content-aware models VBPR, MMGCN, and GRCN.
  • Evaluation Settings: Warm-start data splits each user’s historical interactions into 80% training, 10% validation, and 10% testing.Observed positives are paired with one negatively sampled item.
  • Evaluation Settings: Cold-start evaluation removes interactions for a randomly selected 20% item set, dividing those unseen items equally between validation and testing.
  • Metrics: Performance is measured with Recall@20, NDCG@20, and Precision@20, averaged across testing users.

3.2 Performance Comparison (RQ1)

LATTICE outperforms the compared methods in overall and cold-start evaluations, with explicit latent item relationships complementing collaborative and content-based signals.

  • Overall Performance: 12.5%, 9.8%, and 9.9% are LATTICE’s Recall@20 improvements over the strongest baselines on Clothing, Sports, and Baby, respectively.The reported improvements are significant with p-value ≤0.05.
  • Overall Performance: LATTICE significantly outperforms both collaborative filtering and content-aware methods across the evaluated datasets.The comparison includes CF and content-aware baselines, with Table 2 reporting Recall@20, Precision@20, and NDCG@20.
  • Cold-start Performance: LATTICE outperforms all baselines across Clothing, Sports, and Baby in cold-start settings.Its learned item graphs transfer feedback from relevant neighbors to cold-start items through neighborhood aggregation.
  • Cold-start Performance: MF and LightGCN generally perform poorly in cold-start settings because unseen items lack user-item interactions for representation updates.
  • Cold-start Performance: VBPR generally outperforms CF methods for cold-start items, whereas GRCN performs poorly because cold-start items become isolated in its user-item graphs.

3.3 Ablation Studies (RQ2)

Ablation studies show that LATTICE consistently improves three downstream CF methods over multimodal-feature and original CF variants, supporting latent-structure discovery and the plug-in design.

  • Ablation Results: LATTICE–CF consistently outperforms original CF methods and two comparison variants when combined with MF, NGCF, and LightGCN.
  • Ablation Results: 17.6% average improvements are obtained over CF+feats variants that directly use multimodal features as item side information.
  • Ablation Results: LATTICE–CF applies graph convolutions to item ID embeddings using the same learned structures as LATTICE/feats–CF.This design aims to directly model item affinities.

3.4 Sensitivity Analysis (RQ3)

LATTICE performs best with a moderate number of item neighbors and a balanced skip-connection coefficient, while excessive neighbors or raw-feature weighting can introduce noise.

  • Impact of varied k values: LATTICE improves significantly from k=0 to k=10, supporting the usefulness of item relationships mined from multimodal features.Here, k=0 excludes item relationships and reduces the model to original LightGCN.
  • Impact of varied k values: Performance initially increases with k because aggregating more item neighbors provides additional meaningful information.
  • Impact of varied k values: Performance declines when k becomes too large because unimportant neighbors introduce noise during information propagation.The result motivates kNN sparsification of the learned dense graph.
  • Impact of varied coefficients: With λ=0, performance is poor because the model relies only on transformed high-level features and continually updates the adjacency matrix.
  • Impact of varied coefficients: Performance first rises as λ increases, then deteriorates when raw multimodal features receive excessive weight because those features can be noisy.
  • Impact of varied coefficients: The absence of sharp performance changes across λ values indicates that LATTICE is not highly sensitive to selecting this coefficient.All tested models surpass the baselines according to the cited analysis.

4 RELATED WORK

Related work combines collaborative filtering with multimodal side information and graph learning, but LATTICE explicitly discovers fine-grained semantic item relationships from multimodal features.

  • Multimedia Recommendation: CF-based multimedia recommenders extend collaborative filtering by adding multimodal item features as side information.Examples include VBPR, DVBPR, Sherlock, DeepStyle, and ACF.
  • Multimedia Recommendation: Existing multimedia methods generally use multimodal features directly for each item rather than discovering relationships among items.LATTICE instead mines fine-grained item-item relationships from those features.
  • Graph Structure Learning: Graph neural networks aggregate neighborhood information, but graph noise can propagate through repeated updates and affect many node embeddings.
  • Graph Structure Learning: Graph structure learning jointly optimizes graph structures and representations through metric learning, probabilistic modeling, or direct optimization.
  • Graph Structure Learning: NeuralSparse learns k-neighbor subgraphs by selecting at most k neighbors for each node, while other methods learn adaptive or probabilistic graphs.
  • Graph Structure Learning: Because item-item relations are underexplored in personalized recommendation, LATTICE uses metric learning to represent edge weights as distances between item nodes.This formulation accommodates multimodal content when measuring semantic relationships.

5 CONCLUSION

LATTICE mines latent item relationships from multimodal features, propagates neighbor affinities through graph convolutions, and supports downstream collaborative filtering. Experiments on three public datasets demonstrate its effectiveness.

  • LATTICE learns and fuses modality-aware item graphs, then uses graph convolutions to inject high-order neighbor affinities into item representations.The enriched representations are combined with downstream collaborative filtering methods for recommendation.
  • Empirical results on three public datasets demonstrate the effectiveness of LATTICE.
Loading 2104.09036v2…