PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Memory Network for Recommendation Systems

Travis Ebesu, Bin Shen, Yi Fang

arXiv:1804.10862v2cs.IR

TL;DR

Existing deep collaborative-filtering methods focus on latent factors while overlooking nonlinear neighborhood-based approaches. CMN unifies global latent factors and local neighborhoods with memory, associative addressing, and attention, and experiments on three public datasets show improvements over competitive baselines.

  • Problem

    Existing deep collaborative-filtering architectures integrate latent factor models but omit nonlinear integration of neighborhood-based approaches.

  • Method

    CMN combines user and item memories, associative addressing, neural attention, and nonlinear interaction to learn user-item-specific neighborhoods and ranking scores.

  • Results

    Comprehensive experiments on three public datasets demonstrate significant improvements over competitive baselines.

  • Takeaways & Limitations

    Attention visualizations provide insight into CMN’s recommendation process and suggest that higher-order transitive relations may be present.

  • Takeaways & Limitations

    The study leaves top-K neighbor prefiltering techniques for future work.

Abstract

from arXiv · show

Recommendation systems play a vital role to keep users engaged with personalized content in modern online platforms. Deep learning has revolutionized many research fields and there is a recent surge of interest in applying it to collaborative filtering (CF). However, existing methods compose deep learning architectures with the latent factor model ignoring a major class of CF models, neighborhood or memory-based approaches. We propose Collaborative Memory Networks (CMN), a deep architecture to unify the two classes of CF models capitalizing on the strengths of the global structure of latent factor model and local neighborhood-based structure in a nonlinear fashion. Motivated by the success of Memory Networks, we fuse a memory component and neural attention mechanism as the neighborhood component. The associative addressing scheme with the user and item memories in the memory module encodes complex user-item relations coupled with the neural attention mechanism to learn a user-item specific neighborhood. Finally, the output module jointly exploits the neighborhood with the user and item memories to produce the ranking score. Stacking multiple memory modules together yield deeper architectures capturing increasingly complex user-item relations. Furthermore, we show strong connections between CMN components, memory networks and the three classes of CF models. Comprehensive experimental results demonstrate the effectiveness of CMN on three public datasets outperforming competitive baselines. Qualitative visualization of the attention weights provide insight into the model's recommendation process and suggest the presence of higher order interactions.

1 INTRODUCTION

CMN addresses a gap in deep collaborative filtering by integrating neighborhood-based structure with latent factors through memory and attention. The resulting hybrid architecture models user-item relations and adaptive neighborhoods nonlinearly.

  • Collaborative filtering includes memory-based, latent-factor, and hybrid approaches with complementary local and global structures.
  • Existing deep collaborative-filtering architectures emphasize latent factor models while omitting nonlinear neighborhood-based integration.
  • The output module combines adaptive neighborhood information with user and item memories through nonlinear interactions to derive recommendations.
  • Stacking memory components enables deeper reasoning over increasingly precise user neighborhoods.
  • CMN uses external memory, associative addressing, and neural attention to identify and weight user neighborhoods for implicit feedback.

2 RELATED WORK

Prior recommendation research applies deep architectures to latent factors, content, context, and attention, but CMN targets collaborative filtering with memory-based neighborhood reasoning. Memory networks provide external memory and attention mechanisms suited to tracking relations and selecting relevant information.

  • Deep recommendation models include autoencoders, neural matrix factorization, CNNs, RNNs, and other nonlinear architectures.
  • Existing attention-based recommender methods commonly use additional content or context, whereas CMN studies attention for collaborative filtering alone.
  • Memory-augmented neural networks combine external memory with a controller that reads, writes, or erases memory representations.
  • Memory components increase capacity and support internal knowledge representations for long-term dependencies and reasoning.
  • Attention focuses processing on selected subsets of information rather than treating all information uniformly.

3 COLLABORATIVE MEMORY NETWORK

CMN maintains user, item, and neighborhood memory states to combine local neighborhood structure with global latent-factor structure. Associative addressing selects similar users, attention weights them adaptively, and stacked hops extend the architecture.

  • CMN maintains internal user-specific memory, item-specific memory, and a collective neighborhood state.
  • The architecture jointly applies nonlinear interactions to neighborhood-based local structure and latent-factor global structure.
  • Associative addressing selects semantically similar users based on the current item.
  • Neural attention learns adaptive nonlinear weights so more similar users contribute more strongly to the output.
  • Stacking multiple hops produces a deeper CMN architecture for more complex neighborhood reasoning.

3.1 User Embedding

CMN represents users and items in separate memory matrices and computes item-specific user preference similarities over an interaction-defined neighborhood. Associative addressing uses these similarities to select relevant internal memories.

  • The user memory matrix M stores user preference embeddings, while the item memory matrix E stores item-specific attributes.
  • Each memory cell has dimensionality d, with P user slots and Q item slots.
  • The user preference vector q^u_i contains target-user agreement similarities with neighborhood users for item i.
  • For implicit feedback, N(i) contains users who interacted with item i; explicit-feedback users can instead be represented by R(i).
  • Associative addressing combines target-user compatibility and neighbor support for the item to identify the most similar internal memories.

3.2 Neighborhood Attention

CMN learns an adaptive, item-specific weighting over neighborhood users rather than relying on fixed similarity heuristics or a predefined neighbor count. Its external memory stores user-specific neighborhood information, while attention produces the collective neighborhood representation.

  • 3.2 Neighborhood Attention: The attention mechanism learns nonlinear weights over neighborhood users, emphasizing influential users and reducing the contribution of less similar users.Traditional methods use Pearson correlation or cosine similarity and require specifying how many users to consider.
  • 3.2 Neighborhood Attention: External memory stores long-term information about each user’s role, and attention retrieves relevant values after associative addressing identifies similar users.The associative addressing scores act as keys, while external-memory embeddings provide the corresponding values.
  • 3.2 Neighborhood Attention: CMN assigns user contributions dynamically according to the target item instead of fixing the neighborhood size, improving the model’s capacity to generalize.The mechanism also reduces the need to encode all neighborhood information in each individual memory slot.

3.3 Output Module

The output module combines local neighborhood information with global user and item memories through nonlinear interactions to produce the recommendation score. This integration is intended to combine neighborhood-based and latent-factor structures while benefiting sparse users.

  • 3.3 Output Module: The output module integrates the collective neighborhood state with user and item memories through a nonlinear interaction before producing the ranking score.The neighborhood represents localized user-item relations, whereas the memories represent global user-item interactions.
  • 3.3 Output Module: An elementwise product combines user and item memories, followed by a linear projection and a skip-connection that shortens the longest path from output to input.The skip-connection is used to encourage information flow and ease learning.
  • 3.3 Output Module: For sparse users, CMN can leverage all users who rated the target item to obtain additional information about user-item relations.The model also adjusts each user’s contribution according to the specific item.

3.4 Multiple Hops

CMN extends to multiple memory hops by iteratively refining the neighborhood representation. Each hop uses information from the preceding hop to reconsider similar users and produce a more precise final neighborhood before scoring.

  • 3.4 Multiple Hops: Each hop queries user and item memories and applies attention to derive the next collective neighborhood state.The first hop may acquire additional information needed for later neighborhood inference.
  • 3.4 Multiple Hops: Starting with the second hop, the previous collective neighborhood guides the search for a representation of community preferences.Stacked modules pass the output of hop h into hop h+1.
  • 3.4 Multiple Hops: The initial query is z0_ui = mu + ei, and nonlinear transformations update the internal state before soliciting the user neighborhood.The hop-specific matrix Wh maps the prior user-preference query to a latent space while incorporating earlier information.
  • 3.4 Multiple Hops: Repeated hops recompute user-neighborhood compatibility and adaptive attention, yielding an iteratively refined summary used by the output module for recommendation.The final scoring module receives the weighted neighborhood vector from the Hth hop.

3.5 Parameter Estimation

CMN is trained for implicit feedback using pairwise preferences: an observed item should outrank an unobserved or negative item. The formulation uses differentiable optimization and shares embedding matrices across hops to reduce parameters.

  • 3.5 Parameter Estimation: Implicit feedback records 1 for observed items and 0 otherwise, so CMN models a user’s preference for observed item i+ over negative item i−.Pairwise triplets (u, i+, i−) avoid assuming that an unobserved item is explicitly disliked.
  • 3.5 Parameter Estimation: CMN uses Bayesian Personalized Ranking as its loss function, approximating the area under the ROC curve.The supplied formulation identifies BPR as the optimization criterion for the pairwise objective.
  • 3.5 Parameter Estimation: The differentiable architecture can be trained with backpropagation, while layerwise weight tying shares embedding matrices across hops to reduce parameter count.The model is not restricted to the logistic sigmoid and can use other pairwise probability functions such as Probit.

3.6 Computational Complexity

CMN’s forward-pass cost depends primarily on the target item’s neighborhood size and embedding dimension rather than the total number of items. Training uses two such passes, while runtime recommendation uses one.

  • Forward-pass complexity: O(d|N(i)| + d^2) is the dominant forward-pass complexity when |N(i)| is comparable to or larger than d.The full expression also includes an additional d-order term in the supplied formulation.
  • Forward-pass complexity: Each additional memory hop adds O(d|N(i)| + d^2) complexity.
  • Training complexity: Training computes separate forward passes for observed positive and negative unobserved items, with parameter updates retaining the same complexity.
  • Efficiency: CMN is efficient because it computes similarities only over target-user neighbors, whose size is often less than or comparable to the total item count Q.Top-K prefiltering could further limit neighbors, but the study leaves that technique for future work.
  • Runtime recommendation: Runtime recommendation requires one network pass to score a user-item pair, and its complexity matches a single training forward pass.The highest-scoring item is recommended.

3.7 Relation to Existing Models

CMN unifies latent-factor and neighborhood-based collaborative filtering components through memory and attention, while connecting its modules to established CF and memory-network formulations. Its output module provides a nonlinear hybrid interaction rather than the simpler combination used by SVD++.

  • Connections to CF models: CMN components correspond to latent-factor, neighborhood-based, hybrid, and memory-network perspectives on collaborative filtering.The section explicitly relates CMN to all three CF classes and to memory networks.
  • Latent-factor models: Removing the neighborhood term and bias while setting U to the identity reduces CMN’s formulation to a latent-factor model.
  • Latent-factor models: Setting the nonlinear function to identity and constraining v to an all-ones vector recovers matrix factorization, while the pairwise loss recovers BPR.
  • Neighborhood-based models: CMN’s neighborhood module acts as a learned similarity matrix, with attention providing normalization and factorizing user-user similarity as S = CCT.Under its prediction rule, this yields a user-based FISM variant and FISMauc under BPR loss.
  • Hybrid models: Unlike SVD++, CMN combines latent factors and neighborhood terms through user-item-specific attention and complex nonlinear interactions.The user and item memories represent global interactions, while the memory module represents the localized neighborhood component.
  • Memory networks: CMN maps recommendation onto a memory-network analogy in which the user neighborhood is the story and the ranking score is the answer.

4 EXPERIMENTAL RESULTS

Experiments on three public implicit-feedback datasets show that CMN consistently outperforms competitive baselines, while ablations and sensitivity analyses clarify the roles of memory, attention, nonlinearity, hops, and embedding size.

  • Baseline Comparison: CMN variants obtain the best HR and NDCG performance at cutoffs 5 and 10 across Epinions, citeulike-a, and Pinterest.The comparison covers CMN with one, two, and three hops against the listed baselines.
  • Baseline Comparison: CMN improves over CDAE, GMF, and NeuMF, indicating benefits from integrating memory and attention with nonlinear recommendation models.On Epinions, the authors attribute the increase over methods sharing the same loss to the memory component; on citeulike-a, CMN also exceeds NeuMF.
  • Embedding Size: Larger embeddings generally improve Epinions and Pinterest performance when sufficient hops are used, whereas two hops remain competitive with deeper models on citeulike-a.Single-hop models can degrade with increased capacity on Epinions and citeulike-a, and more than two hops generally adds little benefit.
  • Effects of Attention and Nonlinearity: Removing attention generally worsens performance, and the best results require both attention and nonlinear transformations.The linear-with-attention variant often exceeds the nonlinear-without-attention variant, while the authors identify optimization difficulties as a possible explanation for counterintuitive citeulike-a results.
  • Attention Visualization: Attention visualizations show weights shifting across hops toward users with shared items and, in citeulike-a, reveal higher-level transitive interactions.A user without items directly shared with the target can receive high attention when connected to other neighborhood users.

5 CONCLUSION

CMN unifies latent-factor and neighborhood-based collaborative filtering through a Memory Network-inspired hybrid architecture. Experiments show improvements over competitive baselines, while attention visualizations suggest higher-order transitive relations.

  • CMN unifies the strengths of latent factor and neighborhood-based methods for collaborative filtering with implicit feedback.
  • Comprehensive experiments under multiple configurations demonstrate significant improvements over competitive baselines.
  • Attention-weight visualizations provide insight into the recommendation process and suggest that higher-order transitive relations may be present.
Loading 1804.10862v2…