Multiplex Behavioral Relation Learning for Recommendation via Memory Augmented Transformer Network

Lianghao Xia; Chao Huang; Yong Xu; Peng Dai; Bo Zhang; Liefeng Bo

Source-linked AI summary

Multiplex Behavioral Relation Learning for Recommendation via Memory Augmented Transformer Network

Lianghao Xia, Chao Huang, Yong Xu, Peng Dai, Bo Zhang, Liefeng Bo

arXiv:2110.04002v1Oct 8, 2021cs.IRcs.AI

TL;DR

Recommendation methods often overlook multiplex, inter-dependent user behaviors and their type-specific context. MATN combines transformer-based cross-behavior relation learning, memory attention, and gated aggregation to model these signals. It outperforms 12 baselines across two benchmark datasets and major-platform behavior data, while future work includes auxiliary data and temporal modeling.

Problem
Existing methods often focus on singular behavior types, although practical interactions have arbitrary cross-type dependencies and distinct contextual semantics.
Method
MATN combines cross-behavior embedding, memory-augmented representation recalibration, and aggregation of type-specific embeddings for prediction.
Results
MATN outperforms 12 baselines on two benchmark datasets and user behavior data from a major online retailing platform.
Takeaways & Limitations
MATN encodes implicit cross-type behavioral dependencies and preserves individual behavioral contextual information for multiplex recommendation.
Takeaways & Limitations
Future work is needed to incorporate auxiliary data and account for the time dimension of arriving multi-type interactions.

Abstract

from arXiv · show

Capturing users' precise preferences is of great importance in various recommender systems (eg., e-commerce platforms), which is the basis of how to present personalized interesting product lists to individual users. In spite of significant progress has been made to consider relations between users and items, most of the existing recommendation techniques solely focus on singular type of user-item interactions. However, user-item interactive behavior is often exhibited with multi-type (e.g., page view, add-to-favorite and purchase) and inter-dependent in nature. The overlook of multiplex behavior relations can hardly recognize the multi-modal contextual signals across different types of interactions, which limit the feasibility of current recommendation methods. To tackle the above challenge, this work proposes a Memory-Augmented Transformer Networks (MATN), to enable the recommendation with multiplex behavioral relational information, and joint modeling of type-specific behavioral context and type-wise behavior inter-dependencies, in a fully automatic manner. In our MATN framework, we first develop a transformer-based multi-behavior relation encoder, to make the learned interaction representations be reflective of the cross-type behavior relations. Furthermore, a memory attention network is proposed to supercharge MATN capturing the contextual signals of different types of behavior into the category-specific latent embedding space. Finally, a cross-behavior aggregation component is introduced to promote the comprehensive collaboration across type-aware interaction behavior representations, and discriminate their inherent contributions in assisting recommendations. Extensive experiments on two benchmark datasets and a real-world e-commence user behavior data demonstrate significant improvements obtained by MATN over baselines. Codes are available at: https://github.com/akaxlh/MATN.

1 INTRODUCTION

Existing recommendation methods often model only one interaction type, although user-item behaviors are multiplex, inter-dependent, and contextually distinct. MATN addresses these challenges by combining cross-behavior relation learning, type-specific contextual modeling, and gated behavior aggregation, outperforming 12 baselines across three datasets.

Motivation: Most recommendation methods focus on singular user-item behavior, despite practical interactions involving inter-dependent types such as viewing, favoriting, cart addition, and purchasing.These behaviors can provide complementary signals for predicting purchases.
Challenges: Arbitrary dependencies between behavior types can vary across users, items, and categories, limiting approaches that model only cascading correlations.The paper identifies comprehensive cross-type dependency modeling as a central challenge.
Challenges: Type-specific behaviors differ in frequency, semantics, and relationships, so their contextual information and contributions to target prediction must be modeled carefully.Examples include frequent page views, delayed purchases after favoriting, and supportive or exclusive behavioral patterns.
Approach: MATN uses a transformer-based dependency encoder, memory attention for type-specific behavior semantics, and gated aggregation to combine behavior-specific representations.The framework preserves cross-type collaborative signals while recalibrating representations with contextual information.
Evaluation: MATN outperforms 12 baselines on two benchmark datasets and user behavior data from a major e-commerce platform.The study also examines interpretability through qualitative case studies and model efficiency across recommendation scenarios.

2 PRELIMINARY

The paper formulates multiplex behavior recommendation using a tensor of user-item interactions indexed by users, items, and behavior types. The task is to infer unknown interactions for a specified target behavior from this tensor.

Behavior Definitions: Target behavior is the interaction type to predict, while source behaviors are other relevant user-item interaction types used as signals.Purchase can be the target, with click, cart addition, and favorite as source behaviors.
Multi-Behavior Tensor: The multi-behavior tensor X represents L behavior types across I users and J items, with each entry indicating whether an interaction occurred.The formulation focuses on implicit feedback, where x_i,j,l = 1 denotes an observed interaction.
Problem Formulation: The recommendation task takes the multi-behavior tensor containing target and source behaviors as input and predicts unknown interactions for target behavior l.The formal objective expresses the probability of x_i,j,l = 1 as a function of X.

3 METHODOLOGY

MATN models multiplex user-item behavior through cross-behavior dependency encoding, type-specific memory transformations, adaptive aggregation, and ranking-based prediction.

Architecture: MATN uses cross-behavior embedding layers, customized representation recalibration, and a forecasting layer to predict user-item interactions.The architecture preserves cross-type dependencies, individual behavioral context, and outputs predicted interaction likelihoods.
Multi-Behavior Dependency Modeling: A shared projection layer maps each behavior’s interaction vector into an initial latent representation using common parameters across behavior categories.The projection provides the starting state for subsequent dependency modeling and representation refinement.
Multi-Behavior Dependency Modeling: Multi-head self-attention lets behavior-specific representations exchange information and identify informative correlations across interaction types.Residual addition preserves both projected behavior features and dependency-aware signals.
Customized Behavioral Context Learning: The memory module applies attention-weighted transformations to refine general behavior embeddings in type-aware latent spaces.Different memory transformations capture behavioral context, while attention over memory matrices is used to reduce overfitting risk.
Multiplex Relation Aggregation Layer: A behavior-type gating mechanism adaptively weights refined representations before a feed-forward network produces the final user representation.The gate learns the distinct contributions of behavior types when modeling user preferences.
The Learning Process of MATN: MATN scores user-item tendencies with a dot product between item embeddings and user representations, then trains with pairwise positive-negative ranking loss.The loss includes weight-decay regularization, and learning proceeds through mini-batch optimization over sampled interactions.

4 EVALUATION

The evaluation tests MATN across benchmark and real-world recommendation datasets, using leave-one-out ranking metrics and comparisons with 12 baselines. It also examines module impact, behavior integration, hyperparameters, interpretability, and scalability.

Research Questions: The experiments investigate comparative performance, module necessity, behavior-type integration, hyperparameter effects, captured relational patterns, and scalability.These questions are organized as RQ1 through RQ6.
Datasets: Experiments use MovieLens, Yelp, and real-world E-Commerce datasets with multiple behavior types and defined target interactions.MovieLens and Yelp distinguish dislike, neutral, and like behaviors; E-Commerce includes page view, add-to-favorite, add-to-cart, and purchase.
Evaluation Protocol: Evaluation uses leave-one-out testing, 99 sampled negative items per ground-truth item, and HR@k and NDCG@k ranking metrics.The latest interaction is held out for testing, while remaining interactions are used for training.
Baselines: MATN is compared with 12 baselines spanning matrix factorization, multi-behavior learning, neural collaborative filtering, auto-encoding, autoregressive, and graph-based recommendation.The baselines include BiasMF, NCF variants, DMF, NMTR, DIPN, AutoRec, CDAE, CF-NADE, CF-UIcA, ST-GCN, and NGCF.
Implementation: MATN uses a 16-dimensional hidden state, two attention heads, eight memory transformations, Adam optimization, learning rate 1e-3, and batch size 32.These settings define the reported implementation configuration.

4.2 Performance Comparison (RQ1)

MATN improves recommendation performance across venue, movie, retail, and click-prediction settings by modeling cross-type behavior dependencies. Its gains are consistent across datasets and ranking cutoffs, with the largest improvement reported for E-Commerce data.

Target Behavior: MATN achieves remarkable performance improvements on Yelp, MovieLens, and E-Commerce recommendation tasks by exploring cross-type behavior dependencies.The comparison covers like behavior on Yelp and MovieLens and purchase behavior on E-Commerce.
Target Behavior: The largest improvement occurs on E-Commerce data, while consistent gains across interaction densities suggest robustness under different sparsity degrees.The authors relate the E-Commerce result to four behavior types and possible purchase-related ordinal relations.
Baseline Limitation: NMTR and DIPN model behavior correlations through singular cascading dependencies, limiting their ability to capture arbitrary cross-type dependencies.The paper identifies this simplification as a source of suboptimal recommendation results.
Overall Click Prediction: MATN achieves the best performance across all datasets when forecasting overall click behavior.This evaluates whether jointly modeling multiple user behavior types benefits overall interaction prediction.
Ranking Robustness: MATN consistently outperforms representative baselines across different Top-K settings on Yelp ranking evaluation.The varying-cutoff comparison reports both HR@K and NDCG@K for click and like prediction.

4.3 Model Ablation Study (RQ2)

The ablation study shows that MATN benefits from modeling cross-type behavior dependencies, type-specific context, and differing behavior importance.

The ablation compares MATN with variants removing the transformer, memory attention, or gating components.
Removing the multi-behavior transformer substantially reduces performance, supporting explicit pair-wise relation learning across behavior types.The result supports the use of the multi-head self-attention architecture for modeling mutual dependencies.
MATN consistently outperforms MATN-M, demonstrating the importance of preserving individual behavior context and semantics.
Replacing MATN’s gating mechanism with average pooling degrades performance because average pooling cannot model differing behavior importance.

4.4 Impact Studies of Multi-Behavior Relation Integration (RQ3)

Integrating multiple interaction behaviors improves purchase prediction, with page views providing especially important information in the e-commerce setting.

The purchase-only MATN_B variant performs worst, indicating positive contributions from page view, add-to-favorite, and add-to-cart behaviors.
MATN using all interaction types consistently outperforms the reduced variants across varying top-k settings, except for a minor top-1 defect.
Removing page-view behavior causes more severe degradation than removing add-to-favorite or add-to-cart behavior.

4.5 Hyperparameter Study of MATN (RQ4)

Hyperparameter studies examine hidden-state dimensionality, memory dimension, and feature-extraction depth, revealing gains followed by limited benefit or overfitting at larger settings.

The study evaluates MATN across hidden-state dimensionality, memory dimension, and neural-network depth.
Hidden State Dimensionality d: Increasing hidden-state dimensionality d from 4 to 16 improves recommendation performance, whereas d ≥ 32 may not improve prediction accuracy.
Memory Dimension M: Increasing memory dimension M initially benefits recommendation, but continued increases can lead to overfitting.
Feature Extraction Network Depth N: Using two hidden layers improves performance, while three or more layers can produce overfitting.

4.6 Case Study on Model Interpretation (RQ5)

The case study visualizes MATN’s learned attention and aggregation weights to interpret behavior relationships and recommendation signals.

The analysis visualizes weights from multi-head self-attention, memory-augmented attention, and multiplex relation aggregation across four sampled cases.
Page views and purchases provide more informative signals for predicting clicks and purchases, respectively.
The study also evaluates MATN’s per-epoch training time against competing recommendation methods.
MATN has comparable time complexity to many baselines, while graph neural network methods show a running-time gap associated with graph convolution and information propagation.

5 RELATED WORK

Prior recommender work uses deep, graph-based, relation-aware, and attention mechanisms to model user–item interactions. The cited figures and table cover hyper-parameter behavior, learned module weights, and computational cost.

Experimental Analysis: Figure 4 studies hyper-parameter effects using HR@10 and NDCG@10.The supplied caption identifies the metrics but not the individual settings or outcomes.
Experimental Analysis: Figure 5 reports learned quantitative weights for MATN’s behavior-relation, memory-attention, and gating modules, while Table 5 investigates computational time cost in seconds.The figure describes a 4 × 4 behavior-relation weight matrix, eight memory dimensions, and four behavior relevance scores.
Deep Collaborative Filtering Techniques: Deep collaborative filtering uses multilayer perceptrons, encoder–decoder networks, and graph neural networks to learn nonlinear or graph-informed user–item representations.These approaches include autoencoder variants and graph models using convolution or message passing.
Relation-aware Recommender Systems: Relation-aware recommender systems incorporate social relations, knowledge graphs, and multiple item relationships as additional recommendation signals.These relations include shared directors, categories, and other item-side connections.
Attention Network for Recommendations: Attention mechanisms differentiate recommendation relations such as item transitions, user connections, and customer-group dynamics.Self-attention is introduced to model pairwise correlations without recurrent models’ rigid order assumptions.

6 CONCLUSION

MATN is proposed as a memory-augmented transformer architecture for cross-behavior collaborative filtering and reports state-of-the-art performance across benchmark and real-world data. Future work concerns adding auxiliary data and modeling the time dimension of multi-type interactions.

Conclusion: MATN incorporates multiple user-behavior relationships into a cross-behavior collaborative filtering framework.The architecture is described as memory augmented and transformer based.
Conclusion: MATN achieves state-of-the-art performance on two benchmark datasets and large-scale user behavior data from a major online retailing platform.The paper also analyzes attentive weights to identify implicit cross-type behavioral dependencies encoded by MATN.
Future Work: Future work includes incorporating auxiliary text data and adapting MATN to time-sensitive multi-type interaction modeling.The paper frames time-sensitive adaptation as a trade-off between accuracy and complexity.