Source-linked AI summary
ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation
Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, Jun Gao
TL;DR
Existing user modeling can lose behavior-specific information by aggregating heterogeneous histories into fixed-size representations, limiting downstream recommendation. ATRank projects behaviors into multiple semantic spaces and uses self-attention and vanilla attention to model relevant interactions. The paper reports better performance and faster convergence, while a unified multi-behavior model performs comparably to optimized individual models.
Problem
Fixed-size sequence encodings and aggregated user features may lose specific behaviors relevant to downstream recommendation, while heterogeneous behaviors are difficult to model.
Method
ATRank projects heterogeneous user behaviors into multiple latent semantic spaces, models their influences with self-attention, and applies vanilla attention between behavior vectors and the ranking item.
Results
ATRank achieves better performance and faster convergence, while one unified model predicts multiple behavior types with comparable performance to highly optimized individual models.
Takeaways & Limitations
Attention can model heterogeneous user behaviors and support unified prediction of multiple user actions for recommendation tasks.
Takeaways & Limitations
The reported framework is currently used for recommendation tasks, and its attention formulation may neutralize relationships across different semantic spaces.
Abstract
from arXiv · showhide
A user can be represented as what he/she does along the history. A common way to deal with the user modeling problem is to manually extract all kinds of aggregated features over the heterogeneous behaviors, which may fail to fully represent the data itself due to limited human instinct. Recent works usually use RNN-based methods to give an overall embedding of a behavior sequence, which then could be exploited by the downstream applications. However, this can only preserve very limited information, or aggregated memories of a person. When a downstream application requires to facilitate the modeled user features, it may lose the integrity of the specific highly correlated behavior of the user, and introduce noises derived from unrelated behaviors. This paper proposes an attention based user behavior modeling framework called ATRank, which we mainly use for recommendation tasks. Heterogeneous user behaviors are considered in our model that we project all types of behaviors into multiple latent semantic spaces, where influence can be made among the behaviors via self-attention. Downstream applications then can use the user behavior vectors via vanilla attention. Experiments show that ATRank can achieve better performance and faster training process. We further explore ATRank to use one unified model to predict different types of user behaviors at the same time, showing a comparable performance with the highly optimized individual models.
Introduction
User behavior modeling must handle heterogeneous histories while preserving the specific behaviors relevant to downstream recommendation. ATRank addresses this with self-attention across multiple semantic spaces and reports stronger performance with faster training.
- Motivation: Fixed-size RNN or CNN encodings struggle to represent both short and long behavior sequences and may lose specific behavior information downstream.RNN methods also have long-term-dependency and parallelization difficulties.
- Motivation: Attention lets recommendation models dynamically reference the small parts of a user history related to the item being ranked.This avoids relying only on one fixed-size representation of the entire behavior sequence.
- Motivation: A one-dimensional attention score can neutralize relationships across semantic spaces because every element of a behavior vector receives the same scalar multiplier.This makes it difficult to preserve highly related semantics while discarding unrelated parts.
- Motivation: Heterogeneous behaviors such as browsing, buying, coupon use, ad clicks, searches, reviews, and video viewing reveal complementary aspects of a user.These behaviors can support more comprehensive user models and better understanding of user intents.
- ATRank: ATRank projects heterogeneous behaviors into multiple latent spaces, applies self-attention among behaviors, and uses vanilla attention with the ranking item.The resulting outputs are fed into a ranking neural network.
- ATRank: Experiments report better performance and faster training for ATRank, while one unified model achieves comparable performance to highly optimized individual models across behavior types.The unified model is further used to predict multiple types of user behaviors simultaneously.
Related Works
Prior recommendation and sequence-modeling work uses multiple behavior features, recurrent encoders, attention, self-attention, and heterogeneous representation learning. These approaches motivate modeling user context directly while addressing sequence length, parallelization, and cross-type behavior challenges.
- Context Aware Recommendation: Industrial recommendation systems commonly extract separate user features for long-term and short-term interests, then build models for each recommendation scenario.The extracted features may be continuous or categorical.
- Context Aware Recommendation: RNN-based recommendation builds context directly from user behaviors but is difficult to parallelize during prediction and compresses histories into a fixed-size aggregated state.This representation may not model short and long sequences well or preserve specific behavior information.
- Attention and Self-Attention: Attention preserves vectors for individual sequence elements so a decoder can reference them at each decoding step.It was introduced in encoder-decoder machine translation and later applied to other tasks, including ads recommendation.
- Attention and Self-Attention: Self-attention models inner relations among data on the encoder side, and prior work reports benefits from projecting each word into multiple spaces.These mechanisms extend attention beyond a single sequence-level representation.
- Heterogeneous Behavior Modeling: Heterogeneous representation learning has been studied in knowledge-graph completion and multimodal learning through projections into relation-type semantic subspaces.This line of work provides a precedent for representing different entity and relation types in specialized spaces.
Framework
ATRank models heterogeneous user behaviors through behavior-specific embeddings, multiple latent semantic spaces, and self-attention before downstream recommendation attention. It preserves behavior-level representations while allowing downstream ranking networks to focus on behaviors relevant to the target item.
- Behavior Representation: ATRank represents each user as heterogeneous behavior tuples containing an action, object, and timestamp.Objects are represented through their belonging features, and the user history is the set of these tuples.
- Behavior Embedding Spaces: Behavior tuples are partitioned into groups by target object type, with group-specific neural networks embedding their raw features.Within each group, behaviors share raw feature spaces and an embedding building block.
- Behavior Embedding Spaces: Temporal and behavior-type information are encoded alongside object features, using bucketized elapsed-time lookups with multiple granularities.The time intervals can grow exponentially, and temporal granularities may differ across behavior groups.
- Latent Semantic Spaces: Behaviors from different groups are projected into K latent semantic spaces so their heterogeneous representations can be compared under common semantics.The projections accommodate different embedding sizes and meanings across behavior types.
- Self-Attention Layer: Self-attention models interactions among behaviors within each semantic space, producing behavior vectors whose intensities reflect influences from other behaviors.Attention scores are normalized within each space, and a bilinear score function measures behavior impacts.
- Downstream Application Network: The resulting vectors are concatenated and transformed, then vanilla attention combines them with the target behavior embedding for downstream ranking.The downstream network can use point-wise or pair-wise fully connected prediction, with ranking based on the target embedding and user encoding.
Experiment
Experiments evaluate ATRank on single-type Amazon behavior data and heterogeneous multi-behavior recommendation tasks. Results report stronger performance in dense histories, faster convergence than RNN/CNN methods, and comparable performance from one unified multi-task model.
- Single-type behavior: ATRank performs better than competitors on Amazon average user AUC, especially when user behavior is dense.The paper attributes this result to self-attention-based user behavior modeling.
- Single-type behavior: Time encoding through self-attention incorporates temporal information as a replacement for RNN/CNN structures.This conclusion is based on average vanilla-attention scores across time buckets in the Amazon dataset.
- Single-type behavior: ATRank converges much faster than RNN- or CNN-based methods during training on the Amazon Electro test set.The paper relates this to avoiding less-parallelizable recurrent operations and enabling cross-behavior influence in one self-attention layer.
- Multi-type behavior: All2one models outperform one2one models by using all behavior types as user history for separately predicted behavior types.The evaluated behavior groups include item, coupon, and query behaviors.
- Multi-type behavior: The all2all model performs three tasks with one model and achieves comparable performance with highly optimized individual models.It uses all user behavior types to predict any behavior type at the same time.
- Case study: Self-attention scores vary across eight latent semantic spaces, with some spaces emphasizing specific behaviors and others aggregating broader behavior patterns.The heatmap uses row-normalized scores, making the score matrix asymmetric.
- Case study: Vanilla-attention spaces differ in focus: some highlight the most correlated behaviors, while others average across all behaviors.Vanilla-attention space numbers need not correspond to self-attention space numbers because a feedforward network performs another nonlinear projection.
Conclusion
ATRank models heterogeneous user behaviors with attention across multiple semantic spaces and improves recommendation training and performance. A unified multitask version predicts different user actions with performance comparable to optimized individual models.
- ATRank models heterogeneous user behaviors using self-attention across multiple semantic spaces.The framework is evaluated on recommendation tasks.
- ATRank achieves faster convergence while obtaining better performance in experiments.
- A unified ATRank model predicts all types of user actions with performance comparable to highly optimized individual models.
- A case study provides insight into how attention works in ATRank.