Source-linked AI summary
Next Item Recommendation with Self-Attention
Shuai Zhang, Yi Tay, Lina Yao, Aixin Sun
TL;DR
Anticipating a user’s next interaction is central to personalized recommendation, while recent actions reflect near-term intent. AtRec models item relationships across interaction sequences and learns user representations with metric learning; it outperforms state-of-the-art models across twelve benchmark datasets.
Problem
Recent interactions reflect near-term user intent, making their temporal preferences important to understand for next-item recommendation.
Source: §1 · p. 1, §3.2 · p. 3Method
AtRec attends to item relationships across a user’s interaction sequence to represent short- and long-term interests, then trains those representations with metric learning.
Source: §1 · p. 1Results
Across twelve benchmark datasets, AtRec outperforms current state-of-the-art models on all datasets by 2.82% to 45.03% (13.96% on average) in standard retrieval metrics.
Source: §1 · p. 2Takeaways & Limitations
Analysis indicates that AtRec captures the importance of recent actions while incorporating short-term intent and long-term preference for next-action prediction.
Source: §6 · p. 9The paper identifies sparsity as a target for future work, proposing user and item side information as a way to address it.
Source: §6 · p. 9
Abstract
from arXiv · showhide
In this paper, we propose a novel sequence-aware recommendation model. Our model utilizes self-attention mechanism to infer the item-item relationship from user's historical interactions. With self-attention, it is able to estimate the relative weights of each item in user interaction trajectories to learn better representations for user's transient interests. The model is finally trained in a metric learning framework, taking both short-term and long-term intentions into consideration. Experiments on a wide range of datasets on different domains demonstrate that our approach outperforms the state-of-the-art by a wide margin.
1 INTRODUCTION
Past interactions provide a basis for predicting users’ next choices, but RNNs and CNNs do not explicitly model item-item interactions across a user’s history. The paper proposes self-attention and metric learning to represent short-term and long-term interests, reporting broad benchmark gains.
1 INTRODUCTION
- RNNs and CNNs do not explicitly capture item-item interactions across a user’s history, although fine-grained item relationships can matter for understanding context.Historical interaction data, including clicks, purchases, and views, has enabled recommender systems; past interactions are generally predictive of the next one.Source: §1 · p. 1
- The proposed model explicitly relates items across the current interaction window, combining local dependencies with global, long-range representations to form a user representation.Self-attention weights the historical items by relevance when forming that representation.Source: §1 · p. 1
- Training uses metric learning to draw the user’s self-attentive representation closer to the prospective positive item.The framework is presented as modeling both temporary and long-lasting user intents.Source: §1 · pp. 1–2
- AtRec outperforms current state-of-the-art models such as Caser and TransRec on all twelve datasets, by 2.82%–45.03% and 13.96% on average in standard retrieval metrics.The paper also reports hyperparameter and ablation studies and qualitative visualizations of learned attention matrices.Source: §1 · p. 2
2 RELATED WORK
The related work surveys sequence-aware recommenders, deep neural recommendation models, and neural attention. It contrasts prior temporal and sequential approaches with the paper’s use of self-attention and metric learning.
2.1 Sequence-aware Recommender Systems
- Sequence-aware recommenders model temporal rating dynamics or rank next items using Markov chains and metric embeddings; some temporal methods are designed specifically for rating prediction.Prior work combines matrix factorization with Markov chains or combines similarity methods with Markov chains for personalized ranking.Source: §2.1 · p. 2
2.2 Deep Neural Network for Recommendation
- Deep recommendation uses MLPs, CNNs, autoencoders, and RNNs; sequential neural models capture behavior but may struggle with sparse data, while this work combines self-attention with metric learning for short- and long-term inclinations.Session-based recommendation differs because user identity is usually unknown, leading to different model structures and learning procedures.Source: §2.2 · p. 2
2.3 Neural Attention Models
- Neural attention has been applied across tasks, while this work uses self-attention to model dependencies and the importance of short-term behavior patterns.The accompanying module illustration maps embeddings of the latest L interactions to self-attentive representations.Source: §2.3 · pp. 2–3, Figure 1 · p. 3
3 THE PROPOSED MODEL: ATTREC
AttRec combines self-attention for short-term intent with metric learning for long-term preference. It predicts future items from interaction histories by modeling relationships across a recent sequence and user–item distances.
3 THE PROPOSED MODEL: ATTREC
- AttRec combines a self-attention module for short-term intent with collaborative metric learning for long-term user preference.The two components contribute to the recommendation list.Source: §3 · p. 3, §3.4 · p. 5
3.1 Sequential Recommendation
- Sequential recommendation predicts the next items a user will interact with from her earlier consumption trajectory.The history is an ordered sequence of items the user has interacted with.Source: §3.1 · p. 3
3.2 Short-Term Intents Modelling with Self-Attention
- Self-attention matches a user's recent interaction sequence against itself, weighting item values to capture relationships between items regardless of their distance.Query, key, and value are formed from recent interaction histories, and the attention output is a weighted sum of the values.Source: §3.2 · pp. 3–4
- Sinusoidal time embeddings are added to query and key so the attention model retains sequential patterns.The time embedding uses sinusoids at different frequencies and is added before the nonlinear transformation.Source: §3.2 · p. 4
3.3 User Long-Term Preference Modelling
- Long-term preference is modeled with user and item latent factors, whose Euclidean distance is expected to be smaller for liked items and larger otherwise.Source: §3.3 · p. 4
3.4 Model Learning
- The recommendation score sums Euclidean-distance-based short-term and long-term scores, with a weighting factor controlling their contributions.Candidate items are ranked in ascending order by this score.Source: §3.4 · pp. 4–5
- Training uses pairwise ranking with sampled negative items and a margin-based hinge loss to distinguish positive from negative user–item pairs.The objective also uses L2 loss to control model complexity.Source: §3.4 · p. 4
4 EXPERIMENTS
The experiments compare AttRec with established baselines across benchmark datasets and use ablations to examine the model and key hyperparameters.
4.1 Datasets Descriptions
- Evaluation spans timestamped MovieLens variants, seven Amazon categories, LastFM, and MovieTweetings; ratings become implicit feedback, and sparse datasets are filtered for users and cold-start items.MovieLens datasets are denser than the others.Source: §4.1 · p. 5
4.2 Evaluation Metrics
- Each user's latest item is held out for testing and the second-latest for tuning; evaluation uses HR@50 and mean reciprocal rank.HR@50 checks whether the target ranks in the top 50, while MRR uses the reciprocal of its rank.Source: §4.2 · pp. 5–6
4.3 Compared Models
- Baselines range from pairwise ranking to Markov, hierarchical, metric-embedding, translation, and convolutional sequential recommenders.Source: §4.3 · p. 6
4.4 Implementation Details
- AttRec and the baselines use 100-dimensional latent vectors; sequence and target lengths vary by dataset, and regularization, dropout, and weighting factors are tuned.Source: §4.4 · p. 6
4.5 Performance Comparison
- Among baselines, Caser performs well on MovieLens and MovieTweetings but poorly on sparse datasets; HRM usually exceeds PRME, with no clear HRM–TransRec winner.Source: §4.5 · p. 6
- AttRec achieves the best performance on all 12 benchmark datasets, including dense MovieLens and sparse Amazon and MovieTweetings datasets.The comparison evaluates both prediction accuracy and ranking quality.Source: §4.5 · p. 6, Table 2 · p. 7
5 MODEL ANALYSIS AND DISCUSSION
The analysis examines aggregation, attention, sequence length, latent dimensions, and runtime. It finds that performance depends on dataset characteristics, while learned attention weights vary across users.
Impact of Self-Attention
- Removing self-attention reduces performance, although AttRec without the module still beats all baselines on the four evaluated datasets.The comparison explicitly removes the self-attention module and replaces the representation with a mean.Source: §5 · p. 7
Impact of Aggregation Method
- Mean aggregation performs desirably on sparse and dense datasets, while the other three strategies underperform especially on sparse data.The authors suggest mean aggregation could retain more information.Source: §5 · p. 8, Table 4 · p. 8
Impact of weight ω
- Setting ω between 0.2 and 0.4 is preferable; short-term-only intent usually performs better than long-term-only preference, and denser sequential signals favor lower ω.The reported effects vary across four datasets.Source: §5 · p. 8, Figure 4 · p. 8
Impact of sequence length L
- Larger sequence lengths benefit MovieLens datasets with over a hundred actions per user, whereas sparse datasets favor smaller L because increasing L reduces training samples.Self-attention can theoretically learn dependencies between distant positions.Source: §5 · p. 8, Figure 4 · p. 8
Impact of Number of Latent Dimensions
- AttRec outperforms all baselines across tested latent dimensions, but larger dimensions do not necessarily improve performance and may overfit.Figure 5 reports HR@50 on Movielens 100K and Amazon Garden.Source: §5 · p. 8, Figure 5 · p. 8
Model Efficiency
- AttRec incurs only a small computational cost over Caser, and reaches its best Movielens 100K performance in fewer than 30 epochs.The authors suggest dual embeddings and Euclidean-distance calculation might account for the added cost.Source: §5 · p. 8, Table 5 · p. 8
6 CONCLUSION
The conclusion presents AttRec as a self-attention-based metric-learning model for sequential recommendation, combining short-term intent and long-term preference. It also identifies side information as future work for addressing sparsity.
Conclusion
- AttRec is a self-attention-based metric-learning approach for sequential recommendation.The model combines user short-term intent with long-term preference to predict next actions.Source: §6 · pp. 8–9
- Self-attention learns short-term intent from recent actions, and the analysis finds that it captures the importance of those actions.The conclusion also describes generalizing self-attention into metric learning for sequence prediction.Source: §6 · p. 9
- The authors identify incorporating user and item side information to address sparsity as future work.They also suggest AttRec may adapt to session-based recommendation or clickthrough prediction.Source: §6 · p. 9