NAIS: Neural Attentive Item Similarity Model for Recommendation

Xiangnan He; Zhankui He; Jingkuan Song; Zhenguang Liu; Yu-Gang Jiang; Tat-Seng Chua

Source-linked AI summary

NAIS: Neural Attentive Item Similarity Model for Recommendation

Xiangnan He, Zhankui He, Jingkuan Song, Zhenguang Liu, Yu-Gang Jiang, Tat-Seng Chua

arXiv:1809.07053v1Sep 19, 2018cs.IR

TL;DR

Item-based collaborative filtering needs recommendation-optimized item similarities, but neural models have been relatively underexplored compared with shallow linear approaches. NAIS adds an attention network to FISM to weight historical items by prediction relevance, and experiments show it significantly outperforms FISM while retaining competitive item-recommendation performance.

Problem
Neural network models for learning item similarities in item-based collaborative filtering have been relatively underexplored, despite the importance of accurate recommendation-optimized similarities.
Method
NAIS extends FISM with an attention network that learns varying importance among a user's historical items, using a smoothed attention design for variable-length histories.
Results
NAIS significantly outperforms FISM and achieves competitive performance for the item recommendation task.
Takeaways & Limitations
NAIS provides a neural approach to item-based collaborative filtering while preserving FISM's high online-prediction efficiency and supporting fast user-representation refreshes.
Takeaways & Limitations
The global attention-weight design ignores the target item's effect on the importance of historical items.

Abstract

from arXiv · show

Item-to-item collaborative filtering (aka. item-based CF) has been long used for building recommender systems in industrial settings, owing to its interpretability and efficiency in real-time personalization. It builds a user's profile as her historically interacted items, recommending new items that are similar to the user's profile. As such, the key to an item-based CF method is in the estimation of item similarities. Early approaches use statistical measures such as cosine similarity and Pearson coefficient to estimate item similarities, which are less accurate since they lack tailored optimization for the recommendation task. In recent years, several works attempt to learn item similarities from data, by expressing the similarity as an underlying model and estimating model parameters by optimizing a recommendation-aware objective function. While extensive efforts have been made to use shallow linear models for learning item similarities, there has been relatively less work exploring nonlinear neural network models for item-based CF. In this work, we propose a neural network model named Neural Attentive Item Similarity model (NAIS) for item-based CF. The key to our design of NAIS is an attention network, which is capable of distinguishing which historical items in a user profile are more important for a prediction. Compared to the state-of-the-art item-based CF method Factored Item Similarity Model (FISM), our NAIS has stronger representation power with only a few additional parameters brought by the attention network. Extensive experiments on two public benchmarks demonstrate the effectiveness of NAIS. This work is the first attempt that designs neural network models for item-based CF, opening up new research possibilities for future developments of neural recommender systems.

1 INTRODUCTION

Item-based collaborative filtering supports interpretable, efficient real-time recommendation, but estimating item similarities accurately remains challenging. NAIS addresses this by using attention to distinguish the importance of historical items and improves on FISM.

Motivation: MF methods can be difficult to retrain for large-scale data in real time because updating user embeddings may require complex online-learning infrastructure.
Motivation: Item-based CF represents users through historically interacted items and recommends items similar to that profile, enabling interpretable and easier real-time personalization.Item similarities can be computed offline, leaving online recommendation to retrieve and score similar items.
Similarity estimation: Heuristic similarities such as cosine similarity and Pearson coefficient lack recommendation-tailored optimization and typically underperform machine-learning methods in top-K accuracy.
Similarity estimation: FISM learns item similarities efficiently with item embeddings but assumes all historical items contribute equally to a user’s preference.The paper notes that incidental or outdated interactions may not reflect future preferences equally.
Proposed approach: NAIS adds an attention network to FISM to learn varying item importance, preserving online efficiency while providing stronger representation power.Experiments on two public benchmarks report a 4.5% relative NDCG improvement over FISM and competitive performance.

2 PRELIMINARIES

Learning-based item-similarity methods optimize recommendation objectives, progressing from direct similarity learning to embedding-based FISM. FISM is efficient and expressive but treats all historical items equally, motivating attention-based extensions.

Item-based CF: Item-based CF predicts a target item from its similarities to items previously interacted with by the user.The model can use explicit ratings or binary implicit-feedback interactions.
Item-based CF: Offline item-similarity computation enables efficient online recommendation through lookups and incremental updates after new user interactions.
Learning-based methods: Heuristic similarity measures lack recommendation-specific optimization, motivating learning-based methods that optimize reconstruction or recommendation-aware objectives.SLIM learns an item-item similarity matrix by reconstructing the user-item interaction matrix.
Learning-based methods: SLIM constrains its similarity matrix with sparsity, non-negativity, and zero diagonal elements, but direct learning over I^2 elements is costly at scale.It also cannot capture transitive relations between items that have not been co-rated.
FISM: FISM represents items with low-dimensional embeddings and parameterizes similarity as an inner product, avoiding SLIM’s item-similarity constraints.Its user representation aggregates historical-item embeddings and can be trained with standard recommendation losses.
FISM: FISM’s representation ability is limited by equal treatment of historical items, a limitation that NAIS addresses with a neural attention network.

3 NEURAL ATTENTIVE ITEM SIMILARITY MODEL

NAIS extends FISM with an attention network that assigns prediction-dependent importance to historical items. Its smoothed attention design addresses user-history variability while retaining efficient online personalization.

NAIS Model: NAIS builds on FISM by learning varying importance for interacted items through an attention mechanism, improving expressiveness while preserving online prediction efficiency.The model is designed for implicit-feedback recommendation and uses attention to learn item-to-item interactions.
Model Designs: Design 1 assigns each historical item a global trainable weight, but cannot condition that weight on the target item being predicted.Fixing all weights to 1 recovers FISM, whereas global weights imply a static user representation.
Model Designs: Design 2 assigns weights to item pairs, but unseen item pairs cannot provide reliable attention estimates because their pair-specific weights are unobserved.This creates a generalization problem whenever two items never co-occurred in training data.
Model Designs: Design 3 parameterizes attention with target and historical item embeddings, allowing an MLP to estimate weights even for item pairs absent from training.The embedding-based parameterization supplies shared information for estimating attention across item pairs.
Attention Design: Standard softmax attention can underperform FISM because highly variable user-history lengths create large attention-weight variance that harms item-embedding optimization.MovieLens histories have mean 166 and maximum 2313, illustrating the normalization imbalance for active users.
NAIS Model: NAIS smooths the softmax denominator with exponent β∈[0,1], reducing punishment for active users; its prediction cost is O(ak|R+_u|), while user histories enable parameter-free online refresh.β=1 recovers softmax, whereas β<1 suppresses the denominator; online updates aggregate item embeddings without changing model parameters.

4 EXPERIMENTS

The experiments evaluate NAIS through attention effectiveness, comparisons with recommendation baselines, qualitative analyses, pre-training, and hyper-parameter studies on MovieLens and Pinterest. NAIS achieves strong recommendation performance while adding attention-related training cost and sensitivity to normalization choices.

Experimental design: Experiments answer whether attention improves recommendations, how NAIS compares with state-of-the-art methods, and how its key hyper-parameters affect performance.
Experimental design: Evaluation uses leave-one-out testing with 99 sampled negatives per user, reporting Hit Ratio and NDCG at position 10.
Efficiency: NAIS training takes roughly 0.8 times the training time of FISM additionally, while NAIS-concat is slightly slower than NAIS-prod.
Attention effectiveness: NAIS-prod and NAIS-concat improve over FISM by 6.3% and 3.6% relative NDCG on MovieLens and Pinterest, respectively, at convergence.
Qualitative analysis: Attention weights become more variable during training, and this increasing differentiation coincides with improved performance over FISM.
Pre-training: Both NAIS variants benefit significantly from FISM pre-training and converge faster than with random initialization.
Performance comparison: The two NAIS methods achieve the highest NDCG and HR scores on both datasets, with significant improvements over other methods at p < 10^-3.
Performance comparison: FISM outperforms ItemKNN by about 10% relative improvement, while user-based models outperform FISM on MovieLens and FISM outperforms them on Pinterest.

5 RELATED WORK

Related work spans matrix-factorization and learning-to-rank approaches for collaborative filtering, alongside emerging neural recommendation models. NAIS is distinguished from prior attentive collaborative filtering by tailoring attention to item-based CF and variable-length interaction histories.

Learning objectives: Implicit-feedback recommenders typically formulate collaborative filtering as personalized ranking rather than explicit-rating prediction.These methods must account for missing data or sampled negative feedback during optimization.
Learning objectives: Learning-to-rank methods use pointwise or pairwise objectives, with neural models requiring SGD-based optimization and sampled negatives for efficiency.Pairwise methods compare observed interactions with sampled unobserved feedback by maximizing score margins.
Neural recommendation: Deep neural recommendation research commonly uses neural networks either to extract features from auxiliary data or to learn user–item interactions.The cited related work frames DNNs as models for complex functions and high-level feature extraction.
Attentive collaborative filtering: NAIS differs from user-based Attentive Collaborative Filtering by adapting attention to item-based CF and smoothing softmax normalization for variable-length histories.The paper reports that standard softmax performs poorly when user interaction-history lengths vary substantially.

6 CONCLUSION

The conclusion presents NAIS as a neural extension of item-based collaborative filtering that models unequal historical-item importance. It reports improved performance over FISM and identifies deeper architectures as future work, while retaining a deliberately simple second-order design.

Conclusion: NAIS assigns different importance to historical items because they do not contribute equally to predicting a user’s preference.The model revisits FISM and introduces attention mechanisms to enhance its representation ability.
Conclusion: A smoothed softmax variant addresses the large-variance problem in user-history lengths.The conclusion describes this variant as simple and effective and reports empirical validation of the NAIS methods.
Conclusion: NAIS significantly outperforms FISM while achieving competitive performance for item recommendation.This is the paper’s reported experimental conclusion for the evaluated recommendation task.
Future work and scope: The current design models only pairwise, or second-order, item interactions to keep online personalization simple.The authors propose fully connected or convolutional layers as possible extensions for higher-order and nonlinear feature interactions.
Conclusion: The paper identifies neural item-based CF as an initial direction for future neural recommender-model research.It characterizes the work as the first design of neural network models for item-based CF, to the authors’ knowledge.