Source-linked AI summary
A Neural Autoregressive Approach to Collaborative Filtering
Yin Zheng, Bangsheng Tang, Wenkui Ding, Hanning Zhou
TL;DR
Collaborative filtering needs accurate, practical methods for predicting user preferences, while RBM-CF suffers from inaccuracy and impractically long training. CF-NADE adapts neural autoregressive modeling to CF with shared parameters, factorization, ordinal cost, and deep extensions; experiments show it outperforms state-of-the-art methods on three benchmarks.
Problem
RBM-CF provides a neural collaborative-filtering approach but suffers from inaccuracy and impractically long training because its training is intractable.
Method
CF-NADE is a feed-forward autoregressive CF architecture with parameter sharing, a factored scalable version, ordinal cost, and an efficiently optimized deep extension.
Results
CF-NADE outperforms state-of-the-art performance on MovieLens 1M, MovieLens 10M, and Netflix benchmarks.
Takeaways & Limitations
A single-hidden-layer CF-NADE outperforms all baselines on MovieLens 10M and Netflix, while adding another hidden layer yields slight improvements.
Takeaways & Limitations
The reported running times include GPU data-transfer overheads, and the authors note that faster implementations remain possible.
Abstract
from arXiv · showhide
This paper proposes CF-NADE, a neural autoregressive architecture for collaborative filtering (CF) tasks, which is inspired by the Restricted Boltzmann Machine (RBM) based CF model and the Neural Autoregressive Distribution Estimator (NADE). We first describe the basic CF-NADE model for CF tasks. Then we propose to improve the model by sharing parameters between different ratings. A factored version of CF-NADE is also proposed for better scalability. Furthermore, we take the ordinal nature of the preferences into consideration and propose an ordinal cost to optimize CF-NADE, which shows superior performance. Finally, CF-NADE can be extended to a deep model, with only moderately increased computational complexity. Experimental results show that CF-NADE with a single hidden layer beats all previous state-of-the-art methods on MovieLens 1M, MovieLens 10M, and Netflix datasets, and adding more hidden layers can further improve the performance.
1. Introduction
Collaborative filtering predicts user preferences from prior ratings and similar users, forming a core recommender-system technology. The paper motivates CF-NADE as a neural alternative to RBM-CF that addresses RBM-CF’s accuracy and training-time limitations.
- Collaborative filtering predicts a user’s item preferences from previous ratings and decisions made by similar users.
- Memory-based CF is easy to implement and explain but performs poorly on large-scale, sparse real-life data.
- Model-based CF learns from historical data and can recognize complex preference patterns; matrix factorization is especially popular and successful.
- RBM-CF models rating distributions with a two-layer undirected generative graph but requires variational approximation or MCMC because training is intractable.
- NADE estimates high-dimensional binary-vector distributions tractably by modeling conditionals with shared parameters and optimizing through backpropagation.
- CF-NADE adapts NADE to collaborative filtering and proposes parameter sharing, factorization, ordinal cost, and deep extensions, evaluated on three benchmarks.
2. Related Work
Related collaborative-filtering work centers on matrix-factorization models and neural-network approaches. CF-NADE is positioned as a neural autoregressive model that supports variable-length rating vectors, unlike standard NADE’s fixed-length binary vectors.
- Matrix-factorization CF commonly assumes that the partially observed user–item matrix has low rank.
- Matrix factorization represents users and items with lower-dimensional latent-factor vectors whose correlations support recommendation.
- Prior neural CF methods include RBM-CF and AutoRec, with RBM-CF applied in the Netflix Prize and AutoRec achieving state-of-the-art performance on some benchmarks.
- CF-NADE uses a neural autoregressive architecture for variable-length vectors, whereas NADE handles binary vectors of fixed length.
3. NADE for Collaborative Filtering
CF-NADE models each user’s sparse rating vector autoregressively, with user-specific visible units and shared parameters. The section adds rating-parameter sharing and matrix factorization to address rare ratings and large parameter counts.
- The Model: CF-NADE uses one user-specific model per user, with visible units matching that user’s rated items while weights and biases are tied across users.
- The Model: A training case is an ordered vector of a user’s ratings, where the ordering indexes rated items and each rating lies on the K-star scale.
- The Model: CF-NADE models the rating-vector probability by the chain rule and computes each conditional with a neural network hidden representation.
- Sharing Parameters Between Different Ratings: Different ratings initially use separate connection matrices and biases, but rare-rating parameters may be insufficiently optimized.
- Sharing Parameters Between Different Ratings: Parameter sharing lets multiple ratings of the same item use shared weights, functioning as a form of regularization whose advantage is confirmed experimentally.
- Dealing with Large-Scale Datasets: With H = 500 on Netflix, the unfactored Wk and Vk parameterization requires around 89 million free parameters, making learning problematic for massive datasets.
- Dealing with Large-Scale Datasets: Factoring Wk and Vk into lower-rank matrix products reduces the parameter count; setting J = 50 lowers it from 89 million to about 9 million.
4. Traing CF-NADE with Ordinal Cost
CF-NADE’s ordinal cost incorporates the ordered relationships among ratings rather than treating them as unrelated labels. It combines ordinal and regular costs to train the model.
- Ordinal preference modeling: The ordinal cost models ratings as ordered preferences, increasing toward the observed rating from below and decreasing above it.For an observed rating k, preferences over ratings 1 to k should increase monotonically, while preferences from k to K should decrease monotonically.
- Ordinal preference modeling: The basic CF-NADE objective misses ordinal information because it treats each rating as a separate label.Maximizing the regular conditional only ensures that the observed rating has the largest probability among possible ratings.
- Ordinal preference modeling: The lower and upper ordinal rankings are ydown = (k, k −1, . . . , 1) and yup = (k, k + 1, . . . , K).The boundary-specific ordinal terms are omitted when the true rating is 1 or K.
- Ordinal preference modeling: The ordinal ranking loss is defined over ratings, unlike other learning-to-rank CF methods that define ranking losses over items.The model fits rankings induced by the observed rating and previous ratings through the conditional distribution of the lower and upper rankings.
- Hybrid objective: The hybrid training objective combines regular cost Creg and ordinal cost Cord, with λ controlling the weight assigned to Cord.The final objective is Chybrid = (1 −λ)Creg + λCord.
5. Extending CF-NADE to a Deep Model
The paper extends CF-NADE from one hidden layer to a deep architecture by training on randomly sampled item orderings and split points. Additional layers use the shared hidden representation and increase computation moderately.
- Deep architecture: CF-NADE is extended to multiple hidden layers following an efficient deep extension of NADE.The deep variant is motivated by the potential for better performance from deeper neural architectures.
- Training procedure: Randomly sampling an item ordering and split point trains CF-NADE across possible orderings for each user.The ratings before the split form the input context, while subsequent items are predicted conditionally.
- Training procedure: Equation 19 predicts ratings after the split as though each were at the split position, averaging over possible choices of that position.The scaling factors account for the number of elements and possible split-position items in the sum.
- Deep architecture: Each training update relies on one hidden representation h(r_mo<i), allowing more hidden layers with moderately increased computational complexity.New layers are added as in a regular deep feed-forward neural network, and final conditionals are computed from the deepest representation.
- Computational complexity: For one input, a deep CF-NADE requires O(K ˆDH + H^2L) operations.Here ˆD is the average number of ratings per user, H is the number of hidden units per layer, and L is the total number of hidden layers.
- Implementation: The same randomly ordered training procedure can train both single-hidden-layer and multiple-layer CF-NADE models efficiently on GPUs.The authors use this procedure throughout their experiments.
6. Experiments
Experiments on MovieLens 1M, MovieLens 10M, and Netflix evaluate CF-NADE variants against strong baselines using test RMSE. Parameter sharing, ordinal costs, factorization, and depth are associated with improved performance, while larger hidden layers provide diminishing returns.
- Experimental setup: Across three benchmarks, CF-NADE is evaluated against strong baselines using average test RMSE over five random splits.The datasets contain approximately 10^6, 10^7, and 10^8 ratings, respectively.
- MovieLens 1M: CF-NADE-S consistently outperforms regular CF-NADE as ordinal weight λ increases, with both models' test RMSE decreasing monotonically.The experiments fix CF-NADE-S and λ = 1 thereafter.
- MovieLens 1M: I-CF-NADE-S achieves RMSE 0.830 with one hidden layer and 0.829 with two hidden layers on MovieLens 1M.The item-based variant substantially outperforms the user-based variant in this comparison.
- MovieLens 1M: Increasing hidden units improves I-CF-NADE-S performance on MovieLens 1M, but the gains diminish as model size grows.Deep CF-NADE models also perform better than shallow ones.
- MovieLens 10M: U-CF-NADE-S reaches RMSE 0.772 with one hidden layer on MovieLens 10M and outperforms the baselines, with slight improvement from additional depth.It also outperforms I-AutoRec, despite the weaker performance of U-AutoRec relative to I-AutoRec.
- Netflix: U-CF-NADE-S achieves RMSE 0.804 with one hidden layer and 0.803 with a deep architecture on Netflix, outperforming all baselines.The Netflix experiment uses the factored U-CF-NADE with J = 50.
7. Conclusions
CF-NADE is a feed-forward autoregressive architecture for collaborative filtering that combines parameter sharing, factorization, ordinal optimization, and deep extensions. Across three real-world benchmarks, it outperforms state-of-the-art collaborative-filtering methods, but the reported results rely on explicit ratings.
- CF-NADE is a feed-forward autoregressive architecture for collaborative filtering tasks, inspired by RBM-CF and NADE.
- Parameter sharing between ratings improves performance, while factorizing a large matrix into two lower-rank matrices reduces parameters for scalability.
- An ordinal cost incorporates the ordered nature of user preferences into CF-NADE optimization.
- CF-NADE extends to a deep model with only a moderate increase in computational complexity.
- CF-NADE outperforms state-of-the-art collaborative-filtering methods on three real-world benchmark datasets.
- All reported results rely on explicit user ratings, whereas implicit feedback such as watching, searching, or browsing is not always available in the same form.