Source-linked AI summary
Learning Distributed Representations from Reviews for Collaborative Filtering
Amjad Almahairi, Kyle Kastner, Kyunghyun Cho, Aaron Courville
TL;DR
Sparse rating matrices can overfit, and prior work suggests reviews may help regularize collaborative filtering. This paper compares neural review models with LDA-based approaches and finds that BoWLF performs best, while the RNN does not improve performance. The authors also show that data splits matter for interpreting model comparisons.
Problem
The paper asks how different review models affect review-based regularization of collaborative filtering, a setting where sparse rating data can overfit.
Source: §1 · p. 1Method
The paper compares a bag-of-words Paragraph Vector model and an RNN review model with LDA-based approaches for regularizing collaborative filtering.
Source: §1 · p. 1Results
27 of 28 datasets: BoWLF achieves state-of-the-art results; LMLF outperforms HFT but not BoWLF as dataset size increases.
Source: §5 · p. 7Takeaways & Limitations
In this application, controlling review-model flexibility is important, and absolute performance should not be compared across different data splits.
Source: §1 · p. 2The authors leave the effect of language modeling on prediction performance for further investigation and present the RNN explanation as a possibility.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Recent work has shown that collaborative filter-based recommender systems can be improved by incorporating side information, such as natural language reviews, as a way of regularizing the derived product representations. Motivated by the success of this approach, we introduce two different models of reviews and study their effect on collaborative filtering performance. While the previous state-of-the-art approach is based on a latent Dirichlet allocation (LDA) model of reviews, the models we explore are neural network based: a bag-of-words product-of-experts model and a recurrent neural network. We demonstrate that the increased flexibility offered by the product-of-experts model allowed it to achieve state-of-the-art performance on the Amazon review dataset, outperforming the LDA-based approach. However, interestingly, the greater modeling power offered by the recurrent neural network appears to undermine the model's ability to act as a regularizer of the product representations.
1. INTRODUCTION
This paper studies how different natural-language review models affect review-based regularization of collaborative filtering. On Amazon reviews, the bag-of-words model outperforms the LDA-based approach, while the RNN does not improve performance; the paper also examines how train/test splits affect comparisons.
1. INTRODUCTION
- The study investigates whether natural-language reviews can serve as side information to regularize collaborative-filtering matrix factorization.Prior work had shown improved rating prediction from incorporating reviews.Source: §1 · p. 1
- The paper compares two neural review models with LDA-based approaches: a more flexible bag-of-words model and an RNN whose added modeling power does not improve regularization performance.The models are compared with current state-of-the-art LDA-based approaches.Source: §1 · pp. 1–2
- The proposed bag-of-words model outperforms the LDA-based approach on Amazon reviews, while the RNN does not improve performance as a regularizer.The authors conclude that controlling model flexibility matters in this application.Source: §1 · p. 2
- For a fixed split, relative performance conclusions generalize across splits, but absolute performance comparisons across different splits are highly problematic.Source: §1 · p. 2
2. MATRIX FACTORIZATION FOR COLLABORATIVE FILTERING
The section describes matrix factorization as predicting ratings from user and product representations, then explains how sparsity motivates regularization and review prediction as an auxiliary task.
2. MATRIX FACTORIZATION FOR COLLABORATIVE FILTERING
- Each unobserved rating is predicted from user and product latent factors, with global, user-specific, and product-specific biases.The model estimates parameters by minimizing mean-squared error between predicted and observed ratings.Source: §2 · p. 2
2.1 Taming the Curse of Data Sparsity
- More than 99.999% of Amazon ratings are missing, and matrix factorization can overfit observed ratings, harming generalization to held-out user–product pairs.Source: §2.1 · p. 2
- Adding a regularization term to the cost is a common response to overfitting, with weight decay among the widely used choices.Parameters are estimated by minimizing CR(θ) + λΩ(θ), where λ is a regularization coefficient.Source: §2.1 · p. 2
- Reviews can serve as an auxiliary task because they often explain ratings and describe features affecting users’ opinions, and the paper explores this approach to improve generalization.Source: §2.1 · p. 2
3. REGULARIZING WITH EXTRA DATA
This section uses reviews as additional information for rating prediction, jointly modeling ratings and review text. It introduces bag-of-words and recurrent review models, then compares the bag-of-words model with HFT.
3.1 Reviews as Extra Data
- Reviews can accompany ratings and provide information that may help rating prediction generalize.The section motivates using reviews as extra data for rating models.Source: §3.1 · p. 2
3.2 Natural Language Review Modeling
- The framework jointly models user–product ratings and reviews, sharing some rating-model parameters with the review model.It optimizes the rating and review objectives as a convex combination, with α as a hyperparameter.Source: §3.2 · pp. 2–3
3.2.1 BoWLF: Distributed Bag-of-Word
- BoWLF represents reviews as bags of words and combines this review model with matrix factorization for rating prediction.The word distribution is formed from a product representation by an affine transformation followed by softmax.Source: §3.2.1 · p. 3
3.2.2 LMLF: Recurrent Neural Network
- LMLF combines matrix factorization with an RNN language model that processes reviews as ordered word sequences.The implementation uses an LSTM.Source: §3.2.2 · p. 3
3.3 Related Work: LDA-based Approach
- HFT is an LDA-based review model that represents a review as a mixture of multinomial distributions over words.Its topic proportions are tied to the product representation rather than sampled from LDA’s top-level Dirichlet distribution.Source: §3.3 · p. 4
3.4 Comparing HFT and BoWLF
- Unlike HFT’s mixture, BoWLF projects the product representation onto the probability simplex, allowing its elements to suppress probabilities for selected words.The section frames the distinction as product-of-experts versus mixture-of-experts and links BoWLF’s flexibility to modeling negative word correlations.Source: §3.4 · p. 4
4. EXPERIMENTS
Across 28 Amazon product categories, BoWLF generally outperforms competing rating-prediction models, while LMLF’s stronger review modeling does not translate into better ratings. Results also show that split choice affects absolute comparisons, although model rankings remain consistent when the split is held fixed.
4.1 Dataset
- The evaluation uses the Amazon Reviews dataset, with approximately 35 million ratings and reviews from 6,643,669 users and 2,441,053 products.The dataset contains 28 categories, and reviews average 110 words.Source: §4.1 · pp. 4–5
4.2 Experimental Setup
- Experiments use per-category splits with 80% of ratings for training, a 5,000-word vocabulary, rating MSE, and review-model NLL.The remaining ratings are split evenly into validation and test sets; the approaches are compared with MF, HFT, and RMR.Source: §4.2 · p. 5
4.3 Rating Prediction Results
- As dataset size increases, BoWLF improves over HFT and LMLF modestly improves over HFT.BoWLF and LMLF generally perform better than other methods, especially on larger datasets.Source: Figure 1 · p. 5, Figure 2 · p. 5, §4.3 · p. 5
- BoWLF outperforms all other models except in Jewelry, with 20.29% improvement over MF and 5.64% over HFT across categories.The results are reported as test-data MSE; Table 1 covers all 28 categories.Source: §4.3 · p. 5, Table 1 · p. 6
- BoWLF always outperforms LMLF in rating prediction; its word distribution depends linearly on product representations, unlike LMLF’s nonlinear mapping.The authors suggest the nonlinear mapping may make LMLF’s product-related structure harder for the rating model to decode linearly [15].Source: §4.3 · p. 6
4.4 Impact of Training / Test Data Split
- Across five random folds, split-to-split variance can mislead model selection, but relative model performance remains consistent when comparisons use the same split.The authors conclude that a single random split can support evaluation if held constant across models; standardization is needed for comparisons across research groups.Source: Figure 3 · p. 6, §4.4 · p. 7, Figure 4 · p. 7
4.5 Effect of Language Model
- LMLF has better test-data NLL than HFT and BoWLF, but does not improve rating prediction over BoWLF.The authors suggest LSTM nonlinearity helps model reviews without necessarily yielding linearly decodable product representations.Source: Figure 5 · p. 7, §4.5 · p. 7
- BoWLF’s product neighborhoods qualitatively group similar products more closely than HFT’s broader neighborhoods, consistent with its ability to model peakier word distributions.For example, BoWLF groups tomato soup with other soups; the authors caution that firm conclusions from the small set of neighbors are ill-advised.Source: §4.5 · p. 7
5. DISCUSSION
BoWLF delivers strong Amazon rating-prediction results, while LMLF's relative performance improves with dataset size. The discussion also connects product-neighborhood quality to regularization and cautions that results from different data splits are not directly comparable.
5. DISCUSSION
- LMLF outperforms HFT as dataset size increases but remains slightly behind BoWLF; the authors suggest its nonlinear encoding may be difficult for a linear rating model to use.They note that this explanation and the effect of language modeling on prediction require further investigation.Source: §5 · pp. 7–8
- Results from models trained on different data subsets cannot be directly compared, so the discussion emphasizes standardized splits and cross-validation.The paper reports five-fold cross-validation in its split analysis.Source: §5 · p. 7
- Product neighborhoods provide a qualitative diagnostic: their quality appears to correlate with regularization, and BoWLF's neighborhoods correspond with human analysis.Table 2 compares cosine-similarity nearest neighbors for HFT, BoWLF, and LMLF on Gourmet Foods.Source: Table 2 · p. 8, §5 · p. 8
- BoWLF improves over basic matrix factorization by 20.29% on average and over HFT by 5.64%, achieving state-of-the-art results on 27 of 28 datasets.Its performance is also reported as superior across dataset splits.Source: §5 · pp. 7–8