Source-linked AI summary
Joint Deep Modeling of Users and Items Using Reviews for Recommendation
Lei Zheng, Vahid Noroozi, Philip S. Yu
TL;DR
Sparse ratings make recommendation difficult, so DeepCoNN jointly learns user behaviors and item properties from review text through coupled neural networks. It achieves an overall 8.3% improvement over state-of-the-art baselines across Yelp, Beer, and Amazon datasets.
Problem
Collaborative filtering struggles with sparse ratings, making recommendations difficult for users or items with few ratings.
Method
DeepCoNN jointly models user behaviors and item properties from reviews using two parallel CNN-based neural networks coupled by a shared layer.
Results
8.3% improvement was attained overall against state-of-the-art baselines across Yelp, Beer, and Amazon datasets.
Takeaways & Limitations
DeepCoNN effectively alleviates sparsity, with the greatest MSE reduction when users or items have only one review.
Takeaways & Limitations
The paper identifies prior review-based approaches as limited by lexical similarity, bag-of-words representations, and topic-modeling scalability and cold-start issues.
Abstract
from arXiv · showhide
A large amount of information exists in reviews written by users. This source of information has been ignored by most of the current recommender systems while it can potentially alleviate the sparsity problem and improve the quality of recommendations. In this paper, we present a deep model to learn item properties and user behaviors jointly from review text. The proposed model, named Deep Cooperative Neural Networks (DeepCoNN), consists of two parallel neural networks coupled in the last layers. One of the networks focuses on learning user behaviors exploiting reviews written by the user, and the other one learns item properties from the reviews written for the item. A shared layer is introduced on the top to couple these two networks together. The shared layer enables latent factors learned for users and items to interact with each other in a manner similar to factorization machine techniques. Experimental results demonstrate that DeepCoNN significantly outperforms all baseline recommender systems on a variety of datasets.
1. INTRODUCTION
The introduction identifies sparsity as a major limitation of collaborative filtering and motivates using review text to model users and items jointly. It presents DeepCoNN, a coupled neural-network model that uses learned user and item representations to predict ratings and reports improved accuracy across evaluated datasets.
- Motivation: Collaborative filtering struggles to recommend items or users with few ratings because of data sparsity.The sparsity problem occurs when users rate only an insignificant fraction of available items.
- Motivation: Review text can alleviate sparsity by providing information beyond users’ numeric ratings and explaining the reasons behind those ratings.Most collaborative-filtering techniques model users and items primarily from numeric ratings.
- DeepCoNN: DeepCoNN jointly models user behavior and item properties with two coupled neural networks using reviews written by users and reviews written for items.A shared top layer connects the parallel networks so learned user and item representations interact for rating prediction.
- DeepCoNN: DeepCoNN represents reviews with pre-trained word embeddings to capture semantic information and sentimental attitudes for rating prediction.The paper contrasts this representation with traditional bag-of-words techniques used by competing topic-modeling approaches.
- Contributions: DeepCoNN is presented as the first deep model to jointly represent users and items from reviews using neural networks.The introduction also states that the model is scalable and suitable for online learning scenarios requiring continuous updates with new data.
- Empirical findings: DeepCoNN outperforms all compared baselines in prediction accuracy on Yelp, Amazon, and Beer datasets, with larger gains for users and items having fewer ratings.The reported results indicate that leveraging review text both improves overall performance and alleviates sparsity.
2. METHODOLOGY · 2.1 Definition and Notation
DeepCoNN jointly learns user behaviors and item properties from review text to estimate ratings. The methodology defines review-based training tuples and summarizes the paper’s mathematical notation.
- 2. METHODOLOGY: DeepCoNN learns hidden latent factors for users and items from review text to estimate ratings given by users.
- 2. METHODOLOGY: The model uses a CNN-based architecture with two parallel neural networks for users and items.
- 2. METHODOLOGY: One network models user behaviors from reviews written by the user, while the other models item properties from reviews written for the item.
- 2. METHODOLOGY: A shared layer at the top couples the two networks so they can jointly predict ratings.
- 2.1 Definition and Notation: The training set T contains N tuples, each representing a user, item, rating, and review text.
- 2.1 Definition and Notation: Each tuple is denoted (u, i, rui, wui), where rui is the rating and wui is the review text.
- 2.1 Definition and Notation: Table 1 summarizes the mathematical notations used throughout the paper.
2.2 Architecture
The proposed model architecture uses convolution, max-pooling, and fully connected layers, with an objective function measuring rating prediction error from latent factors produced by Netu and Neti. Netu and Neti differ only in their inputs and use the same processing structure.
- Architecture: The architecture includes convolution, max-pooling, and fully connected layers.
- Architecture: The objective function measures rating prediction error using latent factors produced by Netu and Neti.
- Architecture: Netu and Neti differ only in their inputs, so the process illustrated for Netu is applied to Neti with similar layers.
2.3 Word Representation
DeepCoNN represents review text with word embeddings to capture semantic information. Reviews are organized into word-vector matrices that preserve word order, unlike bag-of-words representations.
- Word Representation: DeepCoNN uses word embeddings to exploit the semantics of reviews, mapping dictionary words to n-dimensional distributed vectors.The embedding function is parameterized and maps words from the dictionary M to distributed representations.
- Word Representation: For each user, the model builds a matrix of word vectors from all reviews written by that user.The matrix is formed by applying the corresponding c-dimensional word-vector lookup to each word in the user’s review document and concatenating the vectors.
- Word Representation: The word-vector matrix preserves word order, providing an advantage over bag-of-words techniques.The representation retains the sequence of words in the review matrix.
2.4 CNN Layers
The CNN layers apply convolutional filters to word-vector windows, use ReLU activations, and then max pooling to extract fixed-size text features. These pooled features are passed through a fully connected layer to produce user and item representations.
- 2.4 CNN Layers: Each convolutional neuron applies a filter K_j ∈ ℜ^{c×t} to a window of t word vectors.The convolution operation is performed for each kernel in the convolutional layer.
- 2.4 CNN Layers: The model uses Rectified Linear Units (ReLUs) as the convolutional activation function.The passage notes that ReLU-based deep convolutional networks train faster than equivalents using tanh units.
- 2.4 CNN Layers: Max pooling selects the highest-valued feature from each feature map and reduces convolutional results to a fixed-size vector.This pooling scheme accommodates text of varied length.
- 2.4 CNN Layers: Outputs from the max-pooling layer pass through a fully connected layer, yielding user and item CNN outputs x_u and y_i.The resulting representations are treated as features for the corresponding user and item.
2.5 The Shared Layer
The shared layer maps user and item outputs into a common feature space and couples them by concatenating their representations and estimating ratings with a Factorization Machine.
- 2.5 The Shared Layer: The shared layer maps user and item outputs into the same feature space so their representations become comparable.It is introduced on top to couple Netu and Neti.
- 2.5 The Shared Layer: The model concatenates xu and yi into a single vector ˆz for rating estimation.The concatenated vector is defined as ˆz = (xu, yi).
- 2.5 The Shared Layer: The Factorization Machine estimator uses a global bias, variable-strength terms, and second-order interactions between variables.The interaction term is defined as ⟨ˆvi, ˆvj⟩ = P|ˆz| f=1 ˆvi,f ˆvj,f.
2.6 Network Training
DeepCoNN is trained by minimizing its objective using derivatives computed through the network and chain rule. Optimization uses RMSprop on shuffled minibatches, with dropout applied to fully connected layers to reduce overfitting.
- Training minimizes Eq. 7, with derivatives of J taken with respect to z and other parameters computed using the differentiation chain rule.
- RMSprop optimizes parameters over shuffled minibatches by adaptively controlling step sizes according to gradient magnitude.The method scales each weight update using a running average of its gradient norm.
- Dropout is applied to the fully connected layers of both networks to prevent overfitting.The learning rate is denoted by λ, while ϵ provides numerical stability.
2.7 Some Analysis on DeepCoNN
DeepCoNN analyzes reviews without topic modeling, instead using word embeddings and convolution operations to exploit word order. It is also designed to scale with training data and support updates using new data.
- DeepCoNN avoids topic modeling and uses word embeddings to represent reviews as a matrix of word vectors while preserving word order.
- Convolution operations exploit the internal structure of review data and provide an efficient mechanism for using word order in text modeling.
- DeepCoNN scales with training-data size and can be trained or updated with new data because it is based on neural networks.Item or user latent factors can be updated independently from historical data.
3. EXPERIMENTS
Experiments evaluate DeepCoNN on three review datasets using MSE against rating-based, topic-modeling, and deep-learning baselines. DeepCoNN achieves the best reported performance and is particularly beneficial when users or items have few ratings.
- Datasets: The evaluation uses Yelp, Amazon, and Beer review datasets, each containing more than half a million reviews.Yelp contains more than 1M reviews, Amazon more than 143.7 million, and Beer almost 3 million.
- Datasets: Yelp and Amazon are extremely sparse, with customers providing fewer than six review-rating pairs on average, while reviews average fewer than 150 words across datasets.The passage links this sparsity to potential deterioration in recommender-system performance.
- Evaluation metric: Performance is measured with Mean Square Error (MSE), selected because related works commonly use the same metric.MSE compares observed and predicted ratings over the total number of observations.
- Baselines: Baselines span rating-only models MF and PMF, topic-modeling models LDA, CTR, and HFT, and the deep model CDL.MF uses only the rating matrix, while the review-aware methods exploit review-derived representations or jointly model ratings and reviews.
- Performance evaluation: DeepCoNN beats the best baseline on Yelp, Beer, and Amazon, gaining 8.3% improvement on average.CDL outperforms all topic-modeling recommender systems, while DeepCoNN benefits from joint modeling and semantic information in reviews.
- Cold-start analysis: DeepCoNN yields positive MSE reductions versus MF across Yelp, Beer, and Amazon Music Instruments, with larger reductions for users or items having fewer ratings.The findings indicate that review text is especially valuable when rating information is limited.
4. RELATED WORKS
Related work falls into review-text methods for rating prediction and deep-learning recommender systems. DeepCoNN differs by using review histories to predict ratings while addressing limitations of lexical, bag-of-words, topic-modeling, and review-ignoring approaches.
- Earlier review-based methods mainly predicted ratings for an existing review, whereas this paper predicts ratings from a user’s review history for recommendation.
- Review-based approaches exploited aspects such as price, service, and sentiment, while HFT used topic modeling to discover latent aspects from item or user reviews.
- These methods relied on lexical similarity and bag-of-words representations, ignoring semantic meaning and word order in reviews.
- Topic-modeling approaches also suffer from scalability problems and cannot handle new users and items.
- Most neural recommender approaches either use only rating matrices or content other than review text, unlike DeepCoNN.A prior review-text method combined matrix factorization for user and item latent factors with an RNN.
5. CONCLUSION
DeepCoNN jointly models users and items from review text through two neural networks coupled by a shared layer, enabling latent-factor interaction for rating prediction. It improves over state-of-the-art baselines across Yelp, Beer, and Amazon, especially under sparse review conditions.
- Conclusion: DeepCoNN exploits user reviews for buying and rating behavior and item reviews for features and properties.Its two deep neural networks model users and items from review text.
- Conclusion: A shared layer maps user and item representations into a common feature space, allowing latent factors to interact similarly to matrix factorization.The interaction is used to predict the corresponding rating.
- Conclusion: 8.5% and 7.6% improvements over state-of-the-art baselines were achieved on Yelp and Beer, respectively, while Amazon gained 8.7% on average.Overall, the proposed model attained an 8.3% improvement across all three datasets.
- Conclusion: The two networks separately learn user and item latent factors and cooperate to improve rating prediction, while word embeddings help capture review-text semantics.This was shown by comparisons with random or TF-IDF review representations.
- Conclusion: DeepCoNN reduces MSE more than MF for users and items with few reviews or ratings, with the greatest reduction when only one review is available.These experiments support its ability to alleviate the sparsity problem.