Source-linked AI summary
Collaborative Deep Learning for Recommender Systems
Hao Wang, Naiyan Wang, Dit-Yan Yeung
TL;DR
Sparse ratings limit collaborative filtering, while sparse auxiliary information can weaken CTR's learned representations. CDL addresses these challenges with a hierarchical Bayesian model that jointly learns deep content representations and collaborative filtering, and experiments report significant advances over the state of the art.
Problem
Sparse ratings degrade CF performance, and CTR's learned representation may be ineffective when auxiliary information is very sparse.
Method
CDL is a hierarchical Bayesian model that jointly performs deep representation learning for content and collaborative filtering for the ratings matrix.
Results
CDL significantly outperforms the state of the art across three real-world datasets from different domains.
Takeaways & Limitations
CDL simultaneously learns effective content features and captures item and user similarity, with learned representations potentially usable beyond recommendation.
Takeaways & Limitations
The paper identifies replacing bag-of-words with more powerful representations and incorporating other side information as possible extensions for further performance gains.
Abstract
from arXiv · showhide
Collaborative filtering (CF) is a successful approach commonly used by many recommender systems. Conventional CF-based methods use the ratings given to items by users as the sole source of information for learning to make recommendation. However, the ratings are often very sparse in many applications, causing CF-based methods to degrade significantly in their recommendation performance. To address this sparsity problem, auxiliary information such as item content information may be utilized. Collaborative topic regression (CTR) is an appealing recent method taking this approach which tightly couples the two components that learn from two different sources of information. Nevertheless, the latent representation learned by CTR may not be very effective when the auxiliary information is very sparse. To address this problem, we generalize recent advances in deep learning from i.i.d. input to non-i.i.d. (CF-based) input and propose in this paper a hierarchical Bayesian model called collaborative deep learning (CDL), which jointly performs deep representation learning for the content information and collaborative filtering for the ratings (feedback) matrix. Extensive experiments on three real-world datasets from different domains show that CDL can significantly advance the state of the art.
1. INTRODUCTION
Recommender systems face sparse ratings and weak content representations, motivating tightly coupled models that combine collaborative filtering with deep content representation learning. The paper proposes CDL, a hierarchical Bayesian framework that jointly learns both sources and reports state-of-the-art performance.
- Sparse ratings significantly reduce CF prediction accuracy and prevent recommending new products without rating information.
- CTR tightly couples topic modeling with collaborative filtering, but its learned representation can be ineffective when auxiliary information is very sparse.
- Deep learning can automatically learn effective content representations, whereas CF better captures item similarity and implicit relationships.
- CDL is a hierarchical Bayesian model that tightly couples deep representation learning for content with collaborative filtering for ratings, enabling two-way interaction.
- CDL simultaneously extracts deep content features and captures item and user similarity, with representations potentially usable beyond recommendation.
- The paper uses collaborative filtering as a probabilistic deep-learning target and derives both MAP and sampling-based algorithms for CDL.
2. NOTATION AND PROBLEM FORMULATION
The paper formulates recommendation as predicting unobserved entries in an implicit-feedback rating matrix using item content. Content is represented with bag-of-words vectors, while the SDAE uses clean and corrupted inputs for representation learning.
- The task uses implicit feedback in training and testing, representing items with a J-by-S bag-of-words content matrix and users' feedback with an I-by-J binary rating matrix.
- Given part of the rating matrix and item content, the model predicts the remaining ratings, and the formulation also supports recommendation tasks beyond movies and articles.
- In the SDAE, the clean content matrix is the input, while a noise-corrupted matrix is used as the corresponding corrupted input.
3. COLLABORATIVE DEEP LEARNING
CDL is a hierarchical Bayesian model that integrates deep representation learning for item content with collaborative filtering for ratings. Its middle layer bridges both information sources, while extreme coupling settings reduce to weaker two-step or encoder-only variants.
- CDL hierarchically integrates an SDAE for content representations with collaborative filtering for the ratings matrix.The model is defined through a generative process covering network layers, latent item and user vectors, and ratings.
- 3.1 Stacked Denoising Autoencoders: SDAE learns representations by reconstructing clean input from corrupted input, with encoder and decoder halves surrounding a middle representation layer.The Bayesian formulation treats clean input, corrupted input, layer outputs, weights, and biases within the network.
- 3.3 Collaborative Deep Learning: The middle layer X_L/2 and latent offset ϵ_j connect content learning with item representations used by collaborative filtering.This bridge supports learning feature representations while capturing item and user similarity relationships.
- 3.4 Maximum A Posteriori Estimates: The MAP procedure alternates updates to user vectors, item vectors, network weights, and biases, with back-propagation used for layer parameters.A sampling-based Bayesian algorithm is also provided, while confidence parameters weight observed ratings in the updates.
- 3.3 Collaborative Deep Learning: When λ_n/λ_v approaches positive infinity, CDL becomes a two-step model that feeds SDAE representations directly into CTR.When the ratio approaches zero, the decoder essentially vanishes and the model becomes an encoder-only degenerate case.
- 3.5 Prediction: For a new item without training ratings, the item-specific offset ϵ_j is set to 0 when calculating predicted ratings.Predictions use point estimates of user vectors, network parameters, and offsets.
4. EXPERIMENTS
Experiments evaluate CDL against four baselines on three real-world datasets under sparse and dense implicit-feedback settings. CDL improves recall and mAP, while its gains depend on dataset sparsity and model depth.
- Datasets: Experiments use three real-world datasets from CiteULike and Netflix, representing different domains, scales, and sparsity levels.The datasets include citeulike-a, citeulike-t, and Netflix, with Netflix containing ratings and movie plots.
- Evaluation: Sparse and dense settings use one or ten observed items per user for training, respectively, with five random repetitions averaged.Recall is used for implicit feedback, and mAP is additionally reported at cutoff 500.
- Baselines and settings: CDL is compared with CTR, DeepMusic, CMF, and SVDFeature using recall@M under sparse and dense settings.The comparisons use a 2-layer CDL, while CDL complexity is controlled through the number of layers.
- Quantitative comparison: In the sparse setting, 2-layer CDL outperforms CTR by 4.2%∼6.0% on citeulike-a, 10.4%∼13.1% on citeulike-t, and 1.9%∼5.9% on Netflix.In the dense setting, the corresponding margins are 3.3%∼4.6%, 4.7%∼7.6%, and 1.5%∼2.0%.
- Quantitative comparison: The mAP of CDL is almost or more than twice that of CTR in the sparse settings.For citeulike-t and Netflix, recall increases with depth, whereas citeulike-a begins to overfit beyond two layers.
- Qualitative comparison: Qualitative examples indicate that CDL more accurately captures article topics and user preferences, while also modeling word co-occurrence and relations better.For one user, CDL achieves precision of 100%, compared with 30% for CTR.
5. COMPLEXITY ANALYSIS AND IMPLEMENTATION
The analysis gives CDL’s update and epoch complexities, then reports practical runtimes from GPU/C++-accelerated experiments, including on Netflix.
- Updating user representations costs O(K2J + K3), with K the learned-representation dimension and J the number of items.
- Updating item representations costs O(K2I + K3 + SK1), where I is the user count, S the vocabulary size, and K1 the first-layer output dimension.
- A complete epoch has total time complexity O(JSK1 + K2J2 + K2I2 + K3).
- Each epoch takes about 40 seconds for the first two datasets and about 60 seconds for Netflix using GPU/C++ acceleration.The first two datasets use about 200 epochs per run, while Netflix uses about 100.
6. CONCLUSION AND FUTURE WORK
The paper concludes that CDL jointly learns content representations and ratings-based collaborative filtering, while identifying richer content models and additional side information as future directions.
- CDL achieves state-of-the-art performance by jointly learning deep content representations and collaborative filtering from the ratings matrix.
- CDL is presented as the first hierarchical Bayesian model bridging state-of-the-art deep learning models and recommender systems.
- The framework may incorporate more powerful content representations than bag-of-words, including models such as convolutional neural networks.
- Incorporating other side information is identified as another possible way to improve performance through CDL’s Bayesian formulation.
A. BAYESIAN TREATMENT FOR CDL
The Bayesian treatment derives a sampling-based learning algorithm for CDL. Under a limiting condition, the sampling procedure becomes a Bayesian generalized form of back-propagation.
- The paper derives a sampling-based algorithm for the Bayesian treatment of CDL.
- The sampling algorithm is described as a Bayesian and generalized version of back-propagation.
- For λs approaching infinity with adaptive rejection Metropolis sampling, sampling for W+ becomes a Bayesian generalized version of back-propagation.
- Figure 7 illustrates the sampling interpretation by drawing the next sample in the region under a gradient-defined line.