Source-linked AI summary
RecVAE: a New Variational Autoencoder for Top-N Recommendations with Implicit Feedback
Ilya Shenbin, Anton Alekseev, Elena Tutubalina, Valentin Malykh, Sergey I. Nikolenko
TL;DR
Collaborative-filtering models must handle large parameter counts, sparse feedback, and difficult regularization, while existing autoencoder approaches provide only partial solutions. RecVAE improves Mult-VAE with a new architecture, composite prior, user-dependent β, and alternating training, and is reported to outperform prior autoencoder models across classical datasets. Its scope is bounded by mixed comparative results against EASE and negative results for a symmetric item-embedding variant.
Problem
Matrix factorization has many parameters, requires optimization for new users or items, and can overfit sparse feedback, while existing autoencoder approaches only partially address these issues.
Method
RecVAE improves Mult-VAE with a new encoder architecture, composite latent-code prior, user-specific β, alternating updates, and an encoder mapping feedback to user embeddings.
Results
RecVAE significantly outperforms previous autoencoder-based models across MovieLens-20M, Netflix Prize, and Million Songs datasets, with results comparable to EASE.
Takeaways & Limitations
The reported improvements make RecVAE a strong autoencoder-based model for top-N recommendations with implicit feedback across classical collaborative-filtering datasets.
Takeaways & Limitations
RecVAE significantly outperforms EASE only on MovieLens-20M, is competitive on Netflix, and its symmetric item-embedding variant was slower, more memory-intensive, and less effective.
Abstract
from arXiv · showhide
Recent research has shown the advantages of using autoencoders based on deep neural networks for collaborative filtering. In particular, the recently proposed Mult-VAE model, which used the multinomial likelihood variational autoencoders, has shown excellent results for top-N recommendations. In this work, we propose the Recommender VAE (RecVAE) model that originates from our research on regularization techniques for variational autoencoders. RecVAE introduces several novel ideas to improve Mult-VAE, including a novel composite prior distribution for the latent codes, a new approach to setting the $β$ hyperparameter for the $β$-VAE framework, and a new approach to training based on alternating updates. In experimental evaluation, we show that RecVAE significantly outperforms previously proposed autoencoder-based models, including Mult-VAE and RaCT, across classical collaborative filtering datasets, and present a detailed ablation study to assess our new developments. Code and models are available at https://github.com/ilya-shenbin/RecVAE.
1 INTRODUCTION
RecVAE extends Mult-VAE for implicit-feedback collaborative filtering with architectural, prior, β-setting, and alternating-training innovations. The resulting model is reported to outperform prior autoencoder-based methods across several datasets.
- RecVAE extends Mult-VAE for implicit-feedback collaborative filtering with a new encoder architecture, composite latent-code prior, user-specific β, and alternating updates.The composite prior combines a standard Gaussian with a previous-iteration latent-code distribution.
- β should depend on each user's available implicit feedback rather than use one fixed value for all users.
- Alternating encoder and decoder updates permit multiple encoder updates per decoder update and corrupted inputs for encoder training with clean decoder inputs.The encoder is more complex, whereas the decoder is a simple single-layer network.
- RecVAE significantly outperforms previous autoencoder-based models and achieves competitive or better results across MovieLens-20M, Netflix Prize, and Million Songs datasets.
2 BACKGROUND AND RELATED WORK
Variational autoencoders provide latent-variable and amortized-inference foundations for collaborative filtering, while CDAE and Mult-VAE adapt autoencoding to corrupted implicit feedback and multinomial recommendation likelihoods. This line of work addresses the parameterization and regularization difficulties of matrix factorization, but some arbitrary-conditioning approaches do not directly apply to implicit feedback.
- Variational autoencoders and extensions: A VAE models complex distributions through latent codes and approximates its intractable marginal likelihood with the evidence lower bound.
- Variational autoencoders and extensions: Amortized inference parameterizes qϕ(z|x) as a function of input, providing regularization and closed-form variational parameters.
- Variational autoencoders and extensions: β-VAE rescales the KL-divergence term in the ELBO with a regularization coefficient β.
- Variational autoencoders and extensions: Denoising VAEs reconstruct inputs from corrupted versions and are described as improving basic VAE performance and posterior robustness.
- Autoencoders and collaborative filtering: CDAE reconstructs a user's feedback vector after randomly removing interactions, using neural encoder and decoder layers with user-specific and shared weights.
- Autoencoders and collaborative filtering: Mult-VAE is the closest prior work, extending VAEs to implicit-feedback collaborative filtering with a multinomial likelihood.
3 PROPOSED APPROACH
RecVAE extends Mult-VAE with architectural changes and three main training or regularization innovations: denoising, a composite prior, feedback-dependent KL scaling, and alternating updates. These changes target stability and generalization while preserving Mult-VAE’s multinomial-likelihood framework.
- Mult-VAE: Mult-VAE models user feedback with a neural decoder and multinomial likelihood, while its variational posterior uses a diagonal Gaussian.The decoder maps latent user representations to item scores, and the likelihood treats interaction histories as multinomial observations.
- Model Architecture: RecVAE changes the inference network with densely connected layers, swish activations, and layer normalization, while retaining a simple linear softmax decoder.The decoder parameters correspond to item embeddings and biases; the encoder maps feedback to user embeddings.
- Composite Prior: The composite prior combines a standard normal distribution with the previous epoch’s approximate posterior to regulate large variational-parameter updates.The posterior component acts as an auxiliary loss, while the standard normal component is described as preventing overfitting; the combination is not mathematically equivalent to separate regularization.
- Rescaling KL Divergence: RecVAE makes β user-specific and proportional to each user’s observed feedback amount, with a shared γ selected by cross-validation.The paper reports that this rescaling approach works better and significantly improves experiments.
- Alternating Training and Regularization by Denoising: RecVAE uses denoising during encoder training but trains the decoder on clean inputs, separating regularization across the two parameter sets.The authors report that denoising helps autoencoder-based collaborative filtering, whereas additional decoder regularization degraded performance.
- Alternating Training and Regularization by Denoising: Training alternates encoder and decoder updates, using multiple encoder updates per decoder update because the encoder is more complex.The separation also permits corrupted inputs for encoder training while retaining clean inputs for decoder training.
4 EXPERIMENTAL EVALUATION
RecVAE is evaluated against classical, learning-to-rank, and autoencoder-based recommenders on three collaborative-filtering datasets. It outperforms prior autoencoder models, while ablations examine which features improve performance, stability, and training behavior.
- Evaluation setup: Recall@k and NDCG@k measure ranking quality by comparing top-k recommendations with held-out user feedback after excluding training items.NDCG@k weights higher-ranked items more heavily than Recall@k.
- Evaluation setup: RecVAE is evaluated on MovieLens-20M, Netflix Prize Dataset, and Million Songs Dataset using preprocessing aligned with Mult-VAE.The processed datasets contain 136,677, 463,435, and 571,355 users, respectively.
- Results: RecVAE outperforms all previous autoencoder-based models across the three datasets, with a large improvement over Mult-VAE.It significantly outperforms EASE only on MovieLens-20M and is competitive with EASE on Netflix Prize Dataset.
- Results: RecVAE confidence intervals indicate that its score differences are significant, while competing-model metrics come from prior work.Table 1 reports the top-N recommendation performance comparisons.
- Ablation study: The ablation study tests the new encoder, composite prior, β rescaling, alternating training, and removing decoder denoising against Mult-VAE.The study evaluates subsets of the proposed features rather than only the complete RecVAE model.
- Ablation study: All proposed features together achieve the best NDCG@100 scores on all three datasets, but feature effects are complementary and dataset-sensitive.β rescaling and alternating training can degrade scores individually yet improve them together; the new architecture facilitates other features.
- Stability analysis: The composite prior stabilizes training: random-user NDCG@100 changes remain positive almost everywhere, unlike the more volatile behavior of a simple Gaussian prior.The analysis interprets these changes as evidence that the composite prior addresses forgetting during encoder updates.
- Negative results: A symmetric autoencoder for item embeddings trained more slowly, required more memory, and failed to reach RecVAE’s results.This negative experiment tested alternating the symmetric model with regular RecVAE while regularizing each encoder using the other model’s embeddings.
5 CONCLUSION
The conclusion presents RecVAE as a combination of architectural, prior-distribution, β-setting, and alternating-training improvements over Mult-VAE. It reports strong collaborative-filtering performance while acknowledging that some theoretical motivations remain incomplete.
- Conclusion: RecVAE combines a new encoder architecture, composite latent-code prior, feedback-dependent β setting, and alternating encoder-decoder updates.These modifications are presented as improvements to the basic Mult-VAE model.
- Conclusion: RecVAE is comparable to EASE and significantly outperforms other models on MovieLens-20M, Netflix Prize Dataset, and Million Songs Dataset.The conclusion summarizes performance across classical collaborative-filtering datasets.
- Limitations and future work: The theoretical motivations for the modifications are sometimes incomplete, and some ideas were primarily motivated by practical improvements.The authors identify comprehensive theoretical analysis as a direction for future research.
A NETWORK ARCHITECTURE
Figure 3 presents the architecture of the inference network ψϕ.
- A NETWORK ARCHITECTURE: The figure depicts the inference network ψϕ.
- A NETWORK ARCHITECTURE: The architecture is associated with the inference network rather than the decoder.
- A NETWORK ARCHITECTURE: The network is parameterized by ψϕ as labeled in the figure.