PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Collaborative Filtering vs. Matrix Factorization Revisited

Steffen Rendle, Walid Krichene, Li Zhang, John Anderson

arXiv:2005.09683v2cs.IRcs.LGstat.ML

TL;DR

Collaborative filtering traditionally combines embeddings with dot products, while neural collaborative filtering uses learned MLP similarities. This paper revisits that comparison, examines how difficult dot products are for MLPs to learn, and considers deployment costs. It concludes that dot products often provide the stronger practical default.

  • Problem

    The paper examines whether learned MLP similarities are superior to traditional dot products for combining embeddings in collaborative filtering.

  • Method

    The authors revisit NCF experiments, empirically study MLP approximation of dot products, and compare dot-product and MLP inference costs.

  • Results

    Carefully configured dot products largely outperform MLP similarities, while MLPs require substantial capacity and data to approximate dot products accurately.

  • Takeaways & Limitations

    Dot products might be a better default embedding combiner because they simplify modeling and learning and support efficient retrieval.

  • Takeaways & Limitations

    The conclusions may differ on other datasets, and the evaluated MLP and NeuMF setups could potentially be improved.

Abstract

from arXiv · show

Embedding based models have been the state of the art in collaborative filtering for over a decade. Traditionally, the dot product or higher order equivalents have been used to combine two or more embeddings, e.g., most notably in matrix factorization. In recent years, it was suggested to replace the dot product with a learned similarity e.g. using a multilayer perceptron (MLP). This approach is often referred to as neural collaborative filtering (NCF). In this work, we revisit the experiments of the NCF paper that popularized learned similarities using MLPs. First, we show that with a proper hyperparameter selection, a simple dot product substantially outperforms the proposed learned similarities. Second, while a MLP can in theory approximate any function, we show that it is non-trivial to learn a dot product with an MLP. Finally, we discuss practical issues that arise when applying MLP based similarities and show that MLPs are too costly to use for item recommendation in production environments while dot products allow to apply very efficient retrieval algorithms. We conclude that MLPs should be used with care as embedding combiner and that dot products might be a better default choice.

1 Introduction

Collaborative filtering commonly combines user and item embeddings with a dot product, but neural collaborative filtering replaces this fixed similarity with a learned MLP. This paper revisits that comparison and finds that dot products can outperform MLP similarities while being easier to learn and deploy.

  • Embedding-based collaborative filtering models commonly combine user and item embeddings into a preference score using a similarity function.
  • Neural collaborative filtering replaces the traditional dot product with a learned similarity function, most commonly an MLP.
  • Carefully configured dot-product baselines largely outperform the MLP similarities evaluated in the revisited NCF experiments.
  • Learning a dot product with an MLP requires substantial model capacity and training data, despite MLPs being universal function approximators.
  • Dot products also offer lower inference cost because efficient maximum inner product search algorithms support retrieval.

2 Definitions

The paper formalizes embedding-combination functions and contrasts dot-product similarities with learned MLP-based alternatives. It also describes NeuMF, which combines an MLP with a weighted generalized matrix-factorization component.

  • The paper studies functions φ that map two d-dimensional embeddings into a single score, such as a user–item affinity.
  • Embeddings may be free model parameters or outputs of networks that transform user or item features.
  • The dot product is the most common embedding combination and is equivalent to matrix factorization when embeddings are free parameters.
  • Adding explicit biases does not increase expressiveness but may provide a useful inductive bias for the problem.
  • An MLP stacks affine transformations and activation functions, and NCF applies it to the concatenation of two embeddings.
  • NeuMF combines an MLP with a weighted generalized matrix-factorization model using separate embedding portions.

3 Revisiting NCF Experiments

Revisiting NCF experiments, the study finds that a properly configured dot-product matrix factorization baseline outperforms MLP-based similarities across the evaluated settings. NeuMF variants and GMF introduce additional issues without overturning this result.

  • Models, loss and training algorithm: The evaluation compares MLP, NeuMF, and matrix factorization using the same embeddings, loss, negative sampling, and varied embedding dimensions.The models differ only in their similarity functions, with d ∈ {16, 32, 64, 96, 128, 192}.
  • Results: Matrix factorization exhibits the best quality across all evaluation metrics and all but one embedding dimension.The experiments use item retrieval on binarized MovieLens 1M and Pinterest implicit-feedback datasets.
  • Matrix Factorization vs MLP: The dot product substantially outperforms MLP on all datasets, evaluation metrics, and embedding dimensions.The comparison uses a carefully configured matrix factorization baseline and finds no evidence that MLP is superior.
  • Matrix Factorization vs NeuMF: NeuMF provides only a minor improvement over direct MLP training and remains substantially worse than matrix factorization overall.The pretrained NeuMF variant performs better than direct combination but exceeds matrix factorization only at one point: HR on MovieLens with d = 192.
  • Matrix Factorization variants: GMF does not improve expressivity over a dot product because its weights can be absorbed into the embedding matrices.Its extra parameters can make L2 regularization ineffective and may lead to vanishing embedding norms, diverging weights, and numerical instability.
  • Further comparison: A non-cherry-picked comparison confirms that matrix factorization substantially outperforms NeuMF on all metrics and datasets.The authors note that the original MLP and NeuMF results likely overestimate test performance because stopping iterations were selected on the test set.

4 Learning a Dot Product with MLP is Hard

The paper empirically investigates how difficult it is for MLPs to learn dot-product similarities, finding that accuracy requires increasing amounts of data as embedding dimension and precision demands grow. Even with substantial training data, approximation errors remain practically large in moderate dimensions.

  • Empirical setup and findings: MLPs can approximate dot products on seen and fresh embeddings when training data and hidden-layer width are sufficient.The experiment evaluates generalization both to embeddings appearing in training and testing and to entirely fresh embeddings.
  • Empirical setup and findings: The sample size needed to learn the dot product scales polynomially with embedding dimension and decreasing target error.The authors report an anecdotal scaling of about O(d/ϵ)^α for 1 ≤ α ≤ 2.
  • Results: Learning the dot product becomes increasingly difficult as the embedding dimension grows.The result is consistent with the theoretical discussion that learning a degree-two polynomial has a dimension-dependent sample complexity.
  • Results: For d = 128 with 128000 users, the approximation error remains above 0.02, exceeding the 0.01 difference characterized as very significant.The synthetic task calibrates noise and scale against rating-prediction RMSE values from Netflix and MovieLens.
  • Results: The experiment indicates that an MLP may pay a significant price to approximate a dot product even when explicitly trained for that target.The authors suggest this difficulty may help explain why dot-product models outperform MLP similarities in their recommender experiments.

5 Applicability of Dot Product Models

Dot products are cheaper to compute and support efficient sublinear retrieval, whereas MLP similarities are too costly for real-time top-N recommendation at large scale.

  • Serving requirements: Real-time context-aware recommenders must retrieve top-scoring items from millions-item catalogues within a few milliseconds.Recommendations may depend on query-time context, preventing offline precomputation for all possible contexts.
  • Computational cost: Dot-product scoring costs O(d), compared with O(d2) for MLP-learned similarity.For n items, total scoring costs are O(dn) versus O(d2n).
  • Retrieval: Approximate nearest-neighbor and maximum-inner-product-search algorithms make dot-product retrieval feasible in typically a few milliseconds.These methods provide efficient sublinear-time retrieval even with millions of items.
  • Conclusion: MLP similarity is not applicable for real-time top-N recommenders, while dot products enable fast retrieval with established nearest-neighbor algorithms.The paper reports that no analogous sublinear techniques are known for MLP-based nearest-neighbor retrieval.

6 Related Work

The paper situates dot-product output layers within broader DNN practice, distinguishes related recommender architectures, and contrasts its focused analysis with broader evaluation studies.

  • Dot products at the Output Layer of DNNs: Many DNN classifiers map inputs to embeddings and combine them with class embeddings through a dot product at the output layer.The resulting scores can serve as logits in a softmax cross-entropy loss.
  • Dot products at the Output Layer of DNNs: These findings align with image-classification and natural-language-processing models that commonly use dot products to combine input and class representations.The paper frames dot-product embedding combination as common beyond recommender systems.
  • MLPs at the Output Layer of DNNs: NeuMF combines an MLP with extra embeddings and an explicit dot-product-like GMF structure, while another proposal replaces NCF’s MLP with an outer product followed by a CNN.The outer-product approach can trivially recover a dot product by summing the diagonal.
  • MLPs at the Output Layer of DNNs: The outer-product proposal uses different data, so its reported numbers cannot establish improvement over a well-tuned dot-product baseline.The paper also notes that it shares MLP applicability issues.
  • Specialized inductive biases: Specialized neural structures are commonly used when their inductive bias better represents the problem’s structure.The cited examples include convolutional, recurrent, and attention-based architectures.
  • Prior analysis: Prior work also studies MLP inefficiency in approximating dot and tensor products, but focuses on tensor products with embedding dimensions d = 1 and d = 2.This provides related evidence in a narrower setting.
  • Evaluation context: The paper’s analysis differs from a meta-study by focusing specifically on learned similarity functions while providing apples-to-apples comparisons.The meta-study covers a broader set of methods and publications.

7 Conclusion

The conclusion recommends dot products as a possible default for embedding combination, citing practical applicability, simpler modeling, and alignment with other DNN fields while cautioning against one-off conclusions.

  • Conclusion: A dot product might be a better default choice than MLP or NeuMF learned similarities for combining embeddings.The paper connects this recommendation to applicability and modeling considerations.
  • Practical implications: Using dot products could make recommender research more industry-relevant because the resulting models are applicable in production settings.The paper links this benefit to the applicability discussion.
  • Practical implications: Dot-product similarity simplifies modeling and learning by avoiding pretraining and the need for large datasets.This is presented as a potential positive effect of shifting research focus toward dot products.
  • Broader alignment: The conclusion also highlights alignment with natural-language-processing and image models, where dot products are commonly used.This connects recommender-system modeling with practices in other research areas.
  • Evaluation caution: The experiments provide further evidence that machine-learning methods are difficult to run properly and that one-off studies can produce wrong conclusions.The paper suggests shared benchmarks may help identify improvements more reliably.

A Experiments from NCF Paper

This appendix section documents the setup used to establish a dot-product baseline for reproducing the NCF experiments.

  • Experimental setup: The setup uses the NCF authors’ code and datasets and includes an implementation of matrix factorization plus a script for generating a tuning split.These materials support the dot-product baseline experiments described for Section 3.3.

A.1 Model and Optimization

The study implements a biased matrix factorization model using embeddings, binary logistic loss, L2 regularization, and uniform negative sampling. Its hyperparameters include embedding dimension and optimization, sampling, regularization, and initialization choices.

  • The matrix factorization model uses user and item embeddings, P ∈ R^M×d and Q ∈ R^N×d, with a bias term.
  • Training casts positive-only implicit data as binary classification and samples m uniformly distributed negative items per positive tuple each epoch.A new set of negative items is drawn in every epoch.
  • The model minimizes binary logistic loss with L2 regularization, using λ as the regularization constant.
  • Optimization uses stochastic gradient descent with learning rate η, while embeddings are initialized from a normal distribution.
  • The dot-product model tunes embedding dimension d, regularization λ, learning rate η, negative samples m, epochs, and initialization standard deviation.The reported embedding dimensions are 16, 32, 64, 96, 128, and 192.

A.2 Hyperparameter Tuning

Hyperparameters are selected on a tuning split that mirrors the final evaluation protocol, then fixed for full-data training and repeated evaluation. The tuning emphasizes embedding dimension, learning-rate convergence, and computational budget.

  • Hyperparameters are chosen on a tuning dataset formed by holding out each user’s last feedback while preserving the remaining cases for tuning training.Models are evaluated on the corresponding tuning test set before final training.
  • Larger embedding dimensions improve matrix factorization quality when other hyperparameters are properly selected, while learning rate and epochs affect convergence.Lower learning rates generally improve quality but require more epochs.
  • Embedding initialization uses a Gaussian distribution with standard deviation 0.1, and tested variations had little effect.
  • The selected settings use 256 epochs for both Movielens and Pinterest, with dataset-specific learning rates, negative samples, and regularization.Movielens uses η = 0.002, m = 8, λ = 0.005; Pinterest uses η = 0.007, m = 10, λ = 0.01.
  • After tuning, models are retrained on the full dataset and evaluated eight times, with mean metrics reported.

A.3 MLP and NeuMF Results

The study compares its results with MLP and NeuMF results from the original NCF paper under shared evaluation conditions. It also clarifies how predictive factor relates to embedding dimension across the models.

  • MLP and NeuMF results are taken from the original NCF paper and are comparable because the studies share evaluation protocols and data splits.
  • The comparison uses NeuMF results from its original Table 2 and MLP-3 results from the original Tables 3 and 4.
  • The original NCF paper calls the reported quantity predictive factor rather than embedding dimension.
  • For a 3-layer MLP, predictive factor k corresponds to input embeddings of dimension d = 2k.
  • For NeuMF, predictive factor k corresponds to embeddings of dimension d = 3k because it combines an independent MLP with another embedding component.
Loading 2005.09683v2…