Source-linked AI summary
Neural Network Matrix Factorization
Gintare Karolina Dziugaite, Daniel M. Roy
TL;DR
The paper asks how to predict missing entries in relational arrays when a fixed inner product may be too restrictive. It replaces that function with a learned feed-forward neural network jointly trained with latent features. NNMF outperforms standard latent-feature methods in the reported benchmarks but remains behind approaches that exploit local graph structure, while its broader potential is left unresolved.
Problem
Matrix factorization predicts missing array entries with a fixed inner product, motivating a learned function for relational data such as collaborative filtering.
Method
NNMF jointly learns a feed-forward neural network and latent row and column features, alternating optimization while using a regularized objective.
Results
NNMF dominates PMF, RFM, and BiasedMF, but is dominated by LLORMA and AutoRec or other approaches that exploit local graph structure.
Takeaways & Limitations
Learned neural interactions can substantially improve latent-feature matrix factorization, but the reported advantage does not extend to methods incorporating local graph structure.
Takeaways & Limitations
The experiments did not exhaustively explore architectures, activation functions, regularization, or validation strategies, so the limits of NNMF were not identified.
Abstract
from arXiv · showhide
Data often comes in the form of an array or matrix. Matrix factorization techniques attempt to recover missing or corrupted entries by assuming that the matrix can be written as the product of two low-rank matrices. In other words, matrix factorization approximates the entries of the matrix by a simple, fixed function---namely, the inner product---acting on the latent feature vectors for the corresponding row and column. Here we consider replacing the inner product by an arbitrary function that we learn from the data at the same time as we learn the latent feature vectors. In particular, we replace the inner product by a multi-layer feed-forward neural network, and learn by alternating between optimizing the network for fixed latent features, and optimizing the latent features for a fixed network. The resulting approach---which we call neural network matrix factorization or NNMF, for short---dominates standard low-rank techniques on a suite of benchmark but is dominated by some recent proposals that take advantage of the graph features. Given the vast range of architectures, activation functions, regularizers, and optimization techniques that could be used within the NNMF framework, it seems likely the true potential of the approach has yet to be reached.
1 INTRODUCTION
The paper studies predicting unobserved entries in relational arrays such as user–movie ratings. It extends latent-feature matrix factorization by learning the function that maps user and movie features to predictions instead of fixing it as an inner product.
- Matrix factorization: Matrix factorization predicts missing array entries by approximating X with the product of two low-rank latent-feature matrices.User and movie rows are represented by latent vectors, with rank D much smaller than the array dimensions.
- Matrix factorization: Probabilistic matrix factorization models observed entries with Gaussian noise around inner-product predictions and minimizes prediction error.Regularizing the latent feature vectors often improves performance.
- Motivation: PMF is effective but can be improved for large, sparsely observed arrays through additional modeling choices.The paper identifies sparse collaborative-filtering data as a setting where PMF is easy to improve upon.
- Motivation: BiasedMF adds user, movie, and global effects to model systematic rating biases.These effects can be represented as a special case of PMF with fixed latent-feature entries, creating a strong inductive bias.
- NNMF: NNMF learns a feed-forward neural network function jointly with latent features, replacing the fixed inner product used by PMF and BiasedPMF.The network weights and latent features are learned from the data together.
2 MODEL
The model assigns latent features to rows and columns, then uses a feed-forward neural network to produce the mean of each observed array entry. Its inputs combine user-specific, movie-specific, and additional inner-product-derived features.
- Latent representations: Each row and column receives a latent feature vector, while additional latent feature matrices provide further row- and column-level representations.The row and column feature collections are denoted jointly by (U, V).
- Observation model: The model treats network weights and latent features as unknown parameters and assumes independent entries with means determined by the neural network.The network is a feed-forward function fθ.
- Network inputs: The neural network has 2D + D′ real-valued inputs and one real-valued output.Inputs include user-specific features, movie-specific features, and inner products between K-dimensional latent vectors.
3 LEARNING
Learning minimizes a regularized objective over network weights and latent features. The procedure alternates gradient-based optimization of the network and latent features, with a particular experimental parameterization and regularization choices.
- Objective: The training objective jointly optimizes neural-network weights and latent features with a regularization penalty.The objective is interpretable as penalized Gaussian log likelihood or a maximum a posteriori estimate with independent Gaussian priors.
- Objective: The regularization parameter λ controls ℓ2 penalties on latent features within the objective.The objective also uses the Frobenius norm for the data-fitting term.
- Model configuration: The experiments used K = 1 and large D′, whereas D′ = 1 gives a simpler matrix-factorization modification with less dramatic improvements.The simpler model combines a rank-K prediction with 2D additional features.
- Optimization: Training alternated between optimizing network weights with fixed latent features and optimizing latent features with fixed network weights.Both stages used gradient descent over the entire dataset, with RMSProp adjusting the learning rate.
4 RELATED WORK
Related work places NNMF among latent-feature and graph-feature approaches for relational data, connecting it to Gaussian-process matrix models, collaborative-filtering systems, and neural tensor methods.
- Matrix and relational models: NNMF replaces the Gaussian-process function in the Random Function Model with a parametric feed-forward neural network over latent representations.The authors report better performance when some input dimensions undergo an elementwise product before entering the network.
- Knowledge graphs: Knowledge graphs represent relational facts as entity–relation–entity triples, making them comparable to extremely sparse arrays or a single three-dimensional array.Unlike collaborative filtering, knowledge-graph methods must learn many relations using shared representations under limited data per relation.
- Neural tensor methods: The Neural Tensor Network is related to NNMF because it combines latent feature vectors with a tensor product and a single-layer neural network.Its formulation uses feature vectors, linear weights, biases, a weight matrix, a third-order tensor, and elementwise tanh.
- Collaborative filtering: Collaborative-filtering methods such as LLORMA and I-AutoRec combine latent features with graph-derived information rather than relying solely on global low-rank structure.LLORMA uses local low-rank approximations selected using graph features, while I-AutoRec uses observed ratings to represent relationships among movies.
- Collaborative filtering: I-AutoRec feeds a user’s observed ratings into an autoencoder, and its missing entries are typically replaced with 3.It achieves state-of-the-art performance and slightly outperforms LLORMA on some benchmarks, although strict comparison requires controlling model parameters and datasets.
- Empirical comparisons: NNMF outperformed other latent-feature methods and I-RBM but was outperformed by LLORMA and I-AutoRec across graph-link-prediction and movie-rating benchmarks.The authors suggest this gap may reflect either neural-network optimization choices or information in missing-rating patterns that NNMF does not use.
5 EXPERIMENTS
NNMF was evaluated on graph and collaborative-filtering benchmarks against latent-feature, neural, and graph-aware methods. It generally outperformed standard latent-feature baselines, but stronger local or graph-structure methods remained competitive or superior.
- Datasets: NNMF was evaluated on two graph datasets and two MovieLens collaborative-filtering datasets.The experiments used NIPS, Protein, MovieLens 100K, and MovieLens 1M.
- Evaluation protocol: The evaluation averaged performance over five randomly subsampled test sets, with validation data reserved from the remaining training data.Validation fractions differed across graph and MovieLens datasets.
- Results: NNMF dominated PMF, RFM, and, to a lesser extent, NTN across the reported comparisons.The paper notes that PMF’s higher-dimensional configuration outperformed the lower-dimensional RFM comparison.
- Results: NNMF outperformed BiasedMF, although its margin narrowed on the sparsely observed MovieLens datasets.Adding bias-correction terms improved NNMF by approximately 0.003, a potentially non-robust gain.
- Results: NNMF beat global low-rank LLORMA but not local LLORMA, and AutoRec also outperformed NNMF.The authors note that architecture, composition, and training choices might have affected the comparison.
- Architecture experiments: Using four hidden layers on MovieLens, fewer units per layer yielded better results, but additional experiments were needed to assess overfitting.The comparison used 50 versus 20 units per layer under different feature dimensions.
6 DISCUSSION
NNMF performs strongly among latent-feature models but remains behind methods that use local graph structure. The authors caution that the explored configurations do not establish the model’s full potential.
- Discussion: NNMF achieves state-of-the-art results among latent-feature models but is dominated by approaches using local graph structure.The comparison therefore separates strength within the latent-feature family from performance against graph-aware approaches.
- Discussion: The experiments did not exhaustively explore neural architectures, activation functions, regularization, or cross-validation strategies.The authors also warn that extensive adaptive configuration searches could overfit and reduce interpretability.