PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta Matrix Factorization for Federated Rating Predictions

Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Dongxiao Yu, Jun Ma, Maarten de Rijke, Xiuzhen Cheng

arXiv:1910.10086v4cs.IRcs.LG

TL;DR

Federated recommender systems must protect privacy while fitting mobile limits and exploiting collaborative information that local model fine-tuning handles poorly. MetaMF generates private item embeddings and rating-prediction models through a server-side meta network, using collaborative memory and rise-dimensional generation. Across four benchmark datasets, it achieves competitive performance with matrix-factorization methods and higher rating-prediction performance than existing federated methods, while limitations remain for privacy leakage and cold-start users.

  • Problem

    Existing federated recommender systems impose substantial mobile resource demands and do not effectively exploit collaborative filtering among users and devices.

  • Method

    MetaMF uses collaborative memory and a meta recommender to generate private item embeddings and rating-prediction models, with rise-dimensional generation reducing embedding-generation difficulty.

  • Results

    MetaMF achieves competitive performance with state-of-the-art matrix-factorization methods and higher rating-prediction performance than existing federated recommendation methods across benchmark experiments.

  • Takeaways & Limitations

    MetaMF provides a federated architecture that exploits collaborative filtering while using smaller device-side rating-prediction models suited to mobile constraints.

  • Takeaways & Limitations

    MetaMF may leak personal information through server-generated embeddings, models, and updates, and currently cannot handle cold-start users well.

Abstract

from arXiv · show

Federated recommender systems have distinct advantages in terms of privacy protection over traditional recommender systems that are centralized at a data center. However, previous work on federated recommender systems does not fully consider the limitations of storage, RAM, energy and communication bandwidth in a mobile environment. The scales of the models proposed are too large to be easily run on mobile devices. And existing federated recommender systems need to fine-tune recommendation models on each device, making it hard to effectively exploit collaborative filtering information among users/devices. Our goal in this paper is to design a novel federated learning framework for rating prediction (RP) for mobile environments. We introduce a federated matrix factorization (MF) framework, named meta matrix factorization (MetaMF). Given a user, we first obtain a collaborative vector by collecting useful information with a collaborative memory module. Then, we employ a meta recommender module to generate private item embeddings and a RP model based on the collaborative vector in the server. To address the challenge of generating a large number of high-dimensional item embeddings, we devise a rise-dimensional generation strategy that first generates a low-dimensional item embedding matrix and a rise-dimensional matrix, and then multiply them to obtain high-dimensional embeddings. We use the generated model to produce private RPs for the given user on her device. MetaMF shows a high capacity even with a small RP model, which can adapt to the limitations of a mobile environment. We conduct extensive experiments on four benchmark datasets to compare MetaMF with existing MF methods and find that MetaMF can achieve competitive performance. Moreover, we find MetaMF achieves higher RP performance over existing federated methods by better exploiting collaborative filtering among users/devices.

1 INTRODUCTION

MetaMF addresses federated recommendation’s mobile-resource constraints and limited collaborative filtering by generating compact, user-specific recommendation components on the server. Experiments show competitive matrix-factorization performance and stronger results than existing federated methods.

  • Federated recommender models can heavily consume mobile storage, RAM, energy, and bandwidth while failing to fully exploit collaborative-filtering information.
  • MetaMF targets federated rating prediction with a framework designed for mobile environments and performance comparable to centralized state-of-the-art methods.
  • MetaMF uses collaborative memory to form user-specific collaborative vectors, then generates private item embeddings and rating-prediction models through a meta recommender.
  • The rise-dimensional generation strategy reduces generation difficulty by combining a low-dimensional item-embedding matrix with a rise-dimensional matrix.
  • Experiments on four benchmark datasets find competitive performance with state-of-the-art matrix-factorization methods and higher rating-prediction performance than existing federated methods.
  • The paper contributes MetaMF, collaborative-memory and meta-recommender modules, rise-dimensional generation, and experiments evaluating effectiveness and efficiency.

2 RELATED WORK

Related work spans federated recommendation, matrix factorization, and meta learning. MetaMF differs by generating small, private matrix-factorization models and item embeddings for individual users while using a server-side meta network.

  • Federated Recommender Systems: Federated recommender systems preserve local data but face difficulty exploiting collaborative filtering across users and devices.
  • Federated Recommender Systems: Prior federated approaches commonly fine-tune shared models locally or do not focus on reducing local-model size for rating prediction.
  • Federated Recommender Systems: Unlike earlier federated work focused on ranking or same-sized global and local models, MetaMF uses a large server-side meta network and a small device-side rating-prediction model.
  • Matrix Factorization: Traditional matrix-factorization and neural recommendation methods generally share item embeddings and models while learning personalized user representations.
  • Matrix Factorization: MetaMF instead generates non-shared, small models and item embeddings for individual users.
  • Meta Learning: MetaMF differs from prior meta-learning recommendation work by directly generating private matrix-factorization models for rating prediction.

3 META MATRIX FACTORIZATION

MetaMF separates a shared server-side meta network from private device-side prediction models. Its collaborative memory combines user-specific embeddings with shared information so generated models can exploit cross-user structure.

  • Overview: Rating prediction estimates an unknown user-item rating from users, items, and divided training, validation, and test ratings.
  • Overview: MetaMF contains collaborative memory, a meta recommender, and a private prediction module, with the first two forming a shared meta network.
  • Collaborative Memory Module: The collaborative-memory module produces a user’s collaborative vector from a user embedding and shared memory that fuses information from all users.
  • Overview: The meta recommender uses the collaborative vector to generate a private rating-prediction model and private item-embedding matrix for that user.
  • Federated Deployment: The server deploys collaborative memory and the meta recommender, while mobile devices receive prediction modules and compute local losses and gradients.
  • Federated Deployment: Placing the parameter-heavy meta network on the server and the smaller prediction module on devices supports mobile environments with limited resources.
  • Collaborative Memory Module: Shared memory lets MetaMF assign similar collaborative vectors to users, enabling federated collaborative filtering while learning user representations.

3.4 Meta Recommender Module

The meta recommender generates private item embeddings and private RP models for each user from a collaborative vector. Rise-dimensional generation reduces the cost of producing high-dimensional item embeddings.

  • The MR module generates private item embeddings and an RP model from each user’s collaborative vector.
  • Private Item Embeddings: Rise-dimensional generation decomposes each private item embedding matrix into low-dimensional and rise-dimensional matrices before multiplication.The low-dimensional embedding size s is much smaller than the item embedding dimension d_i.
  • Private Item Embeddings: O(s × n + d_i × s) parameters replace O(d_i × n) parameters for generating each item embedding matrix.The generated matrices differ across users.
  • Private RP Model: The private RP model is generated layer by layer, with weights and biases reshaped to match each layer’s input and output dimensions.Generation parameters are not shared across RP-model layers, and MetaMF returns different MLP parameters to different users.

3.5 Prediction Module

The prediction module uses each user’s generated item embedding matrix and RP model to estimate ratings for items. The final RP-model layer returns a scalar predicted rating.

  • The prediction module estimates a user’s rating for an item using the generated item embedding matrix and RP model.
  • The RP model applies its sequence of generated layers to the item embedding, with the final layer returning the scalar predicted rating.

3.6 Loss

MetaMF formulates rating prediction as regression and trains its server-side parameters with regularized federated learning. Item embeddings and RP-model parameters are generated outputs rather than trainable parameters.

  • The rating-prediction task is formulated as a regression problem.
  • L2 regularization is added to the training objective to reduce overfitting.
  • Only MetaMF’s trainable parameters Θ are optimized directly; generated item embeddings and RP-model parameters are excluded from Θ.
  • Federated backpropagation trains the framework from decentralized user data, with devices uploading gradients and the server updating shared parameters.

4 EXPERIMENTAL SETUP

The experiments evaluate MetaMF against conventional, deep-learning, and federated rating-prediction baselines on four datasets using MAE and MSE. Statistical differences are tested with paired t-tests.

  • The study asks whether MetaMF matches state-of-the-art MF performance and assesses the contribution of private item embeddings and RP models.
  • Experiments use Douban, Hetrec-movielens, Movielens1M, and Ciao, splitting each user’s data into 80% training, 10% validation, and 10% test.
  • Baselines: The comparison includes conventional methods such as NMF, PMF, SVD++, and LLORMA; deep methods including RBM, AutoRec, and NCF; and federated methods.
  • Evaluation Metrics: MAE and MSE evaluate rating-prediction performance on held-out test ratings.
  • Evaluation Metrics: Observed differences are tested using a two-sided paired t-test with p < 0.01.
  • Implementation Details: The implementation uses 32-dimensional user and item embeddings, a 128-dimensional collaborative vector, and an 8-dimensional low-dimensional item embedding.

5 EXPERIMENTAL RESULTS

MetaMF achieves competitive rating-prediction performance despite federation, outperforming FedRec while using smaller models. Its privacy-performance trade-off varies with dataset sparsity and the extent of private user data.

  • MetaMF is comparable to NCF on Douban and Hetrec-movielens, despite being federated while most baselines are centralized.
  • MetaMF performs poorly on Movielens1M and Ciao because these datasets provide fewer ratings per user.The average rating counts are 357, 405, 166, and 38 across the four datasets, respectively.
  • MetaMF significantly outperforms FedRec on all datasets with smaller user/item embeddings and rating-prediction model scale.The authors attribute this advantage to MetaMF’s meta network, which exploits collaborative filtering across users and devices.
  • MetaMF achieves a better balance between privacy protection and rating-prediction performance, although federation can reduce performance relative to centralized methods.
  • Generating private item embeddings and private rating-prediction models contributes to MetaMF’s overall performance.MetaMF outperforms MetaMF-SI on most datasets and consistently outperforms MetaMF-SM.
  • Private item embeddings have a greater performance impact than private rating-prediction models in the reported ablations.MetaMF-SI outperforms MetaMF-SM on all datasets.

6 ANALYSIS

MetaMF maintains high capacity at smaller model scales by generating private embeddings and prediction models, but its benefits are constrained by sparse user data and large item sets. Visualizations show user-specific generated parameters on most datasets, with Ciao as the exception.

  • 6.1 Model Scale Analysis: MetaMF matches NCF with smaller item embeddings, fewer layers, and smaller layer sizes at the best reported settings.The compared settings are MetaMF (32,, [128, 512]) and NCF (256, [256, 128, 64]).
  • 6.1 Model Scale Analysis: At small model scales, MetaMF significantly outperforms NCF.The authors attribute this to private embeddings and prediction models encoding each user’s preferences with fewer parameters.
  • 6.1 Model Scale Analysis: MetaMF performance deteriorates earlier than NCF as model scale increases.Insufficient private data can cause overfitting, while generating larger embeddings for many items becomes difficult.
  • 6.2 Weights and Embeddings: Figure 2 visualizes generated weights and item embeddings after t-SNE reduction and normalization, with each point representing one user.
  • 6.2 Weights and Embeddings: MetaMF generates different weights and item embeddings for users on most datasets, indicating user-specific modeling while exploiting collaborative filtering.
  • 6.2 Weights and Embeddings: Ciao is the exception: MetaMF does not learn distinguishable weights and item embeddings, consistent with insufficient private data.

7 CONCLUSION AND DISCUSSION

MetaMF targets federated rating prediction by reducing matrix-factorization model scale while generating private models and embeddings per user. It achieves competitive performance with smaller models, but remains limited by privacy leakage risk and weak cold-start handling.

  • Conclusion and Discussion: MetaMF generates private rating-prediction models and item embeddings for each user with a meta network.The framework is designed for federated rating prediction while reducing matrix-factorization model scale.
  • Conclusion and Discussion: MetaMF reaches state-of-the-art rating-prediction performance despite significantly smaller rating-prediction models and item embedding sizes.The paper reports this outcome in its comparison with state-of-the-art rating-prediction methods.
  • Conclusion and Discussion: MetaMF outperforms FedRec by a large margin by using collaborative filtering in a federated environment.The comparison attributes the advantage to better use of collaborative filtering among users and devices.
  • Conclusion and Discussion: MetaMF may leak personal information through server-generated private embeddings, models, and their updates.The authors identify a more privacy-aware generation network as future work.
  • Conclusion and Discussion: MetaMF does not handle cold-start users well because satisfactory performance requires a certain amount of personalized data.The authors suggest few-shot or zero-shot learning as possible directions for reducing this data requirement.

DATA AND CODE

The authors share all resources used in the paper to facilitate reproducibility.

  • Data and Code: All resources used in the paper are available in the authors’ GitHub repository.The repository URL is https://github.com/TempSDU/MetaMF.
Loading 1910.10086v4…