Source-linked AI summary
Federated Evaluation of On-device Personalization
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, Daniel Ramage
TL;DR
Federated learning preserves on-device data, but personalization requires evaluating whether local refinement helps individual users. The paper extends federated learning with privacy-preserving evaluation tools and reports that personalization benefits users across a large live population. It also examines how training settings and user-data characteristics affect these benefits.
Problem
Personalized refinements may improve some users’ experience while worsening others, and their effects cannot be tested server-side because user data is not collected centrally.
Method
The paper extends federated learning with Federated Personalization Evaluation, using local train/test data, on-device personalization, and server-side aggregation of metric deltas.
Results
Personalization benefits users across a large population, with a best mean relative accuracy increase of 14.5% for B = 5 and L = 0.1.
Takeaways & Limitations
Personalization strategies can be tuned and evaluated privately on live user traffic, while deployment conditions can restrict use to models that improve user experience.
Abstract
from arXiv · showhide
Federated learning is a distributed, on-device computation framework that enables training global models without exporting sensitive user data to servers. In this work, we describe methods to extend the federation framework to evaluate strategies for personalization of global models. We present tools to analyze the effects of personalization and evaluate conditions under which personalization yields desirable models. We report on our experiments personalizing a language model for a virtual keyboard for smartphones with a population of tens of millions of users. We show that a significant fraction of users benefit from personalization.
1 Introduction
The paper motivates on-device personalization as a privacy-preserving way to adapt global models to individual users, while emphasizing the need to prevent harm from overfitting. It extends federated learning to evaluate personalization and identifies conditions for beneficial deployment.
- On-device data can personalize global models without exporting sensitive user data.Fine-tuning occurs locally using standard optimization methods.
- Personalization may improve most users’ experience but can worsen outcomes for some through overfitting.
- The framework evaluates personalization strategies and derives conditions for deploying personalized models only when user experience improves.
- Experiments use an RNN language model for smartphone keyboard next-word prediction and show benefits for a significant fraction of users.
2 Federated Personalization Evaluation
Federated Personalization Evaluation extends federated learning so personalization can be assessed privately on-device rather than server-side. Devices evaluate baseline and personalized models locally, while the server aggregates metric differences to analyze deployment conditions.
- Federated learning keeps user data on devices while transmitting minimal, ephemeral model updates for server aggregation.
- Personalization cannot be tested server-side because training and evaluation data are not collected centrally.
- A gating mechanism accepts or rejects personalized models for inference, alongside calibrated hyperparameters intended to prevent undesired effects.
- FPE splits local data into train and test partitions, evaluates baseline and personalized models, and reports paired metrics to the server.The server computes histograms of metric deltas across devices.
- FPE evaluates personalization benefits and hyperparameters using existing federated-learning infrastructure without user-visible impact.Live inference using personalized models is beyond the paper’s scope.
3 Method
The method trains a federated keyboard language model, then personalizes it locally using configurable optimization and stopping criteria. Personalization is evaluated on temporally separated local data partitions.
- Global Model Training: The global next-word prediction model uses a CIFG recurrent neural network with tied input-embedding and output-projection matrices.
- Global Model Training: Federated training uses client SGD updates and weighted server aggregation across synchronous rounds.
- Global Model Training: Training converges after 3000 rounds, processing 600 million sentences across 1.5 million clients over 4 to 5 days.
- Model Personalization Strategies: A personalization strategy specifies the model graph, initial parameters, client learning rate, train batch size, and stopping criteria.
- Model Personalization Strategies: Local cache data is split temporally into 80% train and 20% test partitions, then trained with SGD until token or epoch criteria are met.
4 Experiments
Experiments measure per-user accuracy changes under different personalization settings and data conditions. Results show substantial gains for some users, sensitivity to learning rate and batch size, and larger benefits for users with more divergent baseline performance.
- Prediction accuracy is measured as correct predictions divided by total tokens, with per-user deltas compared against baseline accuracy.
- 14.5% mean relative accuracy increase occurs with B = 5 and L = 0.1, raising mean accuracy from 0.166 to 0.19.
- 47% of users achieve ≥0.02 accuracy improvement at B = 5 and L = 0.1, compared with 39% at B = 10 and L = 1.0 and 29% at B = 10 and L = 0.1.
- With larger batches, learning rate 1.0 has heavier left and right tails; 39% versus 29% achieve ≥0.02 improvement despite mean gains of 0.015 versus 0.018.
- More user tokens generally produce larger improvements, while learning rate 1.0 performs best with very few tokens and gains saturate for learning rate 0.1.
- Users with baseline accuracy ≤0.1 improve by greater than 0.25, whereas users with baseline accuracy ≥0.2 improve by less than 0.2.
5 Conclusion
The work develops privacy-preserving tools for evaluating and tuning on-device personalization, showing benefits across a large population of live users.
- Personalization benefits users across a large population.The experiments use live traffic to assess personalization outcomes.
- The proposed tools evaluate personalization and analyze results in a privacy-preserving manner.
- Personalization strategies can be tuned to achieve better performance.