Source-linked AI summary
Applied Federated Learning: Improving Google Keyboard Query Suggestions
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, Françoise Beaufays
TL;DR
The paper addresses how to improve mobile keyboard query suggestions while protecting sensitive user data and meeting latency constraints. It applies federated learning to train, evaluate, and deploy a triggering model for Gboard without direct access to underlying user data, and reports improved live click-through rate with some deployment skew and training-population limitations.
Problem
Gboard needs relevant, low-latency suggestions while handling sensitive typed input without direct access to users’ underlying data.
Method
The paper uses federated learning to train, evaluate, and deploy a logistic-regression triggering model that filters baseline query suggestions on-device.
Results
Live deployments improved click-through rate, capturing most expected training-metric improvements but showing a slight drop between expected and actual ∆CTR.
Takeaways & Limitations
The end-to-end production deployment demonstrates a privacy-advantaged way to improve Gboard suggestion quality without direct access to underlying user data.
Takeaways & Limitations
Training was constrained to devices with 2GB of RAM, while deployment covered all devices, creating a device-specification skew between populations.
Abstract
from arXiv · showhide
Federated learning is a distributed form of machine learning where both the training data and model training are decentralized. In this paper, we use federated learning in a commercial, global-scale setting to train, evaluate and deploy a model to improve virtual keyboard search suggestion quality without direct access to the underlying user data. We describe our observations in federated training, compare metrics to live deployments, and present resulting quality increases. In whole, we demonstrate how federated learning can be applied end-to-end to both improve user experiences and enhance user privacy.
1. INTRODUCTION
The paper applies federated learning to Gboard’s global-scale query suggestions, training and deploying a model without direct access to user data. This approach addresses privacy and latency constraints while improving suggestion quality.
- Federated learning trains models on end-user devices while aggregating only ephemeral parameter updates on a centralized server.This contrasts with traditional server-side training, which aggregates user data centrally.
- Gboard’s scale and access to typed input make privacy essential, while mobile typing requires minimal suggestion latency.Gboard had over 1 billion installs in 2018 and must respect the privacy of sensitive user input.
- On-device inference and training through federated learning minimize latency and maximize privacy for keyboard suggestions.
- The paper trains and deploys a federated model for search query suggestions in a commercial, global-scale setting without access to underlying user data.An additional triggering model filters suggestions produced by the baseline model.
- The study reports federated training observations, live-deployment comparisons, and quality improvements while respecting mobile constraints.The paper covers enabling infrastructure, model architecture, federated training, and live inference results.
2. FEDERATED LEARNING AND ON-DEVICE INFRASTRUCTURE
Federated learning keeps data at network edges while clients perform local training and send model updates for server aggregation. Gboard’s infrastructure schedules these tasks on suitable devices, evaluates models on-device, and deploys converged checkpoints.
- Federated Learning: Federated learning keeps data at network edges instead of collecting it centrally, transmitting focused model updates instead.The paper identifies privacy-sensitive, naturally labeled, or excessively large datasets as suitable use cases.
- Federated Learning: In Federated Averaging, selected clients download global parameters, train locally on private data, upload updates, and contribute to server aggregation.
- Privacy Advantages: Basic federated learning transmits parameter deltas ephemerally, while secure aggregation and differential privacy can provide stronger privacy but were not used here.
- System Description: Gboard stores interaction-derived feature-label tuples in an on-device cache for local training and evaluation.Features describe query and context information, while labels are either clicked or ignored.
- System Description: Background jobs run when devices are idle, charging, and on unmetered networks, then upload anonymous updates for server-side aggregation.The client runtime uses TensorFlow Mobile, and trained checkpoints are deployed after convergence.
3. MODEL ARCHITECTURE
The system combines a traditionally server-trained baseline that generates query candidates with a federated triggering model that filters low-quality suggestions. The triggering model scores click likelihood and uses a threshold to control which suggestions appear.
- Model Architecture: The architecture uses a server-trained baseline to generate candidates and a federated triggering model to remove low-quality suggestions.Its stated objective is to improve query click-through rate by filtering baseline outputs.
- Baseline Model: The baseline matches input against an on-device Knowledge Graph subset, then scores candidates with an offline-trained LSTM.The highest-scoring candidate is displayed as a query suggestion.
- Triggering Model: The triggering model is logistic regression that predicts click probability and outputs higher scores for suggestions it considers more likely to be clicked.
- Triggering Model: A threshold τ controls the desired triggering rate: higher thresholds are stricter and reduce the number of displayed suggestions.The model score is in logit space, with click probability obtained through the logistic sigmoid function.
- Triggering Model: Features include past clicks and impressions, the baseline score, and temporal indicators such as day of week and hour of day.These features support personalization and capture temporal patterns in click behavior.
- Triggering Model: Logistic regression was chosen because its convex error function supports training with limited clients and examples, including heavily skewed binary labels.
4. TRAINING WITH FEDERATED LEARNING
Federated training is constrained by device availability and participation requirements, producing strong time-of-day effects and skewed training populations. Without direct access to training examples, model quality was assessed through on-device metrics, synthetic and donated data, coefficient inspection, and live-oriented validation.
- 4.1. Federated Training Requirements: Training clients had to be charging, on an unmetered network, idle, and meet memory, Android-version, and language requirements.Rounds targeted 100 clients, could run with 80, and required 80% reporting within two minutes.
- 4.2. Federated Training: Most training occurred in the evening, so rounds progressed faster at night than during the day as more clients became available.Off-peak periods also faced fewer available clients and contention from other training tasks, making round requirements harder to satisfy.
- 4.2. Federated Training: Daytime training involved skewed populations, while evening training had more examples; consequently, eval loss was highest during the day and training example count highest in the evening.Many daytime clients were in or around India despite the en-US locale restriction, contributing to the population skew.
- 4.2. Federated Training: Federated training introduced diurnal characteristics that are expected to remain inherent challenges even as overall training speed improves.The authors identify globally distributed training examples and clients as continuing sources of difficulty.
- 4.3. Model Debugging Without Training Example Access: Synthetic and donated examples supported architecture and hyperparameter development, integration testing, and confidence that the model learned expected application-generated patterns.These data sources enabled validation without relying on direct access to user training examples.
- 4.3. Model Debugging Without Training Example Access: Coefficient inspection found intuitive category weights, smooth monotone binned-feature weights, and larger absolute weights for more common features.Inspecting weights also exposed an overly broad feature range, which was narrowed to improve precision without changing the number of bins.
5. LIVE RESULTS AND OBSERVATIONS
Live deployment improved CTR relative to the expected training metrics, while threshold selection exposed tradeoffs and several sources of training–deployment skew. Later model iterations incorporated more data, tuning, and features, with the final model adding LSTM-based typed-text featurization.
- Live deployment results: The triggering model’s score is compared with thresholds τ0 < τ1 < τ2, where higher thresholds impose stricter quality bars and reduce suggestion trigger rates.Threshold selection trades off CTR against retained impressions and clicks.
- Live deployment results: Live deployments improved CTR, although actual ∆CTR was slightly lower than the expected ∆CTR from training metrics.The comparison is reported in Table 1.
- Sources of skew: Training and deployment populations differed because training required charging, unmetered networks, and at least 2GB of RAM, whereas deployment had no minimum RAM requirement.These conditions bias training toward users and devices with more favorable environmental and hardware characteristics.
- Sources of skew: Requiring 80% of selected devices to respond further skews training toward higher-end devices with more stable networks.Lower-end devices are described as more unstable from both device and network perspectives.
- Sources of skew: Training and evaluation rounds may overlap, but the expected overlap is <0.1% and is expected to have minimal impact on performance skew.Both rounds select only a small subset of the overall training population.
- Sources of skew: Model drift from training–deployment time offsets may also contribute to differences between expected and actual metrics.The authors expect this delta to narrow as federated learning matures.
- Model iterations: Successive iterations trained longer on more users’ data, tuned hyperparameters, and added features, including an LSTM-based featurization of typed text in the final model.The authors report that these iterations demonstrated effective privacy-advantaged model development without direct access to user data.
6. CONCLUSION
The work applies federated learning end to end to improve keyboard search suggestions without access to underlying user data. It also examines training dynamics, iteration without direct data access, and skew between federated training and live deployment.
- 6. CONCLUSION: The paper trains, evaluates, and deploys a logistic regression model with federated learning to improve keyboard search suggestion quality without underlying user data access.This is presented as an early end-to-end production example of federated learning.
- 6. CONCLUSION: The authors discuss cyclic training, model iteration without direct training-data access, and sources of skew between federated training and live deployments.The work explores a privacy-advantaged path for improving user experience.