Source-linked AI summary
Federated Learning for Emoji Prediction in a Mobile Keyboard
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, Françoise Beaufays
TL;DR
The paper asks whether emoji prediction can be trained for mobile keyboards without centrally collecting users’ text. It combines a recurrent emoji predictor with language-model pretraining, triggering and diversification mechanisms, and federated on-device training. The federated model achieves better performance than a server-trained model, supporting the feasibility of production-quality natural-language models trained while data remains on users’ devices.
Problem
Mobile keyboards need accessible emoji suggestions, but training such models must account for constrained devices and users’ data remaining on-device.
Method
The paper uses a recurrent emoji prediction model with language-model pretraining, separate triggering and diversification mechanisms, and federated learning.
Results
The federated model achieves better Accuracy@1 than a server-trained model in federated evaluation, although its AUC is lower.
Takeaways & Limitations
Federated learning can train an emoji prediction model with sparse, poorly balanced data while keeping users’ data on their devices.
Takeaways & Limitations
AUC is computed only on server logs from short snippets typed in selected apps, which are less representative than client-cache data.
Abstract
from arXiv · showhide
We show that a word-level recurrent neural network can predict emoji from text typed on a mobile keyboard. We demonstrate the usefulness of transfer learning for predicting emoji by pretraining the model using a language modeling task. We also propose mechanisms to trigger emoji and tune the diversity of candidates. The model is trained using a distributed on-device learning framework called federated learning. The federated model is shown to achieve better performance than a server-trained model. This work demonstrates the feasibility of using federated learning to train production-quality models for natural language understanding tasks while keeping users' data on their devices.
1 Introduction
The paper adds text-based emoji predictions to Gboard, while addressing mobile constraints that require low-latency inference.
- Gboard offers emoji predictions based on text previously typed, alongside word predictions.The paper illustrates this behavior with the context “This party is lit.”
- Mobile devices constrain memory and CPU, while keyboard responses are typically expected within 20 ms of an input event.
- A unidirectional RNN supports cached model states at each timestep because it uses only backward-in-time dependencies.Caching is used to reduce inference latency.
2 Federated Learning
Federated learning keeps user data on devices and sends focused model updates to a server, which aggregates local training into a global model.
- Federated Learning keeps users’ data on their devices rather than collecting it centrally, while transmitting minimal and focused model updates.The framework can also be combined with secure multi-party computation and differential privacy.
- FederatedAveraging sends a global model to selected devices, where local SGD updates are computed on client batches.
- The server averages client weights after each training round to produce the next global model parameters.
3 Method
The method combines a compact CIFG recurrent model with language-model pretraining, separate emoji triggering, and probability scaling to diversify candidates.
- Model architecture: CIFG couples the input and forget gates through f = 1−i, reducing parameters per cell by 25% compared with an LSTM.
- Model architecture: The model uses a two-layer CIFG with 256 units per layer and predicts probabilities over 100 emoji from a 10,000-word vocabulary.Inputs use 96-dimensional embeddings.
- Pretraining: Language-model pretraining of all layers except the output projection leads to much faster convergence for the emoji model.The language model achieves Accuracy@1 of 13.7% on the same vocabulary.
- Triggering: A separate triggering decision is treated as a product choice, allowing control over how often emoji predictions are offered.
- Triggering: The model uses an additional <UNK> class and shows emoji predictions only when its probability falls below a threshold.Training truncates sentences without emoji labels to random lengths so the model learns not to predict emoji after mid-sentence tokens.
- Diversification: Because frequent emoji dominate predictions across contexts, the method scales each emoji probability using its empirical training frequency and a live-traffic-tuned factor α.Setting α to 0 removes diversification.
4 Server-based Training
Server-based models are trained on anonymized, opt-in Gboard logs filtered to high-confidence English sentences.
- Server-based training uses anonymized snippets shared by opted-in Gboard users from selected applications.Personally identifiable information is stripped from the logs before further filtering.
- The training logs are filtered to sentences labeled English with high confidence by a language detection model.
5 Federated Training
Federated training uses local client caches under eligibility constraints, with one local epoch per round and convergence typically after 2000 rounds.
- Participating devices must have at least 2 GB of RAM, be in the United States or Canada, and use English (US).Devices must also be idle, charging, and connected to un-metered networks.
- Each client has approximately 400 sentences on average and trains for one epoch in each round.
- The model typically converges after 2000 training rounds.Federated evaluation uses a separate task on a different subset of client devices rather than an explicit train–evaluation split.
6 Evaluation
Model quality is measured with Accuracy@1, while AUC evaluates the triggering mechanism using server-collected logs; all metrics are computed before diversification.
- Accuracy@1 is the ratio of accurate top-1 emoji predictions to examples containing emoji.
- AUC evaluates the quality of the triggering mechanism.
- AUC is reported only on server-collected logs because numerical integration is not straightforward in the federated-learning setting.
- All evaluation metrics are computed before diversification.
7 Federated Experiments
Federated experiments examine client batch size, devices per round, and server optimization before comparing the best federated model with a server-trained model.
- Client caches change continuously, so results from these non-concurrent experiments cannot be compared across experiments.The study nevertheless compares training configurations and then evaluates the best federated model against a server-trained model.
- Large client batch sizes substantially improve model quality because emoji-containing sentences are sparse in client caches.Larger batches produce more accurate gradient updates, especially with heavily imbalanced target classes.
- Accuracy increases with the number of devices per round, but returns diminish beyond K = 500.
- Momentum of 0.9 with Nesterov accelerated gradients improves convergence speed and model performance over SGD.
- The production federated model uses B = 1000, K = 1000, and momentum, while assigning zero weight to 99% of <UNK> examples to balance losses.
- The federated model achieves better Accuracy@1 but lower AUC than the server-trained model in federated evaluation.AUC is computed on server logs, whose short snippets from selected apps are considered less representative of users’ typed text than client caches.
8 Live experiment
Live traffic experiments evaluated emoji prediction models for English-US users in the USA and Canada, with both models improving engagement metrics and the federated model performing better across metrics.
- Average inference latency is around 1 ms using a quantized TensorFlow Lite model.
- The live-traffic experiment covered users in the USA and Canada typing in English (US).
- Both federated and server-trained models significantly increase prediction CTR, total emoji shares, and daily active emoji users over a no-prediction baseline.Table 3 reports relative metric changes with 95% confidence intervals derived using user-bucket jackknife estimates.
- The federated model performs better than the server-trained model on all reported live-traffic metrics.
- The increase in CTR is quite large for both models because emoji are triggered rarely.
9 Conclusions
The paper trains an emoji prediction model with a CIFG-LSTM using federated learning, achieving better performance than a server-trained model despite sparse data and poorly balanced classes.
- Federated learning trains a CIFG-LSTM emoji prediction model without centrally collecting users’ data.
- The federated model achieves better performance than a server-trained model.
- The approach works with sparse data and poorly balanced classes.