Source-linked AI summary
Federated Learning Of Out-Of-Vocabulary Words
Mingqing Chen, Rajiv Mathews, Tom Ouyang, Françoise Beaufays
TL;DR
The paper asks whether useful OOV words can be learned for mobile keyboards without exporting sensitive user text. It trains a character-level recurrent neural network with federated learning and samples words from the learned model. Simulated FL achieves 90.56% precision and 81.22% recall for the top 105 unique words, while on-device experiments demonstrate meaningful OOV learning with 55.8% character-level top-3 accuracy and 2.35 cross entropy.
Problem
Mobile keyboards need to discover useful OOV words, but on-device vocabulary limits and sensitive user content make centralized collection undesirable.
Method
The paper trains a character-based recurrent neural network through federated learning, aggregating local model updates while leaving user text on devices, then samples OOV words from the model.
Results
90.56% precision and 81.22% recall were achieved for the top 105 unique words in simulated FL, while on-device FL reached 55.8% character-level top-3 accuracy and 2.35 cross entropy.
Takeaways & Limitations
The experiments demonstrate the feasibility of learning OOV words through federated learning in both simulated and real on-device settings.
Takeaways & Limitations
The approach does not address unintended memorization risks or the accuracy trade-offs of privacy techniques, and sampled models may require filtering for undesirable typos.
Abstract
from arXiv · showhide
We demonstrate that a character-level recurrent neural network is able to learn out-of-vocabulary (OOV) words under federated learning settings, for the purpose of expanding the vocabulary of a virtual keyboard for smartphones without exporting sensitive text to servers. High-frequency words can be sampled from the trained generative model by drawing from the joint posterior directly. We study the feasibility of the approach in two settings: (1) using simulated federated learning on a publicly available non-IID per-user dataset from a popular social networking website, (2) using federated learning on data hosted on user mobile devices. The model achieves good recall and precision compared to ground-truth OOV words in setting (1). With (2) we demonstrate the practicality of this approach by showing that we can learn meaningful OOV words with good character-level prediction accuracy and cross entropy loss.
1 Introduction
The paper targets privacy-preserving discovery of useful OOV words for mobile keyboards by training a character-level recurrent model through federated learning. It reports strong simulated-FL retrieval and practical on-device learning, while noting remaining privacy and generation-quality concerns.
- Learning frequently typed words helps mobile keyboards incorporate emerging terms and compensate for omissions, especially in low-resource languages.
- OOV words are absent from keyboard vocabularies, limiting prediction and gesture typing while potentially causing incorrect autocorrection.On-device latency and reliability constraints limit vocabulary size, making useful-word discovery important.
- Federated learning trains a character-based recurrent neural network on devices, aggregates model updates, and samples OOV words on servers without transmitting raw user content.The approach is designed to avoid transmitting and storing sensitive user text on centralized servers.
- The probabilistic model may generate unseen character sequences or fail to regenerate interesting training words, motivating evaluation of sampling frequency and distribution fidelity.
- 90.56% precision and 81.22% recall were achieved for the top 105 unique words in simulated federated learning.The result used a public Reddit comments dataset and 108 parallel independent samplings.
- 55.8% character-level top-3 prediction accuracy and 2.35 cross entropy were reached with real on-device Gboard data.The sampled words were reported to be meaningful and included words known to be trending during the experiments.
2 Method
The method combines a character-level LSTM generative model with federated learning to generate OOV words while keeping training data localized. Words are sampled iteratively from the model, and client updates are aggregated into a shared model.
- 2.1 LSTM Modeling: The LSTM uses CIFG gates, peephole connections, and a projection layer to model character sequences.CIFG reduces the number of parameters by 25%, while the projection layer reduces hidden-state dimensionality and speeds training.
- 2.1 LSTM Modeling: The model computes cross-entropy loss between predicted character distributions and one-hot encodings of the true characters.
- 2.1 LSTM Modeling: Sampling applies the chain rule, generating characters sequentially until an end-of-word token is reached.Multiple independent threads start from the start-of-word token and sample from the model’s conditional distribution.
- 2.2 Federated Learning: Federated learning keeps OOV-containing training data on user devices and aggregates locally computed client updates into a shared model.The approach is motivated by the sensitivity of OOV words and avoids transmitting and storing raw user content centrally.
- 2.2 Federated Learning: FederatedAveraging combines client model weights using each client’s data size, with adaptive L2-norm clipping applied to client gradients.The clipping is reported to improve the robustness of model convergence.
3 Experiments
The experiments evaluate OOV-word learning with simulated Reddit FL and on-device FL, using precision/recall for simulated data and character-level metrics when raw device data remain inaccessible.
- Evaluation metrics: Simulated FL evaluates OOV-word quality with precision and recall because ground-truth words and frequencies are available.On-device FL instead uses cross entropy and top-K character-level prediction accuracy because user input remains inaccessible.
- Model parameters: Table 1 compares three federated settings through RNN layers, server learning rate, momentum, and batch size.The settings differ in model architecture and optimization choices, including adaptive clipping and momentum.
- Model parameters: FLSGD_L diverges with a high client-side learning rate, unlike the convergent setting described for FLSGD_L.Federated learning uses both client-side and server-side learning rates, with local minibatch updates on devices.
- Experimental setup: Each training round requires 200 client updates and uses one local epoch in both simulated and on-device FL.These settings apply across both experimental configurations.
- Simulated FL on Reddit data: The Reddit corpus contains 492 million filtered comments from 763 thousand users and 19 million unique filtered OOV words.The corpus is organized by user IDs and spans diverse topics across hundreds of thousands of sub-forums.
- FL on client device data: On-device FL requires participating devices to have at least 2G memory, charging, an unmetered network, and an idle state.Experiments cover American English, Brazilian Portuguese, and Indonesian, with separate regional models.
4 Results
Federated models converge effectively and recover meaningful OOV words across simulated Reddit and on-device settings. The Reddit experiment matches sampled words to ground-truth frequencies, while on-device models achieve useful character-level prediction and cross-entropy performance.
- On-device FL: L converges faster and better than FLSGD in both cross-entropy loss and prediction accuracy.Momentum and adaptive clipping also lead to faster convergence and more stable performance.
- FL simulation on Reddit data: The generative model learns top Reddit OOV-word occurrence probabilities with absolute values and relative ranks close to ground truth.Table 2 compares occurrence probabilities from the Reddit dataset and the generative model.
- FL simulation on Reddit data: 90.56% precision and 81.22% recall are achieved for the top 105 unique words after 3000 rounds with 10^8 independent samplings.Both precision and recall increase with more independent samples and over training rounds.
- On-device FL: 55.8% top-3 character-level prediction accuracy and 2.35 cross entropy are achieved on users’ on-device evaluation data.Training values are 63.9% and 2.01, respectively; experiments in Brazilian Portuguese and Indonesian show a similar pattern across the three settings.
- On-device FL: Sampled OOV words span abbreviations, slang or typos, repetitive forms, foreign words, and names across English, Portuguese, and Indonesian models.The authors report that profanity was also learned and plan improved filtering of unintended typos.
5 Conclusion
The paper presents federated learning of OOV words through a character-based generative model whose learned knowledge is shared by aggregating local gradient updates. Simulated and live on-device experiments demonstrate effective OOV learning in public-corpus and real-world user settings.
- Conclusion: The method discovers OOV words by sampling from a character-based model trained through federated learning.Devices retain local data and transmit learned knowledge through aggregated local-SGD gradient updates.
- Conclusion: 90.56% precision and 81.22% recall are achieved for the top 10^5 unique words in simulated FL on a publicly available corpus.Live experiments on data from three Gboard-user populations also demonstrate effective OOV learning in a real-world setting.