Source-linked AI summary
Federated Learning for Keyword Spotting
David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, Joseph Dureau
TL;DR
Embedded wake word detectors must learn from variable, privacy-sensitive user speech while operating continuously under resource constraints. The paper evaluates federated learning for the Hey Snips detector using crowdsourced speech data and compares standard averaging with Adam-inspired adaptive per-coordinate averaging. The adaptive approach reaches 95% recall at 5 FAH within 100 communication rounds, with 8 MB of upstream communication per client, and the dataset is open sourced.
Problem
Wake word detectors must remain accurate across varied recording conditions and speakers, while speech-data collection for training raises privacy concerns.
Method
The paper evaluates federated learning for a resource-constrained Hey Snips wake word detector using crowdsourced speech data and Adam-inspired adaptive per-coordinate averaging.
Results
95% recall at 5 FAH is reached within 100 communication rounds on the crowdsourced dataset, with 8 MB of upstream communication per client.
Takeaways & Limitations
The Hey Snips dataset is open sourced to support further transparent research on federated learning for speech data.
Takeaways & Limitations
The wake word task requires supervised local data collection and labeling, while the aligner used here cannot be easily embedded.
Abstract
from arXiv · showhide
We propose a practical approach based on federated learning to solve out-of-domain issues with continuously running embedded speech-based models such as wake word detectors. We conduct an extensive empirical study of the federated averaging algorithm for the "Hey Snips" wake word based on a crowdsourced dataset that mimics a federation of wake word users. We empirically demonstrate that using an adaptive averaging strategy inspired from Adam in place of standard weighted model averaging highly reduces the number of communication rounds required to reach our target performance. The associated upstream communication costs per user are estimated at 8 MB, which is a reasonable in the context of smart home voice assistants. Additionally, the dataset used for these experiments is being open sourced with the aim of fostering further transparent research in the application of federated learning to speech data.
1. INTRODUCTION
Wake word detectors continuously listen for predefined keywords to activate voice-assistant interaction, but must operate under tight resource constraints and remain accurate across varied recording conditions and speakers. Federated learning is investigated as a privacy-preserving way to train such models from users’ local speech data.
- Use and requirements: Wake word detection continuously listens for predefined keywords and activates voice input for a voice assistant.Examples include “Hey Siri” and “OK Google”.
- Use and requirements: The detector is designed to run always on device with a 200k-parameter and 20 MFLOPS resource budget.
- Use and requirements: Recording proximity, hardware, room configuration, and speaker variability challenge consistent behavior across usage settings.
- Use and requirements: High accuracy requires high recall while limiting unintended triggers through a low false alarm rate.
- Federated approach: Federated learning trains a central model from users’ local data without uploading the speech data to a central server.Devices perform local training steps, and a parameter server averages their updates into a global model.
2. RELATED WORK
Prior decentralized-learning research largely studied controlled, independently distributed data, while user-specific speech data remained underexplored. This work positions dynamic per-coordinate averaging as a communication-efficient federated optimization strategy for neural speech models.
- Research gap: Earlier decentralized-learning research primarily used controlled cluster or data-center settings with evenly partitioned i.i.d. datasets.
- Research gap: The authors describe this study as the first experiment of its kind on user-specific speech data.
- Research gap: Prior federated optimization work proposed globally and locally scaled SVRG for convex objectives, but its sparse linear-model assumption does not fit neural speech networks.
- Existing approaches: FedAvg improvements have addressed client selection, budget constraints, upload-cost reduction, and robustness to concept drift.
- This work: The paper introduces dynamic per-coordinate gradient updates instead of global averaging and evaluates them with crowdsourced data and models.
3. FEDERATED OPTIMIZATION
The federated procedure samples users, performs local mini-batch SGD, uploads model updates, and aggregates them into a global model. The paper motivates replacing weighted averaging with adaptive per-coordinate updates based on the gradient-like difference between successive models.
- FedAvg procedure: FedAvg begins with a central model and runs synchronous communication rounds over randomly selected users.
- FedAvg procedure: Each selected user performs local mini-batch SGD on a local objective, with work determined by local data size, epochs, and batch size.
- FedAvg procedure: Users upload their completed local model updates, which the server combines into an average weighted by each user’s data count.
- FedSGD and FedAvg: FedSGD is the special case with one full-batch local update, whereas FedAvg allows multiple local updates per user.
- Adaptive averaging: The global averaging step can be interpreted as a gradient update using the difference between the previous global model and users’ local models.
- Adaptive averaging: Adam-inspired first- and second-order moments adapt per-coordinate updates and smooth information from previous rounds involving different user subsets.
- Evaluation: Federated evaluation averages metrics on the server using a weighting scheme similar to parameter averaging.
4. EXPERIMENTS
Experiments evaluate federated training for Hey Snips using a crowdsourced, non-i.i.d. user distribution, comparing averaging strategies, participation rates, local-training configurations, and communication costs. Adam-inspired per-coordinate averaging reaches 95% recall at 5 FAH within 100 communication rounds, with an estimated 8 MB upstream cost per client.
- 4.1. Dataset: The Hey Snips dataset addresses the absence of a reference wake-word dataset and is publicly released for keyword-spotting research.The dataset was collected from 1.8k contributors recording positive wake-word utterances and negative short sentences.
- 4.1. Dataset: Distinct-user splits mimic non-i.i.d., unbalanced federated data and measure generalization to new users.Users contributed to separate training, tuning, and evaluation splits, with 77% used solely for training.
- 4.3. Results: The standard centralized Adam baseline reaches the early-stopping target in 400 steps, whereas standard SGD remains below 87% after 28 epochs.The centralized Adam result is treated as the upper bound of performance.
- 4.3. Results: 10% user participation is selected because using half the users offers limited later-stage gain over 10%, while 1% produces more sensitive gradients.The participation analysis assumes faster convergence with more users when local training does not diverge excessively.
- 4.3. Results: Adam-inspired adaptive per-parameter global averaging accelerates convergence, while standard global averaging performs poorly even after 400 communication rounds.The comparison is conducted in the FedSGD setting with optimized local learning rates.
- 4.3. Results: 63–112 communication rounds are required across tested local-training configurations, with E = 1 and B = 20 yielding an 80% speedup over FedSGD.This best configuration averages 2.4 local updates per participating worker and uses C = 10%, Adam global averaging, ηglobal = 0.001, and ηlocal = 0.01.
- 4.3. Results: At fixed 95% recall, false alarms measure 3.2 FAH on negative test data, 3.9 FAH on Librispeech, 0.2 FAH on news, and 0.6 FAH on TV datasets.False alarms are more common on close-field continuous datasets than on background negative audio sets.
- 4.4. Communication cost analysis: The estimated client upload cost is 8 MB within 100 communication rounds, while the server receives 110 GB across training.The server receives 137 updates per round at C = 10%, and 400-round experiments reach 98% recall / 0.5 FAH with a 32 MB per-user budget.
5. CONCLUSION AND FUTURE WORK
The work applies federated learning to crowdsourced speech data for a resource-constrained wake word detector, reaching the target criterion within 100 rounds at 8MB upstream cost per client. Real-world deployment remains limited by the need for supervised local data collection and labeling.
- 5. CONCLUSION AND FUTURE WORK: 95% recall per 5 FAH was reached within 100 communication rounds, with 8MB of upstream communication cost per client.This used per-coordinate Adam-based averaging instead of standard global averaging.
- 5. CONCLUSION AND FUTURE WORK: The Hey Snips wake word dataset was open sourced for further research on federated learning with speech data.
- 5. CONCLUSION AND FUTURE WORK: Real-life implementation requires local data collection and labeling because the wake word task requires supervision.The frame-labeling aligner used here cannot be easily embedded.
- 5. CONCLUSION AND FUTURE WORK: Memory-efficient end-to-end models could ease local data labeling compared with the presented class-based model.