Source-linked AI summary
Federated Self-Learning with Weak Supervision for Speech Recognition
Milind Rao, Gopinath Chennupati, Gautam Tiwari, Anit Kumar Sahu, Anirudh Raju, Ariya Rastrow, Jasha Droppo
TL;DR
On-device federated continual ASR adaptation must work without human transcripts or a stronger machine transcriber, while respecting privacy and device constraints. The paper combines EMA-teacher self-learning with weak user-derived supervision and historical-data rehearsal. It reports 10% improvement on new use cases and describes rehearsal as mitigating catastrophic forgetting.
Problem
The paper studies continual federated adaptation of on-device RNN-T ASR without human ground-truth transcriptions or a resource-intensive stronger teacher, in a setting where audio is not persisted or sent to the cloud.
Source: §1 · p. 1Method
The approach uses teacher-labeled self-learning with an EMA-updated paired teacher, weak NLU or feedback signals, and cloud rehearsal with historical labeled data.
Source: §2.2 · p. 2, §2.3.2 · p. 3, §5 · p. 4Results
Takeaways & Limitations
The reported approach improves new-use-case performance without strong transcription supervision and uses rehearsal to reduce forgetting on other test sets.
Source: §1 · p. 1, §5 · p. 4The on-device setup assumes the teacher has the same configuration as the student and can be stored and run on-device.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
Automatic speech recognition (ASR) models with low-footprint are increasingly being deployed on edge devices for conversational agents, which enhances privacy. We study the problem of federated continual incremental learning for recurrent neural network-transducer (RNN-T) ASR models in the privacy-enhancing scheme of learning on-device, without access to ground truth human transcripts or machine transcriptions from a stronger ASR model. In particular, we study the performance of a self-learning based scheme, with a paired teacher model updated through an exponential moving average of ASR models. Further, we propose using possibly noisy weak-supervision signals such as feedback scores and natural language understanding semantics determined from user behavior across multiple turns in a session of interactions with the conversational agent. These signals are leveraged in a multi-task policy-gradient training approach to improve the performance of self-learning for ASR. Finally, we show how catastrophic forgetting can be mitigated by combining on-device learning with a memory-replay approach using selected historical datasets. These innovations allow for 10% relative improvement in WER on new use cases with minimal degradation on other test sets in the absence of strong-supervision signals such as ground-truth transcriptions.
1. INTRODUCTION
The paper studies privacy-preserving federated continual adaptation of on-device ASR, where streaming audio cannot be retained and strong transcription supervision is unavailable. It combines self-learning, weak user-derived supervision, and rehearsal to improve new-use-case performance while mitigating forgetting.
1. INTRODUCTION
- On-device ASR adaptation targets deployment-specific acoustic or linguistic content and temporal distribution shifts under memory and compute constraints.The described federated setting avoids sending utterances to the cloud, persistent audio storage, and the need for human ground-truth annotations.Source: §1 · p. 1
- The setting differs from prior work by fine-tuning a well-trained model on streaming device data that is not persisted across rounds.The study also incorporates weak supervision from alternate feedback sources.Source: §1 · p. 1
Contributions
- 10% improvement on new use cases is reported for RNN-T self-learning with a paired teacher updated by exponential moving average.The authors also report that rehearsal training with historical datasets mitigates catastrophic forgetting, while weak NLU or feedback signals improve self-learning.Source: §1 · p. 1
- Rehearsal training with historical datasets mitigates catastrophic forgetting on other test sets during self-training.The authors report results on public and internal datasets.Source: §1 · p. 1
- Weak supervision from NLU semantics or noisy feedback scores improves self-learning through a policy-gradient approach.The signals can derive from user behavior across multiple turns or explicit feedback.Source: §1 · p. 1
2. METHODS
The methods combine federated RNN-T self-learning with an EMA-updated paired teacher, optional cloud rehearsal, and weak supervision from NLU semantics or user feedback. The weak signals are used to update ASR through semantic-cost or policy-gradient objectives.
2.1. RNN-T ASR model architecture
- RNN-T predicts label sequences from acoustic features using an encoder, a prediction network conditioned on prior labels, and a joint network that outputs label probabilities.The architecture is described as used for real-time speech recognition.Source: §2.1 · p. 2
2.2. Federated Self-Learning for ASR
- Each round uses teacher-labeled, confidence-filtered device audio for student updates, aggregates local model deltas, and periodically updates the teacher by EMA of itself and the global student.The assumed teacher has the same configuration as the student so it can run on-device; historical ground-truth batches can also be used for cloud rehearsal updates.Source: §2.2 · p. 2
2.3. Weak supervision
- Weak supervision supplements unlabeled-audio self-learning with user-derived signals, including request cancellation or repetition and recovered NLU slot labels.The experiments use machine-generated NLU semantics and synthetic feedback scores as proxies for user-session information.Source: §2.3 · p. 2
2.3.1. Weak Supervision: NLU semantics
- For weak NLU supervision, semantic cost is the fraction of slots whose tokens are not all present in an ASR hypothesis, and may also include WER against an alternate transcript.The method augments self-learning loss with the weak-NLU loss; its expected hypothesis objective is approximated using the n-best hypotheses.Source: §2.3.1 · p. 3
2.3.2. Weak Supervision: Feedback Scores
- Feedback-score supervision restricts feedback to the hypothesis served to the user and uses policy gradients with potentially noisy semantic or binary sentence-error costs.The output space is approximated by n-best hypotheses, with a size-one sampling approximation for the expectation.Source: §2.3.2 · p. 3
Data
The experiments combine single-pass federated training on internal voice-assistant data with evaluation on internal test sets and the public SLURP dataset.
Data
- The internal training stream contains 4,500 hours from 800K devices, and the experiments make one pass through the data.Persistent audio storage is treated as infeasible.Source: Data · p. 3
- Internal evaluation uses human-transcribed General and Delta test sets, with Delta targeting words more frequent in 2021 than 2020.General includes a 37-hour 2021 test set and older 2020 test sets; Delta is a 22-hour test set.Source: Data · p. 3
- The public evaluation uses a pretrained RNN-T fine-tuned with self-learning and weak supervision on SLURP, tested on its 13K-utterance partition.The model was pretrained on 960 hours of Librispeech audio.Source: Data · p. 3
- Table 2 covers federated self-learning with weak supervision on SLURP, while Table 3 covers voice-assistant data and contrasts several weak-supervision forms.Table 2 also includes corrected-utterance examples; Table 3 reports WERR relative to the initial model's WER.Source: Table 3 · p. 4, Table 2 · p. 4
4. RESULTS
Self-learning improves performance on SLURP and the internal new-use-case test set, with further gains reported from weak supervision. Noisy feedback and rehearsal also show benefits, alongside trade-offs in noise, retention, and teacher-update choices.
Federated self-learning with weak supervision
- 19% improvement is reported for self-learning on public SLURP, with additional gains from NLU feedback scores.The authors note that sparse transcript-token annotations or few slots per utterance limit the gains from weak supervision on SLURP.Source: §4 · p. 4
- 8.3% improvement on the new-use-case test set is reported for ASR self-learning, with further improvement from ASR-based feedback and gains that increase when NLU is included.Similar improvement is reported when NLU feedback scores only the served hypothesis rather than all possible hypotheses.Source: §4 · p. 4
Noisy feedback
- At σ = 0.4, noisy feedback still significantly improves the model on the Delta dataset.The noisy-feedback setup truncates sampled noise to [0, 1] before adding it to the feedback score.Source: §4 · p. 4
EMA hyperparameters and rehearsal training
- Rehearsal reduces regression on the older 2020 test set at the expense of performance on new Delta test sets.Delta results are not comparable across prior tables because computation and catastrophic forgetting differ.Source: §4 · p. 4
- Frequent teacher updates with a lower EMA value improve performance, while updating the teacher after every step led to training divergence.The authors suggest an error feedback loop as the explanation for divergence.Source: §4 · p. 4
5. CONCLUSION
The conclusion reports that paired-teacher self-learning, weak supervision, and rehearsal support federated continual ASR updates without ground-truth transcripts or persistent audio storage.
5. CONCLUSION
- A paired teacher updated by EMA improves RNN-T performance by 10% on new use cases, with larger improvement on public SLURP and performance only 10% away from the fully supervised setting.The conclusion states that these results are obtained without ground-truth human transcriptions, large on-device compute for strong teachers, or sending or persisting audio in the cloud.Source: §5 · p. 4
- Rehearsal with historical transcribed datasets mitigates catastrophic forgetting and error feedback loops.Source: §5 · p. 4
- Machine-generated NLU semantics and simulated noisy interaction feedback further improve self-learning through a policy-gradient approach.Source: §5 · p. 4