FedNST: Federated Noisy Student Training for Automatic Speech Recognition

1. Introduction

• Prior federated ASR methods assume labelled client data, although manually annotating user speech is impractical for real-world applications.Source: §1 · p. 1
• Semi- and self-supervised central training requires storing data centrally, motivating the question of using unlabelled user data for federated ASR.Source: §1 · p. 1
• Existing federated semi-supervised methods for image and audio classification do not directly apply to sequence-to-sequence learning.Source: §1 · p. 1
• A prior semi-supervised ASR approach uploads client speech and relies on high-compute trainers, making it unsuitable for cross-device systems.Source: §1 · p. 1
• FEDNST adapts Noisy Student Training to leverage private, unlabelled speech across more than a thousand clients in a cross-device setting.Source: §1 · p. 1
• 22.5% WERR over training with only labelled data is reported for FEDNST.The contribution passage describes this as a method for federated ASR using private, unlabelled speech.Source: §1 · p. 1
• 2.2% relative difference from fully centralized NST is reported in a comparable setup.Source: §1 · p. 1

2. Background

• End-to-end ASR maps a sequence of input audio frames to a sequence of corresponding labels.Source: §2 · p. 1
• Modern encoder-decoder ASR models use a weighted sum of Seq2Seq and CTC objectives, with CTC aiding early convergence and robustness to noisy conditions.Source: §2 · p. 1
• Noisy Student Training iteratively generates pseudo-labels for unlabelled data with a model trained on labelled data, then trains on both datasets.The described procedure may integrate a language model and apply data filtering or balancing when generating pseudo-labels.Source: §2 · p. 2

3. Proposed Approach

• FEDNST uses labelled data on a central server and unlabelled speech held by clients to improve ASR accuracy while preserving user privacy.Source: §3 · p. 2

3.1. Federated Noisy Student Training

• A server-trained initial model is required; it is sent to clients to pseudo-label their local speech, with a language model and beam search used to improve transcripts.The paper notes that training modern state-of-the-art architectures from scratch with FL is complex.Source: §3.1 · p. 2
• An alternative pseudo-labelling strategy has participating clients relabel their data each round using the latest global model, rather than pseudo-labelling all clients with the initial model.The paper frames pseudo-labelling frequency and reuse of labelled server data as design choices balancing performance and training cost.Source: §3.1 · p. 2
• FEDNST studies whether to mix server-labelled data into FL training, since standard Noisy Student Training mixes labelled and pseudo-labelled samples but supervised FL commonly updates from client data only.Source: §3.1 · p. 2

3.2. Federated Optimization for FEDNST

• Each round updates the same model on the server and sampled clients, aggregates client weight differences with the server update, and passes the resulting pseudo-gradients to the server optimizer.Client updates are computed locally; the process continues for a fixed number of rounds or until validation WER convergence.Source: §3.2 · p. 2

• LibriSpeech was split by speaker into disjoint 480-hour sets A and B; experiments either pretrained on A then federated on B or mixed A and B during federation.Source: §4 · p. 3
• The end-to-end ASR model used a Conformer (S) encoder, transformer decoder, and joint CTC+Seq2Seq objective, with 1,000 federated rounds and 7% client participation.Source: §4 · p. 3
• Models were selected by dev-clean WER, and dev-clean, test-clean, and test-other results used an off-the-shelf language model and beam search of size 10.Source: §4 · p. 3
• To address FL convergence issues with standard Batch Normalization, FedNST uses modified static Batch Normalization that reuses server model θ0’s running statistics, assuming client and server data distributions match.Source: §4 · p. 3
• FedNST uses FEDAVG because adaptive server optimizers showed limited accuracy differences; it uses client-side SGD because Adam performed worse in the stateless-client setting, with α set to 0.5.Source: §4 · p. 3
• Client learning rates decay exponentially between rounds; λ=1000 performed best in the tested setup, where weak decay outperformed strong decay.Source: Figure 1 · p. 3, §4 · p. 3

5.1. Comparison with SotA Methods

• FEDNST’s test-clean WER was within a 2.2% relative difference of NST, and FEDNST achieved a 22.5% relative WER reduction over SLseed.SLseed is trained on labelled data; Table 2 compares WER across supervised, federated, and noisy-student methods.Source: §5.1 · p. 3, Table 2 · p. 4

5.2. Analyses with Different Pseudo-Labelled Data Regimes

• At 0%-labelled, A ∪B had an 8.5% relative WER increase versus 100%-labelled, while reusing A with B improved WER by 0.5; the mixing gain was nearly zero at 25% or more labelled.The authors hypothesize that A regularizes training when most client data is pseudo-labelled.Source: §5.2 · p. 4, Figure 2 · p. 4

5.3. Analyses with Different Labelled Data Regimes

• Reducing labelled data A from 480 toward 120 hours increased WER, and the initial model’s WER correlated with FEDNST’s WER.The analysis used |A| of 120, 240, 360, or 480 hours and fixed |B| at 480 hours.Source: §5.3 · p. 4, Table 3 · p. 4

5.4. Analyses with Varying Pseudo-Labelling Frequency

• Per-round pseudo-labelling achieved WER very similar to the original strategy but increased total federated-experiment time by 10x.The comparison used labelled A and pseudo-labelled B for 50 rounds.Source: Table 4 · p. 4, §5.4 · p. 4

6. Conclusions

• FEDNST applies noisy student training to private, unlabelled user data in federated ASR and improves models in low-labelled-data regimes.The evaluation used LibriSpeech and over 1,000 simulated clients.Source: §6 · p. 4
• 22.5% relative WER reduction over a supervised baseline trained only on labelled server data was achieved on LibriSpeech.The authors report WER comparable to fully centralized NST and supervised training, with no extra communication overhead versus FedAvg.Source: §6 · p. 4
• Future work will test FEDNST on more challenging datasets such as CommonVoice and incorporate other unlabelled-data methods such as Wav2Vec2.0.Source: §6 · p. 4