A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization

Motivation and scope

• The work focuses on federated learning as an alternative to synthetic speech, which still mismatches personalized speech, acoustic environments, and device characteristics.Experiments use public datasets in simulation; no real user data or production telemetry informed the work.Source: §1 · p. 1

Scope

• Differential privacy is not added in this study because its clipping and noise may work against the low-variance server updates used for stabilization.The interaction of DP with pseudo-labeling is left for future study.Source: §1 · p. 2

Teacher choice

• The online teacher matches or beats the global teacher on most data pairs, while a transitioning teacher extends that advantage as the seed model grows stronger.These comparisons depend on stabilizing training; the online teacher had been expected to drift and destabilize training.Source: §1 · pp. 2–3

Server anchoring

• Frequent server updates on labeled seed data are a key stability condition for online and transitioning teachers.SpecAugment strength and batch size jointly govern server-update noise; strong SpecAugment causes divergence, while tuning both stabilizes training.Source: §1 · pp. 2–3

Teacher choice

• Online and transitioning teachers can break asymmetrically under domain shift: swapping seed and client corpora can flip training between divergence and convergence.The findings indicate that the domain gap does not act like a symmetric distance between corpora.Source: §1 · p. 3

Overall comparison

• The resulting recipe closes much of the gap to fully supervised FL and is compared with three existing SSFL-for-ASR methods across source-to-target pairs.The study also examines cohort size, local steps, and pseudo-label generator choice in ablations.Source: §1 · p. 3

2.1 Federated Learning

• In supervised FL, client drift from non-IID data is addressed through proximal regularization, control variates, dynamic regularization, adaptive server optimization, and personalization.This work instead targets semi-supervised FL, where it identifies pseudo-label drift as the dominant instability.Source: §2.1 · p. 3

2.2 Pseudo-labeling: from Centralized SSL to SSFL

• Centralized SSL uses fixed, EMA, and evolving-student pseudo-label sources, while federated SSL typically uses a global teacher; image-focused switching methods assume discrete labels or persistent clients.FedSwitch (Zhao et al., 2024) and BSemiFL do not address sequence-output ASR with large, non-persistent client populations.Source: §2.2 · pp. 3–4

2.3 Server Training and Anchoring

• Federated SSL methods differ in how labeled server data re-anchors training, but these strategies have not been systematically studied for ASR.Examples include FedNST’s frozen seed pseudo-labels (Mehmood et al., 2022), SemiFL’s server fine-tuning (Diao et al., 2022), and Rao et al.’s supervised gradients.Source: §2.3 · p. 4

2.4 Federated Learning in ASR

• Existing FL-ASR demonstrations assume fully labeled clients, whereas this work addresses the semi-supervised setting with labels only at the server.Guliani et al. (2021) demonstrate FedAvg-style end-to-end ASR, and Pelikan et al. (2025) benchmark differentially private FL-ASR.Source: §2.4 · p. 4

3 Semi-Supervised FL for ASR: Problem Setting and Practical Constraints

• Labels-at-server SSFL places a small labeled corpus on the server and unlabeled audio on clients, reflecting the expense of on-device transcription.The objective includes reducing target-domain WER without regressing on the server seed distribution.Source: §3 · pp. 4–5
• ASR pseudo-label errors compound across variable-length sequences, and prior SSFL-for-ASR methods use fixed or global teachers rather than locally updated teachers.FedNST and Rao et al. (2023) leave open whether adaptive local pseudo-labeling improves performance.Source: §3 · p. 5

3.1 Problem Setting

• Training starts from a server-trained seed model, then alternates client pseudo-labeling and local updates with aggregated server updates and a labeled-data pass.Methods differ along the pseudo-label source and server training axes; prior approaches vary from fixed seed labels to EMA teachers and supervised server updates.Source: §3.1 · pp. 5–6
• The paper modifies Diao et al. (2022), originally validated on image classification, as a baseline and reports that it outperforms Static PL, FedNST, and Rao et al. (2023) in the stated LibriSpeech setting.The comparison is for L: LS100 and U: LS860.Source: §3.1 · p. 6
• Global and online teachers differ when clients take multiple local steps: the online teacher adapts to each client, whereas a single step leaves it equivalent to the global teacher.More local steps reduce communication rounds but can let the local model drift, making pseudo-labels stale or inconsistent.Source: §3.1 · pp. 6–7

3.2 Practical Constraints and Scope

• The experimental design assumes no client labels, non-persistent clients, and only one model broadcast per round, excluding accumulated client state and separate teacher–student transmissions.The study also holds LM-guided decoding and confidence-based filtering fixed to isolate the teacher and server-strategy choices.Source: §3.2 · p. 7

4.1 Datasets and Domain-Shift Pairs

• Experiments span LibriSpeech, TED-LIUM, Common Voice, and Fisher, with in-domain and cross-domain pairs of labeled server seeds and larger unlabeled client pools.Server and client speakers are disjoint; only server data is labeled.Source: §4.1 · pp. 7–8
• WER is reported on both server and client dev sets, while the final comprehensive comparison uses test sets.The dev sets are used for analysis and hyperparameter tuning.Source: §4.1 · p. 8

4.2 Model Architecture and Training Configuration

• All experiments use a Transformer encoder with CTC loss, 80-dimensional log-mel features, character-level tokens, and greedy decoding.Language-model-guided decoding and confidence-based filtering are omitted as orthogonal to the two study axes.Source: §4.2 · p. 8
• Client inputs use SpecAugment for training but clean features for pseudo-label generation; clients use SGD and the server uses LAMB.The seed model initializes the student and EMA teacher; dropout rates are reduced during SSFL.Source: §4.2 · p. 8

4.3 Semi-Supervised Federated Learning Configuration

• FedAvg averages sampled clients’ pseudo-gradients equally, and the server applies supervised updates on labeled seed data each round with probability p.The setup caps FL steps at 6,000 and uses fixed local-step counts by dataset; Figure 1 also compares online-teacher baselines and global-teacher performance on LibriSpeech dev sets.Source: §4.3 · p. 8
• Figure 1 compares the online teacher with existing SSFL baselines and shows it consistently outperforming the global teacher on both LibriSpeech dev sets throughout training.The baselines include Static PL, FedNST, Rao et al., and Diao et al. for ASR.Source: Figure 1 · p. 9

5.1 Global vs. Online Teacher

• On LS100 →LS860, online-teacher target WER stays below global from the first few rounds, with final-round ΔWER = 2.88 and source dev-clean within ΔWER = 0.5.Local-model WER falls monotonically through local training; local EMA underperforms online across tested decay rates, favoring the latest adaptation step.Source: §5.1 · pp. 9–10, Figure 2 · p. 10

5.2 Online Teacher Requires Frequent Server Training

• On LS100 →LS860, server training probability p = 0 diverges, p = 0.1 remains unstable at WER = 12.8, and p ≥0.2 converges to WER 10–11.The best converged WER is 10.27 at p = 0.3; the authors report no significant additional benefit from raising p further.Source: §5.2 · p. 10, Figure 2 · p. 10

5.3 Cross-Domain Robustness

• For cross-domain LS100 →CV and LS100 →Ted, online beats global on target WER: CV-dev 39.9 vs. 37.7 and Ted-dev 12.6 vs. 11.6.The gains are not accompanied by regression on the source data.Source: §5.3 · pp. 10–11, Figure 3 · p. 11

5.4 Seed Strength and Transitioning Teacher

• As the seed strengthens from LS100 to LS960 →CV, the online advantage narrows and can reverse; transitioning improves target WER over global by ΔWER = 1.20 (20.9 vs. 19.7).The crossover round depends on seed strength and domain gap; transition is not beneficial for LS100 →CV or LS100 →LS860.Source: §5.4 · p. 11, Figure 4 · p. 12
• Online-teacher behavior is direction-asymmetric: LS →Ted converges with online outperforming global, whereas Ted →LS diverges rapidly.The same asymmetry holds for LS →CV versus CV →LS.Source: Figure 5 · p. 12

5 Pseudo-Label Generation

• The online teacher matches or beats the global teacher in-domain and in most cross-domain settings; a transitioning teacher matches or beats both pure strategies as seed strength grows.The online–global gap ranges from ΔWER = 2.52 at LS100 →CV to −0.61 at LS960 →CV.Source: §5.4 · p. 12

6.1 Online Teacher Breaks Asymmetrically

• Ted →LS diverges with the online teacher while LS →Ted converges, despite LS →CV converging across a larger domain gap; the breakage is directional, not determined by gap magnitude.Matching Ted’s audio-duration distribution to LS with Ted-Long does not prevent divergence.Source: §6.1 · pp. 12–13, Figure 6 · p. 13

6.2 Investigating Instability: Seed Training

• Audio duration, seed SpecAug strength, and seed-checkpoint overfitting do not explain the asymmetric breakage; changing the server-update corpus changes whether the online teacher diverges with the seed fixed.A shorter seed checkpoint delays but does not prevent divergence.Source: §6.2 · pp. 13–14, §6.3 · p. 14

6.3 Investigating Instability: Server Update

• With the seed fixed, the online teacher diverges when server updates use Ted but converges when they use LS, locating the instability in the FL-stage server update.The global teacher converges in both cases.Source: §6.3 · p. 14

6.3.1 SpecAug Strength

• Strong server-update SpecAug diverges with high deletion and near-zero insertion errors, while moderate SpecAug converges; strong augmentation also increases pseudo-label sensitivity and representation drift.The reported deletion-heavy error signature begins around 3.8k FL steps.Source: §6.3.1 · pp. 14–15

6.3.2 Batch Size

• Small batches converge noisily to higher WER, with greater pseudo-label sensitivity and representation drift; dispersion is the primary instability driver, overlap secondary, and SpecAug and batch size must both be stable.Increasing the batch reduces gradient-direction noise; overlap’s destabilizing effect at the smallest batch is recovered with a moderately larger batch.Source: §6.3.2 · pp. 16–17

6.4 Stabilizers Generalize Across Domain-Shift Pairs

• Across four cross-domain pairs, stabilizers restore convergence wherever the online teacher previously diverged and lower WER for global teachers; the stabilized transitioning teacher wins on every converging pair.The tested stabilizers combine a large batch with moderate SpecAug.Source: §6.4 · pp. 17–18, Figure 11 · p. 17

7.1 Comprehensive Comparison

• With the stabilizer, the global teacher improves over Rao et al. (2023) on 8 of 11 pairs and the online teacher on 9 of 11; the online teacher converges on all 11.Without the stabilizer, the online teacher diverges on 5 of 11 pairs; stabilized online training recovers on high-dispersion Ted sources, including Ted →LS (Div. vs. 11.34) and Ted →Fisher (Div. vs. 18.51).Source: §7.1 · p. 18, Table 3 · p. 19

7.2 Transitioning Teacher

• With stabilizers on LS960 →CV, the transitioning teacher at r = 2000 reaches 18.25 WER, versus 18.65 for global and 18.73 for online.The transition captures the global teacher’s early stability and the online teacher’s late adaptivity, but its improvement is modest once stabilizers are applied and is presented as analysis, not a general recommendation.Source: Figure 12 · p. 19, §7.2 · p. 19

7.3 Ablation Studies

• On LS →CV, increasing cohort size improves both stabilized teachers, with no significant difference in trend between them.The ablation tests cohort sizes 64, 256, and 1024 at 20 local steps.Source: Figure 13 · p. 19, §7.3 · p. 20
• On LS →CV, 40 local steps work best for both stabilized teachers; too many steps degrade performance through stale global pseudo-labels or online-teacher overfitting and drift.The ablation compares 20, 40, 80, and 160 steps at cohort size 64.Source: §7.3 · p. 20
• For the online teacher, p = 0.0 diverges in both settings, while p = 0.2 tends to converge and can achieve lower WER than p = 0.5 when it converges.Lowering p too far risks client drift; dropout can mitigate divergence at p = 0.2, and stronger SpecAugment can improve in-domain WER while increasing divergence risk.Source: §7.3 · p. 20

7.4 When to Use What: Practical Guidelines

• Use the online teacher in-domain and the more robust global teacher under domain shift; keep a plain supervised server update between rounds and scale its batch size with seed-data dispersion.Calibrate SpecAugment to source audio duration for the online teacher; the recipe is framed as requiring no per-pair tuning.Source: §7.4 · pp. 20–21

8.1 Limitations

• All four studied corpora are English, so the findings may not transfer to tonal, low-resource, or non-Latin-script languages.Such languages may have different SpecAugment-coverage thresholds and online-versus-global teacher trade-offs.Source: §8.1 · p. 21
• Experiments use one Transformer-CTC architecture, greedy decoding without a language model, and fixed client and server optimizers.Other architectures may express the server-training failure modes differently; effects under LM-fused decoding and Adam-class server optimizers remain untested.Source: §8.1 · p. 22
• The study uses public benchmarks and simulated clients, not real-user data or production deployments, and its large-batch stabilizer requires substantial server compute.Static-dataset cohorts omit deployment churn and failures; constrained server budgets may prevent direct use of the recipe.Source: §8.1 · p. 22
• The FedNST and Rao et al. comparisons are end-to-end, not controlled tests of server-update mechanisms, because their pseudo-label teachers also differ.The authors therefore recommend the simple supervised batch update primarily for simplicity and leave controlled comparison for future work.Source: §8.1 · p. 22

8.2 Future Work

• The recipe has not been tested under differential privacy; DP noise may counter the stabilizer’s low-variance updates, requiring batch-size and SpecAugment retuning.Quantifying this interaction and establishing guarantees under a specified threat model are future work.Source: §8.2 · p. 22
• Whether the online-teacher advantage combines with LM-fused decoding, and whether confidence-based filtering composes with transitioning teachers and stabilizers, remain open questions.The study uses greedy decoding and no pseudo-label filtering.Source: §8.2 · p. 22

9 Conclusion

• A per-client online teacher matched or beat the broadcast global teacher once stabilized, while a transitioning teacher matched or beat both as the seed model grew stronger.The transition was from global to online when the global teacher was better than the online teacher.Source: §9 · p. 22
• Interleaving server training on labeled data between FL rounds was a prerequisite for stability, and the anchor governed convergence more than the seed model itself.Aggressive teacher choices paid off only once the anchor stabilized training.Source: §9 · pp. 22–23
• Anchor stability was highly sensitive to SpecAugment strength and batch size, which control server-update gradient noise and pseudo-label consistency.The stabilization needed depended on seed-data dispersion and overlap with client data.Source: §9 · p. 23
• 20.8% in-domain and 10.0% cross-domain average WER improvements over the strongest prior SSFL method were achieved on 9 of 11 pairs.The recipe also improved over a naive static pseudo-labeling baseline in-domain by roughly 48% (17.7 → 9.3 WER).Source: §9 · p. 23

A Experimental Details

• The listed configurations are choices for public-dataset simulations, not descriptions of a production system or its planned configuration.Defaults apply throughout unless a section specifies a variation.Source: A Experimental Details · p. 27

A.1 Datasets and Splits

• Four English corpora span read audiobooks, prepared talks, crowd-sourced read speech, and conversational telephone speech; client data is partitioned by speaker.Server and client roles vary by pair, and utterances longer than 30 seconds are filtered; standard corpus splits are used, with held-out test sets reserved for the final comparison.Source: §A.1 · pp. 27–28, Table 5 · p. 27

Model Parameters

• The acoustic model is a roughly 255M-parameter, 36-block Transformer encoder with a convolutional front end, trained using CTC and greedily decoded without an external language model.It uses 80-dimensional log-mel features and a 28-character vocabulary plus the CTC blank symbol.Source: Model Parameters · p. 28

Training Parameters

• SpecAugment is enabled from the first training step, while dropout is 0.3 for central seed training and 0.1 during federated training.Client SGD learning rates vary by source corpus; the server uses LAMB with an exponential-decay schedule, and the seed model initializes both student and EMA teacher.Source: Training Parameters · p. 28

Federated Learning Parameters

• Client updates use equally weighted FedAvg; most pairs run 3,000 FL steps, with two in-domain pairs using 2,000.Cohort sizes vary by client corpus, local steps are selected from validation performance except for LS clients, and the server makes a supervised seed-data pass with probability 0.2.Source: Federated Learning Parameters · pp. 28–29

Pseudo-Labeling Parameters

• Pseudo-labels come from a global or online EMA teacher, and a transitioning teacher switches at approximately round 2,000.The EMA decay is 0.99; other experiments use the stated defaults unless specified otherwise.Source: Pseudo-Labeling Parameters · p. 29

B Baseline Method Details

• The appendix frames baseline methods around two choices: the source of pseudo-labels and the server training strategy.L denotes labeled server data, U_i client i's unlabeled data, and θ_0 the seed model trained on L.Source: B Baseline Method Details · p. 29

B.1 FedNST

• FedNST caches beam-search pseudo-labels from the frozen seed model before training, then combines the sample-count-weighted client gradient with a supervised server gradient.The two gradients are mixed with α = 0.5, so the labeled pass re-anchors training every round; regenerating labels each round raised runtime roughly 10× for negligible WER change.Source: §B.1 · pp. 29–30

B.2 Rao et al.

• Rao et al. refresh an EMA paired teacher every round and use rehearsal participants drawn from labeled server data in the same aggregation as sampled real clients.The rehearsal cohort is set to 10% of the FL cohort; this folds supervision into aggregation rather than performing a separate server pass.Source: §B.2 · p. 30

• Each GPU processes C K/G client batches per round plus its share of server work, and both client and FedNST-server work scale as 1/G.Server batches cost (# batches) × 3bS for a common batch size bS.Source: §B.3 · pp. 30–31
• FedNST sweeps the labeled source (nS/G batches per GPU), Rao et al. rehearse ρ clients (ρ nS/G), while our teachers use k data-parallel supervised steps per GPU.The methods differ only in their server term.Source: §B.3 · p. 31
• For a fixed pair, the relative slowdown is independent of GPU count.Source: §B.3 · p. 31
• FedNST costs ≈1.25× per step for LS100 → LS860 and ≈2.34× for LS → CV, and its runs are truncated under the shared 7-day limit.Other methods reach 2000 FL steps for LS100 → LS860 and 3000 for LS → CV within the limit.Source: §B.3 · p. 31

C Quantifying Domain Shift

• The measure quantifies domain-knowledge judgments using symmetrized nearest-neighbor cosine similarity between per-speaker embeddings; higher scores indicate more similar populations and smaller domain shift.Source: C Quantifying Domain Shift · p. 31
• The measure’s symmetry captures overall population distance, not transfer direction; direction-asymmetric instability is instead governed by server-data dispersion.Source: C Quantifying Domain Shift · p. 31
• In-domain pairs score 0.56 and 0.59, above cross-domain pairs at 0.38–0.44; the measure can guide teacher choice by overlap.A high score indicates an in-domain pair favoring the online teacher, while a low score indicates a cross-domain pair favoring the more robust global teacher.Source: C Quantifying Domain Shift · p. 31, Table 7 · p. 32