PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework

Dhruv Guliani, Francoise Beaufays, Giovanni Motta

arXiv:2010.15965v2cs.LGcs.DC

TL;DR

Per-user federated speech recognition trains on non-IID client data, a setting associated with degraded model quality. The paper varies the degree of non-IID-ness and evaluates cost with CFMQ; its experiments report that tuning and variational noise can recover quality at lower cost than the IID baseline.

  • Problem

    Per-user federated training uses non-IID client data rather than the IID sampling typical of centralized training, and non-IID training is an open problem associated with sub-optimal results.

    Source: §1 · p. 1

  • Method

    The paper varies non-IID-ness through client data sampling, measures federated cost with CFMQ, and tests hyperparameter changes and federated variational noise.

    Source: §1 · p. 1, §2.2 · p. 2, §4.2.1 · p. 3

  • Results

    Federated ASR can match server-trained quality, and the tuned non-IID experiments report better quality at lower CFMQ than the IID Baseline.

    Source: §1 · p. 1, §4.3.2 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    The degree of non-IID-ness can be adjusted at computational cost to make experimental setups fit a desired quality-performance trade-off.

    Source: §5 · p. 4

    The study assumes abundant server resources and memory, limiting its cost analysis to settings where those resources are not constrained.

    Source: §2.3 · p. 2

Abstract

from arXiv · show

We propose using federated learning, a decentralized on-device learning paradigm, to train speech recognition models. By performing epochs of training on a per-user basis, federated learning must incur the cost of dealing with non-IID data distributions, which are expected to negatively affect the quality of the trained model. We propose a framework by which the degree of non-IID-ness can be varied, consequently illustrating a trade-off between model quality and the computational cost of federated training, which we capture through a novel metric. Finally, we demonstrate that hyper-parameter optimization and appropriate use of variational noise are sufficient to compensate for the quality impact of non-IID distributions, while decreasing the cost.

1. INTRODUCTION

The paper investigates on-device federated ASR without sending raw user data to central servers, where per-user training encounters non-IID client data. It introduces ways to reason about non-IID training and its computational cost, and reports that federated ASR can match server-trained model quality.

  • The study examines speaker-split ASR training, where per-user client distributions differ from the IID sampling typical of centralized training.Non-IID training has been reported as sub-optimal across domains and remains an open federated-learning problem.Source: §1 · p. 1
  • The paper provides a mental model for reasoning about differences between IID and non-IID training.Source: §1 · p. 1
  • It introduces a general federated-learning cost function for measuring the computational cost of model quality.Source: §1 · p. 1
  • Federated ASR models can achieve the same quality as server-trained models.Source: §1 · p. 1

2. METHODOLOGY

The methodology uses FedAvg for federated optimization and treats the degree of non-IID-ness as adjustable at a computational cost. It introduces CFMQ to compare federated experiments by cost alongside model quality.

2.1. Federated Averaging Algorithm

  • FedAvg performs local optimization on participating clients and then updates the global model on the server; the paper uses it in all experiments.Clients are sampled for each round, and their updates are weighted-averaged before the server update.Source: §2.1 · p. 1

2.2. Understanding non-IID Data

  • Random client selection with one local example can yield effectively IID aggregated contributions, but is expected to sharply increase rounds, training time, and communication.The paper frames non-IID-ness as adjustable at the expense of computational cost.Source: §2.2 · p. 2

2.3. The Cost of Federated Model Quality

  • CFMQ combines communication and local-computation costs to compare experiments when used with a quality metric.The cost function considers convergence time, local optimization, client participation, and communication payload; α adjusts the relative importance of its cost components. The study assumes abundant server resources and memory.Source: §2.3 · p. 2

3. MODEL AND DATA

The paper uses an RNN-T speech recognizer and evaluates it on Librispeech, whose speaker-split data exhibit several sources of non-IID variation.

3.1. Model Architecture

  • The RNN-T has 122M trainable parameters and predicts label probabilities from acoustic data using LSTM audio and label encoders, a concatenating fully connected layer, and an output softmax.Inputs are 128-dimensional log-mel filterbank energies, and outputs are drawn from 4096 word-pieces.Source: §3.1 · p. 2

3.2. Librispeech Corpus

  • Librispeech contains 960 hours of training utterances from 2,338 speakers and 21 hours of evaluation audio across Dev, DevOther, Test, and TestOther, with WER as the metric.The “Other” sets are intended to be more difficult to recognize.Source: §3.2 · p. 2
  • The corpus is split by speaker to represent devices, with non-IID variation from voice, vocabulary, recording quality, and utterance counts.Figure 2 shows the utterance distribution across speakers.Source: §3.2 · p. 2, Figure 2 · p. 2

4. EXPERIMENTS

The experiments show that non-IID federated training initially degrades WER, while data-limiting and Federated Variational Noise recover quality. Hyper-parameter changes then yield better quality at lower CFMQ than the IID Baseline.

4.1. Baseline and Federated Training

  • The initial non-IID configuration showed substantial WER degradation across all evaluation sets relative to the Baseline.The Baseline used a linear ramp-up learning-rate schedule, SpecAugment, and Variational Noise; federated training used FedAvg with speaker-split data.Source: §4.2 · p. 3, §4.1 · p. 3, Table 1 · p. 3

4.2.1. Limiting Per-speaker Data

  • Data-limiting reduced relative WER degradation from over 40% to less than 30% across all evaluation sets.The method randomly sampled a subset of examples per participating speaker per round, while the full per-speaker dataset was seen over multiple rounds.Source: §4.2.1 · p. 3, Table 2 · p. 3

4.2.2. Federated Variational Noise

  • Federated Variational Noise (FVN) adds client-specific Gaussian noise during local optimization; increasing its standard deviation linearly during training let E7 exceed Baseline quality.FVN adapts Variational Noise to FL’s two-step optimization and was reported as critical to recovering non-IID quality degradation.Source: §4.2.2 · p. 3, Table 3 · p. 3
  • With FVN, model quality changed minimally without per-client data limits, despite quality degrading as per-client data volume increased without that noise.The authors interpret these results as evidence that FVN prevents client drift.Source: §4.2.2 · p. 3, Table 4 · p. 4

4.3.1. Quality-Cost Analysis

  • Figure 3 contrasts number of rounds with CFMQ as cost measures and mean WER on the more challenging Other evaluation sets as quality; E7 matched E8 quality at lower CFMQ.E8 required more local optimization steps because it had no per-client data limits.Source: §4.3.1 · p. 4, Figure 3 · p. 4

4.3.2. Reducing the Cost of non-IID Model Quality

  • E9 and E10 achieved better quality and lower CFMQ than the IID Baseline; both shortened learning-rate ramp-up and added exponential decay, while E10 also increased SpecAugment.The study did not revisit or refine the Baseline.Source: §4.3.2 · p. 4, Table 5 · p. 4

5. CONCLUSION

The conclusion frames non-IID-ness as adjustable through random client sampling, creating a flexible cost-quality trade-off. It reports that double optimization produced a better ASR model at lower cost than baseline Adam-SGD.

5. CONCLUSION

  • Double optimization produced a better state-of-the-art ASR model at lower cost than the baseline Adam-SGD model.Source: §5 · p. 4
  • Random client data sampling can adjust the degree of non-IID-ness, creating a flexible trade-off between model quality and cost.Source: §5 · p. 4
  • Recovering quality in federated training initially likely raises cost, but optimizer configuration, hyperparameter tuning, and regularizers can achieve IID-level quality at relatively low cost.Source: §5 · p. 4
  • Federated learning's non-IID data have been considered a potential drawback, but the conclusion presents them as a tunable aspect of training.Source: §5 · p. 4

Related papers

Loading 2010.15965v2…