PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Robust and Multilingual Speech Representations

Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, Aaron van den Oord

arXiv:2001.11128v1cs.CLcs.LGeess.AS

TL;DR

Speech representations had been evaluated mainly on read English, leaving their robustness to domain shifts and transferability to other languages under-tested. The paper learns representations from diverse, noisy speech and evaluates them across domains and languages, finding improvements in both settings. The authors conclude that large-scale unsupervised representations can improve low-resource speech recognition.

  • Problem

    Prior work mainly evaluated speech representations on read English and similar domains, leaving robustness to domain shifts and transferability to other languages under-tested.

    Source: §1 · p. 1

  • Method

    The authors learn speech representations from up to 8,000 hours of diverse, noisy speech and evaluate them on domain-shift and multilingual recognition tasks.

    Source: §1 · p. 1

  • Results

    The representations improve out-of-domain robustness and recognition across 25 phonetically diverse languages, including tonal and low-resource languages.

    Source: §4.3 · p. 5, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that large-scale unsupervised representations can improve low-resource speech recognition.

    Source: §4.4 · p. 6

    The large-scale pretraining data may contain some examples from the target languages, although no extra data was added specifically for them.

    Source: §4.4 · p. 6

Abstract

from arXiv · show

Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. However, most research has been focused on evaluating the representations in terms of their ability to improve the performance of speech recognition systems on read English (e.g. Wall Street Journal and LibriSpeech). This evaluation methodology overlooks two important desiderata that speech representations should have: robustness to domain shifts and transferability to other languages. In this paper we learn representations from up to 8000 hours of diverse and noisy speech data and evaluate the representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. We find that our representations confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets and the features likewise provide improvements in 25 phonetically diverse languages including tonal languages and low-resource languages.

1 Introduction

Prior unsupervised speech representations were mainly evaluated on read English, leaving their robustness to domain shifts and transferability across languages under-tested. This paper evaluates representations learned from diverse, noisy speech and reports advantages on both dimensions.

1 Introduction

  • The evaluation focus on read English and similar domains leaves robustness to domain shifts and transferability to other languages untested.The paper identifies both as desired properties of speech representations.Source: §1 · p. 1
  • Representations are learned from 8,000 hours of diverse, noisy speech and evaluated for robustness and transfer across datasets and languages.The experiments compare domain transfer and recognition across 25 languages.Source: §1 · pp. 1–2
  • In language transfer, the representations significantly outperform standard features and representations pretrained only on clean English data.The language experiments include low-resource languages such as Amharic, Fongbe, Swahili, and Wolof.Source: §1 · pp. 1–2
  • The authors summarize three patterns: scale matters, unsupervised representations improve downstream robustness, and multilingual training supports transfer across languages.These are the paper’s stated conclusions about representation learning.Source: §1 · p. 2

2 Contrastive Predictive Coding: CPC

Contrastive predictive coding learns speech representations by maximizing mutual information between context and future latent representations. Its InfoNCE objective approximates this quantity using positive and noise samples.

2 Contrastive Predictive Coding: CPC

  • CPC learns features predictive of future observations by maximizing mutual information between context and future latent representations.The prediction horizon affects the structure captured, from local patterns to slower, more global structures such as phonemes, words, and utterances.Source: §2 · p. 2
  • Because mutual information is intractable for high-dimensional data, InfoNCE uses a noise-contrastive loss as a lower bound.The sample set contains one positive sample and N − 1 negatives drawn from a noise distribution.Source: §2 · p. 2
  • The model scores context–future representation pairs with a standard log-bilinear scoring function.The loss sums InfoNCE terms across prediction steps, with negatives uniformly sampled from representations in the same audio signal.Source: §2 · p. 2

3 Methods

The method trains an unsupervised acoustic feature extractor, freezes its parameters, and uses its representations to train supervised speech recognizers. The paper uses bidirectional CPC and evaluates the representations with character-level ASR models.

3 Methods

  • An unsupervised feature extractor is trained first; its frozen representations then serve as inputs to speech-recognition models.The labeled recognition data may differ from or be smaller than the unlabeled pretraining dataset.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Unsupervised learning with bi-directional CPC

  • Bidirectional CPC shares an encoder across directions, learns separate forward and backward context representations, and concatenates them for downstream tasks.The objective sums the forward and backward InfoNCE losses; the architecture also uses dense skip connections.Source: §3.1 · p. 3

3.2 Semi-supervised speech recognition

  • The downstream recognizer predicts characters with a CTC-based model, while pretrained representation parameters remain fixed.The paper uses a smaller DeepSpeech2 model and TDNNs; it reports that a linear layer followed by a CTC decoder lacks sufficient recognition capacity.Source: §3.2 · pp. 3–4

4 Experiments and Results

The experiments train CPC representations on diverse speech and evaluate their robustness to domain shifts and their transfer across languages. The results favor large, diverse pretraining across domain transfer, low-resource African languages, multilingual recognition, and English recognition in both low- and high-data settings.

4.1 Datasets

  • The pretraining corpus combines diverse speech datasets and uses audio without transcripts; evaluation uses transcribed datasets and standard train-test splits.The corpus spans recording conditions, noise levels, speaking styles, and languages, totaling about 8,000 hours; multilingual evaluation includes four African languages with 20 hours or less of transcribed speech each and 21 phonetically diverse languages.Source: §4.1 · pp. 4–5

4.2 Unsupervised Representation Learning

  • Unsupervised representation learning uses bidirectional CPC with 12 prediction steps and 10 negative samples, trained on randomly cropped 9.35-second audio windows.Training used Adam and stopped after 4.2 million steps based on LibriSpeech development-set recognition performance.Source: §4.2 · p. 5

4.3 Robustness

  • Large-scale CPC features consistently outperform other representations across evaluation sets, and are more robust than representations trained only on read English.The domain-transfer experiments train models on Wall Street Journal, LibriSpeech, or Tedlium and evaluate them on other datasets.Source: §4.3 · p. 5, Table 2 · p. 6

4.4 Low-resource Languages

  • CPC-8k outperforms other features on four low-resource African languages, while English-only LibriSpeech features underperform basic spectrogram features.The authors suggest that diverse linguistic inputs are crucial for a phonetic space that generalizes across languages; no extra data was added specifically for these languages.Source: §4.4 · p. 6, Figure 2 · p. 6

4.5 Multilingual Transfer

  • CPC-8k improves performance over spectrogram baselines in all 25 evaluated languages; Sundanese WER falls from 27.85 to 11.49.The multilingual evaluations use the same configuration across languages.Source: §4.5 · pp. 6–7

4.6 Control: English Speech Recognition

  • CPC-8k representations significantly improve TDNN recognition on LibriSpeech with both 10% and 100% of the labeled data, including with language-model beam search.The authors note that the results show improvements in both low- and high-data regimes.Source: §4.6 · p. 7

5 Related Work

Unsupervised learning has improved representations across modalities, while speech-representation work has often emphasized phonetic structure or in-domain performance. The paper situates its contribution against evaluations of whether learned speech representations transfer across domains and languages.

5 Related Work

  • Unsupervised learning has resurged across modalities, improving results especially in low-data regimes, while pretrained NLP representations can outperform state-of-the-art systems even with high data.The passage also notes unsupervised learning’s earlier role in deep-network resurgence across speech and other application areas.Source: §5 · pp. 7–8
  • Recent unsupervised speech representation work has examined phone recognition and related audio classification, while some studies evaluated in-domain speech-recognition WER.The authors argue that these evaluations do not assess robustness to domain shift or suitability across languages.Source: §5 · p. 8
  • ZeroSpeech challenges assess whether learned representations correlate with phonetic structure across languages and adapt to new speakers; Kahn et al. (2019b) reported English phonetic correlations without supervised-recognizer evaluation.Kahn et al. used contrastive predictive coding on 60,000 hours of English speech.Source: §5 · pp. 8–9
  • Data augmentation and self-training advanced supervised English speech recognition, and the authors suggest these methods may be orthogonal to universal representation learning and jointly improve results.The passage proposes assessing their robustness impact with the paper’s evaluation methods.Source: §5 · p. 9

6 Conclusion

The paper introduces unsupervised acoustic representations learned from up to 8000 hours of diverse, noisy speech. It reports robustness gains under domain shifts and consistent improvements across 25 phonetically diverse languages, including tonal and low-resource languages.

6 Conclusion

  • The method discovers acoustic representations from up to 8000 hours of diverse and noisy speech data.The paper describes the method as unsupervised representation learning.Source: §6 · p. 9
  • The learned representations improve robustness to domain shifts versus standard acoustic representations and representations pretrained on smaller, narrower datasets.The evaluation includes both in-domain speech recognition and transfer tasks where models are evaluated out of domain.Source: §6 · p. 9
  • The representations yield consistent improvements on 25 phonetically diverse languages, including tonal and low-resource languages.The authors say this suggests progress toward models that implicitly discover phonetic structure from large-scale unlabeled audio.Source: §6 · p. 9

Related papers

Loading 2001.11128v1…