PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Speech Recognition

Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, Michael Auli

arXiv:2105.11084v3cs.CLcs.SDeess.AS

TL;DR

Speech recognition usually requires labeled speech, limiting its availability across the world’s languages. wav2vec-U trains without labeled data by segmenting unlabeled audio with self-supervised representations and learning phoneme mappings adversarially, achieving strong benchmark results and multilingual viability.

  • Problem

    Speech recognition systems require transcribed speech, which is unavailable for most of the world’s nearly 7,000 languages.

  • Method

    wav2vec-U uses wav2vec 2.0 representations to embed and segment unlabeled audio, then learns mappings to phonemes through adversarial training and unsupervised model selection.

  • Results

    wav2vec-U reduces TIMIT PER from 26.1 to 11.3, achieves 5.9 WER on Librispeech test-other, and performs across multiple additional languages.

  • Takeaways & Limitations

    Using only unlabeled speech audio and unlabeled text can support speech recognition for more languages, including low-resource languages.

  • Takeaways & Limitations

    The approach requires language-specific text phonemization tools, whose quality varies and which are unavailable for some languages; its segmentation also uses fixed-size units despite variable phoneme durations.

Abstract

from arXiv · show

Despite rapid progress in the recent past, current speech recognition systems still require labeled training data which limits this technology to a small fraction of the languages spoken around the globe. This paper describes wav2vec-U, short for wav2vec Unsupervised, a method to train speech recognition models without any labeled data. We leverage self-supervised speech representations to segment unlabeled audio and learn a mapping from these representations to phonemes via adversarial training. The right representations are key to the success of our method. Compared to the best previous unsupervised work, wav2vec-U reduces the phoneme error rate on the TIMIT benchmark from 26.1 to 11.3. On the larger English Librispeech benchmark, wav2vec-U achieves a word error rate of 5.9 on test-other, rivaling some of the best published systems trained on 960 hours of labeled data from only two years ago. We also experiment on nine other languages, including low-resource languages such as Kyrgyz, Swahili and Tatar.

1. Introduction

Speech recognition typically depends on transcribed speech, limiting coverage to roughly 125 of nearly 7,000 languages. wav2vec-U addresses this gap by using unlabeled audio and adversarial learning, achieving strong results across benchmarks and languages.

  • Nearly 7,000 languages exist, but speech recognition technology is available for only about 125 because transcribed speech is generally required.
  • wav2vec-U introduces unsupervised speech recognition using self-supervised wav2vec 2.0 representations and adversarially learned mappings to phonemes.The method segments audio with k-means and trains without labeled speech data.
  • 26.1 to 11.3 PER: wav2vec-U improves the TIMIT phoneme error rate relative to the next best known unsupervised approach.
  • 5.9 WER: wav2vec-U achieves this score on Librispeech test-other, alongside evaluations on six additional European and three non-European low-resource languages.

2. Background

Supervised ASR models transcribe speech using paired speech-text data, often decomposing the task into acoustic, pronunciation, and language models. Hybrid systems align speech frames to phoneme sequences and decode the most probable text sequence.

  • ASR maps a speech waveform to a text transcript, typically learning from paired speech and text examples.
  • Hybrid systems model speech and text jointly with acoustic, pronunciation, and language components.The pronunciation model maps words to phoneme sequences, while the language model scores text sequences.
  • Acoustic-model probabilities marginalize over possible HMM alignments, or approximate them with the highest-probability alignment using forward-backward or Viterbi computation.
  • Hybrid decoding searches for the most probable text sequence by combining acoustic, pronunciation, and language-model probabilities, often using WFSTs.A weighting factor can balance acoustic and language-model contributions, with its value selected on development data.
  • Acoustic and language models are often trained separately with maximum-likelihood objectives, while additional unpaired text can improve language-model training.

2.3 End-to-End Systems for Supervised ASR

End-to-end ASR models directly parameterize the conditional distribution of text given speech. CTC trains by marginalizing over frame-level alignments and can also integrate with pronunciation and language models for decoding.

  • End-to-end ASR directly models pθ(Y | X), including CTC, RNN-T, and sequence-to-sequence architectures.
  • CTC predicts a distribution over text symbols and a blank symbol at each input frame, defining probabilities over possible alignments.
  • CTC computes text-sequence probabilities by marginalizing over alignments with dynamic programming and optimizes the resulting posterior likelihood.
  • During decoding, CTC selects a probable alignment and maps it to text by removing repeated consecutive units and blanks.
  • CTC can compose with pronunciation and language models in a WFST or predict phonemes for decoding with standard pronunciation and language models.

2.4 Self-Training for Semi-Supervised ASR

Semi-supervised ASR combines labeled speech with unpaired speech by generating pseudo-transcriptions. These pseudo-labels are then used with real paired data to retrain the model, often iteratively.

  • Semi-supervised ASR provides unpaired speech alongside a labeled dataset.
  • A seed ASR model trained on labeled data transcribes unpaired speech to create a pseudo-labeled dataset.
  • The generated pseudo-labels and real paired data can train an ASR model with supervised or separately weighted objectives.
  • Self-training may repeat pseudo-label generation and retraining across multiple iterations to improve performance.

3. Speech and Text Representations

The framework builds unsupervised speech representations, segments audio, and maps segments to phonemes, with representation choice strongly affecting performance. It uses wav2vec 2.0 context features, silence preprocessing, clustering-based segmentation, and phonemized text with inserted silence markers.

  • Speech audio representations: Self-supervised wav2vec 2.0 representations embed unlabeled speech, while audio silences are removed before downstream processing except for TIMIT.Context representations are taken from the Transformer network; silence removal improves the mapping between audio representations and transcriptions.
  • Speech audio representations: Block 15 provides the best supervised phoneme classification, achieving 7.5 PER on Librispeech dev-other.Blocks 15–19 achieve below 9 PER, whereas most early and final blocks perform poorly; block 15 also generalizes well across eight MLS languages.
  • Audio segmentation: The unsupervised model’s Viterbi outputs produce more balanced and accurate phoneme-boundary recovery than raw k-means segmentation.K-means has high precision but low recall, while combining neighboring segments with the same predicted label creates larger segments closer to human boundaries.
  • Text preprocessing: Phonemized text is augmented with silence markers because unlabeled phoneme sequences otherwise lack silence tokens present in speech.Adding SIL at sentence boundaries improves accuracy, and inserting SIL between words at rate 0.25 yields the best final accuracy.

4. Unsupervised Learning

The model learns to map unlabeled speech segments to phonemes by adversarially matching generator outputs with phonemized text, while unsupervised criteria guide model selection. Its design combines segment processing, silence labeling, regularization, and a cross-validation metric that avoids labeled development data.

  • Model architecture: The generator maps segment representations to phonemes, collapsing consecutive segments with identical argmax predictions so the output sequence can be shorter than the input.For T input segments, the generator produces M phonemes with M ≤ T.
  • Model architecture: The phoneme inventory includes SIL, preventing the model from repurposing an ordinary phoneme for silence and improving compatibility with later language-model decoding.Without SIL, silence labeling interfered with subsequent decoding and reduced performance.
  • Model architecture: The discriminator distinguishes phonemized unlabeled text from generator outputs derived from unlabeled speech segments.Real text uses one-hot phoneme vectors, while generator outputs represent phoneme distributions over segments.
  • Training objective: Training uses GAN objectives with gradient, segment smoothness, and phoneme diversity penalties, alternating discriminator and generator updates on speech and text batches.Smoothness addresses correlated neighboring segments, while diversity penalizes low vocabulary usage.
  • Model selection: The unsupervised cross-validation metric combines language-model negative log-likelihood with vocabulary usage to reject fluent but degenerate transcriptions.It supports early stopping, seed selection, and hyper-parameter selection without labeled development data.
  • Model selection: 0.3–1.2 higher PER is observed for the unsupervised metric than for labeled development-set selection on TIMIT, enabling development without labeled data at a small accuracy drop.The comparison covers GAN and self-training settings on TIMIT core-dev/test.

5. Experimental Setup

The experiments evaluate wav2vec-U across English, multilingual European, and low-resource non-European speech benchmarks using unlabeled audio and text. They specify dataset scales, phonemization choices, model configurations, decoding procedures, and unsupervised hyper-parameter tuning.

  • Datasets: The evaluation spans TIMIT, Librispeech, Multilingual LibriSpeech, ALFFA Swahili, and CommonVoice Kyrgyz and Tatar.These datasets cover English, six European languages, Swahili, and two low-resource languages.
  • Datasets: TIMIT contains about five hours of audio and supports matched and unmatched settings controlling whether unlabeled text includes the audio transcriptions.The standard train/dev/test split is used for the matched setup.
  • Datasets: Librispeech provides 960 hours of transcribed training audio, while unsupervised training uses only speech audio and a 635-million-word unlabeled text corpus.Experiments evaluate dev-other/clean and test-clean/other, with Libri-Light also providing 53.2k hours of speech.
  • Datasets: Multilingual LibriSpeech uses 100 hours of speech audio per language for six selected European languages.The corpus contains read audiobooks and supplies language-model data as unlabeled text.
  • Datasets: Kyrgyz and Tatar experiments use 1.8 and 4.6 hours of speech audio, respectively, alongside community-corpus and news text.Swahili uses 9.2 hours of training speech and ALFFA and news text data.
  • Model configuration: The generator is a single non-causal convolution, and the discriminator uses three causal convolution blocks; both are trained with fixed learning rates for 150k steps.Each model is updated 75k times, using batches of 160 audio and 160 text samples.
  • Model configuration: Hyper-parameters are tuned with the unsupervised cross-validation metric across gradient, smoothness, and diversity penalty ranges, using five seeds per configuration.The procedure trains 40 models per language.
  • Decoding: Decoding uses WFSTs with 4-gram or Transformer language models, tuning acoustic scale and blank or silence-emission weights against fluency and faithfulness.The Transformer decoder uses beam 50 for tuning and beam 500 for testing on Librispeech.

6. Results

Experiments evaluate wav2vec-U across English, multilingual, and low-resource benchmarks, comparing unsupervised training and self-training under varied data conditions. The method improves substantially over prior unsupervised systems and performs competitively on Librispeech while requiring far less unlabeled data than the full corpus.

  • 6.1 Comparison to Supervised Speech Recognition on Librispeech: WER 5.9 on Librispeech test-other is achieved by wav2vec-U Large with self-training and a Transformer language model.The result uses unsupervised speech recognition and is reported on the noisy test set.
  • 6.2 Comparison to Prior Unsupervised Work: 57% relative PER reduction on TIMIT all-test is reported for wav2vec-U versus Chen et al. (2019) in the matched setup.The comparison covers both matched and unmatched training settings.
  • 6.3-6.4 Multilingual and Low-resource Languages: wav2vec-U is evaluated on six European MLS languages and three low-resource languages, including Swahili, Kyrgyz, and Tatar.The low-resource experiments use between 1.8 and 9.2 hours of unlabeled audio, with reduced self-training for Kyrgyz and Tatar.
  • 6.5 Self-training Strategies: Self-training with an HMM yields substantial improvements, whereas a second HMM iteration adds much smaller gains and HMM re-segmentation does not improve over one iteration.Direct fine-tuning from GAN labels also performs very well, but repeated fine-tuning does not improve performance.
  • 6.6 Amount of Unlabeled Data Needed: 9.6h of unlabeled speech and about 3,000 unlabeled text sentences achieve performance similar to using substantially more data on Librispeech dev-other.These findings come from controlled data-amount experiments.

7. Related Work

The paper builds on self-supervised speech representation learning, semi-supervised speech recognition, unsupervised speech recognition, and unsupervised machine translation. Its approach is especially connected to adversarial mappings and automatic speech segmentation.

  • Semi-supervised Speech Recognition: Semi-supervised speech recognition commonly uses self-training, iterative filtering, knowledge distillation, data augmentation, or cycle-consistency.These methods exploit unlabeled speech or synthetic paired data alongside supervised objectives.
  • Self-supervised Learning for Speech: Self-supervised speech research includes contrastive predictive coding, language-model-style pre-training, and fixed-size speech representations for recognition tasks.These approaches motivate learning speech representations without transcription labels.
  • Unsupervised Speech Recognition: Prior unsupervised speech recognition mapped speech segments to phonemes with adversarial learning, but earlier systems relied on human or automatic segmentation and labeled cross-validation.Automatic segmentation was later refined with HMMs, while cross-validation still used labeled data.
  • Unsupervised Speech Recognition: Speech segmentation research spans clustering, Bayesian methods, efficient approximations, and self-supervised phoneme-boundary detection.This body of work provides alternatives to simple segmentation techniques.
  • Unsupervised Machine Translation: Unsupervised machine translation showed that adversarial games and shared neural representations can align spaces without seed supervision.These ideas extend from word embedding alignment to fully unsupervised sentence translation.

8. Discussion

The discussion identifies phonemization and segmentation as important boundaries for wav2vec-U. Both components can limit applicability or leave room for improvement across languages and speech-unit durations.

  • Phonemization: Phonemizer availability and quality constrain wav2vec-U across languages, and different English phonemizers produce different performance.Future directions include broader phonemizer coverage and graphemic units such as letters.
  • Segmentation: The simple segmentation method may be improved because wav2vec 2.0 uses fixed-size, fixed-stride units while phonemes have variable durations.The paper suggests learning variable-sized representations during pre-training.

9. Conclusion

The paper presents wav2vec-U as a framework for speech recognition from unlabeled speech and text. It reports strong English results, multilingual viability, and a large TIMIT improvement over prior unsupervised work.

  • Conclusion: wav2vec-U embeds and segments speech with wav2vec 2.0 representations, maps segments to phonemes adversarially, and uses an unsupervised metric for model selection.The framework does not require labeled data for building speech recognition models.
  • Conclusion: PER 11.3 on TIMIT versus 26.1 for the previous best unsupervised approach demonstrates the reported improvement.The method is also evaluated on several non-English and low-resource languages.
  • Conclusion: Using only unlabeled speech audio and text is reported to lower the effort required to build speech technology for more languages.The conclusion connects this capability to languages beyond English, including low-resource settings.

Appendix A. Hyperparameter Ablations

This appendix section examines how wav2vec-U performance varies across data settings, preprocessing steps, cluster sizes, PCA sizes, and phoneme-set choices.

  • The ablation varies data settings to assess their effect on the method.
  • Pre-processing steps are tested as factors in the ablation.
  • Cluster sizes and PCA sizes are varied to evaluate their influence.
  • The full phoneme set is included as an additional ablation condition.
Loading 2105.11084v3…