PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Cross-lingual Representation Learning for Speech Recognition

Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael Auli

arXiv:2006.13979v2cs.CLcs.LGcs.SDeess.AS

TL;DR

Speech recognition often lacks labeled data across languages, motivating representations learned from multilingual unlabeled audio. XLSR extends wav2vec 2.0 with shared quantized speech representations and contrastive pretraining, then fine-tunes the model for recognition. Multilingual pretraining outperforms monolingual pretraining and prior work, while one multilingual model remains competitive with individually fine-tuned models.

  • Problem

    Cross-lingual speech systems commonly require labeled data in multiple languages, although transcribed speech is scarce and costly to annotate.

  • Method

    XLSR extends wav2vec 2.0 by learning shared quantized latent speech representations across languages from unlabeled audio and fine-tuning the Transformer.

  • Results

    Multilingual pretraining outperforms monolingual pretraining and prior work, with the largest improvements on low-resource languages.

  • Takeaways & Limitations

    Fine-tuning across languages enables a single multilingual speech recognition model competitive with individually fine-tuned models, while related languages share more representation capacity.

  • Takeaways & Limitations

    The fine-tuning scripts were not adapted to the large-scale English supervised data.

Abstract

from arXiv · show

This paper presents XLSR which learns cross-lingual speech representations by pretraining a single model from the raw waveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over masked latent speech representations and jointly learns a quantization of the latents shared across languages. The resulting model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly outperforms monolingual pretraining. On the CommonVoice benchmark, XLSR shows a relative phoneme error rate reduction of 72% compared to the best known results. On BABEL, our approach improves word error rate by 16% relative compared to a comparable system. Our approach enables a single multilingual speech recognition model which is competitive to strong individual models. Analysis shows that the latent discrete speech representations are shared across languages with increased sharing for related languages. We hope to catalyze research in low-resource speech understanding by releasing XLSR-53, a large model pretrained in 53 languages.

1 INTRODUCTION

XLSR addresses the scarcity of labeled speech by learning cross-lingual representations from unlabeled multilingual audio. It builds on wav2vec 2.0 and evaluates whether multilingual pretraining and fine-tuning improve speech recognition.

  • Supervised cross-lingual speech systems require labeled data in multiple languages, although transcribed speech is scarcer and costly to annotate.
  • XLSR learns shared discrete speech representations across languages from unlabeled data, extending wav2vec 2.0.
  • The approach uses a shared quantization module and contrastive Transformer training to create multilingual speech units from raw audio.
  • Multilingual pretraining outperforms monolingual pretraining in most evaluated cases, except for resource-rich languages.

2 APPROACH

XLSR extends wav2vec 2.0 with a shared cross-lingual quantized speech vocabulary. It trains contextual representations by predicting quantized targets for masked speech using contrastive learning, with multilingual sampling controlling language balance.

  • XLSR learns one set of quantized latent speech representations shared across languages.
  • A convolutional encoder maps raw audio to latent representations, while a Transformer produces contextual representations and quantization supplies training targets.
  • Product quantization selects entries from two codebooks with 320 entries each, concatenating them into each quantized representation.
  • The contrastive task identifies the true quantized latent for a masked time step among 100 distractors.
  • A codebook diversity penalty encourages use of all entries, supplemented by an L2 penalty on feature-encoder outputs.
  • During multilingual pretraining, sampling probabilities are based on language data sizes and an upsampling factor that controls resource balance.

3 EXPERIMENTAL SETUP

The experiments evaluate XLSR across read and conversational speech benchmarks, using multilingual and monolingual pretraining conditions. They also vary model capacity, fine-tuning setup, and multilingual data coverage.

  • CommonVoice provides read speech in 38 languages, with evaluation on ten languages using one hour of labeled training data per language.
  • BABEL contains conversational telephone speech; models are pretrained on ten languages and evaluated on four additional languages for cross-lingual transfer.
  • BABEL evaluation reports character error rate using development data as the test set because the official evaluation set was unavailable.
  • XLSR-53 combines Multilingual LibriSpeech, CommonVoice, and BABEL to cover 53 languages.
  • The Base architecture has 12 Transformer blocks, whereas Large has 24 blocks and larger model and feed-forward dimensions.
  • Fine-tuning adds a task-specific classifier and uses CTC loss while keeping feature-encoder weights fixed.
  • CommonVoice comparisons use English-only, monolingual, and multilingual pretraining on 1350 hours, with low-resource upsampling controlled by α.
  • BABEL comparisons include monolingual models for 14 languages and multilingual models trained on 650 hours across ten languages.

4 RESULTS

The experiments show that multilingual pretraining improves cross-lingual speech recognition, especially for low-resource languages, while multilingual fine-tuning can produce one strong model across languages. The analysis also links representation sharing to language relatedness.

  • Multilingual pretraining significantly outperforms prior work and monolingual models, with the largest improvements on low-resource languages.
  • Fine-tuning one multilingual model across languages achieves strong performance while avoiding separate models for each language.
  • The discrete representations share model capacity across languages, particularly among related languages.

4.1 EFFECTIVENESS OF UNSUPERVISED CROSS-LINGUAL REPRESENTATION LEARNING

Unsupervised multilingual pretraining consistently improves speech recognition over monolingual training and prior systems across CommonVoice and BABEL. The learned representations also transfer to unseen languages, while evaluation covers both in-pretraining and cross-lingual settings.

  • 13.6 PER on average: XLSR-10 achieves a 49% relative PER reduction versus XLSR-Monolingual on CommonVoice.
  • Evaluation compares CommonVoice PER and BABEL CER using multilingual, monolingual, and prior baselines, with Table 1 covering one- versus ten-language pretraining and fine-tuning.
  • 72% relative PER reduction: XLSR-10 Large improves over the prior m-CPC system on CommonVoice.
  • 16% relative word error reduction: XLSR improves over the BLSTM-HMM baseline on BABEL, while XLSR-10 Large reduces CER by 38% relative to multi-BLSTMP+VGG.
  • 29 to 22.8 average CER: multilingual pretraining improves performance on four BABEL languages absent from pretraining.

4.2 UNDERSTANDING CROSS-LINGUAL TRANSFER LEARNING

Cross-lingual transfer is strongest for low-resource languages, while multilingual training can interfere with high-resource languages unless capacity and sampling are adjusted.

  • Cross-lingual transfer learning improves low-resource language understanding: Cross-lingual representation learning is particularly effective for low-resource languages.On CommonVoice, XLSR-10 reduces PER over XLSR-Monolingual by 67% on Swedish, 72% on Turkish, 72% on Kyrgyz, and 64% on Tatar.
  • Cross-lingual transfer learning improves low-resource language understanding: On BABEL, multilingual models outperform monolingual models on every language, with the largest gains on the four lowest-resource languages.These languages are Georgian, Kurmanji, Tokpisin, and Swahili.
  • Cross-lingual transfer learning improves low-resource language understanding: XLSR-53 transfers strongly from MLS to CommonVoice, improving Spanish and French by 4.8 and 5.9% absolute PER over XLSR-10.Both datasets contain read speech, and XLSR-53 also reduces BABEL character error rate by 0.8%.
  • The transfer-interference trade-off: high-resource vs. low-resource: Multilingual models improve low-resource languages but perform worse on high-resource languages because capacity must be shared across languages.This is described as a transfer-interference trade-off.
  • The transfer-interference trade-off: high-resource vs. low-resource: 15.3 versus 13.6 average PER: balanced sampling with α = 0.5 outperforms sampling the true language distribution with α = 1.The balanced model allocates more capacity to low-resource languages; French is the sole exception.
  • The transfer-interference trade-off: high-resource vs. low-resource: Increasing model capacity reduces the gap between multilingual and monolingual models for high-resource languages.The XLSR-10 Large model reduces PER by 9.6% relative to Base on CommonVoice.

4.3 SUPERVISED MULTILINGUAL FINE-TUNING: ONE MODEL FOR ALL LANGUAGES

Multilingual fine-tuning aims to produce one speech recognition model for many languages, but its effectiveness depends on model capacity and supervision volume.

  • Supervised multilingual fine-tuning: one model for all languages: Multilingual fine-tuning trains one model on labeled data from all languages instead of separate models for each language.The study compares fine-tuning all languages jointly (#ft=10) with language-specific fine-tuning (#ft=1).
  • Supervised multilingual fine-tuning: one model for all languages: A shared phoneme vocabulary reduces the modeled phonemes from 474 to 182, while multilingual fine-tuning remains competitive with monolingual fine-tuning on CommonVoice.The Base model obtains 13.6 average PER with monolingual fine-tuning, compared with 14.1 and 13.8 PER for separate and shared multilingual vocabularies.
  • Supervised multilingual fine-tuning: one model for all languages: On BABEL, multilingual fine-tuning decreases Base-model performance from 24.9 to 27.9 average CER, but XLSR-10 Large reaches 23.7 average CER.The Large model approaches the 23.2 average CER of monolingual Large fine-tuning and enables one model for many languages.

4.4 ON THE ROLE OF LANGUAGE SIMILARITY ON CROSS-LINGUAL TRANSFER

Language similarity influences cross-lingual transfer, and the learned discrete latent representations reflect similarities among related languages without fully recovering language families.

  • On the role of language similarity on cross-lingual transfer: Language similarity is studied through cross-lingual transfer and clustering in the multilingual token embedding space.The analysis examines how shared discrete speech representations are allocated across languages.
  • On the role of language similarity on cross-lingual transfer: With 5 hours of Italian data, adding 50 hours from another language improves performance overall, with related Spanish producing the largest improvement.Distant languages such as Kabyle or Chinese are less effective; fine-tuning uses 1 hour of Italian labeled data.
  • Analyzing the shared discrete speech representations: The analysis constructs language vectors from normalized frequencies of shared discrete tokens, computes Jensen-Shannon similarity, then clusters and projects languages.K-Means is performed before two-dimensional PCA to avoid loss of information during clustering.
  • Analyzing the shared discrete speech representations: The model shares more discrete tokens among similar languages, grouping examples such as Basque, Catalan, Spanish, and Italian.Other groupings include English, German, and French, plus Arabic and Kabyle.
  • Analyzing the shared discrete speech representations: BABEL clusters include Bengali and Assamese, while the analysis does not consistently recover full language-family structure.Italian is closer to Spanish, the most effective language in the preceding transfer experiment.

5 CONCLUSION

XLSR learns unsupervised cross-lingual speech representations that improve recognition, especially for low-resource languages, and support competitive multilingual models.

  • Conclusion: Multilingual pretraining improves over monolingual pretraining and prior work, with the largest improvements on low-resource languages.The representations are learned from raw waveforms.
  • Conclusion: Fine-tuning on multiple languages at once produces a single multilingual speech recognition model competitive with individually fine-tuned models.The model shares capacity across languages, particularly related languages.
  • Conclusion: Analysis shows that discrete latent speech representations share capacity across languages, especially among related languages.This connects the learned representation structure with observed cross-lingual transfer.
Loading 2006.13979v2…