Source-linked AI summary
Unsupervised pretraining transfers well across languages
Morgane Rivière, Armand Joulin, Pierre-Emmanuel Mazaré, Emmanuel Dupoux
TL;DR
The paper addresses whether unsupervised pretraining can transfer phoneme representations across languages despite limited supervised resources. It modifies CPC, transfers English-pretrained features to low-resource languages, and finds performance that is competitive with supervised pretraining and can exceed it with more unlabelled data. These results support unsupervised pretraining for low-resource languages.
Problem
Learning phoneme representations for many low-resource languages remains difficult, while supervised cross-lingual pretraining requires massive same-domain supervised corpora.
Method
The paper modifies CPC, pretrains phoneme representations on English Librispeech, and transfers them to low-resource languages while evaluating frozen features with linear classifiers.
Results
Unsupervised pretraining improves over training from scratch, nearly matches supervised pretraining, and can surpass it when scaled to larger unlabelled datasets.
Takeaways & Limitations
Unsupervised pre-trained features can transfer across languages and domains using extremely low-resource datasets, unaligned labels, and a simple linear model.
Abstract
from arXiv · showhide
Cross-lingual and multi-lingual training of Automatic Speech Recognition (ASR) has been extensively investigated in the supervised setting. This assumes the existence of a parallel corpus of speech and orthographic transcriptions. Recently, contrastive predictive coding (CPC) algorithms have been proposed to pretrain ASR systems with unlabelled data. In this work, we investigate whether unsupervised pretraining transfers well across languages. We show that a slight modification of the CPC pretraining extracts features that transfer well to other languages, being on par or even outperforming supervised pretraining. This shows the potential of unsupervised methods for languages with few linguistic resources.
1. INTRODUCTION
The paper asks whether unsupervised CPC pretraining can transfer phoneme representations across languages without massive supervised corpora. A modified CPC model transfers well to low-resource languages, approaching or sometimes exceeding supervised pretraining.
- Low-resource languages often lack supervised resources for learning phoneme representations.
- Supervised cross-lingual pretraining can transfer across distant languages but requires massive same-domain supervised corpora.
- The study evaluates whether unsupervised audio representation methods can improve phoneme representations in cross-lingual low-resource settings.
- The modified CPC model pretrains English phoneme representations on Librispeech and transfers them to low-resource Common Voice languages.
- Unsupervised pretraining outperforms target-language training from scratch, nearly matches supervised pretraining on the same corpora, and can surpass it with more unlabelled data.
2. RELATED WORK
Prior work improves low-resource speech recognition by sharing supervised multilingual components and finetuning them for target languages. This paper builds on contrastive unsupervised learning, particularly CPC, to learn transferable audio features without assumptions about training-sample structure.
- Supervised multilingual approaches share representations or model parameters across languages and then finetune on the target language.
- Contrastive speech-learning methods learn audio features by discriminating between time windows or predicting future representations.
- CPC applies a contrastive objective to sequences, predicting forward representations rather than future audio inputs.
- CPC makes no assumption about the nature or number of training data samples.
3. APPROACH
The approach modifies CPC to learn transferable phoneme features, evaluates their cross-lingual linear separability without finetuning, and stabilizes training by replacing batch normalization. CPC predicts future representations while contrasting them with distant negatives.
- CPC constructs representations from encoded audio windows and a sequence model, then predicts the next K future representations.
- The contrastive objective pulls temporally nearby representations closer and pushes randomly sampled distant windows away.
- The encoder is a five-layer convolutional network and the sequence model is a one-layer GRU, with each feature representing 10ms of 16kHz audio.
- 3.2.1. Stabilization of the training: Replacing batch normalization with channel-wise normalization prevents information leakage across sequence windows and stabilizes CPC training.
- The CPC objective may not directly yield linearly separable phoneme representations because future representations encode time windows rather than phonemes.
- After pretraining, the model is frozen and a linear classifier evaluates phoneme separability by classifying concatenated windows.
4. EXPERIMENTAL SETTING
Experiments pretrain models on English Librispeech and transfer frozen features to Common Voice languages using small speaker-separated datasets. Phoneme separability is also evaluated on English, Mandarin, and French Zerospeech2017 benchmarks.
- Models are pretrained on the 100h and 360h clean Librispeech splits, with supervised pretraining using aligned phone labels for Librispeech-100h.
- Frozen features are transferred to Common Voice languages, using phonemized text and speaker-based train, validation, and test splits.
- Common Voice target-language training sets contain either 1 or 5 hours of speech.
- 4.0.3. Measuring phoneme separability on Zerospeech2017: Zerospeech2017 experiments cover English, Mandarin, and French and report ABX scores measuring phoneme discriminability.
5. RESULTS
The modified CPC model improves within-language phoneme representations and transfers effectively across low-resource languages. Its performance approaches supervised pretraining, especially with more unlabelled data, while fine-tuning yields further gains but remains slightly behind supervised pretraining.
- 5.1. Within-language results: 3.4 points improvement in phoneme classification over the original CPC implementation follows the proposed modifications.The modified model also reaches state-of-the-art performance on English phoneme discriminability and surpasses the original CPC model.
- 5.2. Cross-lingual transfer of phoneme features: 5.4 points average improvement over original CPC is obtained by modified CPC on 100 hours of Librispeech.The comparison uses linear classifiers on pretrained phoneme features for cross-lingual phoneme classification.
- 5.2. Cross-lingual transfer of phoneme features: 1 hour of target-language speech is sufficient for pretrained features to improve performance across all evaluated Common Voice languages.The evaluation compares frozen pretrained features with a model trained from scratch on each 1-hour target dataset.
- 5.2. Cross-lingual transfer of phoneme features: 360 hours of unlabelled Librispeech lets modified CPC match supervised pretraining, compared with a 1.3-point gap using 100 hours.The result demonstrates the benefit of scaling unsupervised pretraining data.
- 5.2. Cross-lingual transfer of phoneme features: English features transferred to French and Mandarin achieve ABX scores competitive with leading within-language systems.Table 4 compares across- and within-speaker ABX scores against the best systems and supervised toplines on the Zerospeech leaderboard.
- 5.2.1. Impact of finetuning phoneme features: Around 7 performance points are gained by fine-tuning pretrained features, while the proposed approach remains 1.5 points below supervised pretraining on average.These results average PER over five target languages using 5 hours of speech per language.
6. CONCLUSION
The study concludes that unsupervised pretraining can produce cross-lingually usable phoneme features at low cost and with extremely limited target-language resources. Larger unlabelled datasets substantially narrow the remaining gap with supervised pretraining, while broader ASR use remains for future work.
- 6. CONCLUSION: Unsupervised or supervised pretraining in one language produces features usable across other languages and domains.The conclusion frames this cross-lingual and cross-domain usability as a central result.
- 6. CONCLUSION: Extremely low-resource datasets and unaligned labels can support matching features to phonemes with a simple linear model.The authors also state that these features can be trained at low cost.
- 6. CONCLUSION: Larger amounts of unlabelled data can greatly reduce the small performance gap between unsupervised and supervised pretraining.Supervised pretraining tends to remain better, but the conclusion emphasizes that the difference can be substantially reduced.
- 6. CONCLUSION: The evaluation used only a linear separation layer for phoneme classification, leaving broader low-resource and other ASR applications for future work.The authors explicitly state that they did not optimize phone error rates in the low-resource languages.
S1. SUPPLEMENTARY METHODS
The supplementary methods describe ablation experiments comparing a reimplementation of the original CPC model with the authors’ improvements.
- S1. SUPPLEMENTARY METHODS: The ablations compare the reimplemented original CPC model with the improvements introduced by the authors.These experiments are designed to isolate the effects of the modified model components.
S1.1. Changing the normalization method
One supplementary ablation changes the normalization method to stabilize CPC training. Its effect is evaluated through phoneme discriminability on the English Zerospeech2017 test set.
- S1.1. Changing the normalization method: Batch normalization is replaced with layer normalization to make CPC training more stable.The resulting performance is reported in Table S1.
- S1.1. Changing the normalization method: Table S1 evaluates the normalization change using within- and across-speaker ABX scores on English Zerospeech2017.ABX scores measure phoneme discriminability in the stated test setting.
S1.2. Choosing the right predictor design
The experiments compare alternative predictor designs for CPC pretraining, testing their effect on phoneme discriminability within and across speakers on English Zerospeech2017.
- The predictor-design comparison is motivated by the possibility that a simple prediction network leaves substantial prediction work to the autoregressive network.The authors therefore examine whether more complex architectures improve the output features.
- The study evaluates multiple predictor designs using within- and across-speakers ABX scores on the English Zerospeech2017 test set.The comparison targets phoneme discriminability under different predictor architectures.
S2. SUPPLEMENTARY RESULTS
Supplementary results evaluate CPC features trained on Libri-light 60K for cross-lingual phoneme transfer and compare them with Bottleneck features and supervised features.
- CPC features trained on Libri-light 60K beat both Bottleneck and supervised features on all languages except one.Figure S1 displays the comparison between CPC and Bottleneck features.
- Libri-light 60K CPC features are compared with Bottleneck and supervised features for phoneme transfer across ten languages.The evaluated languages include Dutch, Spanish, French, Italian, Kyrgyz, Russian, Swedish, Turkish, Tatar, and Mandarin.
- The comparison uses Phone Error Rate from linear classifiers trained on frozen pretrained features with 1h of Common Voice speech per target language.Bottleneck features were trained on 1070h of Babel data, while the supervised baseline used LibriSpeech 100h clean.