Source-linked AI summary
Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
Qiantong Xu, Alexei Baevski, Michael Auli
TL;DR
Speech technology remains unavailable for many languages, while existing unsupervised approaches can ignore labeled data in related languages. This paper fine-tunes a multilingual wav2vec 2.0 model and maps phonemes with articulatory features to recognize unseen languages. The method significantly improves over prior zero-shot work and performs comparably to unsupervised recognition while using one model for multiple unseen languages.
Problem
Many languages lack speech technology because current systems require substantial transcribed speech, while unsupervised approaches may ignore labeled data in related languages.
Method
The paper fine-tunes the multilingual XLSR-53 wav2vec 2.0 model and maps training-language phonemes to target languages using articulatory features.
Results
The approach significantly improves over prior zero-shot work and performs comparably to unsupervised speech recognition while transcribing multiple unseen languages with one model.
Takeaways & Limitations
A simple multilingual zero-shot phoneme recognizer can use related-language labels to recognize multiple unseen languages without task-specific architectures.
Abstract
from arXiv · showhide
Recent progress in self-training, self-supervised pretraining and unsupervised learning enabled well performing speech recognition systems without any labeled data. However, in many cases there is labeled data available for related languages which is not utilized by these methods. This paper extends previous work on zero-shot cross-lingual transfer learning by fine-tuning a multilingually pretrained wav2vec 2.0 model to transcribe unseen languages. This is done by mapping phonemes of the training languages to the target language using articulatory features. Experiments show that this simple method significantly outperforms prior work which introduced task-specific architectures and used only part of a monolingually pretrained model.
1. INTRODUCTION
The paper addresses limited speech technology coverage by using labeled data from related languages to transcribe unseen languages with a simple zero-shot multilingual approach. It reports improvements over prior zero-shot work and performance comparable to unsupervised recognition while supporting multiple unseen languages with one model.
- 1. INTRODUCTION: The method builds a global phoneme recognizer from training-language phonemes and uses a language model to decode unseen-language phoneme sequences.Articulatory-feature lexicons map phonemes between training and target vocabularies.
- 1. INTRODUCTION: The approach fine-tunes cross-lingually pretrained representations on labeled multilingual data instead of using monolingually trained representations or only a feature extractor.The pretrained representations were estimated on 53 languages, and the full pretrained model is used.
- 1. INTRODUCTION: Results show significant improvements on unseen languages over prior work, while cross-lingual pretrained representations are more effective.Experiments cover 42 CommonVoice, 19 BABEL, and six MLS languages.
- 1. INTRODUCTION: Zero-shot transfer performs comparably to unsupervised approaches while transcribing multiple unseen languages with a single model.The unsupervised comparison uses separate unsupervised models for each language, whereas this approach uses labeled related-language data.
2. APPROACH
The approach fine-tunes multilingual self-supervised speech representations for phoneme recognition and maps training-language phonemes to unseen target languages using articulatory features. Two many-to-one lexicon constructions are compared, with tr2tgt used by default.
- 2. APPROACH: At inference, the fine-tuned model is tested on unseen languages by mapping training-vocabulary phonemes to target-language phonemes.The model is simultaneously fine-tuned for phoneme recognition across multiple training languages.
- 2. APPROACH: The system uses XLSR-53, a wav2vec 2.0 model pretrained on speech data from 53 languages.XLSR-53 combines a convolutional feature encoder with a Transformer context network.
- 2.2. Phoneme Mapping: Articulatory-feature vectors address target-language out-of-vocabulary phonemes by describing sounds through major class, manner, place, and laryngeal attributes.Each attribute is represented as positive or negative.
- 2.2. Phoneme Mapping: The method computes Hamming distances between articulatory-feature vectors to construct many-to-one phoneme-mapping lexicons.The tr2tgt lexicon maps training phonemes to closest target phonemes and covers remaining target phonemes using closest training phonemes.
- 2.2. Phoneme Mapping: The tgt2tr lexicon maps each target phoneme to training phonemes with zero articulatory-feature distance, while tr2tgt is used unless otherwise specified.Both lexicon strategies are compared experimentally.
3. EXPERIMENTAL SETUP
The experiments evaluate the approach across multilingual speech corpora with phonemized transcriptions, a pretrained XLSR-53 model, and lexicon-based beam-search decoding. The setup spans CommonVoice, BABEL, and MLS, including zero-shot language splits.
- 3.1. Datasets: Experiments use multilingual corpora including MLS, CommonVoice, and BABEL, with audio standardized to 16 kHz.MLS includes six languages, CommonVoice uses 42 languages, and BABEL includes 21 languages.
- 3.1. Datasets: Table 1 defines the CommonVoice and BABEL dataset splits and highlights the six BABEL languages used by prior work.The highlighted languages support comparison with that earlier study.
- 3.1. Datasets: CommonVoice uses 26 training languages, Italian for validation, and 13 test languages, with each test language matched to a training language family.The selected 42 languages are supported by the phonemizer and use official train, development, and test splits.
- 3.1. Datasets: BABEL contributes 21 conversational telephone-speech languages, including Cantonese and Lao for testing and 19 languages for training.Italian serves as the validation language.
- 3. EXPERIMENTAL SETUP: Transcriptions are normalized and converted to phonemic annotations with Espeak or Phonetisaurus, depending on the corpus.Espeak is used on MLS, while Phonetisaurus is used on BABEL; both are compared with CommonVoice data.
- 3. EXPERIMENTAL SETUP: Models fine-tune pretrained XLSR-53 with a classifier over the joint training-language vocabulary and decode with a lexicon, external 6-gram language model, and beam search.The decoder uses beam size 50 and restricts predictions to valid phones in the target dictionary.
4. RESULTS
Experiments compare the proposed zero-shot system with unsupervised ASR, prior zero-shot work, and alternative pretraining and lexicon choices. Results favor multilingual pretraining and the proposed approach while showing similar lexicon trends across phonemizers.
- 4.1. Comparison with unsupervised method: Zero-shot transfer performs on par with wav2vec-U on MLS while using a simpler training and inference pipeline.The comparison uses XLSR-53 representations and evaluates phoneme error rate on held-out MLS languages.
- 4.2. Comparison to other zero-shot work: 317 hours of labeled BB-6 data outperform prior work using nearly 1.5K hours on supervised languages.The result supports using the full pretrained model rather than only the feature extractor.
- 4.3.2. Comparison of lexicon and phonemizers: Beam-search decoding consistently improves performance across languages and settings, while unseen-language accuracy follows similar trends across phonemizers.These observations are reported alongside the CommonVoice results summary.
- 4.3.1. Effect of multilingual pretraining: Multilingual XLSR-53 pretraining performs better than English monolingual w2v LV-60K pretraining on every CommonVoice test language.Both pretraining-based approaches also substantially outperform training without pretraining, even with up to 20 times more labeled data.
- 4.3.2. Comparison of lexicon and phonemizers: The tr2tgt lexicon is slightly better than tgt2tr on average, and both Espeak and Phonetisaurus show the same trend.Different phonemizers can produce different phoneme sequences from the same word transcriptions.
5. CONCULUSION
The paper investigates zero-shot cross-lingual phoneme recognition with a cross-lingually pretrained self-supervised model. Fine-tuning the full model outperforms prior work and matches unsupervised speech recognition while using related-language labels to transcribe multiple unseen languages.
- 5. CONCULUSION: The approach fine-tunes a large cross-lingually pretrained model for zero-shot cross-lingual phoneme recognition.It is compared with prior work using a monolingually pretrained feature extractor and task-specific architectures.
- 5. CONCULUSION: Multilingual pretraining substantially improves accuracy over no pretraining and outperforms monolingual pretraining.The conclusion reports this pattern even when moderate labeled data is used.
- 5. CONCULUSION: The approach performs on par with unsupervised speech recognition while using labeled data from related languages and one model for multiple unseen languages.The unsupervised comparison uses a method that requires separate models for each target language.
Appendices
The appendices document language-support statistics and CommonVoice evaluation results. They also provide a figure describing language correlations in CommonVoice.
- Appendices: Table 6 reports CommonVoice language statistics and phonemizer support, marking languages potentially poorly supported by Espeak.Languages marked with an asterisk were manually removed from either training or testing.
- Appendices: Table 7 reports Babel language statistics and the languages supported by Espeak and Phonetisaurus.The table concerns phonemizer coverage rather than model performance.
- Appendices: Table 8 compares PER on a subset of CommonVoice test languages across pretrained models and per-language training-hour limits.The default beam-search lexicon is tr2tgt; tgt2tr appears only in columns marked with an asterisk.
- Appendices: Figure 1 shows correlations between pairs of languages in the CommonVoice dataset.The supplied passage identifies the figure's subject but does not report numerical values or a specific comparison outcome.
A. DATASET DETAILS
The dataset appendix summarizes CommonVoice and BABEL language metadata, including language families, split durations, and phonemizer resources. CommonVoice entries are ordered by split and grouped by family.
- A. DATASET DETAILS: The appendix lists language codes, names, families, split durations, phonemizer identifiers, and finite-state transducers for CommonVoice and BABEL.CommonVoice training data are subsampled for high-resource languages to reduce bias, and Espeak covers more listed languages than Phonetisaurus in the supplied passage.
- A. DATASET DETAILS: Table 6 orders languages first by split and then groups them by language family.The ordering covers training, validation, and test splits.
B. LANGUAGE CORRELATION
Language phoneme vocabularies commonly overlap across language pairs, supporting zero-shot transfer, although tone-marked Vietnamese and Chinese vocabularies are more isolated.
- The correlation measure compares two languages using their phoneme vocabularies.
- IPA phoneme symbols are commonly shared across different languages, which supports zero-shot transfer learning.
- Vietnamese and Chinese language groups appear isolated because their phoneme symbols include tones.
- Tone distinctions such as ou1 through ou6 are intrinsically hard to learn and predict.
C. FULL COMPARISON ON COMMONVOICE
CommonVoice results show that beam-search decoding consistently improves performance across languages and settings, while unseen-language accuracy follows similar trends across phonemizers.
- Beam-search decoding consistently improves model performance for all languages across all settings.
- Accuracy trends on unseen languages are similar across phonemizers.
- The CommonVoice results are summarized in Table 8.
D. FULL RESULTS ON BABEL
On BABEL, the proposed method improves over prior zero-shot work on supervised languages and generalizes better when CommonVoice and BABEL data are combined.
- The proposed method performs better than prior work on supervised languages when fine-tuned on the same BABEL subset.
- Mixed CommonVoice and BABEL training data generalize better to unseen languages than either dataset alone.
- The method surpasses prior work even when that work uses an additional learned language encoder.
- Table 9 compares phonetic token error rates on unseen Cantonese and Lao test languages.