Source-linked AI summary
Meta Learning for End-to-End Low-Resource Speech Recognition
Jui-Yang Hsu, Yuan-Jui Chen, Hung-yi Lee
TL;DR
End-to-end ASR requires costly paired speech-transcription data that most languages lack. The paper applies MAML to multilingual ASR pretraining, treating languages as tasks and learning an initialization for fast adaptation. MetaASR significantly outperforms MultiASR across all target languages, while the authors describe the experiments as initial and call for broader evaluation.
Problem
End-to-end ASR requires costly paired speech-transcription data, and most languages lack enough of it.
Source: §1 · p. 1Method
MetaASR uses MAML to learn shared-encoder initialization parameters across language tasks by simulating language-specific learning before meta-updating.
Source: §2.2.2 · pp. 2–3Results
MetaASR significantly outperforms MultiASR across all target languages on the IARPA BABEL dataset.
Source: §1 · p. 2Takeaways & Limitations
The model-agnostic approach may extend to other network architectures and applications beyond speech recognition.
Source: §5 · p. 4The experiments are initial; extensive evaluation across more language and corpus combinations remains future work.
Source: §5 · p. 4
Abstract
from arXiv · showhide
In this paper, we proposed to apply meta learning approach for low-resource automatic speech recognition (ASR). We formulated ASR for different languages as different tasks, and meta-learned the initialization parameters from many pretraining languages to achieve fast adaptation on unseen target language, via recently proposed model-agnostic meta learning algorithm (MAML). We evaluated the proposed approach using six languages as pretraining tasks and four languages as target tasks. Preliminary results showed that the proposed method, MetaASR, significantly outperforms the state-of-the-art multitask pretraining approach on all target languages with different combinations of pretraining languages. In addition, since MAML's model-agnostic property, this paper also opens new research direction of applying meta learning to more speech-related applications.
1. INTRODUCTION
End-to-end ASR depends on costly paired speech-transcription data, which most languages lack in sufficient quantity. The paper proposes meta-learning for multilingual pretraining and reports significant gains over MultiASR across target languages.
1. INTRODUCTION
- CTC supports end-to-end recognition of character sequences directly from speech without advance framewise phoneme alignment.The introduction also presents figures illustrating the multilingual CTC architecture and the distinction between MultiASR and MetaASR parameter learning.Source: §1 · p. 1, Figure 1 · p. 1, Figure 2 · p. 2
- End-to-end ASR requires costly paired speech-transcription data, which most languages lack in sufficient quantity.Multilingual transfer learning/pretraining initializes target-language fine-tuning from other-language sources.Source: §1 · p. 1
- Multilingual pretraining shares an encoder across languages to learn language-independent representations, with language-specific heads.Prior work also explored language-adversarial and hierarchical variants.Source: §1 · p. 1
- Meta-learning targets fast adaptation to unseen data, but had not been applied to multilingual pretraining for speech recognition.Earlier applications included neural machine translation, dialogue generation, and speaker adaptive training.Source: §1 · pp. 1–2
- MetaASR significantly outperforms MultiASR across all target languages on the IARPA BABEL dataset.The paper presents MAML as model-agnostic and does not add modules or require phoneme-level annotations.Source: §1 · p. 2
2. PROPOSED APPROACH
The approach combines a shared multilingual CTC encoder with language-specific heads and uses MAML to learn encoder initializations for fast adaptation across language tasks.
2.1. Multilingual CTC Model
- A shared encoder feeds a language-specific head, and each language's paired speech and transcript are trained with a CTC objective.The encoder produces hidden-state sequences that the corresponding language head maps to prediction sequences.Source: §2.1 · p. 2
2.2. Meta Learning for Low-Resource ASR
- MAML treats languages as tasks and learns shared-encoder initialization parameters intended to enable fast fine-tuning on a target language.The learned initialization is obtained from source tasks and then used to adapt the target task's encoder and head.Source: §2.2 · p. 2
2.2.1. Learn: Language-specific learning
- Language-specific learning minimizes the CTC loss from an initial shared-encoder parameter set using gradient descent, as in MultiASR.The starting parameters may be randomly initialized or taken from a pretrained model.Source: §2.2.1 · p. 2
2.2.2. MetaLearn
- Each meta-learning episode simulates language-specific learning on task training subsets, evaluates adapted parameters on held-out subsets, then updates the shared-encoder initialization toward broad task adaptation.Unlike MultiASR, MetaASR explicitly models the learning process on unseen languages before updating the model.Source: §2.2.2 · pp. 2–3
3. EXPERIMENT
The experiments use IARPA BABEL speech to test adaptation from six non-target languages to four target languages, with full and limited language packs.
3. EXPERIMENT
- Experiments use six BABEL languages for non-target pretraining and four for target adaptation, with each language represented by a Full Language Pack and a Limited Language Pack containing 10% of FLP.The non-target languages are Bengali, Tagalog, Zulu, Turkish, Lithuanian, and Guarani; targets are Vietnamese, Swahili, Tamil, and Kurmanji.Source: §3 · p. 3
- The shared encoder uses an Espnet recipe, 80-dimensional Mel-filterbank and 3-dimensional pitch features, a six-layer VGG extractor, and a six-layer bidirectional LSTM.Each LSTM direction has 360 cells; acoustic windows are 25 ms with a 10 ms stride.Source: §3 · p. 3
- MetaASR uses one SGD language-specific learning step per episode and ignores the second-order derivative term, following a first-order MAML approximation.The authors cite prior work reporting that omitting this term does not affect performance too much.Source: §3 · p. 3
- Both systems are pretrained for 100K steps; target-language fine-tuning runs for 18 epochs on FLP or 20 on LLP, with beam-20 decoding and 5-gram language-model rescoring.The other three target languages' LLP sets are used as validation data to select the pretraining step.Source: §3 · p. 3
4. RESULTS
On Swahili LLP, MetaASR’s adaptation performance improves as pretraining continues, while MultiASR saturates early and later degrades. On LLP, MetaASR also outperforms MultiASR on three target languages, though Tamil is a boundary case.
Learning Curves
- MetaASR’s Swahili LLP adaptation performance improves throughout pretraining without degrading, while MultiASR saturates early and later degrades.The learning curves cover pretraining on Bn, Tl, Zu and on Bn, Tl, Zu, Tr, Lt, Gn.Source: §4 · p. 4, Figure 3 · p. 4, Figure 4 · p. 4
Impact on Training Set Size
- MetaASR outperforms MultiASR on Vietnamese, Swahili, and Kurmanji LLP across the evaluated pretraining-language sets.Table 2 reports CER for all four target languages’ LLP.Source: §4 · p. 4, Table 2 · p. 4
- On LLP, the gap over no-pretrain is smaller than on FLP, and Tamil’s pretrained weights perform worse than random initialization.Source: §4 · p. 4
5. CONCLUSION
The paper proposes meta learning for multilingual pretraining in speech recognition and reports initial evidence of its potential. It also identifies possible applications beyond speech recognition based on MAML’s model-agnostic property.
- The paper proposes a meta-learning approach to multilingual pretraining for speech recognition.Source: §5 · p. 4
- Initial experimental results indicate the approach’s potential for multilingual pretraining.Source: §5 · p. 4
- The authors identify sequence-to-sequence models and applications beyond speech recognition as possible extensions, citing MAML’s model-agnostic property.Source: §5 · p. 4