Source-linked AI summary
Pre-training on high-resource speech recognition improves low-resource speech-to-text translation
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, Sharon Goldwater
TL;DR
Low-resource speech-to-text translation lacks the resources required by traditional pipelines. The paper transfers parameters from high-resource ASR into ST, obtaining gains across language settings, including a four-hour Mboshi-French task.
Problem
Traditional ST requires transcribed source audio and parallel text, resources often unavailable for low-resource languages.
Method
Pre-train an encoder-decoder model on high-resource ASR, then transfer and fine-tune its parameters for low-resource ST, including encoder-only transfer across mismatched languages.
Results
7.1 BLEU was achieved on Mboshi-French ST with 4 hours of parallel data, while Spanish-English ST improved from 10.8 to 20.2 BLEU with 300 hours of English ASR.
Takeaways & Limitations
Most of the benefit comes from transferring the encoder, and mismatched-language ASR pre-training can still improve low-resource ST.
Takeaways & Limitations
The study leaves shared character or subword vocabularies for both low-resource ST languages as future work and uses non-state-of-the-art ASR pre-training.
Abstract
from arXiv · showhide
We present a simple approach to improve direct speech-to-text translation (ST) when the source language is low-resource: we pre-train the model on a high-resource automatic speech recognition (ASR) task, and then fine-tune its parameters for ST. We demonstrate that our approach is effective by pre-training on 300 hours of English ASR data to improve Spanish-English ST from 10.8 to 20.2 BLEU when only 20 hours of Spanish-English ST training data are available. Through an ablation study, we find that the pre-trained encoder (acoustic model) accounts for most of the improvement, despite the fact that the shared language in these tasks is the target language text, not the source language audio. Applying this insight, we show that pre-training on ASR helps ST even when the ASR language differs from both source and target ST languages: pre-training on French ASR also improves Spanish-English ST. Finally, we show that the approach improves performance on a true low-resource task: pre-training on a combination of English ASR and French ASR improves Mboshi-French ST, where only 4 hours of data are available, from 3.5 to 7.1 BLEU.
1 Introduction
The paper addresses low-resource speech-to-text translation, where conventional pipelines require unavailable transcribed source audio and parallel text. It proposes ASR pre-training and parameter transfer, reporting gains even with mismatched languages.
- Low-resource ST often lacks the transcribed source audio and parallel text required by traditional ASR–MT pipelines.
- 150+ hours of translated audio produced 47.3 BLEU, whereas 20 hours produced 5.3 BLEU for Spanish-English ST.
- The approach pre-trains on high-resource ASR, then transfers and fine-tunes parameters for low-resource ST.
- 20.2 BLEU versus 10.8 was obtained for Spanish-English ST using 300 hours of English ASR and 20 hours of ST data.
- The encoder accounts for most of the gain, and French ASR also improves Spanish-English ST despite differing from both ST languages.
2 Method
The method uses a shared attention-based encoder-decoder architecture, initializing ST models from ASR parameters. Transfer can include the decoder when vocabularies are shared or only the encoder across mismatched languages.
- ASR and ST use the same attention-based encoder-decoder architecture, enabling convenient parameter transfer.
- English ASR initializes Spanish-English ST, while French ASR initializes Mboshi-French ST with a shared decoder vocabulary.
- When ASR and ST languages differ, only encoder parameters transfer because decoder softmax dimensions depend on the vocabulary.
- Encoder-only transfer remains flexible because speech inputs can be preprocessed identically across languages.
3 Experimental Setup
The experiments cover English and French ASR, low-resource Spanish-English ST, and four-hour Mboshi-French ST. Speech uses normalized MFCCs, while target text uses BPE subwords and attention-based decoding.
- 300 hours of English ASR and 20 hours of French ASR provide the pre-training corpora.
- Spanish-English ST uses 160 hours of telephone speech, with low-resource subsets ranging from 2.5 to 50 hours.
- Mboshi-French ST contains around 4 hours of speech, with 5K training utterances and a designated development set used as test data.
- Speech is converted to 13-dimensional MFCCs and normalized at the speaker level.
- BPE uses 1000 frequent subword sequences, retaining open vocabulary with 1.9M tokens and just over 1K types.
- The speech encoder combines strided CNN layers with a three-layer bidirectional LSTM, and the decoder uses attention over encoder outputs.
- Models are trained with dropout, weight decay, Adam optimization, feature noise, frame dropping, and decoder-label corruption.
- BLEU is reported for all models, but the paper notes that low-resource BLEU can poorly reflect partial semantic usefulness.
4 ASR results
The paper pre-trains English and French ASR models and evaluates them with development-set WER. These models were intentionally stopped before state-of-the-art performance to prioritize transfer experiments.
- English and French ASR models are evaluated using development-set WER on Switchboard train-dev and GlobalPhone dev, respectively.
- ASR pre-training stopped after about 30 epochs, or three days, to focus on transfer experiments.
- The reported ASR results are far from state of the art because of limited tuning and training time.
5 Spanish-English ST
English ASR pre-training substantially improves Spanish-English speech-to-text translation in low-resource settings, with benefits extending to very small training sets and arising mainly from the speech encoder.
- 27.8 BLEU versus 23.3 shows transfer remains effective for Spanish-English ST with 50 hours of training data.Precision rises from 54% to 58%, while recall rises from 51% to 56%.
- Transfer learning raises precision and recall by 10 to 20 points when ST training data falls below 10 hours.With only 2.5 hours, the model reaches around 30% precision/recall and improves by more than 10 points over the naive baseline.
- 9.1 BLEU with 5 hours of transferred ST training matches the 10.8 BLEU of a 20-hour model without transfer learning.The comparison indicates that fine-tuning an English ASR model can produce similar results with one quarter as much ST data.
- Analysis: Ablations show that speech encoder parameters account for most transfer gains, although transferring all parameters is most effective.The authors hypothesize that the encoder learns language-independent acoustic features that normalize speaker and channel variability.
- Analysis: English ASR pre-training becomes more beneficial as ST data decreases, while its gains tail off with more ST training data.For 20-hour ST, 100-hour ASR improves performance by almost 6 BLEU points and 300-hour ASR by almost 9 points; for 50-hour ST, improvements appear only with 300-hour ASR.
6 Mboshi-French ST
With only 4 hours of Mboshi-French speech-translation data, transferring ASR parameters improves BLEU over baseline models, with the strongest results from combining English encoder and French decoder parameters.
- 4 hours of Mboshi-French data make this a clearly challenging speech-translation setting.
- ASR models alone perform much worse than naive frequency-based baselines on Mboshi-French.The naive baselines predict the 8 or 10 most frequent French training words.
- Transferring French or English ASR parameters improves BLEU while producing precision and recall comparable to naive baselines.French transfer uses encoder and decoder parameters, whereas English transfer uses the encoder only.
- The combined en+fr configuration achieves the best scores on all metrics by pairing the English encoder with the French attention and decoder parameters.This combines the larger encoder training set with the decoder’s matching language.
7 Conclusion
The paper shows that ASR pre-training can substantially improve low-resource speech translation, including a 9-BLEU-point Spanish-English gain and a 7.1 BLEU result for Mboshi-French. Encoder transfer provides most of the benefit, while decoder transfer from ASR did not work and further multilingual or language-model pre-training remains open.
- The method improves Spanish-English ST by 9 BLEU points using 20 hours of parallel data and 300 hours of English ASR data.The pre-trained model also reaches higher BLEU in a couple of hours, versus more than a day for the baseline.
- With only 4 hours of Mboshi-French parallel data, combining an English encoder and French decoder achieves 7.1 BLEU and about 25% precision and recall.The authors report this as an effective result on a real low-resource language.
- Encoder transfer provides most of the benefit, although transferring both encoder and decoder parameters works better than transferring either alone.The authors infer that a large mismatched-language ASR corpus may outperform a smaller corpus matching the output language.
- The approach remains bounded by unresolved modeling choices, including multilingual speech features and alternatives to ASR-based decoder transfer.The paper identifies pre-trained language-model decoders and shallow fusion as future directions.