Source-linked AI summary
Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J. Weiss, Yuan Cao, Chung-Cheng Chiu, Naveen Ari, Stella Laurenzo, Yonghui Wu
TL;DR
End-to-end speech translation is attractive but lacks abundant parallel speech-to-translation data, limiting quality. The paper uses pre-trained ASR and MT components plus TTS- and MT-generated synthetic pairs, finding that weakly supervised and unlabeled data can substantially improve ST, while freezing the encoder helps avoid synthetic-speech overfitting.
Problem
End-to-end ST performance is limited by the scarcity and expense of large parallel speech-to-translation datasets.
Method
The paper separately pre-trains ST encoder and decoder components on ASR and MT, then fine-tunes with synthetic speech-to-translation pairs from weakly supervised and unlabeled data.
Results
Synthetic-data fine-tuning outperforms multi-task learning, enables high-quality ST using only weakly supervised datasets, and unlabeled monolingual data improves performance.
Takeaways & Limitations
High-quality TTS and MT systems can expand ST training data beyond direct parallel corpora, while freezing the pre-trained encoder helps prevent overfitting to synthetic audio.
Takeaways & Limitations
Multi-task learning can limit performance because shared components compromise among tasks and task-specific updates may yield a sub-optimal joint solution.
Abstract
from arXiv · showhide
End-to-end Speech Translation (ST) models have many potential advantages when compared to the cascade of Automatic Speech Recognition (ASR) and text Machine Translation (MT) models, including lowered inference latency and the avoidance of error compounding. However, the quality of end-to-end ST is often limited by a paucity of training data, since it is difficult to collect large parallel corpora of speech and translated transcript pairs. Previous studies have proposed the use of pre-trained components and multi-task learning in order to benefit from weakly supervised training data, such as speech-to-transcript or text-to-foreign-text pairs. In this paper, we demonstrate that using pre-trained MT or text-to-speech (TTS) synthesis models to convert weakly supervised data into speech-to-translation pairs for ST training can be more effective than multi-task learning. Furthermore, we demonstrate that a high quality end-to-end ST model can be trained using only weakly supervised datasets, and that synthetic data sourced from unlabeled monolingual text or speech can be used to improve performance. Finally, we discuss methods for avoiding overfitting to synthetic speech with a quantitative ablation study.
1. INTRODUCTION
End-to-end speech translation avoids several cascade-model costs but is constrained by scarce parallel speech-to-translation data. The paper addresses this bottleneck by scaling weakly supervised training and synthetic data.
- End-to-end ST can avoid ASR–MT error compounding, use prosodic cues, reduce latency, and lower memory and computational requirements.
- Large parallel speech-to-translation datasets are expensive to obtain and often limit end-to-end ST performance.
- Multi-task learning can limit individual-task performance because shared components compromise across tasks and updates are task-specific.
- Using up to 100 million examples, the paper shows synthetic weakly supervised data can improve ST beyond multi-task learning.The experiments include 1.3K hours of translated speech and 49K hours of transcribed speech.
- High-quality ST can be trained without fully supervised data, while synthetic data from unlabeled monolingual text or speech can further improve performance.
2. RELATED WORK
Prior work moved from cascaded ASR–MT systems toward end-to-end ST, using multi-task learning, pre-training, and synthetic data to exploit weak supervision. Figure 1 summarizes the paper’s pre-training and fine-tuning design.
- Early speech translation systems cascaded ASR and MT, whereas recent work increasingly trains a single end-to-end ST model.
- Multi-task learning jointly trains ST with ASR and/or MT, while encoder and decoder pre-training can further improve performance.
- Synthetic TTS data was initially used when direct parallel data was unavailable, but this paper uses high-quality multispeaker TTS to improve ST even with supervised data.
- Figure 1 distinguishes pre-trained, frozen, and ST-fine-tuned components in the end-to-end model.
- MT synthetic data resembles knowledge distillation, whereas TTS-generated speech inputs more closely resemble MT back-translation.
3. MODELS
The paper builds separate ASR, MT, and ST sequence-to-sequence models with shared English/Spanish subword representations. The ST model combines speech encoding with text decoding and attention.
- The system uses separate ASR, MT, and ST sequence-to-sequence models, each composed of an encoder, decoder, and attention module.
- All three models share a 16K-token English/Spanish Word Piece Model for text representation.
- The ASR model uses an acoustic encoder with 80-channel log-mel inputs, while the MT model uses bidirectional encoding and residual decoder connections.
- The best ST model uses an 8-layer bidirectional LSTM encoder, an 8-layer residual unidirectional LSTM decoder, and 8-head additive attention.
4. SYNTHETIC TRAINING DATA
The paper converts weakly supervised text and speech resources into synthetic speech-to-translation training pairs. TTS supplies speech for MT text, while MT supplies translated targets for ASR transcripts.
- Large-scale parallel speech-to-translation data is expensive and its scarcity limits end-to-end ST quality.
- The method synthesizes input speech from parallel MT text and translated targets from ASR transcripts.
- Modern TTS systems can produce natural speech with varied speakers and generate novel voices from speaker embeddings.
- The paper uses a LibriSpeech-trained multispeaker TTS model with Griffin-Lim vocoding and samples speaker embeddings to diversify synthetic voices.
- MT synthesis translates ASR transcripts into the target language using Google Translate, producing final predictions as training targets.
5. EXPERIMENTS
Experiments show that pre-training, multi-task learning, and synthetic data substantially improve end-to-end speech translation, with synthetic mixtures outperforming cascaded models. Freezing the pre-trained encoder and using diverse synthetic speech helps avoid overfitting, while unlabeled monolingual data also provides gains.
- Experimental setup: The experiments use 1M speech-to-translation examples alongside 70M MT sentences and 29M transcribed ASR utterances, evaluated on read and conversational speech.The evaluation includes 10.8K in-domain read-speech examples and 8.9K out-of-domain conversational recordings.
- Baseline models: Pre-training and multi-task learning each improve end-to-end ST, and their combination slightly outperforms the cascaded model.These methods increase the amount of training data seen by two orders of magnitude.
- Encoder adaptation: Freezing the pre-trained ASR encoder improves out-of-domain generalization, with gains saturating after approximately three additional encoder layers.The frozen encoder outperforms the fully trainable model even without extra layers.
- Synthetic training data: Synthetic MT and TTS data substantially improve both evaluation sets, and their mixture produces complementary gains that significantly outperform the cascaded model.MT synthetic data helps out-of-domain performance particularly because natural speech and greater diversity better match that evaluation setting.
- Synthetic training data: A model trained only on synthetic data achieves very good performance and can outperform training with both synthetic and collected data on the out-of-domain set.The result supports training high-quality end-to-end ST using weakly supervised data converted into synthetic parallel examples.
- Synthetic speech robustness: Fully fine-tuning the encoder hurts performance by overfitting to synthetic speech, while single-speaker TTS performs worse than multi-speaker TTS, especially out of domain.Combining real and synthetic speech avoids the single-speaker penalty on the in-domain read-speech evaluation.
- Unlabeled monolingual data: Unlabeled monolingual text or speech can be converted into synthetic speech-to-translation examples that improve a vanilla ST model despite using up to 1M real examples.Synthetic text produces larger gains than synthetic speech, while the latter remains below the cascaded model and the best pre-trained results.
6. CONCLUSIONS
The paper presents weakly supervised synthetic-data training for end-to-end speech translation, combining separate ASR and MT pre-training with synthetic fine-tuning. It reports gains over multi-task learning, robustness measures for synthetic speech, and improvements from unlabeled speech or text.
- The proposed procedure fine-tunes an end-to-end ST model with synthetic training data after separately pre-training its encoder and decoder on ASR and MT.
- The approach outperforms multi-task learning on a large-scale English speech-to-Spanish translation task.
- High-quality multispeaker TTS and a frozen pre-trained encoder help avoid overfitting to synthetic audio.
- The model can be trained using only synthetic data derived from available ASR or MT training sets.
- Unlabeled speech or text can improve an end-to-end ST model when a small fully supervised corpus is available.