Source-linked AI summary
End-to-End Speech Translation with Knowledge Distillation
Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, Chengqing Zong
TL;DR
End-to-end speech translation must combine speech recognition and text translation despite scarce paired speech–target-text data and generally lower performance than text MT. The paper distills knowledge from a text translation teacher into an end-to-end ST student, improving BLEU by 2.72 points in one reported setting and showing gains across similar and dissimilar language pairs. The results support the feasibility of compact end-to-end speech translation while indicating that it remains below the pipeline system in the English-Chinese setting.
Problem
End-to-end speech translation is difficult because paired source speech and target text are scarce, leaving its performance generally inferior to text translation.
Method
The paper trains a text translation model as teacher and trains an end-to-end speech translation model to learn from both ground-truth translations and teacher outputs.
Results
Knowledge distillation increases BLEU from 14.30 to 17.02, and experiments show significant improvements on both English-French and English-Chinese speech translation.
Takeaways & Limitations
Teacher-guided end-to-end speech translation can approach pipeline performance and remains feasible for both similar and dissimilar language pairs.
Takeaways & Limitations
In the English-Chinese experiments, end-to-end speech translation does not outperform the pipeline system.
Abstract
from arXiv · showhide
End-to-end speech translation (ST), which directly translates from source language speech into target language text, has attracted intensive attentions in recent years. Compared to conventional pipeline systems, end-to-end ST models have advantages of lower latency, smaller model size and less error propagation. However, the combination of speech recognition and text translation in one model is more difficult than each of these two tasks. In this paper, we propose a knowledge distillation approach to improve ST model by transferring the knowledge from text translation model. Specifically, we first train a text translation model, regarded as a teacher model, and then ST model is trained to learn output probabilities from teacher model through knowledge distillation. Experiments on English- French Augmented LibriSpeech and English-Chinese TED corpus show that end-to-end ST is possible to implement on both similar and dissimilar language pairs. In addition, with the instruction of teacher model, end-to-end ST model can gain significant improvements by over 3.5 BLEU points.
1. Introduction
End-to-end speech translation avoids the delay, redundancy, and error accumulation of conventional pipelines but typically underperforms because paired speech–target-text data are scarce. The paper addresses this gap by distilling knowledge from a text translation teacher into an end-to-end speech translation student.
- Motivation: Conventional speech translation uses ASR transcripts followed by text MT, introducing time delay, parameter redundancy, and error accumulation.End-to-end ST instead directly generates target translations from raw audio with a compact encoder-decoder model.
- Problem: End-to-end ST generally performs worse because speech paired with target-language text is extremely scarce.Existing pretraining and multi-task approaches provide only limited improvements and do not fully exploit text data.
- Problem: The paper identifies the performance gap between end-to-end ST and text MT as a central opportunity for using MT to guide ST.The authors frame leveraging MT knowledge as important for improving speech translation.
- Approach: Knowledge distillation trains a student to imitate a teacher’s output probabilities, which are smoother and easier to learn than ground-truth text.The paper applies this teacher–student principle to transfer knowledge from text translation to speech translation.
- Contribution: The proposed method trains a text translation teacher on parallel text, then trains an end-to-end ST student using both ground-truth translations and teacher outputs.Experiments on 100h English-French and 542h English-Chinese data show improvements and performance approaching the traditional pipeline.
2. Related Work
Prior work established end-to-end speech-to-text translation through alignment, pretraining, and multi-task learning, while knowledge distillation supplied the paper’s teacher–student foundation. These approaches motivate exploiting text data more directly for speech translation.
- End-to-end speech translation: Earlier studies explored direct speech-to-text translation without source-language transcripts, including audio-pattern clustering and speech–translation alignment.Some work focused on alignment rather than directly predicting final translations.
- Pretraining and multi-task learning: Pretraining and multi-task learning improved end-to-end speech translation, including encoder pretraining on higher-resource ASR data.These studies mainly emphasized acoustic-encoder pretraining rather than fully exploiting text data.
- Knowledge distillation: Knowledge distillation trains a smaller student to mimic a larger teacher by minimizing differences between their predictions.It originated in model compression and later spread to image classification, speech recognition, and natural language processing.
3. Models
The paper uses Transformer-based encoder-decoder models for ASR, speech translation, and machine translation, with speech and text-specific input processing. Knowledge distillation trains the speech translation student to match both ground-truth translations and teacher output probabilities.
- Core Module of Transformer: Transformer provides a shared encoder-decoder architecture for ASR, ST, and MT models.It relies on self-attention, stacked encoder and decoder layers, residual connections, and layer normalization.
- ASR/ST Model: The speech encoder converts raw audio into normalized, stacked acoustic features before Transformer processing.Log-Mel filterbanks, mean-variance normalization, frame stacking, downsampling, linear projection, and positional encodings are applied.
- MT Model: The MT encoder instead receives embedded source-language token sequences, distinguishing it from the ASR/ST speech input.The MT model is a Transformer baseline whose encoder input is a source sentence represented as tokens.
- Knowledge Distillation: The MT model serves as a teacher while the end-to-end ST model serves as a student.The teacher receives the source sentence corresponding to the speech input, and both models produce target-language sentences.
- Knowledge Distillation: The student is trained against ground-truth translations and the teacher’s output distribution through a distillation loss.Teacher probabilities are described as smoother and associated with smaller gradient variance; λ trades off the two loss terms.
4. Experiments
Experiments evaluate ASR, MT, and speech translation on English-French Augmented LibriSpeech and English-Chinese TED data. Knowledge distillation improves end-to-end ST, narrowing its gap with pipeline systems and supporting use across similar and dissimilar language pairs.
- Datasets: The experiments use 100 hours of Augmented LibriSpeech and an English-Chinese TED corpus to evaluate speech translation across language-pair types.The Augmented LibriSpeech training setup uses aligned references and Google Translate references; the TED data tests dissimilar-language translation.
- ASR and MT: Transformer improves Augmented LibriSpeech ASR by 0.92 WER and MT by 4.1 BLEU compared with.The reported gains are attributed to Transformer sequence-to-sequence modeling and the use of subword units.
- English-French ST: The ST evaluation compares pipeline, directly trained end-to-end, pretrained, and knowledge-distilled settings, with MT serving as the KD teacher.The KD setting trains end-to-end ST using instruction from the MT model.
- English-French ST: 2.72 BLEU points separate the KD and non-KD end-to-end models, increasing BLEU from 14.30 to 17.02 on Augmented LibriSpeech.The improvement reduces the gap between end-to-end ST and the pipeline system.
- English-Chinese ST: On English-Chinese, knowledge distillation significantly improves pretrained end-to-end ST, although it does not outperform the pipeline system.The experiment indicates that compact end-to-end ST remains possible for dissimilar language pairs.
- Teacher-model analysis: Increasing the distillation-loss weight λ improves ST performance on Augmented LibriSpeech.λ = 0 corresponds to the pretrained end-to-end model, whereas λ = 1 uses only teacher outputs rather than ground-truth translations.
5. Conclusions
The paper presents knowledge distillation from an MT model to improve end-to-end ST. Across two language pairs, the method yields significant improvements and brings end-to-end performance closer to pipeline systems, though it does not surpass them.
- Knowledge distillation transfers knowledge from an MT model to improve end-to-end ST.
- Experiments on two language pairs show significant improvements in end-to-end ST with MT-model instruction.
- End-to-end ST does not outperform the pipeline system but shows potential to approach its performance.