Source-linked AI summary
Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, shen huang, Qi Ju, Tong Xiao, Jingbo Zhu
TL;DR
Scarce speech-to-translation data and mismatches between ST, ASR, and MT encoders make pretraining challenging. SATE stacks acoustic and textual encoding, with an adaptor and multi-teacher distillation. It achieves comparable or better performance than cascaded ST when large-scale ASR and MT data are available, with reported state-of-the-art BLEU scores on two tasks.
Problem
Speech-to-translation data is scarce, and ST uses an encoder unlike the MT-like encoder assumed by its commonly initialized MT decoder.
Source: §1 · pp. 1–2Method
SATE stacks an acoustic encoder and an MT encoder through an adaptor, with multi-teacher distillation to preserve pretraining knowledge.
Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 4Results
Takeaways & Limitations
The authors conclude that SATE makes it straightforward to incorporate pretrained models into ST while bridging the gap between pretraining and fine-tuning.
Source: §7 · p. 9The adaptor leaves sequence length unchanged despite a length inconsistency; shrinking it risks information incompleteness, an issue left for future work.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
Encoder pre-training is promising in end-to-end Speech Translation (ST), given the fact that speech-to-translation data is scarce. But ST encoders are not simple instances of Automatic Speech Recognition (ASR) or Machine Translation (MT) encoders. For example, we find that ASR encoders lack the global context representation, which is necessary for translation, whereas MT encoders are not designed to deal with long but locally attentive acoustic sequences. In this work, we propose a Stacked Acoustic-and-Textual Encoding (SATE) method for speech translation. Our encoder begins with processing the acoustic sequence as usual, but later behaves more like an MT encoder for a global representation of the input sequence. In this way, it is straightforward to incorporate the pre-trained models into the system. Also, we develop an adaptor module to alleviate the representation inconsistency between the pre-trained ASR encoder and MT encoder, and develop a multi-teacher knowledge distillation method to preserve the pre-training knowledge. Experimental results on the LibriSpeech En-Fr and MuST-C En-De ST tasks show that our method achieves state-of-the-art BLEU scores of 18.3 and 25.2. To our knowledge, we are the first to develop an end-to-end ST system that achieves comparable or even better BLEU performance than the cascaded ST counterpart when large-scale ASR and MT data is available.
1 Introduction
Speech-to-translation data is scarce, and pretrained end-to-end ST had not outperformed cascaded ST when ASR and MT data were much larger. SATE stacks acoustic and textual encoding to use pretrained models together.
1 Introduction
- Speech-to-translation data is scarce, and pretrained end-to-end ST had not outperformed cascaded ST when ASR and MT data were orders of magnitude larger.The comparison is described in the introduction as referring to Table 1; the supplied caption identifies its MuST-C En-De BLEU scores and restricted versus unrestricted settings.Source: §1 · p. 1, Table 1 · p. 1
- ST combines acoustic and textual encoding, so ASR initialization misses translation-wide context while MT-decoder initialization assumes an MT-like encoder.The authors identify these as modeling deficiency and representation inconsistency, respectively.Source: §1 · pp. 1–2
- SATE achieves state-of-the-art results of 18.3 BLEU on LibriSpeech En-Fr and 25.2 on MuST-C En-De, and with large-scale ASR and MT data is comparable to or better than cascaded ST.The authors say it is the first end-to-end system to beat the strong cascaded system in unrestricted speech translation tasks.Source: §1 · p. 2
- SATE cascades an ASR encoder through an adaptor into an MT encoder and uses multi-teacher knowledge distillation to preserve pre-training knowledge during fine-tuning.The adaptor is described as informative and adaptive.Source: §1 · p. 2
2 Related Work
Prior speech translation work used cascaded and end-to-end systems, while scarce paired data prompted approaches including data augmentation, multitask learning, and pretraining.
2 Related Work
- Cascaded ST reuses off-the-shelf ASR and MT systems, but component errors and long-pipeline latency are drawbacks [Ney, 1999; Mathias and Byrne, 2006; Schultz et al., 2004].The passage says cascaded systems were and remain popular in practical ST.Source: §2 · p. 2
- Scarce annotated speech-to-translation data limits end-to-end training; data augmentation can be time-consuming, while multitask learning requires careful losses and more complex architectures [Pino et al., 2019, 2020; Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Berard et al., 2018; Sperber et al., 2019; Dong et al., 2021].Source: §2 · p. 2
- Pretraining commonly initializes the encoder from ASR and the decoder from MT, with variants in training, fine-tuning, shrinking, and adversarial regularization [Berard et al., 2018; Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020a,b; Liu et al., 2020; Alinejad and Sarkar, 2020].The authors say pretrained ST systems still underperform cascaded models, motivating investigation of the reasons and remedies.Source: §2 · p. 2
3 Why is ST Encoding Difficult?
The section shows that ST encoders combine acoustic and textual processing rather than simply substituting for ASR or MT encoders. Its attention behavior and response to CTC placement point to complementary roles for the two types of encoding.
3 Why is ST Encoding Difficult?
- The shared encoding-decoding view treats ST, ASR, and MT as instances of one architecture, allowing components to be pretrained and reused across them.Source: §3 · pp. 2–3
- ST and ASR encoders favor local attention, whereas MT attention is more global for word sequences, especially in deeper layers.The reported localness measure sums attention weights within a small surrounding window, set to 10% of sequence length.Source: §3 · p. 3, Figure 1 · p. 3
- Moving CTC away from upper layers yields more global attention and slightly higher BLEU, while WER increases.The authors report that CTC favors locally attentive models and that only lower layers are then trained in the standard ASR manner.Source: §3 · p. 3
- The authors conclude that acoustic encoding is needed for speech input, while textual encoding produces a representation better suited to the decoder.Source: §3 · p. 3
4 The Method
SATE stacks acoustic and textual encoding to make speech-translation representations decoder-friendly and suitable for pre-training, using an adaptor and multi-teacher distillation.
4 The Method
- The method is designed to produce decoder-friendly representations while keeping the encoder natural for pre-training.SATE is presented as a new speech-translation encoding method addressing these requirements.Source: §4 · p. 3
4.1 Stacked Acoustic-and-Textual Encoding
- SATE uses an acoustic encoder followed by an adaptor and globally attentive textual encoder, with CTC and translation supervision and roles suited to ASR and MT pre-training.The textual encoder's output is fed to the decoder for translation.Source: §4.1 · p. 4
4.2 The Adaptor
- The adaptor fits acoustic-encoder output to the textual encoder using an adaptive soft CTC-based representation and an acoustic-information-preserving mapping.The soft token is the expectation of textual-encoder embeddings over the CTC distribution; the adaptor keeps sequence length unchanged, while shrinking is described as causing information incompleteness and left for future work.Source: §4.2 · p. 5
4.3 Multi-teacher Knowledge Distillation
- Multi-teacher knowledge distillation is introduced to preserve pre-trained knowledge during fine-tuning by training the ST model to mimic teacher distributions.The stated mimicry objective minimizes cross-entropy between teacher and student distributions.Source: §4.3 · p. 5
5 Experiments
Experiments compare SATE with cascaded and end-to-end systems on LibriSpeech En-Fr and MuST-C En-De under restricted and unrestricted data settings. SATE achieves reported BLEU gains over cascaded baselines on both tasks.
5.1 Datasets and Preprocessing
- Restricted experiments use only ST data for ASR and MT; unrestricted experiments add 960 hours of LibriSpeech ASR data and 10M English-French and 18M English-German sentence pairs.The speech-translation datasets are 100 hours for LibriSpeech En-Fr and 400 hours for MuST-C En-De.Source: §5.1 · p. 6
- Speech features are speed-perturbed and represented by 80-channel log-mel filterbanks with pitch; text uses shared-vocabulary BPE with 10,000 merge operations.Utterances longer than 3,000 frames are removed.Source: §5.1 · p. 6
5.2 Model Settings
- SATE uses a 12-layer acoustic encoder and 6-layer textual encoder; unrestricted ASR and ST use wider Conformers, and all models use 6-layer decoders.Inference averages the five best development-set checkpoints and uses beam search with beam size 4; SacreBLEU is case-sensitive.Source: §5.2 · p. 6
5.3 Results
- 1.9 BLEU points over the cascaded baseline on restricted MuST-C En-De, achieving new state-of-the-art performance.The comparison uses SpecAugment for both the cascaded system's ASR model and the compared system.Source: §5.3 · pp. 6–7, Table 2 · p. 7
- 3.7 BLEU points of improvement with SATE in unrestricted MuST-C En-De, with MTKD and SpecAugment reaching 28.1 BLEU points and comparable performance to the cascaded counterpart.The setting uses large-scale ASR and MT data while ST data is scarce.Source: §5.3 · p. 7, Table 2 · p. 7
- 2.0 BLEU points over the cascaded baseline and 18.3 BLEU points for the new state-of-the-art result on restricted LibriSpeech En-Fr; unrestricted SATE exceeds the cascaded counterpart by 0.2 BLEU points.Combining the proposed methods yields the restricted-setting result.Source: §5.3 · p. 7, Table 3 · p. 7
6 Analysis
The analysis compares SATE’s performance and speed with a cascaded system, examines the pre-trained modules and adaptor, and describes how encoding localness differs across layers. It reports a 1.9-BLEU-point performance improvement and up to 1.69× speedup over the cascaded baseline.
6.1 Model Performance vs. Speedup
- 1.9 BLEU points of improvement and up to 1.69× speedup over the cascaded baseline are reported for SATE.For vanilla E2E ST, Table 4 reports a 1.91× inference speedup over the cascaded counterpart, alongside a 0.5 or 0.6 BLEU-point gap with or without pre-training.Source: §6.1 · pp. 7–8, Table 4 · p. 8
6.2 Effects of Pre-trained Modules
- Removing the pre-trained ASR encoder hurts most on noisy MuST-C, while MT pre-training matters more on LibriSpeech and the MT decoder helps more than the encoder.Complete integration requires only slight parameter updates, preserving pre-training knowledge; ASR representations also reduce degradation when textual encoding starts from scratch.Source: §6.2 · p. 8, Table 5 · p. 8
6.3 Effects of The Adaptor
- Fusing soft and mapped adaptor representations achieves the best performance; soft alone shows no obvious MuST-C gain, while mapping alone improves slightly.The comparison is on the development set under the unrestricted setting.Source: §6.3 · p. 8, Table 6 · p. 8
6.4 Impact on Localness
- SATE’s lower layers act like an ASR encoder, while its upper layers capture global representations like an MT encoder.The comparison concerns the localness of the vanilla E2E ST model and pre-trained SATE.Source: Figure 4 · p. 8, §6.4 · p. 9
7 Conclusion
The conclusion presents SATE as a way to incorporate pre-trained models into speech translation, with an adaptor and multi-teacher distillation to bridge pre-training and fine-tuning. Results on LibriSpeech and MuST-C show superiority over comparison systems and comparable or better performance than cascaded ST when large-scale ASR and MT data is available.
7 Conclusion
- SATE stacks acoustic and textual encoding to incorporate pre-trained models into speech translation.The paper also introduces an adaptor and multi-teacher knowledge distillation to bridge pre-training and fine-tuning.Source: §7 · p. 9
- Results on LibriSpeech and MuST-C demonstrate the method’s superiority over comparison systems.Source: §7 · p. 9
- With large-scale ASR and MT data available, the method achieves performance comparable to or better than the cascaded counterpart.Source: §7 · p. 9