PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum Pre-training for End-to-End Speech Translation

Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, Zhenglu Yang

arXiv:2004.10093v1cs.CLcs.SDeess.AS

TL;DR

ASR-based pre-training focuses on transcription, while the encoder also needs linguistic and semantic knowledge for speech translation. The paper proposes a curriculum of transcription, utterance-understanding, and cross-language mapping courses before ST fine-tuning, and reports significant improvements over baselines. It also reports that pre-training with unlabeled speech did not help its final ST task.

  • Problem

    ASR pre-training mainly focuses on transcription and does not capture linguistic knowledge or sentence semantics needed for translation.

    Source: §1 · p. 1

  • Method

    The method pre-trains with ASR for transcription, FMLM for utterance understanding, and FBLT for cross-language word mapping before ST fine-tuning.

    Source: §1 · p. 2

  • Results

    The authors report that curriculum pre-training significantly improves performance and that their model significantly outperforms baselines.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that the courses teach syntactic and semantic knowledge during pre-training and expect curriculum pre-training could be adopted in other NLP tasks.

    Source: §5 · p. 9

    Pre-training with unlabeled speech did not help the final ST task, yielding 16.85 BLEU, below the base model without extra-data pre-training.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

End-to-end speech translation poses a heavy burden on the encoder, because it has to transcribe, understand, and learn cross-lingual semantics simultaneously. To obtain a powerful encoder, traditional methods pre-train it on ASR data to capture speech features. However, we argue that pre-training the encoder only through simple speech recognition is not enough and high-level linguistic knowledge should be considered. Inspired by this, we propose a curriculum pre-training method that includes an elementary course for transcription learning and two advanced courses for understanding the utterance and mapping words in two languages. The difficulty of these courses is gradually increasing. Experiments show that our curriculum pre-training method leads to significant improvements on En-De and En-Fr speech translation benchmarks.

1 Introduction

The paper argues that ASR-focused pre-training leaves the encoder without linguistic and semantic knowledge needed for translation, and proposes progressively harder courses before ST fine-tuning. The authors report significant improvements from this curriculum pre-training.

1 Introduction

  • ASR pre-training mainly teaches transcription and does not capture linguistic knowledge or sentence semantics that are essential for translation.The encoder must transcribe speech, extract source syntax and semantics, and map them to a semantic space for decoding.Source: §1 · p. 1
  • The method uses ASR for transcription, then FMLM and FBLT to teach utterance understanding and cross-language word mapping before ST fine-tuning.FMLM predicts masked words from speech segments; FBLT predicts corresponding target-language words.Source: §1 · p. 2
  • The paper proposes three pre-training courses—transcription, understanding, and mapping—and reports that both advanced courses help speech translation.The authors frame these courses as helping the encoder generate features needed by the decoder.Source: §1 · p. 2
  • The proposed curriculum pre-training leads to significant improvements.Experiments cover LibriSpeech En-Fr and IWSLT18 En-De speech translation tasks.Source: §1 · p. 2

2 Related Work

Prior speech-translation research used cascaded and end-to-end systems and several ways to improve end-to-end performance. This paper distinguishes its approach by applying curriculum learning to pre-training tasks that progressively teach the encoder different kinds of knowledge.

2.1 Speech Translation

  • Cascaded speech translation chains ASR and MT, while end-to-end systems convert source audio directly to target text and avoid error propagation and high latency.The related-work section describes both approaches as responses to speech translation.Source: §2.1 · p. 2
  • Prior end-to-end ST work explored pre-training, multitask learning, attention-passing, and knowledge distillation, but did not explicitly guide the encoder to learn linguistic knowledge.The paper contrasts its pre-training curriculum with these approaches and with encoder stacking, whose gap it describes as hard to bridge.Source: §2.1 · p. 2

2.2 Curriculum Learning

  • Curriculum learning moves from simple patterns to more complex ones, while many prior studies organized training-data difficulty within single-task learning.The cited discussion notes applications across shape recognition, object classification, and question answering.Source: §2.2 · pp. 2–3
  • This method applies curriculum learning to pre-training by increasing task difficulty, rather than by making ST examples or model structures progressively harder.The process has an ASR transcription course followed by FMLM and FBLT courses for understanding and cross-language word mapping.Source: §2.2 · p. 3

3 Method

The method pre-trains the encoder through progressively more linguistically demanding tasks before fine-tuning it for speech translation. Transcription comes first, followed by source-language understanding and cross-lingual word mapping.

3.1 Overview

  • Three stages train the encoder for transcription, source-language understanding, and bilingual word mapping before speech-translation fine-tuning.ASR and FMLM use the bottom N encoder blocks, while FBLT uses all Ne blocks.Source: §3.1 · p. 3

3.2 Elementary Course: Transcription

  • The elementary course trains an end-to-end ASR model whose N encoder blocks initialize the bottom N blocks of the speech-translation encoder.Its objective combines CTC and cross-entropy losses; the CTC loss encourages frame-wise speech-to-word alignment.Source: §3.2 · p. 4

3.3 Advanced Courses: Understanding and Word Mapping

  • The advanced courses add tasks for understanding source-language content and mapping source words to target-language words.FMLM predicts masked source words, while FBLT predicts corresponding target words for speech segments aligned to complete source words.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 4

3.3.1 Frame-based Masked Language Model

  • FMLM masks speech segments aligned to sampled transcript words and trains the encoder to predict the masked words.The task uses a 15% mask ratio, fills masked segments with the utterance mean, and computes its loss at the N-th encoder layer.Source: §3.3.1 · p. 4

3.3.2 Frame-based Bilingual Lexicon Translation

  • FBLT predicts target-language counterparts for speech segments aligned to source words, using word alignments from parallel data or a bilingual lexical translation table.Segments with no target counterpart receive zero loss; when multiple target words correspond, the method uses a KL-divergence loss.Source: §3.3.2 · pp. 4–5

4.1 Data and Preprocess

Experiments use LibriSpeech En-Fr and IWSLT En-De speech-translation data, with task-specific filtering and preprocessing.

LibriSpeech En-Fr

  • LibriSpeech En-Fr uses 100 hours of clean training speech and 90k training instances, with 2,048 test utterances.The training instances combine aligned references with Google Translate references.Source: §4.1 · p. 5

IWSLT En-De

  • IWSLT En-De contains 271 hours of data; filtering low-alignment-quality utterances leaves 137k utterances, and tst2013 supplies 993 test utterances.The development set uses 2,000 segments from the ST-TED corpus.Source: §4.1 · p. 5

Data Preprocessing

  • Preprocessing uses 80-dimensional log-Mel filterbanks plus three-dimensional pitch features, with speed perturbation and speech-transcript forced alignment.Features use a 10 ms step and 25 ms window; utterances longer than 3,000 frames are discarded.Source: §4.1 · p. 5

4.2 Baselines

The experiments compare against ASR-only encoder pre-training and reported end-to-end and cascaded speech-translation baselines. Base and expanded settings distinguish use of task-specific corpora from additional ASR or MT data.

4.2 Baselines

  • The base setting uses only the corpus described for each task, while the expanded setting permits additional ASR and/or MT data.Results are reported in case-insensitive BLEU unless noted.Source: §4.2 · pp. 5–6

4.2.1 End-to-End ST Baselines

  • End-to-end baselines include conventional ASR-only encoder pre-training, alongside selected literature results and task-specific expanded-setting systems.For IWSLT, ESPnet results serve as the base baseline; the multilingual and TCEN-LSTM models are expanded baselines.Source: §4.2.1 · p. 6

4.2.2 Cacased Baselines

  • Cascaded baselines pair ASR and MT systems, with expanded Transformer-based systems also implemented for LibriSpeech and IWSLT.For LibriSpeech, the expanded system uses a pre-trained Transformer ASR model and an LSTM MT model.Source: §4.2.2 · p. 6

4.3 Implementation Details

The implementation uses Transformer models, staged pre-training, and SpecAugment during ASR pre-training.

Model configuration

  • Speech-translation models use 12 Transformer encoder blocks and six decoder blocks, with model dimensions of 256 by default.The expanded LibriSpeech setting uses dimension 512 and eight attention heads.Source: §4.3 · p. 6

Training

  • The model is trained for 50 and 20 epochs in its two pre-training phases, then for another 50 epochs on the final speech-translation task, totaling 120 epochs.Training uses 4 Tesla P40 GPUs, batch size 64 per GPU, and the Adam optimizer.Source: §4.3 · p. 6

Regularization and decoding

  • SpecAugment is applied during ASR pre-training, and decoding uses beam size 10 with length penalty 0.2.The augmentation uses frequency masking F=30, mF=2 and time masking T=40, mT=2.Source: §4.3 · p. 6

4.4 Experimental Results

The curriculum method outperforms end-to-end ASR-pre-training baselines on LibriSpeech En-Fr and IWSLT En-De. Against cascaded systems, it is comparable or better in the base setting, while a gap remains when additional MT data is used.

4.4.1 Comparison with End-to-End Baselines

  • 1.7 BLEU: on LibriSpeech En-Fr, the method improves over the Transformer+ASR pre-train baseline and beats previous works without decoder pre-training.Table 1 compares systems on the LibriSpeech En-Fr test set; its caption identifies 100h as the base-setting ASR data size unless labeled.Source: §4.4.1 · pp. 6–7, Table 1 · p. 7
  • 0.9 to 2.2 BLEU: on IWSLT En-De, the method improves over the ASR-pre-training baseline in the base setting, depending on segmentation.In the expanded setting, combining with decoder pre-training further improves performance and beats other expanded baselines.Source: §4.4.1 · p. 7

4.4.2 Comparison with Cascaded Baselines

  • In the base setting, the end-to-end model achieves comparable or better results than cascaded systems on the two-task comparison.With more ASR data in the LibriSpeech expanded setting, it remains competitive; with additional MT data, a gap from cascaded systems remains.Source: §4.4.2 · p. 7

4.5 Analysis and Discussion

Ablations support the value of both advanced pre-training tasks and the two-phase curriculum, while hyperparameter tests favor N = 8. The explored unlabeled-speech pre-training setup underperformed the base model.

Ablation Study

  • Removing either FMLM or FBLT lowers performance, and the two-phase curriculum is needed; joint ASR, FMLM, and FBLT training also degrades ST performance.Without phase 1 ASR pre-training, advanced-task accuracy drops; without phase 2, the setup reduces to simple ASR pre-training.Source: §4.5 · pp. 7–8, Table 4 · p. 8

Hyper-parameter N

  • N = 8 gives the best performance; N = 6 makes ST training difficult to converge, while N = 10 and N = 12 yield 16.47 and 16.14 BLEU, respectively.The authors attribute the weaker N = 10 and N = 12 results to too few blocks for FBLT, and suggest that N = 6 places valuable source-language knowledge too far from the decoder.Source: §4.5 · p. 8

Unlabeled Speech Data

  • Unlabeled-speech pre-training with masked filterbank prediction did not help: final ST scored 16.85 BLEU, below the base model without extra-data pre-training.The authors leave how to use unlabeled speech as an open question.Source: §4.5 · pp. 8–9

5 Conclusion and Future Work

The paper concludes that curriculum pre-training combines ASR with two frame-based tasks to teach syntactic and semantic knowledge, and reports significant gains over baselines. Future work will explore additional data and applying the approach to other NLP tasks.

Conclusion

  • The curriculum combines ASR loss with frame-based masked language model and bilingual lexicon translation losses to teach syntactic and semantic knowledge during pre-training.The authors report that the model significantly outperforms baselines.Source: §5 · p. 9

Future Work

  • Future work will explore unlabeled speech and large bilingual text data, and whether curriculum pre-training can be adopted for other NLP tasks.Source: §5 · p. 9

Related papers

Loading 2004.10093v1…