PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end Speech Translation via Cross-modal Progressive Training

Rong Ye, Mingxuan Wang, Lei Li

arXiv:2104.10380v2cs.CL

TL;DR

Limited parallel speech-text data remains a challenge for end-to-end speech translation. XSTNet jointly trains speech translation, ASR, and MT with speech or text inputs, and improves on the previous state-of-the-art method by 3.2 BLEU.

  • Problem

    End-to-end speech translation has limited parallel speech-text data, and effectively using unlabeled or other parallel MT corpora remains an open problem.

    Source: §1 · p. 1

  • Method

    XSTNet accepts speech or text input and jointly trains ST, ASR, and MT using a self-supervised Wav2vec2.0 audio encoder and progressive multi-task training.

    Source: §1 · p. 1, §3.4 · p. 2

  • Results

    XSTNet improves on the previous state-of-the-art method by 3.2 BLEU.

    Source: §1 · p. 1

  • Takeaways & Limitations

    XSTNet achieves a significant improvement on the speech-to-text translation task compared with state-of-the-art models.

    Source: §6 · p. 4

    The external MT corpus used is much larger than the speech-translation corpus.

    Source: §3.1 · p. 1

Abstract

from arXiv · show

End-to-end speech translation models have become a new trend in research due to their potential of reducing error propagation. However, these models still suffer from the challenge of data scarcity. How to effectively use unlabeled or other parallel corpora from machine translation is promising but still an open problem. In this paper, we propose Cross Speech-Text Network (XSTNet), an end-to-end model for speech-to-text translation. XSTNet takes both speech and text as input and outputs both transcription and translation text. The model benefits from its three key design aspects: a self-supervised pre-trained sub-network as the audio encoder, a multi-task training objective to exploit additional parallel bilingual text, and a progressive training procedure. We evaluate the performance of XSTNet and baselines on the MuST-C En-X and LibriSpeech En-Fr datasets. In particular, XSTNet achieves state-of-the-art results on all language directions with an average BLEU of 28.8, outperforming the previous best method by 3.2 BLEU. Code, models, cases, and more detailed analysis are available at https://github.com/ReneeYe/XSTNet.

1. Introduction

End-to-end speech translation faces limited parallel speech-text data, motivating XSTNet’s joint training of speech translation, ASR, and MT. The model improves on the previous state-of-the-art method by 3.2 BLEU.

  • Limited parallel speech-text data makes end-to-end speech translation training challenging, with only a few hundred hours for English-German in MuST-C.Source: §1 · p. 1
  • XSTNet jointly trains ST, ASR, and MT, using a self-supervised Wav2vec2.0 audio encoder, external MT data, and progressive multi-task training.It supports either audio or text input and shares a Transformer module.Source: §1 · p. 1
  • XSTNet improves on the previous state-of-the-art method by 3.2 BLEU.Source: §1 · p. 1

2. Related Work

Prior work has advanced end-to-end speech translation through neural sequence models and training strategies including pretraining, multi-task learning, self-training, and meta-learning. The closest related work combines two pretrained modules, while this paper emphasizes training strategy.

  • End-to-end ST demonstrated its potential without intermediate transcription, and recent Seq2Seq models have reported impressive results.Source: §2 · p. 1
  • Pretraining, multi-task learning, self-training, and meta-learning have further improved end-to-end ST performance.Source: §2 · p. 1

Self-supervised Pretraining

  • The most related work incorporates two pretrained modules for multilingual ST, whereas this work focuses more on training strategy.Source: §2 · p. 1

3. Proposed Method: XSTNet

XSTNet uses aligned speech, transcript, and translation data to train speech translation, recognition, and text translation together. Its design combines a speech encoder and shared Transformer with progressive training that uses external MT data in both stages.

3.1. Problem Formulation

  • Each speech-translation example is a speech-transcript-translation triple, pairwise decomposed into ASR, speech-translation, and MT datasets; external MT pairs are much larger than the in-domain corpus.The external MT dataset contains source and target text pairs.Source: §3.1 · p. 1

3.2. Speech Encoder

  • Wav2vec2.0 processes raw speech, and two stride-2 convolution layers reduce its representation's time dimension fourfold to match text-sequence lengths.The resulting audio embedding has the Transformer hidden size.Source: §3.2 · pp. 1–2

3.3. Encoder-Decoder with Modality and Language Indicators

  • XSTNet uses a text-token lookup table and a shared Transformer encoder-decoder for speech and text inputs, with indicators distinguishing tasks and modalities.Audio receives an [audio] token, while text inputs are prefixed with a language ID; the decoder's initial language ID specifies the output language.Source: §3.3 · p. 2, Figure 1 · p. 2

3.4. Progressive Multi-task Training

  • External MT data first pre-trains the Transformer, then remains in fine-tuning alongside speech translation, recognition, and in-domain MT data for joint negative-log-likelihood optimization.The procedure is progressive because external MT data continues from pre-training into fine-tuning.Source: §3.4 · p. 2

4. Experiments

Experiments evaluate XSTNet on MuST-C and LibriTrans, with comparisons spanning end-to-end systems, auxiliary MT and ASR tasks, and cascaded models. The reported results include gains over speech-transformer baselines and better translation than the compared cascades.

4.1. Datasets

  • MuST-C covers English speech translated into eight languages, while LibriTrans provides an En-Fr test set for evaluation.MuST-C is evaluated on tst-COMMON; LibriTrans is evaluated on its test set.Source: §4.1 · p. 2

4.2. Experimental Setups

  • Setup uses jointly tokenized En/X subwords with a 10k vocabulary, averages the last 10 best-dev BLEU checkpoints, decodes with beam 10, and compares eight end-to-end baselines plus W-Transf.MuST-C scores use case-sensitive detokenized BLEU; LibriTrans reports both uncased-tokenized and cased-detokenized BLEU.Source: §4.2 · pp. 2–3, §4.3 · p. 3

4.3. Main Results

  • +3.9 BLEU on average over the current speech transformer; XSTNet (Base) gains around +2 BLEU over W-Transf. across directions, while Expand adds +1.3 over Base.The authors attribute the additional +1.3 BLEU to external MT data and the progressive training procedure.Source: Table 1 · p. 3, Table 2 · p. 3, §4.3 · p. 3

4.4. Results on Auxiliary MT and ASR Tasks

  • Progressive training outperforms other strategies on auxiliary MT and ASR; adding speech-to-text data further improves MT, which the authors attribute to speech information.Table 3 compares auxiliary MT strategies, and Table 4 reports auxiliary ASR word error rate.Source: §4.4 · p. 3, Table 3 · p. 3, Table 4 · p. 3

4.5. Comparsion with Cascaded Baselines

  • XSTNet-Expand translates better than the compared Transformer ASR-to-MT cascades, which the authors say avoids the error-propagation problem of cascaded models.One strong cascade uses W-Transf. for ASR and a Transformer-base MT model trained on DMT ∪ DMT-ext.Source: §4.5 · p. 3

5. Analysis

External MT pre-training and continued progressive use of WMT improve training outcomes, while multi-task training converges faster and generalizes better; increasing external MT data also raises BLEU scores.

5.1. The Influence of Training Procedure

  • Removing external WMT pre-training reduced average performance by 1.2 BLEU; retaining WMT during fine-tuning and multi-task fine-tuning also yielded better BLEU.Source: Table 5 · p. 4, §5.1 · p. 4

5.2. Convergence Analysis

  • WMT-pretrained progressive multi-task XSTNet converges faster than the model without pre-training, while multi-task training is described as more robust with better generalization under small data.Source: Figure 2 · p. 4, §5.2 · p. 4

5.3. Influence of Additional MT Data

6. Conclusion

XSTNet accepts speech and text inputs and jointly trains speech translation, ASR, and MT; the paper also proposes progressive multi-task training and reports significant improvement over state-of-the-art models.

  • XSTNet accepts bi-modal inputs and jointly trains ST, ASR, and MT, using a progressive multi-task training algorithm.Source: §6 · p. 4
Loading 2104.10380v2…