A Comparative Study on End-to-end Speech to Text Translation

Motivation

• End-to-end models translate speech directly into target text, but require paired speech-to-translation data that is difficult to acquire.Proposed responses include multi-task learning, pre-training components, and synthetic data using weakly supervised pairs.Source: §1 · p. 1

Challenges

• Multi-task learning may compromise optimization across tasks, while pre-training and synthetic-data approaches require previously trained models, additional compute, and paired data.Unsupervised systems use independent monolingual speech and text corpora, but their performance trails the other approaches.Source: §1 · p. 1

Contributions

• The study compares end-to-end speech translation methods, tests auxiliary CTC loss, and evaluates pre-training schemes on IWSLT En→De and LibriSpeech En→Fr.It also reports that directly coupling pre-trained encoder and decoder components can hurt performance and presents a way to incorporate them more effectively.Source: §1 · p. 1

2. NETWORK ARCHITECTURES

• End-to-end models learn representations implicitly, unlike cascaded ASR-then-MT systems that use the source sequence as a pivot.The section considers several kinds of end-to-end approaches.Source: §2 · p. 1

2.1. Direct Model

• The direct model encodes speech with bidirectional LSTMs and predicts target tokens with an attention-based LSTM decoder.Three time-dimension max-pooling layers reduce the speech sequence to T′ = T/8; attention weights form a context vector from encoder representations.Source: §2.1 · p. 2

2.2. Mutli-Task Model

• Multi-task variants share a speech encoder or text decoder, while tied cascade and triangle models stack decoders and differ in which states the translation decoder attends to.One-to-many training combines translation and transcription losses with λ = 0.5; the many-to-one model shares a text decoder across speech and text encoders.Source: §2.2 · pp. 2–3

CTC objective

• CTC uses a special blank state that can occur at any time to represent that no token is currently recognized.The blank is modeled as its own token and is distinct from whitespace.Source: §3 · p. 3
• CTC adds a transcription-prediction objective to speech translation training, although translation does not necessarily require monotonic alignment between speech and target sequences.The CTC module is applied on top of the speech encoder during training.Source: §3 · p. 3
• The CTC objective combines the speech-translation and transcription log-likelihoods for the same speech input.CTC predicts source-language transcription tokens, while the decoder predicts target-language tokens.Source: §3 · p. 3

Multi-task use

• In multi-task setups, CTC is also applied to the transcription decoder because the authors expect it to help the ASR subtask.The multi-task objective is rewritten to include the CTC transcription likelihood.Source: §3 · p. 3

4. PRE-TRAINING

• ASR encoder plus MT decoder is a common initialization, while using an ASR model for both ST encoder and decoder also works well.The cited text attributes the latter approach to prior work and the former to.Source: §4 · p. 3
• Here, pre-training means training on a high-resource task and then fine-tuning the model parameters on ST data.Source: §4 · p. 3
• Table 1 presents training data statistics.Source: Table 1 · p. 3

5.1. Datasets and Metrics

• IWSLT uses 390h of speech and extra weakly supervised ASR and MT data, while LibriSpeech uses 200h of speech and ST data only in the multi-task scenario.Source: §5.1 · pp. 3–4
• Evaluation removes punctuation from English transcriptions but retains target punctuation, and reports case-sensitive IWSLT versus case-insensitive LibriSpeech BLEU and TER.The paper notes that the case-insensitive LibriSpeech evaluation is intended for comparability, though the BLEU script used by earlier work is unclear.Source: §5.1 · p. 4, §5.2 · p. 4

5.2. Models

• ST models use six-layer BLSTM encoders, max-pooling for speech, 1024 hidden units, one-layer LSTM decoders, and additive attention.The ASR and MT models also serve as cascade components and as sources for ST pre-training.Source: §5.2 · p. 4
• Training uses Adam with a 0.0008–0.0001 learning rate, dropout of 0.3, label smoothing of 0.1, and beam size 12.The learning rate decays by 0.9 after six consecutive checkpoints; maximum sequence length is 75 sub-words.Source: §5.2 · p. 4
• Table 3 covers MT BLEU and TER using ground-truth source text, while Table 4 covers cascaded ST BLEU and TER.Source: Table 3 · p. 4, Table 4 · p. 4

6. RESULTS

• The cascade baseline reaches 24.43% BLEU and 62.52% TER on IWSLT, and 15.74% BLEU and 70.59% TER on LibriSpeech.Pure MT trained on ground-truth source text scores 31.50% BLEU on IWSLT and 18.22% BLEU on LibriSpeech.Source: §6 · p. 4

6.1. Network Architectures

• On IWSLT, one-to-many gains average 1.98% BLEU and 2.13% TER, and many-to-one gains 3.29% BLEU and 2.61% TER; neither improves LibriSpeech.The IWSLT multi-task models use additional ASR and MT data; the LibriSpeech models use only ST data.Source: §6.1 · p. 4, Table 5 · p. 5
• Tied cascade and triangle underperform direct ST; the authors suggest that extra auxiliary-task parameters are discarded and two stacked decoders make tied-method optimization harder.Cascade models generally perform slightly better than triangle models.Source: §6.1 · pp. 4–5, Table 5 · p. 5

6.2. CTC Loss

• On IWSLT direct ST, CTC improves test results by 1.64% BLEU and 2.09% TER, and dev results by 3.06% BLEU and 3.50% TER; many-to-one performance degrades.The authors report a smaller CTC impact on LibriSpeech.Source: §6.2 · p. 5, Table 5 · p. 5

6.3. Pre-training

• ASR-encoder pre-training boosts the direct model but worsens many-to-one, while pre-training the MT text decoder prevents reasonable performance in all tested cases.The authors attribute the MT-decoder difficulty to its prior training with a text encoder rather than a speech encoder.Source: §6.3 · p. 5, Table 6 · p. 6
• Pre-training both encoder and decoder with ASR yields results above the non-pre-trained baseline in almost all cases; for direct ST, results are comparable to ASR-encoder-only pre-training.The tested pre-training schemes are shown in Table 6.Source: §6.3 · pp. 5–6, Table 6 · p. 6
• For direct ST, jointly trained adapter layers add up to 6.08% BLEU and 7.36% TER on average at most; the approach improves other architectures except tied cascade.The adapter-equipped approach performs better for triangle than tied cascade.Source: §6.3 · p. 6, Table 6 · p. 6
• On IWSLT, the adapter-equipped direct model beats the IWSLT2018 evaluation winner by 0.67% BLEU; on LibriSpeech, it beats LSTM and transformer models but trails knowledge distillation.The comparison uses the authors’ best setup.Source: §6.3 · p. 6, Table 7 · p. 6

• The study evaluates end-to-end speech translation architectures when moderate speech-translation data and weakly supervised ASR or MT pairs are available.Source: §7 · p. 6
• Pre-training network components and adding an intermediate encoder-decoder layer provide performance boosts, with the extra layer allowing better joint learning.Source: §7 · p. 6
• Auxiliary CTC loss can improve end-to-end speech translation performance and speed convergence.Source: §7 · p. 6