Source-linked AI summary
Sequence-to-Sequence Models Can Directly Translate Foreign Speech
Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, Zhifeng Chen
TL;DR
The paper asks whether speech can be translated directly into another language’s text without explicit source transcription. It uses an attention-based recurrent encoder-decoder and reports that the end-to-end model beats an ASR–MT cascade, with further gains from multi-task shared-encoder training.
Problem
The conventional approach pipelines source-language ASR into machine translation, motivating direct end-to-end speech translation.
Method
An attention-based recurrent encoder-decoder predicts target-language text directly from speech, while auxiliary ASR training shares the encoder.
Results
About 1.8 BLEU points separate the end-to-end model from the baseline cascade on Fisher/test, with multi-task training adding 1.4 BLEU points or more on all Fisher datasets.
Takeaways & Limitations
Direct speech-to-text translation matches speech recognition’s computational footprint, and jointly training multiple decoder networks regularizes the encoder and improves translation.
Takeaways & Limitations
A performance gap of about 10 BLEU points remains relative to results assuming perfect ASR, indicating room for acoustic-modeling improvement.
Abstract
from arXiv · showhide
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during training. We apply a slightly modified sequence-to-sequence with attention architecture that has previously been used for speech recognition and show that it can be repurposed for this more complex task, illustrating the power of attention-based models. A single model trained end-to-end obtains state-of-the-art performance on the Fisher Callhome Spanish-English speech translation task, outperforming a cascade of independently trained sequence-to-sequence speech recognition and machine translation models by 1.8 BLEU points on the Fisher test set. In addition, we find that making use of the training data in both languages by multi-task training sequence-to-sequence speech translation and recognition models with a shared encoder network can improve performance by a further 1.4 BLEU points.
1. Introduction
The paper argues that sequence-to-sequence models can directly translate speech into another language’s text, avoiding explicit source transcription. An end-to-end model is motivated by joint optimization, robustness to ASR errors, lower inference latency, and low-resource data use.
- Direct speech translation: Sequence-to-sequence models can directly translate audio in one language into text in another without using source-language text as the target.The model processes log mel filterbank features with a recurrent encoder and predicts target-language text through attention.
- Motivation: End-to-end training jointly adjusts all parameters to optimize the final speech-translation objective.
- Motivation: A cascaded ASR–MT system can combine errors poorly because the translation model is not trained on the ASR errors it receives.
- Motivation: A single end-to-end model can reduce inference latency relative to a cascade and use audio paired with transcripts in another language.The latter advantage is especially relevant to low-resource settings and data such as captioned videos.
- Findings: Experiments on conversational Spanish-to-English speech translation show that source transcripts are unnecessary for training, while multi-task supervision can improve performance.The end-to-end model also outperforms a cascade of independently trained seq2seq ASR and NMT models.
2. Related work
Prior speech-translation systems commonly used ASR outputs or lattices before translation, while newer approaches explored unsupervised or sequence-to-sequence alternatives. This work extends integrated end-to-end modeling to direct translation of real speech.
- Earlier approaches: Early speech-translation approaches supplied ASR lattices to translation models, preserving speech-recognition uncertainty.
- Earlier approaches: Other integrated systems combined acoustic and translation models in stochastic finite-state transducers and decoded translated text with Viterbi search.
- Cascaded systems: Prior Spanish-to-English work improved cascaded speech translation through stronger ASR models and features at the ASR–MT boundary.
- ASR-free alternatives: Recent alternatives avoided ASR by clustering repeated audio patterns or aligning speech with translated text without directly predicting translations.
- Positioning: Compared with similar LAS-like work trained on synthesized speech, this paper trains on a much larger corpus of real speech.
3. Sequence-to-sequence model
The model uses a jointly trained recurrent encoder, attention mechanism, and recurrent decoder to map speech features to output tokens. Multi-task training adds an ASR decoder sharing the encoder, while attention provides soft alignment between input frames and outputs.
- Sequence-to-sequence architecture: A recurrent encoder transforms input feature frames x_1..T into hidden activations h_1..L, optionally at a slower time scale.
- Sequence-to-sequence architecture: The decoder emits one output token per step, conditioned on the previous token and the encoded input sequence.
- Decoder: The decoder is a stacked recurrent network whose LSTM layers emit vectors and update internal states at each time step.
- Attention: Attention summarizes the encoded sequence into a context vector passed to decoder layers, with probabilities defining a soft alignment between input and output sequences.
- Speech model: The speech models use 80-channel log mel filterbank features with delta and delta-delta features, and predict one of 90 output symbols.
- Speech model: The encoder combines convolutional layers, downsampling, convolutional LSTM processing, and bidirectional LSTM layers to produce a 512-dimensional representation.
- Speech model: The decoder combines the previous-token embedding with the attention context and uses stacked unidirectional LSTMs before softmax prediction.
- Multi-task training: Speech translation and recognition models are trained jointly in a multi-task configuration with a shared encoder and separate task supervision.
4. Experiments
Experiments evaluate decoder depth, multi-task sharing, baseline cascades, and end-to-end speech translation on Spanish Fisher and Callhome data. The end-to-end model outperforms the cascade, while multi-task supervision and full encoder sharing further improve performance.
- Experimental setup: Experiments use Spanish Fisher and Callhome telephone conversations, with models trained on the 163-hour Fisher train set and evaluated using WER and BLEU.Spanish transcriptions and English translations are lowercased and stripped of punctuation; BLEU uses four Fisher references and one Callhome reference.
- Tuning decoder depth: Decoder-depth experiments find that speech-translation performance improves as the LSTM decoder grows to four layers, while ASR shows similar gains from deeper decoders.The ASR result suggests decoder architecture merits further investigation in other speech settings.
- Tuning the multi-task model: The one-to-many multi-task strategy, which shares an encoder between speech translation and recognition, performs better than the many-to-one decoder-sharing strategy.Training allocates 75% of steps to speech translation and the remainder to auxiliary ASR.
- Tuning the multi-task model: Sharing all encoder layers yields the best multi-task performance, suggesting a common interlingual subword representation that separate decoders assemble into either language.The sharing comparison varies the final bidirectional LSTM stack while always sharing layers through the convolutional LSTM.
- Speech translation: 1.8 BLEU points separate the end-to-end model from the baseline cascade on Fisher/test, and multi-task training adds 1.4 BLEU points or more across Fisher datasets.The end-to-end model does not use source-language transcripts during training; multi-task training adds them as auxiliary supervision.
- Speech translation: A roughly 10 BLEU-point gap remains between speech translation results and translation from perfect ASR transcripts, indicating substantial room for acoustic-modeling improvement.The baseline MT model also slightly underperforms earlier phrase-based systems, partly because Fisher has less training data and uses character-level targets.
5. Conclusion
The proposed model directly translates speech into text in another language using essentially the same attention-based architecture and computational footprint as speech recognition. It outperforms an ASR-MT cascade without explicitly searching over source-language transcriptions, while multi-language decoder training regularizes the encoder and improves speech translation.
- The model directly translates speech into text in a different language without an explicit source-transcription concept.
- ASR and end-to-end speech translation use the same number of parameters and narrow beam-search decoding algorithm.
- The end-to-end model outperforms an ASR-MT cascade without explicitly searching over source-language transcriptions during decoding.
- Jointly training decoder networks for multiple languages regularizes the encoder and improves overall speech translation performance.