Source-linked AI summary
Direct speech-to-speech translation with a sequence-to-sequence model
Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, Yonghui Wu
TL;DR
Speech-to-speech translation traditionally relies on intermediate text, motivating a direct end-to-end alternative. The paper introduces Translatotron, which predicts translated speech spectrograms and can optionally transfer the source speaker’s voice. Across two Spanish-to-English datasets, it achieves high translation quality but remains below a cascaded ST-and-TTS baseline.
Problem
The paper asks whether speech can be translated directly between languages without an intermediate text representation, rather than using cascaded ASR, MT, and TTS components.
Method
Translatotron is an end-to-end attention-based sequence-to-sequence model that predicts translated speech spectrograms, using transcription-based multitask training and optional speaker conditioning.
Results
Translatotron achieves high translation quality on two Spanish-to-English datasets but does not perform as well as a baseline cascade of speech translation and text-to-speech models.
Takeaways & Limitations
The results demonstrate the feasibility of direct end-to-end speech-to-speech translation as a proof of concept and starting point for future research.
Takeaways & Limitations
Future work is needed to reduce the requirement for speech transcripts during training and improve voice transfer performance.
Abstract
from arXiv · showhide
We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. The network is trained end-to-end, learning to map speech spectrograms into target spectrograms in another language, corresponding to the translated content (in a different canonical voice). We further demonstrate the ability to synthesize translated speech using the voice of the source speaker. We conduct experiments on two Spanish-to-English speech translation datasets, and find that the proposed model slightly underperforms a baseline cascade of a direct speech-to-text translation model and a text-to-speech synthesis model, demonstrating the feasibility of the approach on this very challenging task.
1. Introduction
The paper investigates direct speech-to-speech translation without an intermediate text representation, contrasting it with conventional cascades. It presents Translatotron1 as an end-to-end proof of concept despite substantial data and alignment challenges.
- Speech-to-speech translation converts speech in one language into speech in another to help break communication barriers.
- Conventional systems commonly cascade automatic speech recognition, text-to-text machine translation, and text-to-speech synthesis.
- Direct models avoid intermediate error compounding and require one decoding step instead of three, reducing computational requirements and inference latency.
- Direct translation can preserve paralinguistic and nonlinguistic information, including speaker voice, emotion, and prosody.
- End-to-end training is difficult because paired speech data are harder to collect and translated spectrograms have uncertain alignment.
- Translatotron1 uses transcription-based multitask training without text during inference, but performs below a cascaded baseline while demonstrating feasibility.
2. Speech-to-speech translation model
Translatotron combines an attention-based sequence-to-sequence spectrogram decoder with a vocoder and optional speaker conditioning. Auxiliary phoneme prediction tasks support training but are excluded from inference.
- The architecture contains an attention-based sequence-to-sequence network, a vocoder, and an optional pretrained speaker encoder.The speaker encoder enables cross-language voice conversion alongside translation.
- The encoder maps 80-channel log-mel spectrograms to hidden states, while the autoregressive decoder predicts 1025-dimensional log-spectrogram frames.
- Eight bidirectional LSTM encoder layers pass final outputs to the primary decoder and intermediate activations to auxiliary phoneme decoders.
- The spectrogram decoder uses Tacotron 2-like components with four-head additive attention and a 32-dimensional pre-net bottleneck.
- Auxiliary source and target phoneme recognition losses help the primary decoder learn attention during training, but auxiliary decoders are not used during inference.
- Experiments primarily use the low-complexity Griffin-Lim vocoder, while WaveRNN is used for speech-naturalness listening tests.
3. Experiments
Experiments on conversational and Fisher Spanish-to-English datasets show that auxiliary phoneme supervision is important for direct S2ST, while performance remains below the cascade baseline. WaveRNN substantially improves perceived naturalness, and source-speaker conditioning enables voice transfer with trade-offs in BLEU and similarity.
- 3.1. Conversational Spanish-to-English: 979k parallel utterance pairs provide 1.4k hours of source speech and 619 hours of synthesized target speech in the conversational dataset.The test set contains 9.6k held-out pairs.
- 3.1. Conversational Spanish-to-English: Without auxiliary losses, the model often generates plausible target-voice speech unrelated to the input instead of translating complete utterances.This behavior is consistent with failing to learn attention to the input.
- 3.1. Conversational Spanish-to-English: Auxiliary phoneme recognition regularizes the encoder and enables attention; using both source and target losses achieves the best quality, but remains 6 BLEU points below the baseline.The performance gap is described as demonstrating potential while leaving room for improvement.
- 3.2. Fisher Spanish-to-English: On the smaller Fisher task, two auxiliary losses provide the strongest performance, while source-only supervision performs very poorly because translation alignment is especially difficult.The Fisher corpus is smaller and more challenging because it contains spontaneous, informal speech.
- 3.2. Fisher Spanish-to-English: Over 5 BLEU points from pre-training the bottom six encoder layers makes this the best direct S2ST model, reaching 76% of baseline performance.Four-head attention also outperforms one-head attention on Fisher, unlike on the conversational task.
- 3.3. Subjective evaluation of speech naturalness: WaveRNN vocoders raise naturalness ratings into the “Very Good” range above 4.0, although S2ST models still score below the synthetic ground-truth targets.Naturalness was evaluated using 5-point MOS listening tests with 1k examples per dataset.
- 3.4. Cross language voice transfer: Source-speaker conditioning trails the alternative conditioning settings by 1.8 BLEU points and reduces similarity MOS by 1.4 points, while naturalness MOS remains close.The authors attribute the BLEU difference to a mismatch between training and inference conditioning languages.
4. Conclusions
The paper presents an end-to-end direct speech-to-speech translation model that achieves high translation quality but remains below a cascaded baseline. It also demonstrates source-speaker voice transfer and identifies training and prosody-transfer strategies for future work.
- The model achieves high translation quality on two Spanish-to-English datasets, although it performs below a baseline cascade of speech-to-text translation and text-to-speech models.
- The voice-transfer variant simultaneously transfers the source speaker’s voice to translated speech.
- Future work includes weak supervision or multitask learning for scaling training and transferring prosody and other acoustic factors.