Direct speech-to-speech translation with a sequence-to-sequence model

Ye Jia; Ron J. Weiss; Fadi Biadsy; Wolfgang Macherey; Melvin Johnson; Zhifeng Chen; Yonghui Wu

Source-linked AI summary

Direct speech-to-speech translation with a sequence-to-sequence model

Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, Yonghui Wu

arXiv:1904.06037v2Apr 12, 2019cs.CLcs.LGcs.SDeess.AS

TL;DR

Speech-to-speech translation traditionally relies on intermediate text, motivating a direct end-to-end alternative. The paper introduces Translatotron, which predicts translated speech spectrograms and can optionally transfer the source speaker’s voice. Across two Spanish-to-English datasets, it achieves high translation quality but remains below a cascaded ST-and-TTS baseline.

Problem
The paper asks whether speech can be translated directly between languages without an intermediate text representation, rather than using cascaded ASR, MT, and TTS components.
Source: §1 · p. 1
Method
Translatotron is an end-to-end attention-based sequence-to-sequence model that predicts translated speech spectrograms, using transcription-based multitask training and optional speaker conditioning.
Source: §1 · p. 1, §2 · p. 2
Results
Translatotron achieves high translation quality on two Spanish-to-English datasets but does not perform as well as a baseline cascade of speech translation and text-to-speech models.
Source: §3.1 · p. 3, §3.2 · p. 3, §4 · p. 4
Takeaways & Limitations
The results demonstrate the feasibility of direct end-to-end speech-to-speech translation as a proof of concept and starting point for future research.
Source: §1 · p. 1, §4 · p. 4
Future work is needed to reduce the requirement for speech transcripts during training and improve voice transfer performance.
Source: §4 · p. 4

Abstract

from arXiv · show

We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. The network is trained end-to-end, learning to map speech spectrograms into target spectrograms in another language, corresponding to the translated content (in a different canonical voice). We further demonstrate the ability to synthesize translated speech using the voice of the source speaker. We conduct experiments on two Spanish-to-English speech translation datasets, and find that the proposed model slightly underperforms a baseline cascade of a direct speech-to-text translation model and a text-to-speech synthesis model, demonstrating the feasibility of the approach on this very challenging task.

1. Introduction

The paper investigates direct speech-to-speech translation without an intermediate text representation, contrasting it with conventional cascades. It presents Translatotron1 as an end-to-end proof of concept despite substantial data and alignment challenges.

Speech-to-speech translation converts speech in one language into speech in another to help break communication barriers.Source: §1 · p. 1
Conventional systems commonly cascade automatic speech recognition, text-to-text machine translation, and text-to-speech synthesis.Source: §1 · p. 1
Direct models avoid intermediate error compounding and require one decoding step instead of three, reducing computational requirements and inference latency.Source: §1 · p. 1
Direct translation can preserve paralinguistic and nonlinguistic information, including speaker voice, emotion, and prosody.Source: §1 · p. 1
End-to-end training is difficult because paired speech data are harder to collect and translated spectrograms have uncertain alignment.Source: §1 · p. 1
Translatotron1 uses transcription-based multitask training without text during inference, but performs below a cascaded baseline while demonstrating feasibility.Source: §1 · p. 1

2. Speech-to-speech translation model

Translatotron combines an attention-based sequence-to-sequence spectrogram decoder with a vocoder and optional speaker conditioning. Auxiliary phoneme prediction tasks support training but are excluded from inference.

The architecture contains an attention-based sequence-to-sequence network, a vocoder, and an optional pretrained speaker encoder.The speaker encoder enables cross-language voice conversion alongside translation.Source: §2 · pp. 1–2
The encoder maps 80-channel log-mel spectrograms to hidden states, while the autoregressive decoder predicts 1025-dimensional log-spectrogram frames.Source: §2 · p. 2
Eight bidirectional LSTM encoder layers pass final outputs to the primary decoder and intermediate activations to auxiliary phoneme decoders.Source: §2 · p. 2
The spectrogram decoder uses Tacotron 2-like components with four-head additive attention and a 32-dimensional pre-net bottleneck.Source: §2 · p. 2
Auxiliary source and target phoneme recognition losses help the primary decoder learn attention during training, but auxiliary decoders are not used during inference.Source: §2 · p. 2, Figure 1 · p. 2
Experiments primarily use the low-complexity Griffin-Lim vocoder, while WaveRNN is used for speech-naturalness listening tests.Source: §2 · p. 2

3. Experiments

Experiments on conversational and Fisher Spanish-to-English datasets show that auxiliary phoneme supervision is important for direct S2ST, while performance remains below the cascade baseline. WaveRNN substantially improves perceived naturalness, and source-speaker conditioning enables voice transfer with trade-offs in BLEU and similarity.

3.1. Conversational Spanish-to-English

979k parallel utterance pairs provide 1.4k hours of source speech and 619 hours of synthesized target speech in the conversational dataset.The test set contains 9.6k held-out pairs.Source: §3.1 · p. 3
Without auxiliary losses, the model often generates plausible target-voice speech unrelated to the input instead of translating complete utterances.This behavior is consistent with failing to learn attention to the input.Source: §3.1 · p. 3
Auxiliary phoneme recognition regularizes the encoder and enables attention; using both source and target losses achieves the best quality, but remains 6 BLEU points below the baseline.The performance gap is described as demonstrating potential while leaving room for improvement.Source: §3.1 · p. 3

3.2. Fisher Spanish-to-English

On the smaller Fisher task, two auxiliary losses provide the strongest performance, while source-only supervision performs very poorly because translation alignment is especially difficult.The Fisher corpus is smaller and more challenging because it contains spontaneous, informal speech.Source: §3 · p. 2, §3.2 · p. 3
Over 5 BLEU points from pre-training the bottom six encoder layers makes this the best direct S2ST model, reaching 76% of baseline performance.Four-head attention also outperforms one-head attention on Fisher, unlike on the conversational task.Source: §3.2 · p. 3

3.3. Subjective evaluation of speech naturalness

WaveRNN vocoders raise naturalness ratings into the “Very Good” range above 4.0, although S2ST models still score below the synthetic ground-truth targets.Naturalness was evaluated using 5-point MOS listening tests with 1k examples per dataset.Source: §3.3 · pp. 3–4, Table 4 · p. 3

3.4. Cross language voice transfer

Source-speaker conditioning trails the alternative conditioning settings by 1.8 BLEU points and reduces similarity MOS by 1.4 points, while naturalness MOS remains close.The authors attribute the BLEU difference to a mismatch between training and inference conditioning languages.Source: §3.4 · p. 4

4. Conclusions

The paper presents an end-to-end direct speech-to-speech translation model that achieves high translation quality but remains below a cascaded baseline. It also demonstrates source-speaker voice transfer and identifies training and prosody-transfer strategies for future work.

The model achieves high translation quality on two Spanish-to-English datasets, although it performs below a baseline cascade of speech-to-text translation and text-to-speech models.Source: §4 · p. 4
The voice-transfer variant simultaneously transfers the source speaker’s voice to translated speech.Source: §4 · p. 4
Future work includes weak supervision or multitask learning for scaling training and transferring prosody and other acoustic factors.Source: §4 · p. 4