Source-linked AI summary
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation
Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, Roi Pomerantz
TL;DR
Direct end-to-end speech-to-speech translation had lagged cascade systems, and the original Translatotron also suffered from over-generation. Translatotron 2 combines a speech encoder, linguistic decoder, acoustic synthesizer, and single attention module, alongside a method for preserving source voices. Across three datasets, it improves translation and speech generation quality over the original and approaches cascade S2ST.
Problem
The original direct S2ST model underperformed cascade baselines and suffered from over-generation, including babbling and long pauses.
Source: §1 · p. 1Method
Results
Across three datasets, Translatotron 2 improves translation and speech generation quality over the original, by up to +15.5 BLEU, and approaches cascade S2ST.
Source: §6 · pp. 8–9Takeaways & Limitations
The voice-preservation method retains each speaker’s voice on turns without speaker segmentation and better preserves privacy while mitigating potential voice-cloning misuse for spoofing.
Source: §6 · p. 9ConcatAug may negatively affect naturalness and speaker similarity for strong models.
Source: §5.4.1 · p. 7
Abstract
from arXiv · showhide
We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a linguistic decoder, an acoustic synthesizer, and a single attention module that connects them together. Experimental results on three datasets consistently show that Translatotron 2 outperforms the original Translatotron by a large margin on both translation quality (up to +15.5 BLEU) and speech generation quality, and approaches the same of cascade systems. In addition, we propose a simple method for preserving speakers' voices from the source speech to the translation speech in a different language. Unlike existing approaches, the proposed method is able to preserve each speaker's voice on speaker turns without requiring for speaker segmentation. Furthermore, compared to existing approaches, it better preserves speaker's privacy and mitigates potential misuse of voice cloning for creating spoofing audio artifacts.
1. Introduction
Translatotron 2 targets the quality gap between direct and cascade speech-to-speech translation, improving on the original model while bringing direct translation closer to cascade performance. It also introduces voice preservation on speaker turns without speaker segmentation.
1. Introduction
- The original Translatotron still lagged cascade baselines in translation quality and produced babbling and long pauses, limiting its production practicality.It nevertheless remained the state of the art among end-to-end direct S2ST systems.Source: §1 · p. 1
- Translatotron 2 is an end-to-end direct S2ST model designed to address the performance gap between direct and cascade systems.The introduction describes conventional S2ST as a cascade of ASR, MT, and TTS, and notes that direct speech-to-text translation had outperformed ASR-plus-MT cascades.Source: §1 · p. 1
- Across three datasets, Translatotron 2 improves translation and speech generation quality over the original and approaches cascade S2ST, with gains up to +15.5 BLEU.With ConcatAug on Fisher Spanish-English, the translation-quality gap is reduced from 16.4 to 0.4 BLEU.Source: §1 · pp. 1–2
- Its voice-preservation method retains each speaker’s voice across speaker turns without speaker representation or speaker separation.The authors also report better speaker privacy and mitigation of potential voice-cloning misuse for spoofing audio.Source: §1 · p. 2
2. Related works
Related work spans cascaded S2ST, direct end-to-end models, and discrete-speech representations, with prior systems trading off training complexity, speech quality, and voice preservation. The section also motivates attention to robustness and potential misuse of voice cloning.
2. Related works
- Translatotron was the first direct S2ST model and supported voice preservation, but it substantially underperformed an ST→TTS cascade.It used a separately trained speaker encoder and showed reasonable translation quality and speech naturalness.Source: §2 · p. 2
- Discrete-speech cascade systems use multiple components and training stages, and may not preserve paralinguistic information when that variation is removed from the representation.Some approaches predict discrete target-speech representations before spectrogram or waveform generation.Source: §2 · p. 2
- Kano et al. (2021) trained an end-to-end S2ST model in multiple steps using pretrained ASR, MT, and TTS models as teachers.The end-to-end system uses a cascade of three autoregressive decoders.Source: §2 · p. 2
- Perceptual speech quality had been evaluated by only a few of the reviewed speech-generation approaches, despite its importance to S2ST.Most of the other works focused on translation quality alone.Source: §2 · p. 2
- Attention-based autoregressive speech synthesis can suffer from over-generation, while duration-based upsampling is presented as a more robust alternative.The synthesizer in this work resembles FastSpeech and Non-Attentive Tacotron.Source: §2 · p. 2
- Voice-conversion quality raises concern about spoofing, since systems have been difficult for automatic speaker verification to detect.The section motivates a voice-preservation method intended to avoid potential misuse for creating spoofing artifacts.Source: §2 · p. 2
3. Translatotron 2
Translatotron 2 combines a speech encoder, linguistic decoder, acoustic synthesizer, and one shared attention module to address alignment and speech-generation robustness bottlenecks. Its shared attention supplies aligned linguistic and acoustic information to the synthesizer.
3. Translatotron 2
- The architecture targets suboptimal auxiliary textual supervision, difficult long-sequence translation alignment, and attention-based speech-generation over- and undergeneration.The authors attribute the primary improvement to the high-level architecture rather than the individual component choices.Source: §3 · pp. 2–3
3.1. Speech encoder
- The encoder converts source mel-spectrograms into hidden representations of linguistic and acoustic information using Conformer blocks.The input is first subsampled by a convolutional layer, and SpecAugment is applied during training.Source: §3.1 · p. 3
3.2. Linguistic decoder
- The autoregressive linguistic decoder predicts target-speech phonemes from encoder output accessed through attention.It uses an LSTM stack with Zoneout and label smoothing regularization.Source: §3.2 · p. 3
3.3. Acoustic synthesizer
- The acoustic synthesizer generates a target mel-spectrogram from decoder features and attention context, using predicted durations to upsample its input sequence.An LSTM stack generates the spectrogram, which a residual convolutional block refines; duration supervision uses utterance-level L2 loss rather than per-phoneme duration labels.Source: §3.3 · pp. 3–4
3.4. A single attention
- A single attention module aligns source spectrograms to the shorter target phoneme sequence and supplies per-phoneme acoustic information to the synthesizer.The shared attention synchronizes linguistic and acoustic information, supporting fine-grained retention of source attributes such as speaker voice on turns; the synthesizer does not query attention.Source: §3.4 · p. 4
4. Voice preserving
The voice-preservation approach restricts the model to retaining the source speaker’s voice rather than cloning an arbitrary target voice. It uses cross-lingual synthesized training targets and concatenated examples to support speaker turns without segmentation.
4. Voice preserving
- The approach restricts voice preservation to the source speaker, addressing potential misuse of arbitrary voice cloning and supporting speaker turns without segmentation.The prior Translatotron could use a reference recording or speaker embedding to generate speech in a different speaker’s voice.Source: §4 · p. 4
4.1. Training-time voice transferring
- Training uses parallel utterances with the same speaker’s voice on both sides, with cross-lingual voice-cloning TTS generating target speech because collecting fluent bilingual recordings is difficult.The target speech is synthesized in the source speaker’s voice.Source: §4.1 · pp. 4–5
4.2. Speaker turns
- ConcatAug creates speaker-turn examples by joining randomly sampled source speech, target speech, and target phoneme sequences from two training examples.The concatenated examples contain two speakers on both the source and target sides; the method can also increase content diversity and acoustic-condition complexity.Source: §4.2 · p. 5
- ConcatAug may improve translation quality on small datasets but may reduce naturalness and speaker similarity for models that already perform strongly.The authors suggest this may relate to less-natural augmented utterances and abrupt changes in volume or background noise at artificial turns.Source: §4.2 · p. 5
5. Experiments
Experiments covered two Spanish-to-English datasets and a multilingual dataset, evaluating translation quality, naturalness, and robustness. On the single-speaker datasets, Translatotron 2 improved translation quality over the original and produced more natural speech, approaching cascade systems.
5.1. Translation quality
- +15.5 BLEU on Fisher Es-En and +5.2 BLEU on Conversational: Translatotron 2 outperformed the original, narrowing the gaps to cascade systems to 0.4 and 3.7 BLEU.ConcatAug added +0.5 BLEU on Fisher Es-En.Source: §5.1 · p. 5, Table 2 · p. 6
5.2. Speech naturalness
- Translatotron 2’s predicted translation speech was significantly more natural than the original Translatotron’s, and was on-par with or very close to cascade systems using Tacotron 2 for synthesizing translation speech.MOS scores were reported with 95% confidence intervals on 1,000 randomly sampled predictions.Source: §5.2 · p. 6
- Using a higher-quality neural vocoder at inference produced more natural speech than the lower-quality Griffin-Lim vocoder used to synthesize Conversational training targets.This result was consistent with Jia et al. (2019b).Source: §5.2 · p. 6
5.3. Speech robustness
Translatotron 2 reduced over-generation, preserved speaker similarity, and—with ConcatAug—handled speaker turns. The results also identify limits: training-target quality constrains speaker similarity, and ConcatAug may reduce naturalness and similarity on strong models.
5.3. Speech robustness
- UDR was about 7 times lower than Translatotron on Fisher Es-En and 4 times lower on Conversational; it was about 3 times lower than Conversational training targets.It was about the same as Fisher Es-En targets; the results also suggest robustness to a small ratio of training-target disfluency.Source: §5.3 · p. 6
5.4. Voice preservation
- With voice preservation, both models had about the same speaker-similarity MOS as the original Translatotron, while Translatotron 2 had better translation quality and naturalness.Speaker similarity was capped by the training targets, which were synthesized; the authors suggest human-recorded targets may improve preservation.Source: §5.4 · p. 7
5.4.1. SPEAKER TURNS
- With ConcatAug, matched speaker pairs scored significantly higher in similarity than mismatched pairs, and Translatotron 2 scored higher than Translatotron on matched pairs.Without ConcatAug, the results suggested that both speakers’ voices were imitated regardless of the turn; listening indicated voice changes at sentence boundaries without a blurry transition.Source: §5.4.1 · p. 7
- On strong models, ConcatAug may reduce speech naturalness and speaker similarity, despite enabling speaker-turn preservation and improving translation quality on small datasets.The authors suggest the artificial turns may sound less natural and include abrupt volume changes or background noise.Source: §5.4.1 · p. 7
5.5. Multilingual S2ST
On four high-resource language pairs, Translatotron 2 improved multilingual X→En translation quality over Translatotron. Ablations indicate that the high-level architecture, rather than any one component choice, primarily accounts for the improvement.
5.5. Multilingual S2ST
- +9.4 BLEU on average across four language pairs: Translatotron 2 outperformed Translatotron on multilingual X→En S2ST.The BLEU scores were not directly comparable between S2ST and ST, though their closeness suggested comparable translation quality to the baseline ST model.Source: §5.5 · p. 8
5.5.1. ABLATION STUDIES
- Replacing individual components or removing SpecAugment reduced Translatotron 2 performance by less than 2 BLEU; using those components in Translatotron did not bring it close to Translatotron 2.The results suggest the high-level architecture, rather than individual component choices, primarily accounts for the improvement.Source: §5.5.1 · p. 8, Table 5 · p. 8
5.6. Non-autoregressive synthesizer
A Conformer-based non-autoregressive synthesizer achieved translation quality comparable to the autoregressive NAT synthesizer, but significantly reduced speech naturalness.
5.6. Non-autoregressive synthesizer
- Up to 1.1 BLEU lower: the Conformer-based non-autoregressive synthesizer had comparable translation quality to the autoregressive NAT synthesizer, but significantly worse naturalness.The authors note that this naturalness regression is consistent with observations in TTS and say the approach needs more exploration.Source: §5.6 · p. 8
6. Conclusion
Translatotron 2 improves on the original in translation and speech generation quality, approaching cascade S2ST, and adds voice preservation across speaker turns. The conclusion also identifies several directions for future work.
6. Conclusion
- Up to +15.5 BLEU, Translatotron 2 outperforms the original in translation quality and approaches cascade S2ST while also improving speech generation quality.Results are reported across three datasets.Source: §6 · pp. 8–9
- The voice-preservation method retains each speaker’s voice across turns without speaker segmentation, while better preserving privacy and mitigating potential voice-cloning misuse.Source: §6 · p. 9
- Future work includes simultaneous translation, cross-lingual prosody transfer, unwritten languages, and further quality improvement using self-supervised pre-training and weakly supervised data.Source: §6 · p. 9
A. Table of hyper-parameters
Table 7 presents the model hyper-parameters used in the experiments and distinguishes pre-net dimensions for single-voice translation and voice preservation.
A. Table of hyper-parameters
- Table 7 lists model hyper-parameters used in the experiments.Source: Table 7 · p. 13
- “×n” denotes n layers in the table’s hyper-parameters.Source: Table 7 · p. 13
- The † pre-net is 128-dimensional for single-voice translation and 16-dimensional for voice preservation.Source: Table 7 · p. 13
B. Objective speaker similarity analysis
Objective d-vector similarity is used to assess voice retention, including on speaker turns. The analysis finds evidence of matched-speaker voice preservation with concatenation augmentation, but cautions that encoder-based comparisons can be unreliable.
B. Objective speaker similarity analysis
- The analysis evaluates voice retention using cosine similarity between d-vectors from separately trained speaker encoders.Source: B. Objective speaker similarity analysis · p. 14
- For single-speaker inputs, higher diagonal d-vector similarities indicate that Translatotron 2 preserves the source speaker’s voice in predicted translations.Source: B. Objective speaker similarity analysis · p. 14
- With concatenation augmentation, d-vector similarity is significantly higher for matched than unmatched speaker pairs, indicating speaker separation and voice preservation in translated speech.Without the augmentation, leading and trailing segments have about the same similarity to each source speaker, indicating a single predicted voice.Source: B. Objective speaker similarity analysis · p. 14
- Cross-model d-vector comparisons can misstate speaker similarity because the encoder may respond to recording conditions such as reverb and channel noise.The authors note that reliability also depends on the speaker encoder’s training details, especially its training corpus, and that objective analysis can be less reliable than subjective MOS.Source: B. Objective speaker similarity analysis · pp. 14–15