Source-linked AI summary
Direct speech-to-speech translation with discrete units
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, Wei-Ning Hsu
TL;DR
Direct S2ST seeks to translate speech without an intermediate text sequence, including when languages lack writing systems. The paper predicts self-supervised discrete target-speech units with an S2UT model and adds joint text decoding when transcripts are available, achieving comparable performance to cascaded systems and supporting transcript-free translation.
Problem
Direct S2ST must translate speech without intermediate text while addressing the modeling challenge posed by cross-lingual alignment and target acoustic and linguistic characteristics.
Method
The model predicts HuBERT-based discrete representations of target speech with transformer S2UT and optionally jointly decodes target text using CTC.
Results
The proposed direct S2ST system performs on par with S2T+TTS and is reported as the most efficient during inference among the compared systems.
Takeaways & Limitations
Discrete-unit prediction supports direct speech translation for written and unwritten-language scenarios while enabling consistent speech and text output in one inference pass.
Takeaways & Limitations
The work focuses on content translation and leaves para-linguistic aspects such as voice and prosody for future work.
Abstract
from arXiv · showhide
We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We tackle the problem by first applying a self-supervised discrete speech encoder on the target speech and then training a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the discrete representations of the target speech. When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass. Experiments on the Fisher Spanish-English dataset show that the proposed framework yields improvement of 6.7 BLEU compared with a baseline direct S2ST model that predicts spectrogram features. When trained without any text transcripts, our model performance is comparable to models that predict spectrograms and are trained with text supervision, showing the potential of our system for translation between unwritten languages. Audio samples are available at https://facebookresearch.github.io/speech_translation/direct_s2st_units/index.html .
1 Introduction
Direct S2ST translates speech without intermediate text, addressing the modeling difficulty of target speech while supporting written and unwritten language scenarios. The proposed discrete-unit framework combines target-speech representations, joint speech-text training when transcripts exist, and transcript-free multitask learning.
- Motivation: Direct S2ST avoids intermediate text generation but must model cross-lingual alignment alongside the acoustic and linguistic characteristics of both languages.This creates a performance gap relative to cascaded S2T+TTS systems.
- Discrete units: Self-supervised discrete target-speech units replace mel-spectrogram features and can separate linguistic content from speaker identity or prosodic information.Their discrete form also permits standard sequence-modeling techniques such as beam search.
- Joint training: For written target languages, joint S2ST and S2T training generates speech and text through shared encoder and partially shared decoder components.CTC addresses the length mismatch between speech and text outputs.
- Unwritten languages: For unwritten-language scenarios, the framework trains direct S2ST without text transcripts using discrete representations for both source and target speech.The paper also investigates text-to-speech translation when source text transcripts are available.
2 Related work
Prior work largely combines speech translation with TTS or directly predicts target spectrograms, while discrete-unit approaches target lower-latency and unwritten-language settings. This paper uses HuBERT-based units with transformer S2UT and text decoding.
- Existing systems: Conventional S2ST systems combine cascaded or end-to-end S2T models with TTS, while most speech-translation research focuses on S2T.
- Direct S2ST: Translatotron directly maps source speech mel-spectrograms to target speech spectrogram features but retains a performance gap toward cascaded S2T+TTS.
- Unwritten languages: Some prior systems use synthetic target speech and discrete representations for direct S2ST under unwritten-language conditions.The cited prior work uses VAE-derived discrete representations.
- This work: The proposed approach trains a transformer speech-to-discrete-unit model, adds text decoding from the unit decoder, and selects HuBERT for target unit generation.
- Data constraints: Direct S2ST research remains limited partly because parallel S2ST training data are scarce, motivating experiments with synthetic single-speaker target speech.
3 Model
The model translates source speech into self-supervised discrete target units, optionally producing text jointly, and separately vocodes units into waveform. It uses HuBERT clustering, auxiliary training tasks, and two unit-decoding strategies.
- System architecture: The proposed system combines a transformer speech encoder, discrete unit decoder, auxiliary training tasks, optional target-text CTC decoding, and a separately trained vocoder.The CTC decoder enables joint speech and text generation for written target languages.
- Unit representation: HuBERT and k-means encode target speech as cluster-index sequences sampled every 20 ms from K learned centroids.An utterance is represented as [z1, z2, ..., zT], where T is the number of frames.
- Unit decoding: The stacked strategy predicts r consecutive units per decoding step with a K × r output, whereas the reduced strategy removes consecutive duplicate units.Both strategies reduce training and inference time.
- Auxiliary tasks: Auxiliary attention and decoder modules use encoder intermediate layers during training, with targets including phonemes, characters, subwords, or discrete speech representations.These auxiliary tasks are omitted during inference.
- Text decoding: Target-text CTC decoding conditions on an intermediate discrete-unit-decoder layer, mitigating speech-text length mismatch while relying on transformer layers for reordering.
- Waveform synthesis: A modified HiFi-GAN vocoder converts discrete units into waveform, with reduced outputs additionally using a lightweight duration-prediction module.
4 Experiments
Experiments on Fisher Spanish-English evaluate data preparation, auxiliary-task choices, translation quality, speech naturalness, and inference efficiency. Across written and unwritten settings, discrete-unit S2ST improves translation quality and efficiency relative to spectrogram-based or cascaded alternatives.
- Data and setup: The Fisher Spanish-English experiments use synthetic target speech and HuBERT-derived discrete units, with source phonemes, target phonemes, source characters, or target characters as auxiliary targets.The corpus contains approximately 139k sentences and 170 hours of telephone conversations; HuBERT units are extracted with K = 100 clustering.
- Written target language: 7 BLEU gain: character auxiliary targets outperform phonemes for transformer Translatotron on the test set (4 vs. 5).Characters are used as auxiliary-task targets in subsequent experiments.
- Written target language: 1.2 BLEU: S2UT stacked outperforms transformer Translatotron on the test set (5 vs. 7), while a larger reduction factor speeds training and inference without hurting performance (6 vs. 7).The comparison supports discrete units as easier translation targets than continuous mel-spectrogram features.
- Written target language: 1.2 BLEU gain: S2UT stacked with joint discrete-unit and text CTC training improves dev-set performance (7 vs. 8), while test-set performance remains unchanged.Reduced decoding is more effective than stacked; beam-size-1 decoding improves speech output by 1.4 BLEU and text output by 1.2 BLEU on the test set.
- Written target language: 63% of the gap: the best beam-searched S2UT reduced system bridges this gap between transformer Translatotron and the three-stage cascade, while performing on par with S2T+TTS.It generates consistent speech and text in one inference pass.
- Speech quality: The proposed S2UT reduced system has MOS close to synthetic target speech, while systems predicting all frames tend to have slightly lower MOS.The synthetic target MOS is treated as an upper bound because the model is trained on that speech.
- Unwritten target language: 15.1 BLEU: ASR+T2UT outperforms ASR+T2ST on the test set (15 vs. 16), and exceeds S2T+TTS by 0.8 BLEU (3 vs. 16).These results provide a discrete-unit alternative for two-stage cascaded systems.
5 Conclusion
The paper investigates direct S2ST with self-supervised discrete representations for written and unwritten language scenarios, including joint speech-text training for written languages. It reports efficient inference and identifies real-data evaluation and para-linguistic speech generation as future directions.
- The proposed framework uses self-supervised discrete representations as targets for direct S2ST in both written and unwritten language scenarios.
- For written languages, joint speech and text training performs on par with an S2T+TTS baseline while running more efficiently.
- The framework demonstrates the possibility of translation between two unwritten languages using discrete representations of source and target speech.
- The proposed direct S2ST system with discrete units is the most efficient during inference among the compared direct and cascaded systems.
- Future work includes evaluating the framework with real data and modeling para-linguistic aspects such as voice and prosody.
- The authors plan to release code for reproducing the experiments.
A Model training details
This section identifies the training hyper-parameters used for the direct S2ST models reported in Tables 2 and 3.
- Table 4 lists the training hyper-parameters for the direct S2ST models reported in Tables 2 and 3.
B Examples of model output
Table 5 compares multiple text outputs associated with generated speech under the written-language setup and documents their consistency and mismatch sources.
- Generated speech and CTC-decoded text are consistent, while an auxiliary task may produce inconsistent text through a separate attention module.
- Small mismatches between speech output and CTC-decoded text arise from ASR errors and CTC misspellings.
C Significance test
This section presents paired significance tests for systems evaluated in Table 2 and identifies the threshold used to mark statistically significant p-values.
- Table 6 reports p-values from paired significance tests comparing nine systems in Table 2 on the Fisher test set.
- The tests use paired bootstrap resampling supported by the SACREBLEU tool.
- p-values below 0.05 are marked with an asterisk.