Source-linked AI summary
Textless Speech-to-Speech Translation on Real Data
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, Wei-Ning Hsu
TL;DR
The paper addresses textless speech-to-speech translation on real-world data, including multi-speaker target speech, without text supervision. It uses self-supervised unit-based speech normalization and reports improved translation quality on VoxPopuli and automatically mined S2ST data.
Problem
Textless S2ST has limited prior work, while modeling real target speech is difficult because speakers vary in accents, speaking styles, and recording conditions.
Method
A self-supervised unit-based speech normalizer finetunes a pre-trained speech encoder with multi-speaker input and discrete units from a reference speaker to reduce variation while retaining lexical content.
Results
3.2 BLEU points average gain is obtained on VoxPopuli with 10 minutes of normalization data, and mined S2ST data adds 2.0 BLEU on average over four language directions.
Takeaways & Limitations
The study establishes a textless S2ST technique trained with real-world data that works across multiple language pairs.
Abstract
from arXiv · showhide
We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data. The key to our approach is a self-supervised unit-based speech normalization technique, which finetunes a pre-trained speech encoder with paired audios from multiple speakers and a single reference speaker to reduce the variations due to accents, while preserving the lexical content. With only 10 minutes of paired data for speech normalization, we obtain on average 3.2 BLEU gain when training the S2ST model on the VoxPopuli S2ST dataset, compared to a baseline trained on un-normalized speech target. We also incorporate automatically mined S2ST data and show an additional 2.0 BLEU gain. To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs. Audio samples are available at https://facebookresearch.github.io/speech_translation/textless_s2st_real_data/index.html .
1 Introduction
The paper extends textless direct S2ST to real-world data, addressing multi-speaker target variation with self-supervised speech normalization. It reports gains from normalization and automatically mined S2ST data across multiple language pairs.
- Motivation: Textless direct S2ST avoids generating target-language text, but prior systems often still require text or phoneme annotations during training.Conventional S2ST commonly uses ASR, MT, and TTS in a cascade, while direct S2ST removes text generation as an intermediate step.
- Motivation: More than 40% of the world’s languages lack writing systems, while textless direct S2ST has received limited attention.Previous direct S2ST work mainly relied on synthetic target speech because real S2ST training data was scarce.
- Approach: The paper applies textless S2ST to real S2ST datasets, including VoxPopuli and automatically mined speech data, without text or phoneme data.The setup extends prior discrete-unit S2ST work from synthetic target speech to real data.
- Approach: The proposed speech normalization technique removes multi-speaker speech variation while preserving lexical content using self-supervised discrete units.It is applied to the target speech of real S2ST data.
- Results: Augmenting supervised S2ST data with directly mined S2ST data further improves textless S2ST performance when speech normalization is used.The paper presents this as evidence for the usefulness of mined S2ST data.
- Contribution: The authors identify the technique as the first textless S2ST system trained with real-world data and working across multiple language pairs.This claim is stated as a contribution to the best of the authors’ knowledge.
2 Related work
Related work covers direct S2ST with discrete representations, scarce real S2ST resources and audio mining, and existing speech-normalization methods. The paper positions its unit-based approach as a way to reduce non-lexical variation in real target speech.
- Direct S2ST: Earlier direct S2ST systems either required phoneme data or used VQ-VAE discrete codes for languages without writing systems.These systems learned speech-to-code translation without generating text as an intermediate output.
- Direct S2ST: Lee et al. used self-supervised discrete target representations for textless S2ST, but their experiments used synthetic target speech.The present work follows that design while focusing on real data.
- S2ST data: VoxPopuli provides 17.3k hours of S2ST data covering more than 200 language directions, addressing the scarcity of real S2ST corpora.The corpus consists of European Parliament plenary sessions and simultaneous interpretations.
- Speech normalization: Speech normalization includes single-speaker clean data, silence removal, speech enhancement, and voice conversion approaches.These methods reduce variation not specified by the input in speech-generation settings.
- Speech normalization: The paper proposes self-supervised discrete-unit normalization that maps diverse speech variation to units with little non-textual variation.The method is presented as a novel speech-normalization technique for real S2ST.
3 System
The system discretizes speech into HuBERT units, normalizes multi-speaker target speech with CTC finetuning, translates speech to normalized units, and synthesizes speech with a unit vocoder.
- Discrete units: HuBERT and K-means convert audio into orig-unit sequences of cluster indices over speech frames.The units are derived from learned K cluster centroids.
- Speech normalization: The speech normalizer uses paired same-content audios: multi-speaker audio is input, while reduced reference-speaker orig-unit is the CTC target.Repeated units are removed before CTC finetuning.
- Inference: Only the shaded modules of the S2ST architecture are used during inference.The architecture combines S2UT translation with unit-to-speech conversion.
- Speech normalization: During inference, CTC decoding converts input speech into norm-unit sequences intended to reduce speaker and recording variation.The process treats reference-speaker units as pseudo text while retaining content.
- S2UT model: The S2UT model predicts norm-unit as its target and includes an auxiliary task using reduced source orig-unit.The auxiliary task uses cross-attention and a Transformer decoder attached to an intermediate speech-encoder layer.
- Vocoder: A unit-based HiFi-GAN vocoder converts discrete units into speech, with duration prediction trained separately from S2UT.It can synthesize from orig-unit or norm-unit because both arise from the same K-means process.
4 Experimental Setup
Experiments cover four language pairs and use real S2ST data, multilingual self-supervised speech representations, speech normalization, and unit-based speech generation. Translation quality is measured with BLEU from ASR-decoded outputs, while naturalness is assessed with human MOS tests.
- Experimental scope: Experiments cover Spanish-English, French-English, English-Spanish, and English-French translation directions.All experiments are conducted using fairseq.
- Speech representations: The multilingual HuBERT model combines 13.5k hours of unlabeled English, Spanish, and French speech using one codebook without language information during pre-training.The model is trained from data totaling 4.5k hours per language.
- Speech normalization: Speech normalizers are trained from multi-speaker ASR speech paired with synthetically created reference units, using 10-minute, 1-hour, or 10-hour training sets per language.Filtered VoxPopuli ASR data is supplemented with randomly sampled Common Voice data for Spanish and French when necessary.
- S2ST data: S2UT models use VoxPopuli S2ST data and optionally automatically mined LibriVox S2ST data, with Europarl-ST used for evaluation.The data statistics distinguish language directions and remove overlapping pre-2013 plenary-session speech from VoxPopuli.
- Evaluation: BLEU is computed from ASR-decoded system output after reference-text normalization, and speech naturalness is evaluated with human mean opinion scores.Human listening tests randomly sample 200 utterances for each evaluation.
- Baselines and generation: The unit-based vocoder generates speech from original or normalized unit sequences, while TTS baselines use Transformer TTS or text-to-unit conversion.Vocoder training uses original unit sequences because they retain natural-speech duration information.
5 Results
Speech normalization improves textless S2ST on real multi-speaker data, while mined S2ST data provides further gains across language directions and test domains.
- 5.1 Textless S2ST: 1.5 BLEU improvement is achieved with only 10 minutes of paired data for speech-normalizer training, while 10 hours yields a 4.9 BLEU gain.The gains are measured against the basic setup across four language pairs.
- 5.1 Textless S2ST: Normalized-unit systems improve MOS by 0.85 over orig-unit systems and achieve similar naturalness to Transformer TTS models.Orig-unit systems often produce stuttering, while the ASR evaluation still captures speech content.
- 5.1 Textless S2ST: 3.9 BLEU improvement over the basic setup is obtained on Europarl-ST with normalized units across language pairs.The comparison is between the proposed norm-unit system and the basic orig-unit setup.
- 5.1 Textless S2ST: Mined data produces even larger improvements on CoVoST 2, while target speaker embeddings improve S2ST by 7.1 BLEU on average.The CoVoST 2 comparison reports larger mined-data gains than Europarl-ST; the 7.1 BLEU result is for target speaker embeddings.
- 5.1 Textless S2ST: The proposed textless S2ST systems are 0.6 BLEU behind ASR-derived S2T+TTS systems on Europarl-ST.S2T systems using oracle text and in-domain training data are presented as an upper bound.
- 5.2 Analysis on the speech normalizer: Norm-unit sequences are 15% shorter than reduced orig-unit sequences because long silences and non-speech segments are not encoded.The resulting targets are described as shorter and cleaner for S2UT training.
- 5.2 Analysis on the speech normalizer: Norm-unit has 58% of the unit error rate of reduced orig-unit across paired Common Voice speech, indicating fewer speaker-related variations.The analysis uses 400 same-text audio pairs for each of English, Spanish, and French.
6 Conclusion
The paper concludes that textless S2ST can be trained with real target speech data using self-supervised speech normalization. The approach reduces multi-speaker variation while retaining lexical content and improves translation quality on real and mined data.
- 6 Conclusion: The proposed textless S2ST system is trained with real target speech data using self-supervised discrete units.Experiments use VoxPopuli and automatically mined speech data.
- 6 Conclusion: Speech normalization reduces multi-speaker speech variation while retaining lexical content through CTC finetuning of a pretrained speech encoder.The normalizer can be trained with one hour of parallel speech without human annotations.
- 6 Conclusion: Experiments across datasets and languages demonstrate the usefulness of speech normalization for improving textless S2ST translation quality.The paper evaluates VoxPopuli and mined speech data across multiple language pairs.
A mHuBERT Training details
The appendix describes the setup for mHuBERT training and the target labels used across its three training iterations.
- A mHuBERT Training details: Three iterations of mHuBERT training are documented, including the target-label setup used for each iteration.The appendix table lists the training details and target labels.
B Unit-based Vocoder
The appendix reports unit-based vocoder evaluation through WER on language-specific TTS development sets and explains how resynthesis differences reflect vocoder and duration-prediction quality.
- B Unit-based Vocoder: WER differences between original audio, orig-unit, and reduced orig-unit isolate vocoder quality and the additional effect of duration prediction.The comparison is described for the resynthesis evaluation.
- B Unit-based Vocoder: WER is reported on LJSpeech for English and CSS10 for Spanish and French using audio resynthesized from units.These are the language-specific TTS development sets used for unit-based vocoder evaluation.
C Text-to-Unit (T2U)
The T2U model generates audio with a unit-based vocoder and duration prediction, producing high-quality audio suitable as another TTS option.
- The T2U model generates reference target units for speech normalizer training.It is trained with reduced unit sequences as targets.
- During synthesis, the system applies a unit-based vocoder with duration prediction.
- The generated audio is reported to be high quality and suitable as another TTS option.
- Table 10 reports WER on TTS development sets for English, Spanish, and French.The listed sets are LJSpeech for English and CSS10 for Spanish and French.
D Hyper-parameters
The speech normalizers and S2UT models use specified training configurations, including eight GPUs and fixed batch-size settings, with one stated dropout exception.
- Speech normalizer training uses eight GPUs and a maximum total input audio length of 100 seconds per batch.The best hyper-parameters are listed for three languages and three data setups.
- S2UT model training uses eight GPUs, a total batch size of 160k tokens, and dropout of 0.3.The Es-En experiment ID 1 uses dropout 0.1 instead.
- Table 11 reports hyper-parameters for training the speech normalizers.
- Table 12 reports learning rates for S2UT model training.
E Dev BLEU
The paper reports BLEU scores for systems evaluated on the Europarl-ST development sets.
- Europarl-ST development-set BLEU scores are reported for systems from Tables 4 and 5.
- Table 13 presents the BLEU scores on the Europarl-ST development sets.