Source-linked AI summary
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
TL;DR
Cross-lingual speech synthesis must generate another language while preserving an unseen speaker’s characteristics despite limited multilingual speaker data and model capacity. VALL-E X uses a multilingual conditional codec language model prompted by source speech and target text, and supports zero-shot synthesis and speech-to-speech translation with voice, emotion, and background preservation while reducing foreign accents. Its current version is trained on only two languages, with broader language coverage left for future work.
Problem
Cross-lingual synthesis must transfer a speaker’s voice and other speech characteristics across languages, but data scarcity and limited model capacity constrain quality and speaker similarity.
Method
VALL-E X trains a multilingual conditional codec language model to predict target-language acoustic tokens from source-language speech and target-language text prompts.
Results
VALL-E X supports zero-shot cross-lingual text-to-speech and speech-to-speech translation, preserving unseen speakers’ voice, emotion, and background while reducing foreign accents.
Takeaways & Limitations
A single source-language sentence can provide the prompt for target-language speech that retains the source speaker’s characteristics and supports voice-retentive translation.
Takeaways & Limitations
The current VALL-E X model is trained on speech-transcription data from only two languages; exploring more languages is future work.
Abstract
from arXiv · showhide
We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis. Specifically, we extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at \url{https://aka.ms/vallex}.
1 Introduction
Cross-lingual speech synthesis seeks to transfer a speaker’s voice across languages, but data scarcity and limited model capacity impair speaker similarity and preservation of speech characteristics. VALL-E X addresses this with multilingual in-context codec modeling for zero-shot generation, retaining voice, emotion, background, and reducing foreign accents.
- Motivation: Cross-lingual synthesis transfers a speaker’s voice to another language, but quality and speaker similarity remain behind monolingual TTS because of data scarcity and limited model capacity.Conventional systems struggle to transfer voice, speech background, and emotion from source-language speech.
- Approach: The model converts transcriptions to phonemes, speech to acoustic tokens, and trains on concatenated paired sequences from each language.A rule-based G2P tool produces phoneme sequences, while an offline neural codec encoder produces acoustic tokens.
- Approach: VALL-E X is a conditional multilingual codec language model that predicts target-language acoustic tokens from source-language speech and target-language text prompts.It is trained on multilingual, multispeaker, multidomain speech-transcription data.
- Capabilities: One source-language sentence can prompt cross-lingual speech that maintains an unseen speaker’s voice, emotion, and speech background.This multilingual in-context learning framework supports zero-shot synthesis without requiring cross-lingual paired data from the same speakers.
- Capabilities: A language ID guides target-language speaking style and significantly reduces foreign-accent problems in generated speech.Language conditioning helps select suitable acoustic tokens in multilingual generation and accommodates differences such as tonal versus non-tonal languages.
- Evaluation: Experiments cover zero-shot cross-lingual text-to-speech and zero-shot speech-to-speech translation, with improvements over a strong baseline across multiple evaluation dimensions.Reported dimensions include speaker similarity, speech quality, translation quality, speech naturalness, and human evaluation.
2 Related Work
Speech and audio synthesis has advanced substantially, while cross-lingual TTS and speech-to-speech translation address the harder problem of generating another language’s speech. Prior work uses multilingual representations, speaker disentanglement, cascaded pipelines, and end-to-end systems.
- Speech/Audio Synthesis: Neural speech and audio synthesis has progressed through architectures including WaveNet, HiFi-GAN, DiffWave, AudioGen, and AudioLM.Recent systems also use discrete audio representations and acoustic or semantic tokens for generation.
- Positioning: Unlike related voice-cloning work using short recordings, this paper focuses on retaining an unseen source speaker’s voice in synthesized target-language speech.The distinction is the cross-lingual setting rather than monolingual speech prompting.
- Cross-Lingual TTS: Cross-lingual TTS synthesizes another language for a monolingual speaker while attempting to preserve speaker identity, making it more challenging than monolingual TTS.Prior approaches use shared phonemic representations and adversarial objectives to separate speaker identity from speech content.
- Speech to Speech Translation: Speech-to-speech translation converts speech from one language into speech in another language, traditionally through cascaded ASR, machine translation, and TTS modules.Recent research has also explored end-to-end S2ST models.
3 Cross-Lingual Codec Language Model
VALL-E X extends neural codec language modeling to cross-lingual speech synthesis with multilingual autoregressive and non-autoregressive models. It uses source-language prompts and target-language phonemes to generate target acoustic tokens, with language IDs guiding speaking style.
- 3.2 Model Framework: VALL-E X combines multilingual autoregressive and non-autoregressive codec language models for cross-lingual speech synthesis.The autoregressive model generates first-layer acoustic tokens, while the non-autoregressive model generates the remaining layers.
- 3.3 Multi-lingual Training: Paired phoneme and acoustic-token sequences from different languages train the codec language models to learn cross-lingual acoustic conversion.The paired source and target language data are used to optimize both multilingual codec models.
- 3.3 Multi-lingual Training: A language ID guides target-language speaking style and helps alleviate foreign-accent problems.Language IDs are embedded and added to acoustic-token embeddings for the multilingual autoregressive model.
- 3.4 Cross-Lingual Inference: During inference, source phonemes, target phonemes, and first-layer source acoustic tokens condition generation of the target acoustic-token sequence.The autoregressive model generates the first target layer, and the non-autoregressive model predicts the remaining layers before EnCodec decodes the complete sequence into speech.
- 3.3 Multi-lingual Training: The current version is trained on speech-transcription data from two languages, with exploration of additional languages left for future work.This bounds the demonstrated multilingual scope of the model.
- 3.4 Cross-Lingual Inference: VALL-E X supports zero-shot cross-lingual text-to-speech and zero-shot speech-to-speech translation.The inference design uses two-stage decoding to support both task types.
4 VALL-E X Application
VALL-E X applies its cross-lingual codec language model to zero-shot text-to-speech and speech-to-speech translation. Speech-to-speech translation additionally uses a recognition-and-translation model to produce source and target phonemes that condition target speech generation.
- 4.1 Zero-Shot Cross-Lingual Text-to-Speech: VALL-E X is applied to zero-shot cross-lingual text-to-speech and zero-shot speech-to-speech translation.These are presented as two examples of cross-lingual speech generation tasks.
- 4.2 Zero-Shot Speech-to-Speech Translation: For speech-to-speech translation, an additional model synchronously recognizes and translates source speech into source and target phoneme sequences.The recognition-and-translation model supplies the semantic inputs needed by VALL-E X.
- 4.2 Zero-Shot Speech-to-Speech Translation: The improved SpeechUT framework uses speech, phoneme, and decoder components and is pretrained on ASR and machine-translation corpora.It replaces clustering-based hidden units with phonemes as the modality bridge between speech and text.
- 4.3 Evaluation: Evaluation covers speaker similarity, speech quality, translation quality, naturalness, and human judgments.Speaker similarity is measured with an automatic speaker-verification task, while speech quality is assessed using ASR-based criteria.
5 Experiments
Experiments evaluate VALL-E X on zero-shot cross-lingual TTS and bidirectional Chinese↔English S2ST, using automatic and human measures of speaker similarity, translation, naturalness, accent, and emotion.
- Zero-Shot Cross-Lingual TTS: VALL-E X improves English TTS prompted by Chinese speech, raising ASV-Score from 0.30 to 0.36 and reducing ASR-WER from 8.53 to 4.07.The authors also report better speech naturalness than the baseline.
- Zero-Shot Cross-Lingual TTS: Human evaluation on 50 records raises prompt-to-output SMOS from 3.42 to 4.00 and yields a +0.24 CMOS gain over the baseline.These results assess voice similarity and comparative synthesis quality for English TTS prompted by Chinese speech.
- Zero-Shot S2ST: For Chinese→English S2ST, VALL-E X exceeds the baseline in BLEU, with 30.66 versus 27.49, and in naturalness, with 3.54 versus 3.44.With oracle target text, translation performance reaches about 84∼87 BLEU scores.
- Zero-Shot S2ST: Human S2ST evaluation gives VALL-E X 4.12 versus 3.06 SMOS and 3.87 versus 3.81 MOS against the Chinese→English baseline.The source-to-ground-truth speaker-similarity reference remains higher at 4.91 SMOS.
- Analysis: Removing the language ID or supplying the source language ID decreases translation quality, while language IDs guide speech synthesis and foreign-accent control.The accent evaluation labels synthesized English and Chinese speech on a 1-to-5 scale.
- Analysis: VALL-E X maintains emotional consistency to a certain extent in translated speech and offers a promising code-switch synthesis solution without special code-switch optimization.The emotion experiments use source prompts from EmoV-DB, while code-switch capability is reported from training on multiple monolingual speech datasets.
6 Conclusion
The paper presents VALL-E X as a cross-lingual neural codec language model for zero-shot voice-retentive speech generation without cross-lingual paired data from the same speakers.
- 6 Conclusion: VALL-E X transfers the source speaker’s voice into generated target-language speech without requiring cross-lingual paired data from the same speakers.It is trained on large-scale multilingual, multispeaker speech-transcription data and supports zero-shot cross-lingual TTS and voice-retentive S2ST.
A.1.1 Model Pre-training
The speech recognition and translation model is pre-trained with separate speech-side and text-side objectives, then combines them into a final pre-training loss.
- The speech recognition and translation model contains a speech encoder, semantic encoder, and semantic decoder.
- The speech-side objective predicts phonemes from representations produced by the speech encoder and semantic encoder.It uses a speech waveform and corresponding phonemes as inputs.
- The text-side objective performs autoregressive sequence-to-sequence translation from bilingual phoneme sequences.
- The final pre-training objective is Lpt = Lspeech + Ltext.
A.1.2 Model Architecture
The speech recognition and translation model uses the Base SpeechUT architecture with Transformer encoder/decoder components and a convolutional speech pre-net.
- All encoder and decoder components use 6 Transformer layers with relative position bias.
- The model uses an FFN dimension of 3072 and an attention dimension of 768.
- A speech pre-net with several 1-D convolutional layers precedes the speech encoder.The convolutional layers use 512 channels and kernel sizes [10,3,3,3,3,2,2].
- The speech pre-net downsamples the speech waveform by 320 before encoding.
A.1.3 Training Details
The speech recognition and translation model is pre-trained with masked speech inputs and mixed speech-phoneme batches on 32 V100 GPUs for 400K steps.
- The speech mask probability is 8% with a mask length of 10.
- Each GPU processes speech and phoneme batch sizes of 1,400,000 and 3,000, respectively.The speech batch corresponds to 87.5 seconds.
- The maximum learning rate is 5e-4 with 32,000 warm-up steps.
- Pre-training runs on 32 V100 GPUs for 400K steps.