Source-linked AI summary
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
TL;DR
Existing TTS systems struggle with generalization to unseen speakers and often require adaptation or engineered speaker representations. VALL-E treats TTS as conditional language modeling over neural codec tokens, pre-trained on 60K hours of speech, and achieves stronger zero-shot naturalness and speaker similarity while preserving prompt characteristics. Its scope remains limited by synthesis errors and incomplete coverage of accents and speaking styles.
Problem
Existing TTS systems have limited generalization to unseen speakers and may require fine-tuning, engineered features, or complex speaker encoders for zero-shot synthesis.
Method
VALL-E predicts discrete acoustic codec tokens conditioned on phoneme content and a 3-second acoustic prompt, using autoregressive and non-autoregressive language models.
Results
+0.12 CMOS and +0.93 SMOS on LibriSpeech, with further improvements on VCTK, show VALL-E outperforms the state-of-the-art zero-shot TTS system.
Takeaways & Limitations
VALL-E preserves the acoustic environment and speaker emotion of the prompt and produces diverse outputs useful for pseudo-data creation for speech recognition.
Takeaways & Limitations
Synthesis may produce unclear, missed, or duplicated words, while training data insufficiently covers accent speakers and diverse speaking styles.
Abstract
from arXiv · showhide
We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.
1 Introduction
VALL-E reframes zero-shot TTS as conditional language modeling over neural audio codec tokens, using large and diverse speech data to synthesize unseen speakers from short prompts. It outperforms the state-of-the-art zero-shot TTS system in naturalness and speaker similarity while preserving prompt emotion and acoustic environment.
- VALL-E framework: VALL-E generates acoustic tokens from a 3-second enrolled recording and phoneme prompt, then decodes them into personalized speech.The acoustic prompt constrains speaker information, while phonemes constrain content.
- Training data: 60K hours of English speech from over 7000 speakers provide large-scale, diverse training data despite noisier recordings and inaccurate transcriptions.The corpus is LibriLight, whose audio-only data were transcribed with speech recognition.
- Evaluation: +0.12 CMOS and +0.93 SMOS on LibriSpeech, and +0.11 SMOS and +0.23 CMOS on VCTK, measure VALL-E's improvements over the state-of-the-art zero-shot TTS baseline.All test speakers were unseen during training.
- Qualitative findings: VALL-E achieves speech on VCTK that is as natural as human recordings and produces diverse outputs for the same text and target speaker.The diversity is relevant to pseudo-data creation for speech recognition.
- Qualitative findings: VALL-E preserves the acoustic environment and emotion of the acoustic prompt during synthesis.Examples include reverberation and anger.
2 Related Work
Related work spans cascaded and end-to-end zero-shot TTS, spoken generative models, and TTS pre-training. VALL-E differs by combining explicit text control, audio codec tokens, and 60K-hour pre-training for zero-shot TTS.
- Zero-Shot TTS: Zero-shot TTS adapts a system to an arbitrary voice from rare enrolled recordings, motivating speaker adaptation and speaker encoding methods.Earlier approaches often require fine-tuning, speaker-specific parameters, or engineered speaker representations.
- Spoken generative pre-trained models: AudioLM synthesizes speech from audio codecs in a speech-to-speech setting, whereas VALL-E uses text to explicitly control speech content.The distinction is between speech-to-speech generation and TTS.
- TTS pre-training: Previous TTS pre-training used less than 1K hours and earlier token-based systems relied on representations such as VQVAE codes or mel spectrograms.VALL-E uses audio codec codes as intermediate representations and is pre-trained with 60K hours.
- VALL-E: VALL-E is presented as the first TTS framework with strong in-context learning capabilities that avoids fine-tuning, pre-designed features, and a complex speaker encoder.Its language-model formulation uses audio codec codes as the intermediate representation.
3 Background: Speech Quantization
Speech quantization compresses raw audio into discrete representations that are easier to model. VALL-E uses EnCodec and residual vector quantization to encode and reconstruct speech at reduced temporal resolution.
- Motivation: Raw-audio generation is difficult because each timestep may require 65,536 probabilities and audio sequences are extremely long.Speech quantization reduces both value range and sequence length.
- Neural codecs: Neural codecs encode waveforms into discrete acoustic codes and reconstruct high-quality waveforms, including for speakers unseen during codec training.They are particularly effective at low bitrates compared with traditional codecs.
- EnCodec: EnCodec converts 24 kHz audio into embeddings at 75 Hz, a 320-fold sampling-rate reduction, using eight RVQ quantizers with 1024 entries each.This configuration corresponds to 6K bitrate reconstruction.
- EnCodec: More quantizers increase bitrate and reconstruction quality; at 12K bitrate, EnCodec uses 16 quantizers and represents 10 seconds as a 750 × 16 matrix.The convolutional decoder reconstructs the waveform from codes across all quantizers.
4 VALL-E
VALL-E treats zero-shot TTS as conditional language modeling over hierarchical neural codec codes, using phoneme and acoustic prompts to synthesize unseen speakers. An autoregressive first stage and non-autoregressive later stages balance sequence-length flexibility with inference speed.
- Conditional Codec Language Modeling: VALL-E encodes audio as hierarchical discrete codec tokens and models their generation conditioned on phonemes and an acoustic prompt.The codec representation contains multiple quantizers, with earlier quantizers capturing speaker-related properties and later quantizers refining acoustic detail.
- Non-Autoregressive Codec Language Modeling: The non-autoregressive decoder generates tokens for quantizers 2–8 using the phoneme sequence, acoustic prompt, and previously predicted codebooks.The NAR decoder is called seven times, once for each of the seven later quantizers.
- Conditional Codec Language Modeling: Combining AR and NAR decoding supports flexible acoustic sequence lengths while reducing later-stage time complexity from O(T) to O(1).AR decoding handles the first stage’s variable length, while NAR decoding processes subsequent stages in parallel.
- Autoregressive Codec Language Modeling: The autoregressive decoder generates tokens from the first quantizer while conditioning on the phoneme sequence and first-quantizer acoustic prompt.During inference, the enrolled recording’s acoustic tokens serve as a prefix for decoding.
- In-Context Learning via Prompting: Prompting enables VALL-E to synthesize speech for unseen speakers without fine-tuning by using phoneme and acoustic prompts during inference.For voice cloning, the enrolled speech transcription is prepended to the target text and its first-layer acoustic tokens provide the acoustic prefix.
- In-Context Learning via Prompting: VALL-E-continual uses the full transcription and the first 3 seconds of an utterance as prompts to generate a semantically continuous continuation.The enrolled and generated speech are intended to remain continuous in this inference setting.
5 Experiment
Experiments evaluate VALL-E’s robustness, speaker similarity, naturalness, prompt dependence, and qualitative control across LibriSpeech and VCTK settings. Results show strong zero-shot speaker fidelity, improved naturalness, prompt-dependent identity preservation, and preservation of acoustic conditions and output diversity.
- Experimental Setup: VALL-E is trained on 60K hours of unlabeled English speech and evaluated with ASR-derived transcriptions and codec-token modeling.The training corpus contains around 7000 speakers; waveform segments are randomly cropped to 10–20 seconds, and training uses 16 V100 GPUs for 800k steps.
- LibriSpeech Evaluation: VALL-E significantly outperforms YourTTS in robustness and speaker similarity, indicating faithful reproduction of both the input text and enrolled speech.The comparison also includes GSLM and AudioLM, with VALL-E reported as more robust than those speech-to-speech language-model systems.
- Prompt Ablations: Removing the acoustic prompt reduces speaker similarity to 0.236, showing that both NAR and AR acoustic prompts contribute to speaker identity.The NAR ablation finds that acoustic-token prompts improve speaker evaluation quality, while the AR prompt adds further speaker-similarity benefit.
- VCTK Evaluation: VALL-E’s speaker similarity improves with longer prompts and remains stronger than the baseline even when the baseline has seen many training speakers.On VCTK, the baseline observed 97 speakers during training whereas VALL-E observed none of them.
- Qualitative Analysis: Sampling discrete tokens produces varied speech for identical text and speaker prompts, including different durations, speech rates, accents, and emphasis.The diversity is presented as useful for generating pseudo-data for speech recognition.
- Qualitative Analysis: VALL-E preserves acoustic characteristics of the prompt, including reverberation and emotion, in zero-shot synthesis.The authors attribute acoustic-environment consistency to training on large-scale data containing more varied acoustic conditions than the baseline data.
6 Conclusion, Limitations, and Future Work
VALL-E achieves state-of-the-art zero-shot TTS using audio codec codes and 60K hours of pre-training data, while preserving acoustic environment, emotion, and output diversity. Remaining limitations include robustness errors, incomplete voice and style coverage, model-structure constraints, and misuse risks.
- Conclusion: VALL-E achieves new state-of-the-art zero-shot TTS results on LibriSpeech and VCTK while preserving acoustic environment, speaker emotion, and sampling-based output diversity.The model uses audio codec codes as intermediate representations and demonstrates in-context learning in zero-shot scenarios.
- Limitations and Future Work: Some synthesized words may be unclear, missed, or duplicated because the autoregressive phoneme-to-acoustic language model permits disordered attention alignments without explicit constraints.The authors propose non-autoregressive models or modified attention mechanisms as future remedies.
- Limitations and Future Work: 60K hours of training data still do not cover every voice, especially accent speakers, and audiobook-style data provide insufficient speaking-style diversity.The authors identify weaker VCTK performance than LibriSpeech as evidence of insufficient accent coverage.
- Limitations and Future Work: VALL-E currently uses two models to predict codes from different quantizers, while a universal model or full non-autoregressive models remain promising directions.Full non-autoregressive models could speed up inference.
- Broader Impacts: Because VALL-E maintains speaker identity, misuse could enable spoofing voice identification or impersonating a specific speaker.The authors suggest detection models to identify synthesized audio and commit to applying Microsoft AI Principles in future development.