PaperlayerView exact revision on arXiv

Source-linked AI summary

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei

arXiv:2406.05370v2cs.CLcs.SDeess.AS

TL;DR

Zero-shot TTS remains difficult because models must synthesize unseen speakers from short enrolled samples. VALL-E 2 combines repetition aware sampling and grouped code modeling, achieving human parity on LibriSpeech and VCTK while handling difficult repetitive sentences reliably.

  • Problem

    Zero-shot TTS must synthesize speech for unseen speakers from a short enrolled speech sample, and remains challenging despite human-level quality in some single-speaker systems.

  • Method

    VALL-E 2 uses neural codec language modeling with repetition aware sampling for stable decoding and grouped code modeling for shorter, more efficient sequences.

  • Results

    VALL-E 2 achieves human-parity robustness, naturalness, and speaker similarity on LibriSpeech and VCTK, and stably generates high-quality speech for complex or repetitive sentences.

  • Takeaways & Limitations

    The reported capabilities could support speech generation for people with aphasia or amyotrophic lateral sclerosis.

  • Takeaways & Limitations

    The ablation conclusion is based on the current audiobook-domain experiment setting, while real-world unseen-speaker deployment would require speaker approval and synthesized-speech detection.

Abstract

from arXiv · show

This paper introduces VALL-E 2, the latest advancement in neural codec language models that marks a milestone in zero-shot text-to-speech synthesis (TTS), achieving human parity for the first time. Based on its predecessor, VALL-E, the new iteration introduces two significant enhancements: Repetition Aware Sampling refines the original nucleus sampling process by accounting for token repetition in the decoding history. It not only stabilizes the decoding but also circumvents the infinite loop issue. Grouped Code Modeling organizes codec codes into groups to effectively shorten the sequence length, which not only boosts inference speed but also addresses the challenges of long sequence modeling. Our experiments on the LibriSpeech and VCTK datasets show that VALL-E 2 surpasses previous systems in speech robustness, naturalness, and speaker similarity. It is the first of its kind to reach human parity on these benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech, even for sentences that are traditionally challenging due to their complexity or repetitive phrases. The advantages of this work could contribute to valuable endeavors, such as generating speech for individuals with aphasia or people with amyotrophic lateral sclerosis. See https://aka.ms/valle2 for demos of VALL-E 2.

1 Introduction

VALL-E 2 addresses the stability and efficiency challenges of zero-shot TTS with repetition aware sampling and grouped code modeling. Evaluations on LibriSpeech and VCTK report human-parity performance, while complex and repetitive sentences remain stable and high quality.

  • Zero-shot TTS must synthesize speech for unseen speakers from a short enrolled sample and remains challenging despite progress in single-speaker TTS.
  • VALL-E’s random sampling can destabilize output, nucleus sampling can loop indefinitely, and its frame-rate-bound autoregressive model slows inference.Multiple-time sampling can mitigate instability but increases computational cost.
  • VALL-E 2 combines repetition aware sampling with grouped code modeling to stabilize decoding, avoid infinite loops, shorten sequences, and improve efficiency.Repetition aware sampling selects random or nucleus sampling according to token repetition; grouped modeling processes codec-code groups as single frames.
  • VALL-E 2 achieves human-parity robustness, naturalness, and speaker similarity on in-domain LibriSpeech and out-of-domain VCTK evaluations.The authors state that these conclusions are based solely on experiments on LibriSpeech and VCTK.
  • VALL-E 2 stably generates high-quality speech for complex sentences that are difficult to read or contain many repeated phrases.The reported benefits could support speech generation for people with aphasia or amyotrophic lateral sclerosis.
  • VALL-E 2 remains a research project with no current plans for product integration or public access, and misuse risks include voice spoofing and impersonation.Real-world use with unseen speakers would require speaker approval and synthesized-speech detection protocols.

2 Related Work

Related work approaches zero-shot TTS through speaker adaptation, codec language modeling, and non-autoregressive or generative decoding. VALL-E 2 continues the codec-language-modeling line while targeting stable decoding without complex speech-data preparation.

  • Early zero-shot TTS systems often required speaker adaptation, fine-tuning, complex features, or substantial structural engineering.
  • VALL-E-style systems represent speech as discrete neural codec codes and formulate zero-shot TTS as language modeling.
  • Follow-up systems extended codec language modeling across languages, intermediate semantic units, larger models, and larger training datasets.
  • Other systems accelerate inference with confidence-based parallel decoding, diffusion, or flow matching for prompt-conditioned speech synthesis.
  • VALL-E 2 follows codec language modeling while enabling stable decoding without complex preparation such as duration or pitch information, and reports human parity on LibriSpeech and VCTK.

3.1 Problem Formulation: Grouped Codec Language Modeling

VALL-E 2 converts speech into grouped neural codec codes and trains a conditional language model to predict those groups from text. During inference, prompted codec codes from an unseen speaker condition generation of target speech.

  • Grouped codec modeling partitions the codec sequence into fixed-size groups and models each group as one frame, reducing the sequence-length burden.The method also removes the frame-rate constraint of the off-the-shelf codec and mitigates long-context modeling problems.
  • Given audio y and text x, the method encodes speech into an 8-quantizer codec sequence, groups the codes, and trains VALL-E 2 to minimize conditional negative log-likelihood.A few initial silence codes may be clipped so the sequence length is divisible by the group size without removing speech information.
  • The grouped-code objective predicts each current group conditioned on the text and codec groups generated previously.
  • At inference, text and grouped codec codes from an unseen speaker serve as conditions for generating target grouped codes with the requested content and speaker voice.
  • VALL-E 2 training uses an autoregressive and a non-autoregressive Transformer, with the autoregressive model generating grouped codec codes.
  • The generated target codec sequence is converted into a speech waveform using an off-the-shelf neural codec decoder.

3.2 VALL-E 2 Architecture

VALL-E 2 uses a hierarchical codec-language-model architecture with autoregressive and non-autoregressive Transformer components. The autoregressive model generates the first codec code per frame, while the non-autoregressive model generates the remaining codes.

  • The hierarchical architecture contains an autoregressive codec language model followed by a non-autoregressive codec language model.
  • The autoregressive model generates the first codec code for each frame autoregressively, while the non-autoregressive model generates remaining code sequences from preceding codes.
  • Both models use Transformer components comprising text-embedding, code-embedding, and code-prediction layers.

3.3 VALL-E 2 Training

VALL-E 2 trains hierarchical autoregressive and non-autoregressive codec language models from utterance-wise speech-transcription pairs. Grouped embeddings let the autoregressive model predict codec-code groups, while the non-autoregressive model predicts remaining code sequences conditioned on preceding codes.

  • Training setup: Training requires only simple utterance-wise speech-transcription pairs, avoiding forced alignments and additional reference-speaker audio.Audio is converted to codec codes and text to token sequences before training the two models.
  • Autoregressive model training: The autoregressive model predicts the first codec code sequence conditioned on the text sequence.Its grouped variant models groups autoregressively while predicting codes within each group non-autoregressively.
  • Autoregressive model training: Grouped code embeddings partition the code sequence into size-G groups, concatenate each group, and reduce the modeled sequence length to T/G.The resulting group embeddings are combined with text embeddings and special tokens before causal Transformer prediction.
  • Non-autoregressive model training: The non-autoregressive model predicts each remaining code sequence conditioned on text, acoustic-condition codes, and preceding target code sequences.For efficiency, training randomly selects one codec code ID j ∈ [1, . . . , 7] rather than aggregating losses over all j values.
  • Non-autoregressive model training: The non-autoregressive model combines text, code, and codec-code-ID embeddings, then uses full attention to predict the corresponding target sequence.Learnable positional embeddings are added separately to text and code sequences.

3.4 VALL-E 2 Inference

At inference, VALL-E 2 prompts its codec language model with text and an unseen speaker’s speech codes. Repetition-aware sampling stabilizes autoregressive generation, grouped decoding shortens the sequence, and non-autoregressive decoding completes the remaining codes.

  • 3.4 VALL-E 2 Inference: Zero-shot inference concatenates the prompt transcription with the synthesis text and uses the unseen speaker’s encoded speech as the codec prompt.The conditional model then generates target codec codes for personalized speech.
  • 3.4.1 Autoregressive Model Inference: The autoregressive model generates the first target code sequence in groups using grouped codec language modeling.Codes within each group are predicted non-autoregressively while groups are generated autoregressively.
  • 3.4.1 Autoregressive Model Inference: Repetition-aware sampling uses nucleus sampling by default and switches to random sampling when the generated token’s repetition ratio exceeds a threshold.The ratio is computed over a preceding window during autoregressive prediction.
  • 3.4.1 Autoregressive Model Inference: The sampling method combines nucleus-sampling stability with random-sampling avoidance of infinite loops without increasing decoding latency substantially.The additional sampling operation has almost negligible runtime cost compared with model inference.
  • 3.4 VALL-E 2 Inference: After the first code sequence is generated, the non-autoregressive model generates the remaining seven sequences one by one, and the codec decoder synthesizes the waveform.The complete target code matrix is decoded into personalized speech.

4.1 Setups

The evaluation uses LibriSpeech test-clean and the more accent-diverse VCTK corpus, with subjective and objective measures covering similarity, naturalness, robustness, and perceived quality. Training and evaluation procedures also assess multiple prompts and sampling strategies.

  • Datasets and models: VALL-E 2 is trained on Libriheavy, a 50k-hour corpus with around 7000 speakers, using BPE text tokens and 24kHz EnCodec speech tokens.The models share a Transformer-based NAR architecture while AR models vary by group size 1, 2, 4, or 8.
  • Metrics: Subjective evaluation uses SMOS for speaker similarity and CMOS for comparative naturalness, with 20 external native American English speakers.SMOS ranges from 1 to 5, while CMOS compares synthesized speech against ground-truth reference speech on a -3 to 3 scale.
  • Metrics: Objective evaluation measures speaker similarity with SIM, robustness with WER, and overall perceived quality with DNSMOS.Speech continuation is evaluated over the entire utterance rather than only the continuation segment.
  • Evaluation datasets: Zero-shot TTS is evaluated on speaker-disjoint LibriSpeech test-clean and VCTK datasets.LibriSpeech uses 3s-prefix and reference-utterance prompts, while VCTK uses prompts shorter than 3s, 5s, and 10s.
  • Evaluation datasets: LibriSpeech test-clean evaluation uses a 2.2-hour subset containing 40 unique speakers and tests both speech continuation and reference-utterance prompting.For reference prompting, each sample uses another utterance from the same speaker as the prompt.
  • Decoding evaluation: Inference generates the first code sequence with repetition-aware sampling and the remaining seven sequences with greedy non-autoregressive decoding.Experiments report one-sample results and five-sample results sorted by SIM, WER, or metric-wise maximization.

4.2 LibriSpeech Evaluation

On LibriSpeech test-clean, VALL-E 2 improves objective and subjective zero-shot TTS performance while increasing decoding stability and reducing sequence length through grouped code modeling.

  • Objective Evaluation: VALL-E 2 significantly outperforms baseline VALL-E in all objective evaluation settings, including robustness-related WER and DNSMOS measures.The baseline is especially weak under single sampling, whereas VALL-E 2 improves robustness in that scenario.
  • Objective Evaluation: Repetition aware sampling improves decoding stability across group sizes and supports robust speech generation with very small top-p values.The method reduces decoding errors and contributes to VALL-E 2’s strong WER performance.
  • Objective Evaluation: Grouped code modeling improves inference efficiency by shortening codec sequences while maintaining or improving performance, including with group size 4.With group size 2 in the autoregressive model, WER and DNSMOS improve; group size 4 reduces sequence length fourfold while retaining similar or better results than baseline.
  • Subjective Evaluation: In subjective evaluation over 40 speakers, VALL-E 2 surpasses VALL-E in speaker similarity SMOS and speech quality CMOS, also exceeding ground-truth speech performance.The results support human-parity zero-shot TTS performance on the LibriSpeech benchmark.
  • Ablation Study: Removing prompts lowers speaker similarity, while prompts improve robustness by constraining the search space of one-to-many speech synthesis.The prompt remains important in both autoregressive and non-autoregressive components, and explicit acoustic-condition splitting benefits the NAR model.
  • Ablation Study: 10k training examples achieve performance similar to 50k on LibriSpeech test-clean, while fewer than 10k examples degrade performance, especially with reference-utterance prompts.The training-data finding is based on the audiobook-domain experiment setting.

4.3 VCTK Evaluation

On VCTK, VALL-E 2 improves robustness, speaker similarity, and speech quality over VALL-E, while repetition-aware sampling and grouped code modeling improve stability and long-prompt performance.

  • Objective Evaluation: Repetition-aware sampling roughly halves WER in the single-sampling scenario and stabilizes decoding on challenging VCTK accents.Five-time sampling further filters low-quality samples and narrows VALL-E’s WER gap.
  • Objective Evaluation: Grouped code modeling improves WER for longer prompts by reducing sequence length and alleviating long-sequence modeling and attention-alignment problems.The method is especially beneficial when prompts are longer.
  • Decoding Stability: Repetition-aware sampling enhances decoding stability and supports more robust speech generation with relatively small top-p values.This stability pattern is also observed on LibriSpeech.
  • Subjective Evaluation: VALL-E 2 surpasses VALL-E in speaker similarity and speech quality on VCTK, matching or exceeding ground-truth performance with a 3s prompt.The evaluation uses 60 test cases from 60 distinct speakers with diverse accents.
  • Objective Evaluation: Grouped code modeling produces significant speaker-similarity gains with a 10s prompt through improved long-context modeling.The improvement is specifically reported for long prompts.
  • Ablation Studies: Speaker similarity declines significantly without prompt input, while the NAR model also requires text input to synthesize robust speech.The prompt is important for speaker-information modeling in both AR and NAR models.
  • Ablation Studies: The optimal training-data size varies by prompt length and metric; SIM consistently benefits from larger datasets, while the best WER depends on prompt duration.A 3s prompt requires more training data for its best WER than 5s or 10s prompts.

5 Conclusion

VALL-E 2 introduces repetition-aware sampling and grouped code modeling to achieve human-parity zero-shot TTS and reliably synthesize difficult sentences. The paper also notes misuse risks from preserving speaker identity.

  • Conclusion: VALL-E 2 achieves human-parity zero-shot TTS for the first time.The conclusion presents this as the paper’s central contribution.
  • Conclusion: Repetition-aware sampling improves decoding stability, while grouped code modeling improves modeling efficiency.These are described as two simple but effective methods.
  • Conclusion: VALL-E 2 reliably synthesizes complex sentences, including sentences that are challenging to read or contain numerous repeated phrases.The conclusion reports this as an observed capability.
  • Broader Impacts: Because VALL-E 2 maintains speaker identity, misuse could include spoofing voice identification or impersonating a specific speaker.The authors assume users agree to be the target speaker in the experiments.
Loading 2406.05370v2…