Source-linked AI summary
VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
TL;DR
Decoder-only neural-codec language models for zero-shot TTS face alignment-related robustness challenges and costly autoregressive inference. VALL-E R combines phoneme monotonic alignment with codec merging; the authors report stronger robustness and that first-layer-only downsampling preserves quality and intelligibility.
Problem
Decoder-only TTS can suffer alignment degradation on complex or long sequences, while high-rate codec tokens make autoregressive inference computationally costly.
Source: §1 · p. 2Method
VALL-E R uses phoneme prediction and monotonic alignment, plus codec merging to downsample discrete codes in the first layer.
Source: §1 · p. 2, §6 · p. 12Results
The authors report stronger robustness; halving only the first-layer sampling rate leaves audio quality and intelligibility almost unaffected, unlike broader merging that decreases performance without improving inference speed.
Source: §5.4 · p. 10, §6 · p. 12Takeaways & Limitations
VALL-E R supports flexible prosody control while retaining timbre cloning, and its codec-merging approach is reported to improve inference speed without affecting sound quality.
Source: §1 · p. 2, §6 · p. 12Experiments presumed speaker consent, and future applications involving new speakers should ensure consent and implement synthetic-speech detection tools.
Source: §1 · pp. 2–3
Abstract
from arXiv · showhide
With the help of discrete neural audio codecs, large language models (LLM) have increasingly been recognized as a promising methodology for zero-shot Text-to-Speech (TTS) synthesis. However, sampling based decoding strategies bring astonishing diversity to generation, but also pose robustness issues such as typos, omissions and repetition. In addition, the high sampling rate of audio also brings huge computational overhead to the inference process of autoregression. To address these issues, we propose VALL-E R, a robust and efficient zero-shot TTS system, building upon the foundation of VALL-E. Specifically, we introduce a phoneme monotonic alignment strategy to strengthen the connection between phonemes and acoustic sequence, ensuring a more precise alignment by constraining the acoustic tokens to match their associated phonemes. Furthermore, we employ a codec-merging approach to downsample the discrete codes in shallow quantization layer, thereby accelerating the decoding speed while preserving the high quality of speech output. Benefiting from these strategies, VALL-E R obtains controllablity over phonemes and demonstrates its strong robustness by approaching the WER of ground truth. In addition, it requires fewer autoregressive steps, with over 60% time reduction during inference. This research has the potential to be applied to meaningful projects, including the creation of speech for those affected by aphasia. Audio samples will be available at: https://aka.ms/valler.
1 Introduction
Neural-codec language models have made zero-shot TTS promising, but robustness and inference efficiency remain challenges. VALL-E R combines phoneme monotonic alignment and codec merging, with controllable prosody and stated safeguards around voice use.
1 Introduction
- Decoder-only TTS can lose phoneme–audio alignment on complex or long sequences, while high-rate codec tokens make autoregressive inference computationally costly.The introduction notes a 75 Hz sampling rate for Encodec and says prior approaches to robustness or efficiency can add inference or training costs.Source: §1 · p. 2
- VALL-E R adds phoneme prediction during training and monotonic alignment during inference, alongside codec merging to reduce autoregressive steps.The authors describe the alignment strategy as adding almost no computational burden.Source: §1 · p. 2
- Codec merging reduces first-layer code rate without retraining the codec, while phoneme alignment targets robustness and phoneme-based input supports prosody control.The authors also report that the system can clone timbre while adjusting pronunciation prosody for voice conversion.Source: §1 · p. 2
- VALL-E R is a research project with no immediate plans for commercial transition or broader public access; voice replication quality varies with prompt length, clarity, and background noise.The authors identify impersonation and misuse of voice-recognition systems as inherent risks.Source: §1 · p. 2
- The experiments presumed consent from reproduced speakers; future applications involving new speakers should ensure voice-use consent and implement synthetic-speech detection.The paper directs concerns about inappropriate or rights-infringing use to its Report Abuse Portal.Source: §1 · pp. 2–3
2 Related Work
Related work establishes neural-codec language modeling as a basis for zero-shot TTS and surveys alignment approaches for robustness. The paper positions efficiency in decoder-only systems and compatibility with that architecture as open concerns.
2.1 Neural Codec Language Modeling
- VALL-E [Wang et al., 2023a] framed zero-shot TTS as conditional language modeling over discrete codecs, enabling generation for unseen speakers from short speech samples.Subsequent work expanded multilingual and cross-modal capabilities and explored robustness improvements.Source: §2.1 · p. 3
- Prior decoder-only TTS improvements addressed robustness, but the related-work review says autoregressive efficiency remained an unresolved pain point.The review names RALL-E [Xin et al., 2024] among concurrent robustness efforts.Source: §2.1 · p. 3
2.2 Monotonic Alignment
- Monotonic attention and related alignment methods target skipping and repetition, but the surveyed methods use encoder-decoder architectures and are incompatible with decoder-only models.The review contrasts hard attention with windowed or band-constrained approaches that retain more contextual attention.Source: §2.2 · p. 3
3 VALL-E R
VALL-E R pairs codec merging with a decoder-only neural codec language model that learns phoneme–acoustic alignment. Its inference follows phoneme order and supports separate control of prompt timbre and source prosody.
3 VALL-E R
- The system combines codec merging, which improves inference speed without codec retraining, with a monotonic alignment strategy for a decoder-only neural codec language model.The paper presents these as its two main components.Source: §3 · p. 4
3.1 Codec-Merging Approach
- Codec merging downsamples the first quantizer layer so adjacent codes repeat, while leaving the other layers unchanged.The codec architecture inserts a merging module before vector quantization; the described operation downsamples and then repeats the residual representation.Source: Figure 2 · p. 4, §3.1 · p. 4
3.2 Neural Codec LM with Monotonic Alignment
- To adapt monotonic strategies to decoder-only TTS, VALL-E R adds phoneme prediction during training and imposes monotonic alignment during inference.The training and inference process is summarized in Figure 3.Source: §3.2 · pp. 4–5, Figure 3 · p. 6
3.2.1 Training with Phoneme Prediction
- The AR stage predicts first-layer acoustic tokens and aligned phonemes from phoneme input, while the NAR stage iteratively predicts later quantizer layers using phonemes, alignment, and prior layers.The models use teacher forcing in AR training; the paper also shares acoustic embedding and output prediction layer parameters.Source: §3.2.1 · p. 5
3.2.2 Inference with Monotonic Alignment
- At inference, the phoneme pointer can stay on the current phoneme or advance one position; monotonic coverage requires acoustic tokens to follow phoneme order and represent every phoneme.Acoustic-token sampling is used for diversity, while later quantizer layers are generated by greedy search.Source: §3.2.2 · pp. 5–6
3.2.3 Inference for Controlling Prosody
- Replacing self-predicted phonemes with preset phoneme sequences lets users control prosody while retaining the prompt speaker’s timbre, enabling a voice-conversion use case.The stated goal is to preserve the source speech’s linguistic information and prosody while matching the prompt speech’s timbre.Source: §3.2.3 · p. 6
4 Experiments Setup
The experiments use LibriSpeech and compare VALL-E R with established systems under matched training data and architecture. Evaluation covers robustness, speaker similarity, sound quality, subjective judgments, and inference efficiency.
4.1 Dataset
- Experiments use 960 hours of transcribed, multi-speaker English LibriSpeech data, with Montreal Forced Aligner for phoneme alignment and Encodec codes with Vocos as vocoder.Encodec compresses 24 kHz waveforms into eight code layers at 75 Hz.Source: §4.1 · p. 7
4.2 Evaluation Metrics
- Evaluation uses WER for robustness and intelligibility, Spk-Sim for speaker similarity, PESQ and STOI for reconstruction quality, and QMOS, SMOS, and CMOS for human ratings.CMOS assesses synthesized speech against a baseline on a scale from -1 to 1.Source: §4.2 · p. 7
4.3 Baseline
- Robustness comparisons include VALL-E, VALL-T, and ELLA-V; efficiency comparisons include AudioLM, VALL-E, ELLA-V, RALL-E, MusicGen, and other token arrangements.The robustness baselines share LibriSpeech training data and a stacked 12-layer Transformer architecture with VALL-E R.Source: §4.3 · p. 7
4.4 Training Configuration
- The AR and NAR models use 12-layer Transformers and are trained for 400k steps on eight NVIDIA V100 16GB GPUs with AdamW.Training uses 16 attention heads, 1024-dimensional embeddings and hidden states, a 4096-dimensional feed-forward network, and 0.1 dropout.Source: §4.4 · pp. 7–8
5 Results and analysis
VALL-E R improves objective and subjective TTS results, supports phoneme-level prosody control, and reduces inference time through codec merging. Analyses indicate that first-layer downsampling preserves quality better than broader merging, while phoneme alignment supports robust decoding and complete monotonic coverage.
5.1 Zero-shot Text-to-Speech tasks
- VALL-E R approaches Encodec’s WER in continuation and achieves the best WER in cross-sentence synthesis, with speaker similarity comparable to VALL-E and ELLA-V.The evaluation covers continuation and cross-sentence tasks on LibriSpeech test-clean; Encodec is the ground-truth audio passed through its encoder and decoder.Source: Table 1 · p. 8, §5.1 · p. 8
- VALL-E R exceeds VALL-E in subjective quality and intelligibility, while prompt-audio compliance is comparable and naturalness, clarity, and comprehensibility approach Ground Truth.The reported measures include QMOS, SMOS, and CMOS.Source: §5.1 · pp. 8–9, Table 2 · p. 8
5.2 Controllability of VALL-E R
- VALL-E R-Prosody achieves the lowest MCD-DTW-SL by using aligned target phonemes to preserve target prosody while cloning timbre from an acoustic prompt.The evaluation compares generated and target audio for speech-length and alignment similarity.Source: §5.2 · p. 9, Table 3 · p. 9
5.3 Efficiency Comparision
- Reducing first-layer RVQ bitrate from 75 Hz to 37.5 Hz cuts autoregressive steps and increases inference speed by more than two times.The paper reports that VALL-E R also outpaces VoiceBox and CLaM-TTS; Table 4 reports inference times for generating 10 seconds of speech.Source: §5.3 · pp. 9–10, Table 4 · p. 9
5.4 Analysis and Discussion
- Half-rate downsampling of the first codec layer leaves audio quality and intelligibility almost unaffected, whereas extending it to four or eight layers significantly degrades performance without improving speed.Tripling or quadrupling downsampling causes only a slight quality decrease, indicating further efficiency potential.Source: Table 5 · p. 10, §5.4 · p. 10
- At lower top_p, VALL-E R maintains content generation and diversity, whereas VALL-E can favor silent frames and enter loops.Figure 4 presents WER trends as top_p varies during decoding.Source: Figure 4 · p. 10, §5.4 · p. 11
- Attention visualizations show scattered VALL-E attention as audio lengthens, while VALL-E R’s phoneme alignment path is uninterrupted and monotonic, covering phonemes one by one.Figure 5 encodes greater attention weight with brighter color.Source: Figure 5 · p. 11, §5.4 · p. 11
- Ablations show that merged codecs improve inference speed without harming WER or speaker similarity, while removing monotonic phoneme alignment significantly worsens WER.Removing the alignment strategy returns the model to VALL-E.Source: §5.4 · pp. 11–12, Table 6 · p. 12
6 Conclusion
VALL-E R combines codec merging with phoneme prediction and monotonic alignment for zero-shot TTS. The paper reports improved robustness and flexible prosody control.
6 Conclusion
- Codec merging downsamples discrete codes to improve inference speed without affecting sound quality.Source: §6 · p. 12
- Phoneme prediction during training and monotonic alignment during inference help model the correlation between phonemes and audio and improve synthesis robustness.Source: §6 · p. 12
- VALL-E R provides flexible control of prosody.Source: §6 · p. 12
7 Ethics Statement and Broader Impact
VALL-E R preserves speaker identity, which may create risks of voice spoofing or impersonation. The experiments assumed users agreed to be the target speaker.
7 Ethics Statement and Broader Impact
- Preserving speaker identity may enable misuse such as spoofing voice identification or impersonating individuals.Source: §7 · p. 12
- The experiments assumed that users agreed to be the target speaker in synthesized speech.Source: §7 · p. 12
- For real-world use with unseen speakers, the authors say authorization protocols and synthetic-speech detection are imperative.Source: §7 · p. 12