Source-linked AI summary
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering
Yakun Song, Zhuo Chen, Xiaofei Wang, Ziyang Ma, Xie Chen
TL;DR
Zero-shot TTS language models face alignment and autoregressive-control challenges, including phoneme errors and prolonged silence. ELLA-V interleaves phoneme and acoustic tokens and adds phoneme-level control; the authors report higher accuracy and more stable results than VALL-E across decoding strategies.
Problem
Existing zero-shot TTS language models learn phoneme–audio alignment through self-attention and autoregressive decoding can yield repetitions, transpositions, omissions, or extended silence.
Source: §1 · p. 2Method
ELLA-V uses a generalized autoregressive model to generate interleaved phoneme and acoustic tokens, with loss on acoustic and special tokens and local advancement of phoneme-boundary cues.
Source: §1 · pp. 2–3Results
ELLA-V achieves higher accuracy and more stable results than VALL-E across decoding strategies, including greedy decoding.
Source: §1 · p. 3, §5 · p. 10Takeaways & Limitations
The authors report improved synthesis accuracy and robustness without affecting naturalness or speaker similarity.
Source: §4.2 · p. 9
Abstract
from arXiv · showhide
The language model (LM) approach based on acoustic and linguistic prompts, such as VALL-E, has achieved remarkable progress in the field of zero-shot audio generation. However, existing methods still have some limitations: 1) repetitions, transpositions, and omissions in the output synthesized speech due to limited alignment constraints between audio and phoneme tokens; 2) challenges of fine-grained control over the synthesized speech with autoregressive (AR) language model; 3) infinite silence generation due to the nature of AR-based decoding, especially under the greedy strategy. To alleviate these issues, we propose ELLA-V, a simple but efficient LM-based zero-shot text-to-speech (TTS) framework, which enables fine-grained control over synthesized audio at the phoneme level. The key to ELLA-V is interleaving sequences of acoustic and phoneme tokens, where phoneme tokens appear ahead of the corresponding acoustic tokens. The experimental findings reveal that our model outperforms VALL-E in terms of accuracy and delivers more stable results using both greedy and sampling-based decoding strategies. The code of ELLA-V will be open-sourced after cleanups. Audio samples are available at https://ereboas.github.io/ELLAV/.
1 Introduction
Zero-shot TTS language models can lose phoneme–audio alignment and, under autoregressive decoding, produce repetitions, omissions, or prolonged silence. ELLA-V interleaves phoneme and acoustic tokens, adds phoneme-level control, and reports more accurate and stable synthesis than VALL-E.
1 Introduction
- VALL-E-style concatenation leaves phoneme–audio alignment to self-attention, which may be unstable and degrade as generated sequences grow.Autoregressive generation can also produce phoneme repetitions, transpositions, omissions, or extended silence without stopping.Source: §1 · p. 2
- ELLA-V inserts phoneme tokens at corresponding acoustic-sequence positions to help the model capture local alignment between the two modalities.The framework uses a generalized autoregressive model for the first residual-quantizer layer and a non-autoregressive model for the remaining layers.Source: §1 · p. 2
- ELLA-V computes loss only on acoustic tokens and EOP/EOS special tokens, rather than maximizing likelihood over the whole hybrid sequence.The authors report reduced redundant cross-modal alignment computation and say the objective supports fine-grained inference control: after EOP, the user supplies the next phoneme.Source: §1 · pp. 2–3
- Local advance shifts EOP and the next-word phoneme token a few frames forward, giving the model upcoming-phoneme context when predicting current pronunciation.The authors note this context may matter especially for a phoneme's ending.Source: §1 · p. 3
- ELLA-V reports higher synthesized-speech accuracy than VALL-E, with comparable or superior speaker similarity and speech naturalness; its LibriSpeech test-clean WER is 2.28%.The comparison used comparable model configurations and 960 hours of LibriSpeech training data.Source: §1 · p. 3
- ELLA-V performs well across decoding strategies, including greedy decoding, and retains substantially better speech accuracy than the best VALL-E system.Ablations indicate that global advance significantly improves performance, while local advance enhances output stability.Source: §1 · p. 3
2 Related Work
Related work covers generative speech methods, from statistical and end-to-end neural TTS to discrete-audio language models. It highlights zero-shot TTS systems that condition on prompts and synthesize speech without a separate duration model.
2 Related Work
- The paper situates language models among generative AI approaches spanning text, image, video, and audio tasks.It contrasts masked language models, which may require task-specific fine-tuning, with autoregressive models' few-shot and zero-shot capabilities.Source: §2 · p. 3
- Language models for audio use discretized audio tokens; examples include hierarchical audio prediction, text-conditioned audio generation, and language-model-based voice conversion.Other work investigates how language-model word surprisal relates to synthesized prosody.Source: §2 · p. 3
- Zero-shot TTS systems such as VALL-E use discrete audio codes and conditional language modeling, while VALL-E-X extends the approach to cross-lingual voice synthesis.The cited systems use a single utterance from an unknown speaker as a prompt; autoregressive Transformer approaches can generate without an additional duration model.Source: §2 · pp. 3–4
- Earlier speech synthesis progressed from statistical parametric systems and HMM/RNN variants to end-to-end neural models using spectrograms and vocoders.Later end-to-end approaches also used VAE, flow, diffusion, and other techniques.Source: §2 · pp. 3–4
3 Method
ELLA-V treats zero-shot TTS as conditional codec language modeling, rearranging phoneme and acoustic tokens so the model learns their alignment. It combines autoregressive generation of coarse acoustic codes with non-autoregressive generation of finer codes, and adds inference controls and local advance.
3.1 Overview
- ELLA-V maps speech and text prompts into a shared token space, using EnCodec for audio and MFA for phonemes and forced alignment before rearranging the sequence.EnCodec represents 24 kHz waveforms as 75 Hz discrete tokens; the experiments use eight quantizer layers with codebooks of size 1024.Source: §3.1 · p. 4
3.2 Training: Codec Language Model
- The training sequence interleaves each non-silence phoneme with its corresponding acoustic span, placing the phoneme before the span and an EOP marker after it.The phoneme sequence is also prepended as global advance; silence retains its acoustic component without a phoneme token or EOP marker.Source: §3.2 · pp. 4–5, §3.2.1 · p. 6
3.2.1 Generalized Autoregressive (GAR) Codec Language Model
- GAR generates the first quantization layer phoneme by phoneme, predicting EOP to finish each phoneme and EOS to finish the sentence; its output is self-aligned to the phoneme prompt.Training computes loss on acoustic tokens and EOP/EOS, not phoneme tokens; global advance supplies the next phoneme directly.Source: §3.2.1 · p. 6
3.2.2 Non-Autoregressive (NAR) Codec Language Model
- The NAR model uses the previously generated quantizer layers to predict codes for each later layer in parallel, with embeddings from prior layers summed as input.Both GAR and NAR compute loss on target acoustic tokens, while GAR additionally computes loss for EOP and EOS.Source: §3.2.2 · p. 6
3.3 Inference
- At inference, ELLA-V generates speech for a text prompt conditioned on an unseen speaker’s audio prompt, using sampling for GAR and greedy decoding for NAR.After EOP, the next phoneme is appended; GAR can truncate an abnormally long phoneme to avoid possible infinite silence.Source: §3.3 · p. 7
3.4 Local Advance
- Local advance shifts the EOP and next-phoneme tokens forward by several frames, giving the current phoneme’s prediction access to upcoming-phoneme cues.The method is motivated by pronunciation, especially a phoneme’s ending, depending on upcoming phonemes as well as historical context.Source: §3.4 · p. 7
4 Experiments
Experiments compare ELLA-V with a parameter-matched VALL-E baseline on zero-shot TTS continuation and hard cross-speaker synthesis. ELLA-V shows better accuracy and robustness, including across decoding settings.
4.1 Experimental Setup
- ELLA-V was trained on LibriSpeech 960h and evaluated on continuation and hard cross-speaker tasks using objective and subjective measures.The baseline was reproduced on the same 960h training data and matched to ELLA-V at 154.3M parameters; the subjective evaluation included MOS-based comparisons.Source: §4.1 · pp. 7–8
4.2 Results
- 2.28 versus 5.00 WER (ELLA-V versus VALL-E) and +0.10 CMOS favor ELLA-V on continuation, while speaker similarity remains similar.The comparison is reported for the zero-shot TTS continuation task.Source: Table 2 · p. 8, §4.2 · pp. 8–9
- On 100 particularly hard cross-speaker synthesis sentences, ELLA-V has markedly lower WER than VALL-E.The hard cases include challenging phonetic patterns, alliteration, and unusual word combinations.Source: Table 3 · p. 9, §4.2 · p. 9
- The authors attribute VALL-E's infinite-silence tendency to repetitive silence patterns in training that can trap likelihood-based beam search; ELLA-V does not face this problem.Source: §4.2 · p. 9
- Removing either global phoneme information or local phoneme tokens significantly deteriorates synthesis accuracy.The no-global variant has SPK and WER comparable to VALL-E; the authors say combining global and local information may further improve accuracy.Source: §4.2 · pp. 9–10, Table 4 · p. 9
- As top_p decreases, ELLA-V's WER remains better than VALL-E's and its generation remains more stable, including under greedy decoding.Figure 5 reports INF, CUT, and WER trends across top_p values; VALL-E becomes more prone to silence loops as top_p falls.Source: §4.2 · p. 9, Figure 5 · p. 10
5 Conclusion
ELLA-V is a two-stage zero-shot TTS framework that interleaves acoustic and text tokens. Its GAR model supports phoneme-level control and leverages local dependencies, with experiments reporting higher accuracy and stability across top_p settings.
5 Conclusion
- ELLA-V is a two-stage zero-shot TTS framework based on language modeling and interleaved acoustic and text tokens.Source: §5 · p. 10
- Its GAR model provides fine-grained control over synthesized audio at the phoneme level and leverages local dependencies to predict pronunciation.Source: §5 · p. 10
- Experiments report higher accuracy and more stable results under different top_p thresholds for nuclear sampling.Source: §5 · p. 10