PaperlayerView exact revision on arXiv

Source-linked AI summary

RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis

Detai Xin, Xu Tan, Kai Shen, Zeqian Ju, Dongchao Yang, Yuancheng Wang, Shinnosuke Takamichi, Hiroshi Saruwatari, Shujie Liu, Jinyu Li, Sheng Zhao

arXiv:2404.03204v3eess.AScs.AIcs.CLcs.LGcs.SD

TL;DR

Autoregressive LLM-based TTS can produce unstable prosody and omit or repeat words, limiting robustness. RALL-E predicts pitch and duration before speech tokens and uses duration-guided masking; it reduces WER on LibriSpeech test-clean and lowers error on sentences that are hard for VALL-E.

  • Problem

    Autoregressive codec language models can generate unstable prosody and omit or repeat words because text and speech are not strictly aligned.

    Source: §1 · p. 2

  • Method

    RALL-E predicts pitch and duration as intermediate conditions for speech-token generation and uses duration-guided masking to focus attention on corresponding phonemes and prosody tokens.

    Source: §1 · pp. 2–3, §3.3 · p. 6

  • Results

    On LibriSpeech test-clean, WER drops from 5.6% to 2.5% without reranking and from 1.7% to 1.0% with reranking; hard-sentence error rate drops from 68% to 4%.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Subjective tests found better CMOS for RALL-E than VALL-E, while their speaker-similarity scores were similar.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

We present RALL-E, a robust language modeling method for text-to-speech (TTS) synthesis. While previous work based on large language models (LLMs) shows impressive performance on zero-shot TTS, such methods often suffer from poor robustness, such as unstable prosody (weird pitch and rhythm/duration) and a high word error rate (WER), due to the autoregressive prediction style of language models. The core idea behind RALL-E is chain-of-thought (CoT) prompting, which decomposes the task into simpler steps to enhance the robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts prosody features (pitch and duration) of the input text and uses them as intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E utilizes the predicted duration prompt to guide the computing of self-attention weights in Transformer to enforce the model to focus on the corresponding phonemes and prosody features when predicting speech tokens. Results of comprehensive objective and subjective evaluations demonstrate that, compared to a powerful baseline method VALL-E, RALL-E significantly improves the WER of zero-shot TTS from $5.6\%$ (without reranking) and $1.7\%$ (with reranking) to $2.5\%$ and $1.0\%$, respectively. Furthermore, we demonstrate that RALL-E correctly synthesizes sentences that are hard for VALL-E and reduces the error rate from $68\%$ to $4\%$.

1 Introduction

LLM-based codec TTS enables zero-shot speaker cloning, but autoregressive generation can produce unstable prosody and omit or repeat words. RALL-E uses prosody tokens and duration-guided masking to address these robustness problems.

1 Introduction

  • LLM-based codec TTS uses discrete speech tokens and can clone a speaker from a short audio prompt in a zero-shot setting.Source: §1 · p. 1
  • On LibriSpeech test-clean, WER falls from 5.6% to 2.5% without reranking and from 1.7% to 1.0% with reranking; hard-sentence error rate falls from 68% to 4%.The hard-sentence evaluation compares RALL-E and VALL-E on 50 particularly difficult sentences.Source: Table 1 · p. 1, §1 · pp. 2–3
  • Autoregressive codec language models can generate diverse prosody, but may produce unnatural prosody or omit and repeat input words; reranking adds inference time.Source: §1 · p. 2
  • RALL-E predicts pitch and duration as intermediate CoT prompts before speech tokens, then uses duration-guided masking to focus attention on corresponding phonemes and prosody tokens.The method is designed to stabilize speech-token generation and strengthen phoneme–speech-token alignment.Source: §1 · pp. 2–3
  • Comprehensive objective and subjective evaluations report better robustness for RALL-E than VALL-E and two previous works.Source: §1 · p. 3

2 Related work

Related work covers language-model approaches to TTS and methods for improving robustness in autoregressive systems. RALL-E separates duration prediction from speech-token prediction to increase control over generation.

LLM-based TTS

  • LLM-based TTS concatenates text and speech tokens for next-token training and uses large speech datasets and models to support zero-shot in-context learning.Source: §2 · p. 3

Robust autoregressive TTS

  • For encoder-decoder autoregressive TTS, monotonic attention [2] [9] and non-attentive Tacotron [24] are prior approaches to improve robustness.Source: §2 · p. 3
  • ELLA-V [26] and VALL-T [6] entangle duration and speech-token prediction, whereas RALL-E predicts phoneme durations first for greater control; VALL-T also slows training with a forward process for every phoneme.Source: §2 · p. 3

3 RALL-E

RALL-E adds prosody prediction as a chain-of-thought step and uses duration to guide attention, aiming to improve robustness in LLM-based TTS. Although implemented with VALL-E, the method can be applied to any decoder-only autoregressive TTS model.

3 RALL-E

  • RALL-E models prosody and speech-token distributions with a Transformer, using predicted prosody to guide speech generation.The method is presented as applicable beyond its VALL-E implementation to decoder-only autoregressive TTS models.Source: §3 · pp. 3–4

3.1 Preliminary: VALL-E

  • VALL-E converts speech into EnCodec residual-quantization tokens, predicting the first layer autoregressively and the remaining layers non-autoregressively.Its AR and NAR Transformers use different attention masks for speech tokens.Source: §3.1 · p. 4

3.2 Prosody tokens as chain-of-thought prompts

  • RALL-E predicts phoneme-level pitch and duration before speech tokens, then conditions speech-token prediction on those prosody sequences.Pitch is extracted using phoneme–speech alignment, and duration is also predicted before speech generation.Source: §3.2 · p. 5

3.3 Enhancing alignment with duration-guided masking

  • Duration-guided masking confines each autoregressive speech token to a centered window of 2k+1 phoneme and prosody positions.The window accommodates alignment errors and neighboring-phoneme dependence; masking is applied only to the AR Transformer, and duration-guided inference stops at the predicted total duration.Source: §3.3 · pp. 5–6

4 Experiments

RALL-E is evaluated on LibriSpeech, where it improves objective and subjective robustness over VALL-E; ablations and tests on hard sentences further examine its components and scope.

4.1 Setup

  • The English MLS training subset contains about 44K hours from 5,490 speakers; testing uses 1,205 LibriSpeech test-clean utterances with same-speaker prompts.Prompts use the first three seconds of another utterance by the same speaker.Source: §4.1 · p. 6

4.2 Main results

  • 55% relative WER improvement and 41% relative WER-R improvement over VALL-E accompany reductions in all three error types on the 1,205-utterance test set.The error-type values change from 2.8/1.5/1.3 to 1.7/0.6/0.3, respectively.Source: Table 2 · p. 7, §4.1 · p. 7, §4.2 · p. 8
  • RALL-E scores better CMOS than VALL-E and slightly better than ground truth, while its SMOS is similar to VALL-E.RALL-E also has a higher UTMOS score than VALL-E in the objective evaluation.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Ablation study

  • Removing any component degrades robustness; removing duration CoT prompting performs worst even with duration-guided masking, making CoT prompting the most important component.The ablation compares removing pitch, window masking, duration-guided masking, and duration CoT prompting.Source: §4.3 · pp. 8–9, Table 4 · p. 8

4.4 Evaluations on hard sentences

  • On 50 particularly hard sentences, RALL-E reduces sentence error rate from 68% with VALL-E to 4%, with only two mispronunciation errors.Each sentence was synthesized five times and the best sample was selected; errors were manually assessed because WER was imprecise for sentences with many numbers and symbols.Source: §4.4 · p. 9

5 Conclusions

RALL-E uses prosody features as CoT prompts and duration-guided masking to improve robustness in LLM-based TTS. Its evaluations report improvements over VALL-E, including on sentences that are particularly hard for that baseline.

  • RALL-E uses pitch and duration as CoT prompts to stabilize speech-token generation and duration-guided masking to focus attention on corresponding phonemes and prosody features.Source: §5 · p. 9
  • Comprehensive objective and subjective evaluations show significantly improved robustness over VALL-E and two previous works.Source: §5 · p. 9
  • On sentences particularly hard for VALL-E, RALL-E achieves a 4% error rate, approaching non-autoregressive TTS performance.Source: §5 · p. 9

A Window size study

The study varies the duration-guided masking window size k, which controls how many phonemes and prosody features can be attended during decoding. Results favor k = 1 across sampling settings.

  • A window of size k permits attention to 2k + 1 phonemes and prosody features, balancing added context against imperfect duration estimates and alignment learning.Source: A Window size study · p. 12
  • Increasing k from 0 to 1 substantially improves WER in every sampling setting, and the best performance occurs at k = 1.Source: A Window size study · p. 12
  • WER does not improve further for k above 1, while k = ∞ yields the worst WER; more diverse sampling makes robustness problems more evident.Source: A Window size study · p. 12

B Transcripts of the 50 hard sentences

Appendix B lists the 50 hard sentences used in Section 4.4 as numbered transcripts.

Loading 2404.03204v3…