Source-linked AI summary
TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models
Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, Zhou Zhao
TL;DR
Controllable TTS has relied on acoustic-factor inputs or reference speech, while high-quality, large-scale datasets for natural text style prompts are lacking. The paper introduces TextrolSpeech and a multi-stage prompt-programming approach, alongside Salle, a codec-token language-model architecture. Salle surpasses PromptTTS across five style factors and improves MOS similarity by 0.14.
Problem
Controllable TTS has relied on users supplying acoustic-factor values or selecting reference speech, and high-quality, large-scale open datasets with natural text style prompts are lacking.
Source: §1 · p. 1Method
The paper releases TextrolSpeech and develops multi-stage GPT prompt programming and Salle, which autoregressively uses text and style prompts to generate the first layer of acoustic codec tokens.
Source: §2.1 · p. 2, §1 · p. 2, §2.3 · p. 3Results
Salle surpasses PromptTTS across five style factors with 87.6% average accuracy and improves MOS similarity by 0.14.
Source: §4.2.1 · p. 4, §4.2.2 · p. 4Takeaways & Limitations
The authors anticipate TextrolSpeech and Salle can serve as future baselines for text-style control in TTS.
Source: §5 · p. 4The authors note substantial room to improve audio quality and style-control robustness, particularly for emotional effects.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Recently, there has been a growing interest in the field of controllable Text-to-Speech (TTS). While previous studies have relied on users providing specific style factor values based on acoustic knowledge or selecting reference speeches that meet certain requirements, generating speech solely from natural text prompts has emerged as a new challenge for researchers. This challenge arises due to the scarcity of high-quality speech datasets with natural text style prompt and the absence of advanced text-controllable TTS models. In light of this, 1) we propose TextrolSpeech, which is the first large-scale speech emotion dataset annotated with rich text attributes. The dataset comprises 236,220 pairs of style prompt in natural text descriptions with five style factors and corresponding speech samples. Through iterative experimentation, we introduce a multi-stage prompt programming approach that effectively utilizes the GPT model for generating natural style descriptions in large volumes. 2) Furthermore, to address the need for generating audio with greater style diversity, we propose an efficient architecture called Salle. This architecture treats text controllable TTS as a language model task, utilizing audio codec codes as an intermediate representation to replace the conventional mel-spectrogram. Finally, we successfully demonstrate the ability of the proposed model by showing a comparable performance in the controllable TTS task. Audio samples are available at https://sall-e.github.io/
1. INTRODUCTION
The introduction identifies usability and dataset gaps in natural-text controllable TTS. It presents TextrolSpeech and Salle as contributions, alongside a prompt-programming method for expanding style descriptions.
1. INTRODUCTION
- Prior controllable TTS requires acoustic expertise or reference-speech selection, while reference-derived styles are described as unintuitive and weakly generalizable to unseen styles.The introduction identifies the lack of a high-quality, large-scale open text-style-prompt speech dataset as a further gap.Source: §1 · p. 1
- TextrolSpeech is introduced as an open-source, large-scale, multi-speaker speech-emotion dataset enriched with diverse natural-language style descriptions.The paper presents the dataset as intended to drive development of text-controllable TTS systems.Source: §1 · p. 1
- The authors report an efficient prompt-programming methodology that significantly expands the number of text descriptions available for each style group.The introduction describes the method as developed through experiments.Source: §1 · pp. 1–2
- Salle is proposed as a multi-stage, discrete style-token-guided control framework for TTS language models, with in-context capabilities.Source: §1 · p. 2
2. TEXTROLSPEECH DATASET
TextrolSpeech pairs speech with natural-language style descriptions and covers five style factors. The paper describes its audio and prompt preparation, then details a multi-stage GPT prompting approach to produce diverse descriptions.
2.1. Overview of the TextrolSpeech
- TextrolSpeech contains 330 hours of speech paired with 236,220 natural-language style descriptions across five factors: gender, pitch, speaking speed, volume, and emotion.The corpus is split into 200 validation samples, 200 test samples, and the remaining data for training.Source: §2.1 · p. 2
2.2. Data Collection and Text preprocessing
- The corpus combines clean LibriTTS and VCTK audio with curated emotional-speech datasets, labeling samples without emotional content as neutral.Energy, pitch, and speech rate are categorized into high, low, or normal levels using the overall data distribution.Source: §2.2 · p. 2
2.3. Prompt Programming
- A four-stage GPT-3.5-TURBO prompt program defines the task, encourages keyword synonyms and diverse phrasing, limits irrelevant descriptions, and uses hand-crafted few-shot templates.After manual selection, the authors obtained 500 naturally described prompts for each style group.Source: §2.3 · pp. 2–3
3. SALLE
Salle generates speech through two codec-language-model stages: a style-conditioned autoregressive stage predicts the first acoustic-code layer, followed by parallel prediction of later layers. Its design uses the hierarchical structure of codec codes, with later layers representing fine acoustic details.
3. SALLE
- Salle represents speech as a discrete acoustic-code matrix, with T denoting downsampled utterance length and n the number of codes per frame.The experiments set n to 8.Source: §3 · p. 3
- The first code layer often contains primary speaker information, while the following n−1 quantizer layers learn fine acoustic details.Source: §3 · p. 3
- A decoder-only autoregressive model generates first-layer acoustic tokens from the text-style prompt and text embeddings, stopping when it predicts < eos >.The architecture does not require text-speech alignment.Source: §3 · p. 3
- A second, parallelized non-autoregressive model predicts tokens from the second through the last quantizer layer to expedite generation.These later-layer predictions rely on text information and acoustic tokens from previous layers.Source: §3 · pp. 3–4
- During training, the non-autoregressive model randomly uses a layer’s ground-truth codes in each batch to mitigate learned biases.Cross-entropy loss is calculated only at the final quantizer layer.Source: §3 · p. 4
4. EXPERIMENTS
The experiments compare Salle with PromptTTS on style-factor accuracy and subjective speech quality and similarity. Salle achieves 87.6% average accuracy across five factors and improves audio similarity by 0.14, although overall sound quality is not exceptionally high.
4.1. Experiment setup
- The evaluation uses gender and emotion classifiers, labels for pitch, speech rate, and energy, and MOS ratings from 1 to 5 for quality and speaker similarity.The separate gender and emotion classifiers achieve 99.2% and 85% accuracy, respectively.Source: §4.1 · p. 4
4.2.1. Main Results
- Emotion classification accuracy tends to be lower than gender accuracy, which the authors attribute to limited data, neutral-emotion interference, and the difficulty of emotion classification.Source: §4.2.1 · p. 4
- 87.6% average accuracy: Salle surpasses PromptTTS across gender, pitch, speech, volume, and emotion on the TextrolSpeech test set.Source: Table 2 · p. 4, §4.2.1 · p. 4
4.2.2. Speech Quality
- 0.14 improvement in audio similarity: Salle outperforms PromptTTS in MOS-Q and MOS-S subjective evaluation, though overall sound quality is not exceptionally high.The evaluation includes ground-truth audio, codec-reconstructed ground truth, PromptTTS, and Salle.Source: Table 3 · p. 4, §4.2.2 · p. 4
5. CONCLUSION
The paper releases TextrolSpeech, a large-scale, high-quality text-style-prompt speech-emotion corpus, and proposes Salle, a codec language model built on the corpus. The authors identify audio quality and robustness in style control, especially emotional effects, as areas for improvement and anticipate the corpus and model serving as future baselines.
5. CONCLUSION
- TextrolSpeech is released as a large-scale, high-quality text-style-prompt speech-emotion corpus for controllable TTS.The authors describe it as the first corpus of this kind in the field.Source: §5 · p. 4
- Salle is proposed as a straightforward codec language model built upon TextrolSpeech.Source: §5 · p. 4
- The authors identify audio quality and robustness in style control, particularly emotional effects, as areas needing improvement.Source: §5 · p. 4
- The authors anticipate TextrolSpeech and Salle serving as future baselines for text-style control.Source: §5 · p. 4