Source-linked AI summary
PromptTTS: Controllable Text-to-Speech with Text Descriptions
Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, Xu Tan
TL;DR
PromptTTS addresses the limited research and lack of datasets for guiding TTS with natural-language descriptions of speech style and content. It uses separate style and content encoders with a speech decoder, and experiments report precise style control and high speech quality, including 90.31% average style accuracy and a 3.25% improvement over the baseline.
Problem
Prompt-based guidance for TTS has little research, while existing controllable TTS requires acoustic knowledge or suitable reference speech for style control.
Method
PromptTTS extracts style and content representations from corresponding prompt descriptions and uses a speech decoder to synthesize speech from both.
Results
PromptTTS achieves 90.31% average style accuracy on PromptSpeech, outperforming the baseline by 3.25%, while slightly outperforming it in speech quality.
Takeaways & Limitations
PromptTTS provides more user-friendly prompt-based style control and establishes a dataset and evaluation framework for future prompt-guided TTS research.
Abstract
from arXiv · showhide
Using a text description as prompt to guide the generation of text or images (e.g., GPT-3 or DALLE-2) has drawn wide attention recently. Beyond text and image generation, in this work, we explore the possibility of utilizing text descriptions to guide speech synthesis. Thus, we develop a text-to-speech (TTS) system (dubbed as PromptTTS) that takes a prompt with both style and content descriptions as input to synthesize the corresponding speech. Specifically, PromptTTS consists of a style encoder and a content encoder to extract the corresponding representations from the prompt, and a speech decoder to synthesize speech according to the extracted style and content representations. Compared with previous works in controllable TTS that require users to have acoustic knowledge to understand style factors such as prosody and pitch, PromptTTS is more user-friendly since text descriptions are a more natural way to express speech style (e.g., ''A lady whispers to her friend slowly''). Given that there is no TTS dataset with prompts, to benchmark the task of PromptTTS, we construct and release a dataset containing prompts with style and content information and the corresponding speech. Experiments show that PromptTTS can generate speech with precise style control and high speech quality. Audio samples and our dataset are publicly available.
1. INTRODUCTION
PromptTTS addresses the lack of user-friendly, prompt-based style control for speech synthesis by using natural-language descriptions of speech style and content. The work introduces a dataset, system, and evaluation metric for this task.
- Previous controllable TTS methods require acoustic knowledge, explicit style-factor values, or suitable reference speech, making style control time-consuming and less user-friendly.
- PromptTTS accepts a style description and a content description separated by a colon to synthesize speech matching both.For example, a prompt can describe a lady whispering slowly while specifying the utterance to convert to speech.
- The PromptSpeech dataset contains prompts with style and content information alongside corresponding speech recordings.The prompts describe five style factors: gender, pitch, speaking speed, volume, and emotion.
- PromptTTS comprises style and content encoders plus a speech decoder that uses their representations to synthesize speech.
- The proposed evaluation metric measures accuracy between style factors extracted from output speech and those specified by prompts.
2. METHOD
PromptTTS separates style and content representations from a natural-language prompt, conditions content processing on style, and decodes both representations into speech features.
- The style encoder uses BERT to map a style prompt into a style representation that guides the content encoder and speech decoder.The style representation is taken from the BERT [CLS] hidden vector.
- BERT is fine-tuned on auxiliary classification of five style factors so it focuses on style-related information in prompts.
- The content encoder converts the content prompt into phonemes and extracts a content representation conditioned on the style representation.It prepends the style representation to every Transformer block and uses a variance adaptor to predict duration, pitch, and energy.
- The speech decoder concatenates style and content representations and generates a mel-spectrogram consistent with both prompts.
3. DATASET
PromptSpeech was constructed to provide paired prompts and speech for evaluating prompt-guided TTS, with synthesized and real-speech versions covering specified style factors.
- PromptSpeech contains speech recordings paired with corresponding prompts because existing TTS datasets lacked prompts.
- The synthesized version uses commercial TTS API speech covering five style factors: gender, pitch, speaking speed, volume, and emotion.
- The real version uses LibriTTS speech constructed through a similar process but includes only four style factors because LibriTTS lacks emotion.
- Both PromptSpeech versions are split into training and test sets and released publicly for PromptTTS experiments.
4.1. Model Configuration
The model configuration uses a pretrained, fine-tuned BERT style encoder, a Transformer-based content encoder with variance prediction, and a HiFiGAN vocoder for waveform synthesis.
- The style encoder uses a pretrained BERT model with 12 hidden layers and 110M parameters, fine-tuned with P-tuning v2 on five style factors.
- The content encoder contains a variance adaptor and four Transformer blocks, with 256-dimensional style and content representations.The variance adaptor includes duration, pitch, and energy predictors.
- The speech decoder uses Transformer blocks sharing the content encoder architecture and converts output mel-spectrograms to speech with pretrained HiFiGAN.
4.2. Baseline System
The two-stage baseline first predicts style-factor values from style prompts, then converts those values into a style embedding that guides speech synthesis.
- The first stage explicitly predicts style-factor values in style prompts using a BERT-based model fine-tuned with P-tuning v2.
- The second stage converts predicted style-factor values into a style embedding that guides the output speech style.
4.3. Evaluation Metric
Evaluation measures whether output speech matches the style factors specified by prompts, using signal-processing tools or classifiers depending on the factor.
- Evaluation uses accuracy between style factors extracted from output speech and those specified in prompts.
- Signal-processing tools evaluate pitch, speaking speed, and volume, while trained classifiers recognize gender and emotion categories.
- ≥98% classifier accuracy on gender and emotion supports PromptSpeech as distinguishable and the classifiers as an evaluation metric for PromptTTS.
4.4. Results
PromptTTS outperforms the two-stage baseline in style control while maintaining slightly better perceptual speech quality, and fine-tuning is important for style understanding.
- 4.4.1. Main Results: PromptTTS outperforms the two-stage baseline on all five style factors in PromptSpeech and achieves 90.31% average accuracy, 3.25% above the baseline.
- 4.4.2. Ablation Studies: P-tuning v2 slightly outperforms standard BERT fine-tuning, while omitting fine-tuning causes a large performance drop.
- 4.4.3. Performance Analysis: The two-stage baseline reaches 94.97% accuracy after stage one and 91.60% after stage two, with mistakes propagating across stages.
- 4.4.3. Performance Analysis: PromptTTS is end-to-end and uses a latent space to transfer information without cascaded error, supporting its better accuracy.
- 4.4.4. Speech Quality: PromptTTS slightly outperforms the two-stage baseline in perceptual speech quality according to MOS and CMOS evaluation.
5. CONCLUSION
PromptTTS synthesizes speech consistent with a prompt’s style and content, offering a more user-friendly control method than previous controllable TTS approaches. The released PromptSpeech dataset supports this task, while future work targets unseen style factors and improved prompt representations.
- PromptTTS synthesizes speech consistent with prompts in both style and content.
- PromptTTS controls speech generation more user-friendly than previous controllable TTS approaches.
- PromptSpeech is a released dataset of prompts and corresponding speech for prompt-guided TTS.
- Future work will explore zero-shot control of unseen style factors and improve prompt representation pre-training.