Source-linked AI summary
PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions
Reo Shimizu, Ryuichi Yamamoto, Masaya Kawamura, Yuma Shirahata, Hironori Doi, Tatsuya Komatsu, Kentaro Tachibana
TL;DR
Prompt-based TTS has offered natural-language control of speaking style, while prior approaches had limited control over speaker identity. PromptTTS++ adds speaker prompts and models speaker-related embeddings to generate speech; subjective evaluations report better control of specified speaker characteristics than methods without the speaker prompt.
Problem
Prior prompt-based TTS systems used style prompts that describe limited aspects of speaker individuality, and some relied on training-set speaker IDs, restricting generation to those speakers.
Source: §1 · p. 1Method
PromptTTS++ adds a speaker prompt designed to describe identity independently of style and uses mixture density networks to model diverse speaker and style embeddings.
Source: §1 · p. 1, §2.2 · p. 2Results
Subjective evaluations found better control of specified speaker characteristics than methods without a speaker prompt.
Source: §1 · p. 1Takeaways & Limitations
The learned embeddings formed distinct clusters for different speakers with speaker prompts, whereas style-prompt-only embeddings formed approximately two gender-related clusters.
Source: §4.4 · p. 4
Abstract
from arXiv · showhide
We propose PromptTTS++, a prompt-based text-to-speech (TTS) synthesis system that allows control over speaker identity using natural language descriptions. To control speaker identity within the prompt-based TTS framework, we introduce the concept of speaker prompt, which describes voice characteristics (e.g., gender-neutral, young, old, and muffled) designed to be approximately independent of speaking style. Since there is no large-scale dataset containing speaker prompts, we first construct a dataset based on the LibriTTS-R corpus with manually annotated speaker prompts. We then employ a diffusion-based acoustic model with mixture density networks to model diverse speaker factors in the training data. Unlike previous studies that rely on style prompts describing only a limited aspect of speaker individuality, such as pitch, speaking speed, and energy, our method utilizes an additional speaker prompt to effectively learn the mapping from natural language descriptions to the acoustic features of diverse speakers. Our subjective evaluation results show that the proposed method can better control speaker characteristics than the methods without the speaker prompt. Audio samples are available at https://reppy4620.github.io/demo.promptttspp/.
1. INTRODUCTION
Prompt-based TTS systems offer natural-language control of voice characteristics, but prior approaches provide limited control over speaker identity. PromptTTS++ adds a speaker prompt designed to describe identity independently of speaking style, and evaluations report improved control over specified speaker characteristics.
1. INTRODUCTION
- Earlier prompt-based systems used style prompts that primarily reflected prosody and limited speaker individuality, while some systems tied speaker generation to training-set speaker IDs.That speaker-ID basis prevents generating speakers beyond those in the training data.Source: §1 · p. 1
- PromptTTS++ introduces a speaker prompt designed to describe speaker identity approximately independently of the style prompt.It also uses GMM-based mixture density networks to model style and speaker embeddings and a diffusion-based acoustic model.Source: §1 · p. 1
- Subjective evaluations found that PromptTTS++ controlled specified speaker characteristics better than methods without a speaker prompt.The study also reports that speaker prompts alleviate the difficulty of controlling speaker identity beyond gender distinctions seen with style prompts alone.Source: §1 · p. 1
- The work constructs a LibriTTS-R dataset with manually annotated speaker prompts because no suitable database existed for the task.Source: §1 · p. 1
2. METHOD
PromptTTS++ predicts a style embedding from speaker and style prompts and uses it to condition acoustic speech generation. Its method combines a reference encoder, prompt encoder, acoustic model, and training losses.
2. METHOD
- The system uses a reference encoder, prompt encoder, and acoustic model to generate speech from input text and speaker/style prompts.At inference, the predicted style embedding conditions mel-spectrogram generation, followed by vocoding.Source: §2 · p. 1, Figure 1 · p. 2
2.1. Reference encoder
- The reference encoder maps log-scale mel-spectrograms to fixed-dimensional style embeddings that are weighted combinations of learned style tokens.Source: §2.1 · p. 2
2.2. Prompt encoder
- The prompt encoder predicts style embeddings from speaker/style prompts using a pretrained language model and an MDN.The paper calls the learned representations style embeddings, although they also capture speaker factors.Source: §2.2 · p. 2
- A GMM-based MDN models the conditional distribution of embeddings, supporting diverse speaker characteristics and sampling from that distribution.The prompts are concatenated before BERT processing; the resulting prompt embedding passes through linear layers.Source: §2.2 · p. 2
2.3. Acoustic model
- The acoustic model generates mel-spectrograms from content prompts and style embeddings, using a content encoder, variance adaptor, and diffusion decoder.The adaptor predicts duration and pitch; the diffusion decoder denoises noisy mel-spectrograms during training and generates clean ones from noise at inference.Source: §2.3 · p. 2
2.4. Training
- Training minimizes decoder, duration, pitch, and prompt-encoder losses, using weighted variational lower bound, L1, and log-likelihood losses as specified.The reference-encoder output is stop-gradient when optimizing the prompt-encoder loss.Source: §2.4 · p. 2
3. DATASET
The dataset pairs speaker prompts with LibriTTS-R speakers and utterance-level style prompts. Experts annotated speaker characteristics, while style prompts were generated from acoustic statistics.
3. DATASET
- The dataset uses LibriTTS-R, a 585-hour corpus with 2,456 English speakers, and randomly selected 404 speakers for speaker-prompt annotation.Source: §3 · p. 3
- Experts annotated each selected speaker using five randomly chosen audio samples and predefined identity-related words, with room to create descriptions.Examples include young, old, gender-neutral, deep, muffled, raspy, and clear.Source: §3 · p. 3
- Style prompts describe pitch, speaking speed, and energy per utterance, using acoustic statistics and three-level labels for pitch, speed, and loudness.Semantically equivalent variations generated with Llama2 brought the total to 1,349 unique style prompts.Source: §3 · p. 3
4. EXPERIMENTAL EVALUATIONS
The evaluation tests PromptTTS++ on naturalness and prompt-to-speech consistency, comparing it with systems that omit the speaker prompt or MDN and with a reference-speech system. The results favor PromptTTS++ for speaker-prompt consistency and show that adding the speaker prompt separates embeddings by speaker rather than mainly by gender.
4.1. Data
- The LibriTTS-R data were split using 10 evaluation speakers and a validation set comprising 2% of speakers; speaker prompts were annotated for 404 speakers.Training also used data from speakers without speaker prompts, with style prompts supplied instead; preliminary experiments favored this partially annotated full dataset over annotated data alone.Source: §4.1 · p. 3
4.2. Models
- The system uses reference and prompt encoders with a diffusion acoustic model, while comparisons remove the speaker prompt or MDN or replace diffusion with a Transformer decoder.The prompt encoder uses BERT and an MDN; the diffusion decoder is based on non-causal WaveNet.Source: §4.2 · p. 3
4.3. Evaluation
- Evaluation comprised two subjective listening tests: a 5-point naturalness mean opinion score (MOS), rated from Bad to Excellent, and a 4-point prompt-to-speech consistency MOS, rated from Inconsistent to Consistent.Style-prompt consistency was not evaluated because the study focused on controllability of speaker individuality.Source: §4.3 · p. 3
4.4. Results
- All proposed methods significantly outperformed baseline B1 on naturalness, while models without MDN scored better than the other proposed systems.The authors suggest raters preferred averaged voice characteristics; non-MDN models lack sampling capability.Source: §4.4 · p. 4, Table 2 · p. 4
- P1 significantly outperformed systems without the speaker prompt on prompt-to-speech consistency for both training and test speakers, and was on par with reference-speech input.P1 was not statistically different from the model without MDN, suggesting MDN adds sampling capability but does not necessarily improve controllability.Source: §4.4 · p. 4, Table 2 · p. 4
- The t-SNE plots show distinctly clustered embeddings for individual speakers with the proposed method, versus approximately two gender-based clusters without the speaker prompt.The authors interpret this as evidence that style prompts alone make speaker individuality beyond gender difficult to control.Source: §4.4 · p. 4, Figure 2 · p. 4
5. CONCLUSION
PromptTTS++ adds speaker prompts and a manually annotated LibriTTS-R dataset to control speaker identity in prompt-based TTS. The experiments support speaker identity control, while future work includes broader annotations, disentangling speaker and style factors, and novel speaker generation.
5. CONCLUSION
- PromptTTS++ uses speaker prompts and a manually annotated LibriTTS-R dataset, and the experiments demonstrate speaker identity control with the additional prompt.The authors identify larger-scale annotations, disentangling speaker and style factors, and novel speaker generation as future work.Source: §5 · p. 4
- The authors omitted t-SNE plots for non-MDN models because they could not learn complex distributions, making the results difficult to interpret.This limits the embedding-space comparison reported for those models.Source: §5 · p. 4