PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptTTS 2: Describing and Generating Voices with Text Prompt

Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, Lei He, Xiang-Yang Li, Sheng Zhao, Tao Qin, Jiang Bian

arXiv:2309.02285v2eess.AScs.CLcs.LGcs.SD

TL;DR

Text prompts offer a convenient alternative to reference-speech control, but they cannot fully specify voice variability and suitable prompt datasets are costly to create. PromptTTS 2 combines a variation network with LLM-based prompt generation; its reported results show more prompt-consistent, diverse voices and improved attribute accuracy over baselines.

  • Problem

    Text-prompt TTS faces one-to-many voice descriptions that cannot capture all fine-grained speech detail, while prompt datasets are costly and difficult to scale.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    PromptTTS 2 predicts voice variability missing from text prompts with a variation network and uses an SLU model plus an LLM to generate prompts from speech attributes.

    Source: §1 · p. 2

  • Results

    PromptTTS 2 improved accuracy across attributes by an average of 1.79% over baseline systems and generated more prompt-consistent, diverse voices.

    Source: §1 · p. 3, §5.1 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    The pipeline produces high-quality prompts while reducing reliance on vendors, and sampled voice variability offers users more choices in voice generation.

    Source: §1 · p. 3, §6 · p. 10

Abstract

from arXiv · show

Speech conveys more information than text, as the same word can be uttered in various voices to convey diverse information. Compared to traditional text-to-speech (TTS) methods relying on speech prompts (reference speech) for voice variability, using text prompts (descriptions) is more user-friendly since speech prompts can be hard to find or may not exist at all. TTS approaches based on the text prompt face two main challenges: 1) the one-to-many problem, where not all details about voice variability can be described in the text prompt, and 2) the limited availability of text prompt datasets, where vendors and large cost of data labeling are required to write text prompts for speech. In this work, we introduce PromptTTS 2 to address these challenges with a variation network to provide variability information of voice not captured by text prompts, and a prompt generation pipeline to utilize the large language models (LLM) to compose high quality text prompts. Specifically, the variation network predicts the representation extracted from the reference speech (which contains full information about voice variability) based on the text prompt representation. For the prompt generation pipeline, it generates text prompts for speech with a speech language understanding model to recognize voice attributes (e.g., gender, speed) from speech and a large language model to formulate text prompts based on the recognition results. Experiments on a large-scale (44K hours) speech dataset demonstrate that compared to the previous works, PromptTTS 2 generates voices more consistent with text prompts and supports the sampling of diverse voice variability, thereby offering users more choices on voice generation. Additionally, the prompt generation pipeline produces high-quality text prompts, eliminating the large labeling cost. The demo page of PromptTTS 2 is available online.

1 INTRODUCTION

PromptTTS 2 addresses incomplete voice descriptions and costly prompt datasets with a variation network and automated prompt generation. On a 44K-hour speech dataset, it generates voices that better match prompts while supporting sampled voice variability.

1 INTRODUCTION

  • Text prompts cannot capture every detail of voice variability, while prompt datasets are costly to build and tend to be small or inaccessible.The paper identifies these as the one-to-many and data-scale challenges.Source: §1 · p. 2
  • A diffusion-based variation network predicts voice information missing from the prompt and samples different variability representations from Gaussian noise conditioned on that prompt.It predicts the reference-speech representation from the prompt representation; reference speech is used to train the network, not at inference.Source: §1 · p. 2
  • The prompt-generation pipeline recognizes speech attributes with an SLU model, then uses an LLM to compose descriptions from those recognition results.The pipeline combines sentences describing individual attributes into a comprehensive text prompt.Source: §1 · p. 2
  • The authors release an LLM-generated prompt dataset and report high-quality prompts with reduced reliance on vendors.The motivation is that vendor-written prompts are costly and laborious to produce.Source: §1 · pp. 2–3
  • Evaluated on a dataset consisting of 44K hours of speech data, PromptTTS 2 outperformed previous works in generating voices that correspond more accurately to text prompts, while supporting voice-variability control through Gaussian-noise sampling.Source: §1 · p. 3

2 BACKGROUND

Voice modeling has progressed from speaker-limited multi-speaker systems to adaptation from reference speech, while text-prompt TTS offers a more natural interface. PromptTTS 2 targets the limits of prompt detail with a variation network that predicts missing voice information.

2 BACKGROUND

  • Multi-speaker TTS controls voices represented in its dataset; later approaches adapt to new speakers using limited target data or reference speech.Few-shot methods fine-tune on target-speaker data, whereas zero-shot methods model speaker characteristics from a speech prompt.Source: §2 · p. 3
  • Text-prompt TTS avoids requiring a reference clip, but human-constructed prompt datasets can be small or publicly inaccessible because of their cost.The paper proposes LLM-generated prompts to reduce reliance on human labor.Source: §2 · p. 3
  • Adding more prompt detail can only partly alleviate the one-to-many problem, so PromptTTS 2 predicts missing voice variability conditioned on the prompt.The paper frames comprehensive fine-grained description of speech by text prompts alone as impossible.Source: §2 · p. 3

3 PROMPTTTS 2

PromptTTS 2 combines a style-controlled TTS system with a variation network that predicts reference-speech representations from text prompts, and a pipeline that generates text prompts from speech attributes. Its sections also describe prompt construction and the scope of an attribute-control accuracy table.

3.1 OVERVIEW OF TTS SYSTEM

  • The style module extracts fixed-length prompt and reference representations, which guide a TTS module that can use a phoneme-to-speech backbone.Learnable query tokens extract the fixed-length representations; the reference encoder models voice variability not captured by the prompt.Source: §3.1 · pp. 3–4

3.2 VARIATION NETWORK

  • Because reference speech is unavailable at inference, the variation network predicts its representation conditioned on the prompt representation.The network models the reference representation with diffusion, enabling different voice variability samples from Gaussian noise.Source: §3.1 · p. 4, §3.2 · p. 4
  • The variation network estimates noisy-data log-density gradients by predicting the original reference representation given the prompt, noised reference, and diffusion step.Its Transformer Encoder outputs hidden representations for the noised reference and is trained with L1 loss; FiLM layers strengthen diffusion-step awareness.Source: §3.2 · p. 5
  • At inference, the system denoises Gaussian noise into a prompt-conditioned reference representation, then combines it with the prompt representation to guide TTS.The representations guide the TTS module through cross attention.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 TEXT PROMPT GENERATION WITH LLM

  • The prompt-generation pipeline uses an SLU model to recognize speech attributes and an LLM to write prompts from the recognized labels.The LLM-generated prompts can be reused for multiple speech samples with the same labels and are designed for varied vocabulary and sentence formats.Source: §3.3 · p. 5
  • Prompt construction generates attribute keywords and placeholder-based sentences, combines attribute sentences, then samples a combined template and keywords to instantiate prompts.The described attributes include gender, pitch, speed, and volume; placeholders keep an attribute explicit and allow templates to be filled with different class keywords.Source: §3.3 · pp. 5–6, Figure 3 · p. 6
  • Adding an attribute requires defining its classes and tagging the speech dataset for that attribute with an SLU model.The pipeline then uses the LLM and SLU model to generate prompts for the added attribute.Source: §3.3 · p. 6
  • Table 1 reports synthesized-speech accuracy (%) for attribute control by PromptTTS 2 and baseline systems.Source: Table 1 · p. 7

4 EXPERIMENT CONFIGURATION

The experiments use English MLS speech and compare PromptTTS 2 with text-prompt TTS baselines on a shared modified backbone. The configuration follows settings from NaturalSpeech 2 and PromptTTS.

Datasets

  • Experiments use 44K hours of English MLS speech, 20K human-written PromptSpeech prompts, and 20K prompts generated with GPT-3.5-TURBO.The PromptSpeech test set contains 1,305 prompts; gender labels come from an open-source model, while pitch, volume, and speed are recognized with digital signal processing tools.Source: §4 · p. 7

TTS Backbone and Baseline Systems

  • PromptTTS 2 uses the NaturalSpeech 2 backbone and is compared with PromptTTS and InstructTTS after their backbones are modified to match.The fixed-dimension style representations can be integrated into TTS backbones through cross attention.Source: §4 · p. 7

Experiment Details

  • The reference speech encoder and variation network use 6 and 12 layers, respectively, with hidden size 512 and 8 query tokens for each representation.The TTS backbone, prompt encoder, and training configuration follow NaturalSpeech 2 and PromptTTS settings.Source: §4 · p. 7

5 RESULT

PromptTTS 2 improves attribute control and speech quality, while its variation network supports voice diversity and its prompt-generation pipeline produces prompts with higher classification accuracy. Further analyses examine the network’s representations and a Face2Voice extension.

5.2 STUDY OF VARIATION NETWORK

  • PromptTTS 2’s variation-network samples have markedly lower speech similarity than other variation sources, while its other sources show less variability than the baselines.Human experts report conclusions similar to the WavLM-TDNN assessment.Source: Table 3 · p. 8, §5.2 · p. 8

5.1 EFFECTIVENESS OF PROMPTTTS 2

  • 1.79% average improvement in attribute-control accuracy over baseline systems, with higher speech quality also reported.The speech-quality comparison is shown in Table 2.Source: §5.1 · p. 8

5.3 PROMPT GENERATION QUALITY

  • Prompts from the pipeline yield higher intention-classification accuracy than human-authored PromptSpeech prompts or TextrolSpeech prompts.Table 4 reports accuracy on four attributes using the PromptSpeech test set.Source: Table 4 · p. 9, §5.3 · p. 9

5.4 FURTHER ANALYSIS

  • Reference representations cluster by speaker and emotion in the PCA visualization, suggesting they capture voice variability not specified by prompts.Figure 4 plots speech samples, coloring shared speakers on the left and shared emotions on the right.Source: Figure 4 · p. 9, §5.4 · p. 9

5.5 EXTENSION ON FACE2VOICE

  • A preliminary Face2Voice experiment reports voices corresponding more closely to facial images than the baseline method.The paper frames predictable voice tokens as a basis for extensions to other modalities.Source: §5.5 · pp. 9–10

6 CONCLUSION

PromptTTS 2 combines voice-variability modeling with LLM-based prompt generation to address one-to-many and data-scale issues in text-prompt TTS.

6 CONCLUSION

  • PromptTTS 2 models voice variability missing from text prompts and uses an LLM pipeline to generate prompts, addressing one-to-many and data-scale challenges.The variation network supports more detailed voice control through Gaussian-noise sampling; the pipeline avoids vendor reliance and supports adding attributes.Source: §6 · p. 10

A EXAMPLE OF PROMPT GENERATION PIPELINE

The example illustrates a four-stage prompt-generation pipeline: generate attribute keywords, write attribute sentences with placeholders, combine sentences, then sample and fill a prompt.

A EXAMPLE OF PROMPT GENERATION PIPELINE

B DETAILS ON THE TTS BACKBONE

PromptTTS 2 uses a NaturalSpeech 2 backbone with a neural audio codec and latent diffusion model.

B DETAILS ON THE TTS BACKBONE

  • The backbone encodes waveforms into RVQ latent representations, denoises latent speech from Gaussian noise, then decodes the latents back into waveforms.The latent diffusion model uses a duration/pitch predictor and phoneme encoder as its prior.Source: B DETAILS ON THE TTS BACKBONE · p. 14

C SUBJECTIVE TEST ON THE VOICE VARIABILITY IN VARIATION NETWORK

The subjective test measured whether synthesized speech samples were judged to share a style, using five samples per aspect. Its conclusions were similar to those from the WavLM-TDNN metric.

C SUBJECTIVE TEST ON THE VOICE VARIABILITY IN VARIATION NETWORK

D ABLATION STUDY ON PROMPT GENERATION PIPELINE

The prompt-generation ablations examine placeholder use and whether the LLM writes phrases or sentences. The placeholder supports prompt quality, while format variations can marginally improve robustness.

D ABLATION STUDY ON PROMPT GENERATION PIPELINE

E EXTENSION ON FACE2VOICE

The Face2Voice extension predicts reference-speech representations from image representations. In tests against SP-FaceVC, PromptTTS 2 achieved closer face–voice correspondence, and the experiment supports voice generation conditioned on information beyond text prompts.

E EXTENSION ON FACE2VOICE

Loading 2309.02285v2…