PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions

1. Introduction

• Discrete style IDs restrict control to fixed categories, while reference-derived styles are not interpretable and suitable references can be hard to select for arbitrary text.Source: §1 · p. 1
• Natural-language descriptions offer a convenient way to describe desired styles without requiring prior acoustic knowledge.Source: §1 · p. 1
• PromptStyle transfers style to a target speaker with natural-language descriptions through a two-stage approach, rather than requiring expressive recordings from that target speaker.It first trains a reference-based style-transfer TTS model on unannotated data, then fine-tunes with labeled style prompts.Source: §1 · p. 1
• PromptStyle can generate stylistic speech for target speakers from unseen style prompts, drawing on a pre-trained language model’s generalization capability.Source: §1 · p. 2
• Experiments report that cross-speaker style transfer with flexible text prompts achieves high speech quality.Source: §1 · p. 2

2.1. Model Overview

• The model combines a VITS-based TTS system with a style encoder for reference speech and a prompt encoder for natural-language descriptions.Source: §2.1 · p. 2
• Two-stage training first learns a style embedding space from multi-style expressive speech, then fine-tunes the model with natural-language prompts to link semantic and style embeddings.Source: §2.1 · p. 2

2.2. Cross-modal Style Encoder

• The cross-modal style encoder comprises style and prompt encoders that construct a shared space for stylistic and semantic representations.Source: §2.2 · p. 2
• A reference encoder extracts style from Mel-spectrograms, while an adversarial speaker-classification component is used to disentangle speaker and style information.The style encoder is trained to build a style space from expressive speech without style annotations.Source: §2.2 · p. 2
• A pre-trained BERT and adapter turn prompt text into an embedding sized to match the style embedding, capturing their relationship.Source: §2.2 · p. 2

2.3. Training and Inference

• The two embedding spaces allow either reference speech or text descriptions to control style, and annotated-data type and scale can be set for domain needs with time-saving fine-tuning.Source: §2.3 · pp. 2–3

2.3.1. Stage 1: Style transfer by reference Mel

• In stage one, VITS processes phonemes independently of speaker and style, while style conditions the duration predictor and normalizing flow.The stage also uses SSIM loss and a gradient-inverted speaker-classifier cross-entropy loss; the flow is designed as volume-preserving with Jacobian determinant one.Source: §2.3.1 · p. 3

2.3.2. Stage 2: Style transfer by text prompt

• In stage two, cosine similarity loss trains the prompt encoder and TTS system to align prompt embeddings with style embeddings; the style encoder remains frozen.Only BERT’s last attention layer is trained.Source: §2.3.2 · p. 3

3.1. Datasets

• The target was a two-hour single-speaker audiobook with little expressive speech; source data comprised 12 hours from eight female speakers, including six hours with prompts.Nine professional annotators described utterance style while ignoring linguistic content.Source: §3.1 · p. 3

3.2. Model Configuration

• Experiments compared PromptStyle against CST-TTS and GST-MLTTS, using audiobook style-transfer baselines with text prompts or reference speech.The systems were trained for 200,000 steps; StylePrompt fine-tuning used 10,000 steps.Source: §3.2 · p. 3

3.3.1. Subjective Evaluation

• PromptStyle had clearly better speech quality, speaker similarity, and style similarity than the compared models in MOS evaluation.The MOS results report averages and 95% confidence intervals.Source: Table 1 · p. 4, §3.3.1 · p. 4

3.3.2. Objective evaluation

• On t-SNE embeddings of 13 training styles, PromptStyle clustered same-style samples and separated different styles better than GST-MLTTS and MST-TTS.The evaluation used manually selected common styles from the training dataset.Source: §3.3.2 · p. 4

3.4. Ablation Study

• The ablation varied style and speaker conditioning across the posterior encoder, normalizing flow, and decoder, with successive variants removing selected embeddings.PromptStyle removes style embeddings from the decoder; M3 additionally removes them from the posterior encoder.Source: §3.4 · p. 4

3.5. Style Control by Text Prompt

• In ABX tests across quality, speaker similarity, and style similarity, listeners most often chose “No preference” between prompt- and reference-style conditions.The reference-style embedding serves as a performance top line because reference speech contains more direct style information than a text prompt.Source: Figure 3 · p. 4, §3.5 · p. 4

3.4. Ablation Study

• M1 preserved target-speaker timbre but failed at decent style transfer; the authors report difficulty extracting style when both embeddings condition the normalizing flow.Source: §3.4 · p. 4
• Adding speaker embeddings to VITS's decoder improved captured timbre, while style embeddings in the posterior encoder improved expressiveness.Source: §3.4 · p. 4

• PromptStyle is a style-transfer model that uses natural-language prompts to make transfer more controllable and user-friendly.Source: §4 · p. 4
• Two-stage training establishes a relationship between the style-embedding space and the prompt-embedding space.Source: §4 · p. 4
• In expressive audiobook synthesis, PromptStyle transfers style from text prompts while maintaining relatively high stability and speaker similarity.Source: §4 · p. 4