Source-linked AI summary
Controlling Emotion in Text-to-Speech with Natural Language Prompts
Thomas Bott, Florian Lux, Ngoc Thang Vu
TL;DR
TTS must select among many valid prosodic realizations, while reference audio may be unavailable and style-description datasets are costly to create. This paper conditions speech synthesis on natural-language prompts and trains with varied prompts from merged emotional speech and text resources. The authors report prompt emotion transfer alongside preserved speaker identity and perceived naturalness.
Problem
TTS must select among many valid prosodic realizations, while reference audio may be unavailable and style-description datasets are costly to create.
Source: §1 · p. 1Method
The system conditions synthesis on prompt-text embeddings combined with speaker embeddings, using merged emotional speech and text data and varying prompts during training.
Source: §3.1 · p. 2, §3.2 · p. 3, §5 · p. 4Results
Prompt emotions transfer to speech, while speaker identities are preserved and prompt conditioning does not hinder perceived naturalness.
Source: §4.2.1 · p. 3, §4.2.2 · p. 4, §4.3.1 · p. 4Takeaways & Limitations
Users can steer prosody with natural-language prompts, including the text to be read as the prompt, without manually selecting an emotion.
Source: §1 · p. 1, §5 · p. 4Zero-shot voice adaptation is not evaluated; the study leaves out a pretrained speaker embedding function for simplicity.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
In recent years, prompting has quickly become one of the standard ways of steering the outputs of generative machine learning models, due to its intuitive use of natural language. In this work, we propose a system conditioned on embeddings derived from an emotionally rich text that serves as prompt. Thereby, a joint representation of speaker and prompt embeddings is integrated at several points within a transformer-based architecture. Our approach is trained on merged emotional speech and text datasets and varies prompts in each training iteration to increase the generalization capabilities of the model. Objective and subjective evaluation results demonstrate the ability of the conditioned synthesis system to accurately transfer the emotions present in a prompt to speech. At the same time, precise tractability of speaker identities as well as overall high speech quality and intelligibility are maintained.
1. Introduction
The introduction frames prosody control as a way to address TTS’s many valid realizations for a given text, while noting that reference-speech and style-description approaches have practical constraints. The paper proposes prompt-based prosody control that separates speaker identity from utterance prosody and reports emotion transfer with maintained speech quality.
1. Introduction
- A given text admits many valid prosodic realizations, while reference-speech control may be unavailable and style-description datasets are costly to create.Manually supplied descriptions are also limited and tend to follow similar patterns.Source: §1 · p. 1
- Emotional TTS is a controllable-TTS subfield, and prior work extracted prompts from emotional text and matched them to emotion-labeled speech.Automatically generated voice descriptions based on audio properties can limit control granularity.Source: §1 · p. 1
- The approach combines public emotional speech and text datasets, varies prompts across training iterations, and combines prompt and speaker embeddings for prosodic and timbral control.Unlike [21], it does not model speaker identity within the prompt.Source: §1 · p. 1
- The proposed architecture separates speaker voice from prompt-controlled prosody, and evaluation finds prompt emotion transfer while maintaining speaker identity and speech quality.The proposed pipeline can use the text to be read as the prompt, without manual emotion selection.Source: §1 · p. 1
2. Resources
The resources combine publicly available speech corpora, including both emotion-labeled and non-emotional speech, with emotional text prompts for training and evaluation.
2.1. Speech Datasets
- Training speech resources include LJSpeech and LibriTTS-R alongside ESD, RAVDESS, and TESS, whose utterances carry emotion labels.LJSpeech is single-speaker and 24 hours; LibriTTS-R has 2,456 speakers and 585 hours.Source: §2.1 · p. 2
2.2. Text Datasets
- Training prompts come from 10,000 Yelp reviews per emotion, while unseen evaluation prompts come from DailyDialog and Tales-Emotion.The evaluation sources contribute 50 prompts per emotion each; an emotion classifier is used to confirm or ensure prompt quality.Source: §2.2 · p. 2
3. Methods
The system integrates text-prompt and speaker representations into a transformer-based TTS pipeline. Its two-stage training first builds a robust speech system, then emphasizes learning prompt-to-emotion correspondence with varied prompts.
3.1. Prompt Conditioned TTS Architecture
- The TTS pipeline uses a Conformer encoder and decoder, prosody predictors, and a normalizing-flow post-net, with adversarial feedback during training.The prompt and speaker embeddings are concatenated, adapted through a squeeze-and-excitation block, and supplied to the encoder, decoder, and prosody predictors.Source: §3.1 · pp. 2–3
3.2. Prompt Inducing Training Procedure
- Two-stage curriculum training first uses broad speech data for robustness, then trains on emotional speech with randomly selected same-emotion prompts to support generalization.The second stage uses prompts selected from a pool of 10,000 for each emotion category.Source: §3.2 · p. 3
4. Experiments
The experiments evaluate speaker preservation, prompt-driven emotion control, and perceived speech quality. Objective and human assessments indicate that prompts steer prosody while speaker identity and naturalness are maintained.
4.1. Experimental Setup
- The objective comparison includes an unconditioned baseline with the same architecture and EmoSpeech, while testing same-text and different-prompt conditions.The evaluation uses ESD speaker identities and five emotion categories.Source: §4.1 · p. 3
4.2.1. Multi-Speaker Capabilities
- Speaker similarity remains high across ESD speakers, with identities almost perfectly preserved and unaffected by prompt embeddings.The proposed system and baseline both outperform EmoSpeech; the authors suggest multispeaker curriculum training as a likely reason.Source: §4.2.1 · p. 3
4.2.2. Prosody Controllability
- Prompt-conditioned speech aligns predicted emotions with prompt labels under both same-text and different-text prompts, while baseline speech is mostly sadness or neutral.Cramér’s V results are reported as on par with ground truth for the prompt-conditioned system.Source: Table 3 · p. 3, §4.2.2 · p. 4, Figure 3 · p. 4
4.3. Subjective Evaluation
- Human assessments use MOS ratings for naturalness, fluency, and intelligibility, plus prompt-similarity ratings across emotions and two speakers.Both rating scales are five-point scales.Source: Table 4 · p. 3, §4.3.1 · p. 4, §4.3.2 · p. 4
4.3.1. Speech Quality
- Both the baseline and proposed system score slightly but significantly below ground-truth speech in MOS, with no significant difference between them.The authors conclude that prompt conditioning does not hinder perceived naturalness.Source: Table 4 · p. 3, §4.3.1 · p. 4
4.3.2. Emotional Style Transfer
- Strong human ratings show that the same prompt transfers its emotional prosody to utterances with different emotional content.The ratings cover both evaluated speakers.Source: §4.3.2 · p. 4, Table 5 · p. 4
5. Conclusion
The proposed TTS system uses natural-language prompt embeddings to control prosody, integrating them with speaker embeddings throughout the model. Evaluation supports prompt-based prosodic control alongside high speech quality and multi-speaker capability.
- Natural-language prompt embeddings control generated prosody, while merged emotional speech and text data and iteration-varying prompts are used to encourage generalization.Prompt and speaker embeddings are supplied to the encoder, decoder, and prosody predictors.Source: §5 · p. 4