Source-linked AI summary
Contextual Expressive Text-to-Speech
Jianhong Tu, Zeyu Cui, Xiaohuan Zhou, Siqi Zheng, Kai Hu, Ju Fan, Chang Zhou
TL;DR
Expressive TTS commonly relies on fixed labels or reference speech, which can be difficult to specify or find. Contextual TTS instead uses textual scenes to guide speech synthesis, and the authors report correct, natural, expressive speech in synthetic and real-world out-of-domain scenarios.
Problem
Existing expressive TTS methods often use fixed emotion or style labels, which can be difficult to define, or reference speech that may be hard to find.
Source: §1 · p. 1Method
The paper introduces Contextual TTS, using textual context to guide expressive speech synthesis, and constructs a dataset and framework for the task.
Source: §1 · p. 1, §2.1 · p. 2Results
The model synthesizes correct, natural, and expressive speech in synthetic data and real-world out-of-domain scenarios.
Source: §4 · p. 4Takeaways & Limitations
Contextual text offers a more flexible and generic control signal for expressive TTS than previous settings.
Source: §1 · p. 1The authors plan to collect more real-world data to extend the CTTS training dataset.
Source: §4 · p. 4
Abstract
from arXiv · showhide
The goal of expressive Text-to-speech (TTS) is to synthesize natural speech with desired content, prosody, emotion, or timbre, in high expressiveness. Most of previous studies attempt to generate speech from given labels of styles and emotions, which over-simplifies the problem by classifying styles and emotions into a fixed number of pre-defined categories. In this paper, we introduce a new task setting, Contextual TTS (CTTS). The main idea of CTTS is that how a person speaks depends on the particular context she is in, where the context can typically be represented as text. Thus, in the CTTS task, we propose to utilize such context to guide the speech synthesis process instead of relying on explicit labels of styles and emotions. To achieve this task, we construct a synthetic dataset and develop an effective framework. Experiments show that our framework can generate high-quality expressive speech based on the given context both in synthetic datasets and real-world scenarios.
1. INTRODUCTION
Existing expressive TTS often relies on fixed labels or reference speech, while this paper proposes using textual context to guide expressive speech synthesis. It introduces CTTS alongside a dataset and framework, and reports expressive speech in synthetic and zero-shot real-world settings.
1. INTRODUCTION
- Existing approaches use explicit labels or reference speech, but labeling styles and emotions can be difficult and suitable reference speech may be hard to find.Source: §1 · p. 1
- Contextual TTS (CTTS) uses textual scenes to guide speech synthesis by learning emotional, personality, and other information from context.The paper illustrates this with Alice speaking “Go away!” angrily after being hit by John.Source: §1 · p. 1
- The paper constructs a CTTS dataset and develops a framework with an encoder-decoder model and a three-stage training strategy.The framework takes context and content as input and produces speech through a mel-spectrogram output and pre-trained vocoder.Source: §1 · p. 1, Figure 1 · p. 2
- The authors present semantic context as a more flexible and generic way to control expressive TTS than previous settings.Source: §1 · p. 1
- The framework synthesizes speech with accuracy, naturalness, and expressiveness in both constructed and zero-shot real-world settings.Source: §1 · p. 2
2. METHODOLOGY
The methodology frames Contextual TTS (CTTS) as speech synthesis guided by textual context and builds a framework spanning dataset construction, model architecture, and training. It pairs context with content so the model can learn to use contextual information in speech synthesis.
2.1. CTTS Data Construction
- The CTTS dataset joins expressive speech and emotional-text datasets using emotion as the matching key, then combines text and expressive characteristics into templated context.The content text and target speech remain those of the expressive TTS sample; the synthetic data is intended to teach context–content interaction and is evaluated on out-of-domain examples.Source: §2.1 · p. 2
2.2. Model Architecture
- The model encodes context and content, predicts mel spectrograms, and uses a HiFi-GAN vocoder to synthesize speech.The architecture has a context-controlled encoder and an attention-based speech decoder.Source: §2.2 · p. 2, §2 · p. 2
2.2.1. Context-controlled Encoder
- The context-controlled encoder concatenates context-text embeddings with content-phoneme embeddings to represent both sources of information.Its output is intended to capture emotion, personality, scene, and content information from context and text.Source: §2.2.1 · p. 2
2.2.2. Attention-based Speech Decoder
- The attention-based decoder generates mel-spectrogram frames and stop probabilities autoregressively from the encoder output.The decoder is based on Tacotron2 but replaces its two-layer RNN with a Transformer; a pretrained HiFi-GAN vocoder is used downstream.Source: §2.2.2 · pp. 2–3
2.3. Training Process
- Training proceeds through speech-synthesis pretraining, context-embedding initialization from OFA, and training on the synthetic CTTS dataset.The first stage uses LJSpeech with blank context; final training uses mel-spectrogram MSE and stop-probability cross-entropy losses.Source: §2.3 · p. 3
3. EXPERIMENTS
The experiments evaluate contextual expressive speech against context-free and label-conditioned systems, then test whether M-CTTS remains expressive in real-world out-of-domain cases. M-CTTS receives strong emotional-similarity ratings and synthesizes expressive speech in those cases.
3.1. Experimental Setup
- The evaluation used 5,912 CTTS samples from four speakers and two emotions, with MOS judgments from 21 native English speakers and WER for lexical accuracy.The compared systems included contextual M-CTTS, context-free M-TTS, and label-conditioned M-LTTS.Source: §3.1 · p. 3
3.2. Results Analysis
- M-CTTS earned emotional-similarity MOS scores of 3.70 and 3.88 across two emotional speech types, while M-LTTS achieved the best performance.M-CTTS also outperformed M-CTTS-NT, which the authors attribute to the text-understanding ability of its pretrained language model.Source: §3.2 · p. 3, Figure 2 · p. 4
- WER was 0.1342 for M-CTTS and 0.1401 for Ref*, while M-TTS had the lowest WER at 0.0734.The authors note that M-TTS uses no expressive speech data and produces flat-tone speech; emotional connectives can also raise WER.Source: §3.2 · p. 4, Table 2 · p. 4
3.3. Zero-shot Cases in Reality Scenario
- M-CTTS still synthesized expressive speech in out-of-domain real-world scenarios, despite limited Yelp review text and few EmoV-DB speeches.The authors attribute this success to generalizing pretrained language-model text understanding while retaining pretrained TTS ability and fine-tuning prosody and emotion.Source: §3.3 · p. 4
4. CONCLUSION
The conclusion presents CTTS as a text-context-driven setting for expressive speech synthesis and reports correct, natural, expressive output in synthetic and real-world out-of-domain scenarios. It also identifies additional real-world training data as future work.
4. CONCLUSION
- CTTS uses textual context to control expressive speech synthesis and is described as more flexible and better generalizing than previous settings.The proposed model jointly understands context semantics and speech content text while generating expressive speech.Source: §4 · p. 4
- The experiments show correct, natural, expressive speech from context in both synthetic data and real-world out-of-domain scenarios.Source: §4 · p. 4
- The authors identify collecting more real-world data to extend CTTS training as future work.They also propose representing context through modalities beyond text, including video, audio, and images.Source: §4 · p. 4