PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesis

Daisy Stanton, Yuxuan Wang, RJ Skerry-Ryan

arXiv:1808.01410v1cs.CLcs.LGcs.SDeess.ASstat.ML

TL;DR

Text-to-speech must choose among many valid speaking styles for the same text, while explicit style annotations can be difficult, costly, and noisy. This paper adds text-prediction pathways to GST-Tacotron, then reports preferred audiobook synthesis and shared multi-speaker style representations that preserve speaker identity.

  • Problem

    Text-to-speech must produce natural-sounding style from text that admits many valid renderings, while explicit style annotations are difficult to define, costly, and noisy.

    Source: §1 · p. 1

  • Method

    TP-GST augments GST-Tacotron with pathways that predict GST combination weights or style embeddings from text, without requiring explicit style labels or auxiliary inference inputs.

    Source: §1 · p. 1, §3 · p. 2, §3.2 · p. 2

  • Results

    TP-GST audiobook speech was preferred by human raters over baseline Tacotron, and multi-speaker models learned a shared style space while preserving speaker identity.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The reported models predict speaking style from text alone and support multi-speaker synthesis with shared style representations and preserved speaker identity.

    Source: §5 · p. 7

    The study leaves multi-speaker factorization across genders, accents, and languages, larger textual context, and finer-grained or hierarchical temporal resolutions for future research.

    Source: §5 · p. 7

Abstract

from arXiv · show

Global Style Tokens (GSTs) are a recently-proposed method to learn latent disentangled representations of high-dimensional data. GSTs can be used within Tacotron, a state-of-the-art end-to-end text-to-speech synthesis system, to uncover expressive factors of variation in speaking style. In this work, we introduce the Text-Predicted Global Style Token (TP-GST) architecture, which treats GST combination weights or style embeddings as "virtual" speaking style labels within Tacotron. TP-GST learns to predict stylistic renderings from text alone, requiring neither explicit labels during training nor auxiliary inputs for inference. We show that, when trained on a dataset of expressive speech, our system generates audio with more pitch and energy variation than two state-of-the-art baseline models. We further demonstrate that TP-GSTs can synthesize speech with background noise removed, and corroborate these analyses with positive results on human-rated listener preference audiobook tasks. Finally, we demonstrate that multi-speaker TP-GST models successfully factorize speaker identity and speaking style. We provide a website with audio samples for each of our findings.

1. INTRODUCTION

Natural-sounding TTS must choose a speaking style for text that can have many valid spoken renderings. Explicit annotations are difficult to define, costly, noisy, and not necessarily aligned with perceptual quality.

1. INTRODUCTION

  • TTS must produce a natural-sounding speaking style even though the same text permits many valid renderings.Prosody includes pitch, stress, breaks, and rhythm; speaking style includes higher-level characteristics such as emotional valence and arousal.Source: §1 · p. 1
  • Explicit annotations for prosody and style are difficult to define precisely, costly to acquire, noisy, and not necessarily correlated with perceptual quality.Source: §1 · p. 1
  • The paper extends GSTs with two text-based style-prediction pathways that require no additional labels and aim to synthesize expressive speech without auxiliary inference inputs.The introduction also reports speaker-independent variation, background-noise capture, and favorable subjective evaluations.Source: §1 · p. 1

2. RELATED WORK

Prior approaches to prosody and emotion have often depended on explicit annotations or other labeled inputs. The paper contrasts its text-based prediction approach with those requirements and with VAE-Loop’s inference-time latent input.

2. RELATED WORK

  • Prosody annotation and classification methods such as INTSINT, ToBi, Momel, landmark detection, Tilt, and SLAM have largely required explicit annotations.AuToBI automatically detects and classifies prosodic features but requires models pretrained on labeled data.Source: §2 · pp. 1–2
  • Emotion-modeling methods have traditionally required keywords, semantic representations, or labels for training,,,,,.Source: §2 · p. 2
  • A method for predicting acoustic features such as i-vectors [17] from semantic embeddings [16] uses hand-designed features and requires three separately trained models.The stages are an acoustic-feature predictor, a neutral-prosody synthesis model, and a speaker-adaptation model.Source: §2 · p. 2
  • VAE-Loop still requires its latent variable at inference, and its analysis and samples suggest that the variable mainly captures speaker gender and identity rather than prosody or speaking style.Source: §2 · p. 2

3. MODEL

TP-GST augments GST-Tacotron with pathways that predict style from text. One predicts token-combination weights, while the other predicts style embeddings directly.

3. MODEL

  • TP-GST adds two text-prediction pathways to GST-enhanced Tacotron: TPCW predicts GST combination weights, while TPSE predicts style embeddings directly.At inference, either TP-GST pathway can run without auxiliary inputs; traditional GST-Tacotron requires supplied auxiliary inputs.Source: §3 · p. 2

3.1. Text features

  • A 64-unit time-aggregating GRU summarizes Tacotron’s variable-length text-encoder sequence as a fixed-length feature vector for both prediction pathways.Source: §3.1 · p. 2

3.2. Predicting Combination Weights (TPCW)

  • TPCW treats GST attention’s combination weights as text-prediction targets, with an additional cross-entropy loss and stopped gradient flow into the GST layer.The weights are moving targets because the attention and style tokens are updated during backpropagation; at inference, fixed tokens allow weights to be predicted from text alone.Source: §3.2 · p. 2

Encoder

The TPSE pathway predicts a style embedding directly from text features rather than using token-combination weights. Its training target is the GST style embedding.

3.3. Predicting Style Embeddings (TPSE)

  • TPSE predicts the GST style embedding from text features using one or more fully connected layers and an L1 loss, with gradients stopped from flowing back through the GST layer.Source: Encoder · p. 3, §3.3 · p. 4
  • The TPSE output uses tanh to match the style-token and text-encoder activations, a choice reported to lead to better token variation.The hidden fully connected layers use ReLU activations.Source: §3.3 · p. 4
  • At inference, TPSE predicts the style embedding directly from text and does not use the style tokens, which serve only to produce the training target.Source: §3.3 · p. 4

4. EXPERIMENTS

The experiments evaluate TP-GST synthesis in single- and multi-speaker settings. Because objective metrics often do not correlate well with perception, the authors encourage listening to the audio samples.

  • The experiments examine TP-GST synthesis with both single- and multi-speaker models.Source: §4 · p. 4
  • Objective metrics often do not correlate well with perception for generative models.Source: §4 · p. 4
  • The authors use visualizations in some experiments and encourage readers to listen to the provided audio samples.Source: §4 · p. 4

4.1. Single Speaker Experiments

Single-speaker experiments examine learned style-token variation and text-predicted synthesis on expressive audiobook speech. Raters preferred both TP-GST systems to baseline Tacotron, and the text-prediction pathways also removed audible background noise in the reported examples.

4.1. Single Speaker Experiments

  • The single-speaker system was trained on 147 hours of expressive American English audiobook speech read by Catherine Byers.The model used 20 style tokens with 4-headed additive attention and predicted both TPCW and TPSE targets during training.Source: §4.1 · p. 4

4.1.1. Style Token Variation

  • Different TP-GST tokens capture variation in pitch, energy, and speaking rate, with each token producing the same relative F0 and C0 trend across text inputs.Source: §4.1.1 · p. 5

4.1.2. Synthesis with Style Prediction

  • On an unseen audiobook phrase, TP-GST produces a more varied F0 contour and richer spectral detail than baseline Tacotron, fixing its declining-pitch problem without a reference signal at inference.Source: §4.1.2 · p. 5, Figure 3 · p. 5

4.1.3. Subjective Evaluation

4.1.4. Automatic Denoising

  • Both text-prediction pathways removed audible high-frequency background noise found in about 10% of the training recordings.The authors report that about 6% of rater comments mentioned this effect, but did not measure the total number of utterances with and without noise.Source: §4.1.4 · p. 6

4.2. Multiple Speaker Experiments

The multi-speaker TP-GST was trained on expressive and neutral speech from 22 female speakers, conditioning on speaker identity while sharing style tokens. Its tokens capture prosodic variation while preserving speaker identity, and demo samples suggest more expressive synthesis than a multi-speaker Tacotron for an expressive voice.

4.2. Multiple Speaker Experiments

  • The model used 190 hours of American English speech from 22 female speakers, combining expressive audiobook data with neutral-prosody news, web-article, and assistant-style speech.It conditioned on a 64-dimensional speaker-identity embedding.Source: §4.2 · p. 6

4.2.1. Shared Multi-Speaker Style Tokens

  • Shared style tokens capture variation in pitch, energy, and speaking rate while preserving speaker identity, including when synthesizing with neutral-speech voices.Source: §4.2.1 · p. 6, Figure 4 · p. 7

4.2.2. Synthesis with Style Prediction

  • For the expressive audiobook voice, demo samples show more expressive speech from multi-speaker TP-GST than from multi-speaker Tacotron conditioned on the same data.The authors did not run evaluations comparing these models.Source: §4.2.2 · p. 6

5. CONCLUSIONS AND DISCUSSION

TP-GST predicts speaking style from text alone, without explicit style labels during training or input signals at inference, and audiobook speech was preferred by human raters over baseline Tacotron. Multi-speaker models learn a shared style space while preserving speaker identity; broader applications and capabilities remain future directions.

  • TP-GST predicts speaking style from text alone without explicit style labels during training or input signals at inference.Source: §5 · p. 7
  • Human raters preferred TP-GST audiobook speech over baseline Tacotron.Source: §5 · p. 7
  • Multi-speaker TP-GST learns a shared style space while preserving speaker identity for synthesis.Source: §5 · p. 7
  • Further work will test multi-speaker factorization across genders, accents, and languages, larger textual context, and finer-grained or hierarchical style timing.Source: §5 · p. 7
  • The study investigates style prediction only within Tacotron, while suggesting TP-GST could extend to other TTS and conditionally generative models.Source: §5 · p. 7
Loading 1808.01410v1…