Source-linked AI summary
Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ Skerry-Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Fei Ren, Ye Jia, Rif A. Saurous
TL;DR
Expressive TTS needs controllable prosody, but style labels are difficult and costly to obtain. This work adds unlabeled global style tokens to Tacotron, enabling token- or reference-based style control and transfer. GSTs also factorize noise and speaker variation in unlabeled found data, supporting scalable robust synthesis.
Problem
TTS systems struggle to model and control expressive prosody, while explicit style labels are costly and difficult to define reliably.
Method
The model jointly trains shared global style token embeddings with Tacotron, using reference-audio attention to form a style embedding without explicit labels.
Results
GSTs learn interpretable expressive styles, support direct token control and reference-based style transfer, and decompose noise and speaker factors in unlabeled found data.
Takeaways & Limitations
GSTs provide a general approach for interpretable and controllable style modeling that can also uncover latent variation in noisy data.
Takeaways & Limitations
Single-token conditioning can mix attributes such as pitch and speaking rate, and improving GST learning remains open.
Abstract
from arXiv · showhide
In this work, we propose "global style tokens" (GSTs), a bank of embeddings that are jointly trained within Tacotron, a state-of-the-art end-to-end speech synthesis system. The embeddings are trained with no explicit labels, yet learn to model a large range of acoustic expressiveness. GSTs lead to a rich set of significant results. The soft interpretable "labels" they generate can be used to control synthesis in novel ways, such as varying speed and speaking style - independently of the text content. They can also be used for style transfer, replicating the speaking style of a single audio clip across an entire long-form text corpus. When trained on noisy, unlabeled found data, GSTs learn to factorize noise and speaker identity, providing a path towards highly scalable but robust speech synthesis.
1. Introduction
Expressive, controllable prosody is essential for human-like long-form speech, but style modeling lacks objective labels and many TTS systems average away expressive variation. GSTs address these challenges by learning unlabeled style representations within Tacotron.
- Prosody, including intonation, stress, paralinguistic information, and style, is necessary for human-like speech synthesis.
- Style influences intention, emotion, intonation, and speech flow, affecting perception in applications such as audiobooks and newsreaders.
- Style modeling is difficult because correct prosody lacks an objective measure, annotations are costly, and human raters often disagree.
- Many TTS systems learn averaged prosodic distributions, producing less expressive long-form speech and offering limited control over expression.
- GSTs are trained without prosodic labels, learn expressive styles, provide interpretable controls, and support robust synthesis from noisy unlabeled data.
2. Model Architecture
The GST-augmented Tacotron encodes a reference signal into a style embedding that conditions text-to-speech generation. At inference, style can come from selected tokens or an arbitrary audio reference for control and transfer.
- Tacotron predicts mel spectrograms from grapheme or phoneme inputs, which are converted into waveforms by inversion or a neural vocoder.
- The model combines a reference encoder, style attention, style embedding, and Tacotron sequence-to-sequence components.
- 2.1. Training: During training, the reference encoder compresses ground-truth audio prosody into a fixed-length reference embedding.
- 2.1. Training: Attention compares the reference embedding with shared token embeddings and produces weights whose weighted sum conditions the text encoder at every timestep.
- 2.1. Training: The style token layer is jointly trained from Tacotron reconstruction loss, so GSTs require no explicit style or prosody labels.
- 2.2. Inference: Inference supports direct token conditioning for style control or reference-audio conditioning for style transfer, even when the reference transcript differs from synthesized text.
3. Model Details
The implementation retains Tacotron’s core setup while using phoneme inputs, an LSTM decoder, and a compact GST layer. The baseline achieves strong naturalness, and the architecture supports alternative attention mechanisms for style learning and transfer.
- The decoder uses two 256-cell LSTM layers with zoneout probability 0.1 and outputs 80-channel log-mel spectrograms two frames at a time.
- 4.0 mean opinion score (MOS) exceeds the 3.82 MOS reported for Tacotron on the same evaluation set.
- The reference encoder applies six 2-D convolutional layers followed by a 128-unit unidirectional GRU whose final state is the reference embedding.
- The default style token layer uses 10 tokens, each 256-dimensional, to represent a small but rich variety of prosodic dimensions.
- Content-based attention measures similarity between references and tokens, while multihead attention improves style transfer more effectively than simply increasing token count.
4. Model Interpretation
GSTs represent reference embeddings as mixtures of learned basis vectors or soft clusters, functioning as interpretable style representations. They can also be viewed as memory components that store style information extracted from training data.
- GSTs decompose a reference embedding into basis vectors or soft clusters called style tokens.
- Attention scores represent each token’s contribution, allowing the style representation to use similarity-based soft combinations rather than a discrete lookup.
- The GST layer is conceptually similar to a VQ-VAE encoder because both learn quantized representations of their inputs.
- GST embeddings can be interpreted as external memory: reference signals guide writes during training and memory reads occur at inference.
5. Related Work
Prior style models commonly rely on explicit labels or hand-designed acoustic features, whereas GSTs use whole-reference summaries to capture local and global expressive attributes and support clean or noisy data.
- Many existing prosody and speaking-style models require explicit emotion or speaker labels, making supervised training dependent on costly annotations.
- Cluster-based approaches rely on complex hand-designed features and require a separate neutral voice model before partitioned training.
- Reference embeddings can transfer prosody but do not provide interpretable style control and generalize poorly on some style-transfer tasks.
- GSTs summarize the entire reference signal, capturing local and global attributes rather than only local variations primarily corresponding to F0.
- GSTs condition textual input on a disentangled style embedding and apply to both clean recordings and noisy found data.
6. Experiments: Style Control and Transfer
The experiments evaluate GSTs for controlling and transferring speaking style on expressive audiobook speech, while noting that objective metrics may not track perceptual quality well.
- GST control and style-transfer experiments use inference methods evaluated on 147 hours of American English audiobook data.The recordings feature animated, emotive storytelling, including expressive character voices with high dynamic range.
- The audiobook data includes expressive character voices with high dynamic range, creating a challenging setting for prosodic modeling.
- Objective metrics may not correlate well with perception, so the authors use visualizations and encourage listening to demonstration samples.
6.1. Style Control
GSTs enable interpretable style selection, scaling, interpolation, morphing, and improved prosodic variation by conditioning synthesis on learned token embeddings or their combinations.
- Style selection: Replacing the style embedding with an individual token lets GSTs control synthesis without a reference signal.
- Style selection: Individual tokens encode pitch, intensity, speaking rate, and emotion, with consistent relative F0 and C0 trends across different sentences.
- Style selection: A red token produces lower pitch, a green token decreasing pitch, and a blue token faster speaking rate across the plotted sentences.
- Style selection: Tokens may mix attributes reflecting stylistic co-occurrence, such as a low-pitched token also encoding slower speaking rate.
- Style selection: Conditioning on lively-sounding tokens significantly improves prosodic variation in expressive long-form synthesis.
- Style scaling: Scaling token embeddings intensifies style effects: faster-rate tokens compress spectrogram time, while animated-speech tokens increase pitch variation.
- Style scaling: Negative scaling reverses the observed effects, producing slower speaking or calmer speech despite positive softmax values during training.
- Style interpolation and morphing: Manually refined attention weights interpolate styles, while different token conditioning across text segments enables piecewise style control or morphing.
6.2. Style Transfer
GSTs transfer prosodic style from a reference signal to matching or arbitrary target text. They remain robust across reference lengths and outperform a Tacotron baseline in subjective non-parallel style-transfer tests.
- Method: GSTs use a reference signal to guide the choice of token-combination weights for prosodic style transfer, with experiments using 4-head attention.The target transcript need not match the reference transcript for non-parallel transfer.
- Parallel style transfer: In parallel transfer, GST output preserves the reference’s overall spectrotemporal envelopes and perceptually resembles its prosodic style, despite mismatched durations and formant transitions.Direct reference-embedding conditioning achieves closer fine-prosody alignment, while text-only Tacotron does not closely match the reference style.
- Non-parallel style transfer: A 258-character target remains well aligned with GST models conditioned on references of 10, 96, or 321 characters.Both 10-token and 256-token GST models show robust generalization across the three conditioning inputs.
- Non-parallel style transfer: Direct reference-embedding conditioning fails on shorter references and can reduce intelligibility on the longest reference by retaining timing and phonetic information.The per-utterance embedding captures too much source-specific information, hurting generalization.
- Non-parallel style transfer: Raters preferred both GST audiobook outputs over a Tacotron baseline in side-by-side tests using 60 sentences and unseen narrative-style references.The evaluation included many long phrases and two different reference signals.
- Implication: Non-parallel GST transfer enables a source signal to guide robust stylistic synthesis of arbitrary text.This capability is especially relevant to long-form synthesis.
7. Experiments: Unlabeled Noisy Found Data
Experiments on artificially corrupted and real found speech show that GSTs uncover noise and speaker factors without labels, enabling robust synthesis and style transfer from noisy data. The embeddings also support strong noise classification and speaker-discriminative representations.
- 7.1. Artificial Noisy Data: GSTs absorb music, reverberation, background noise, and clean speech into different tokens, allowing clean arbitrary-text synthesis from a clean token.The model was trained on artificially noisified speech and inferred by conditioning on individual tokens.
- 7.1. Artificial Noisy Data: 4.0 MOS: the GST model remains near this naturalness score across all tested noise conditions, unlike baseline Tacotron, which falls to 1.353.Increasing noise required increasing the token-bank size: 10 tokens sufficed at 50% noise, whereas noisier datasets required 20.
- 7.2. Real Multi-Speaker Found Data: In Spanish-to-English transfer, GST synthesis produces intelligible English with a similar pitch range, and its WER is much lower than the multi-speaker baseline.The Spanish accent itself is not preserved.
- 7.3. Quantitative Evaluations: t-SNE embeddings separate artificial-noise samples into two classes and TED samples into 14 speaker clusters, with female and male speakers linearly separable.These visualizations support the organization of style embeddings by noise and speaker variation.
- 7.3. Quantitative Evaluations: GST embeddings recover noise labels with 99.2% accuracy and achieve speaker-classification performance comparable to i-vectors on 431 TED speakers.Both GSTs and i-vectors were evaluated on short utterances averaging 3.75 seconds.
- 7.4. Implications: GST robustness to acoustic and textual noise may simplify found-data mining pipelines and reduce human effort for generating large-scale style annotations.The authors specifically identify segmentation, ASR, and difficult style labeling as targets for simplification.
8. Conclusions and Discussions
The paper concludes that Global Style Tokens provide an unlabeled, interpretable approach to modeling, controlling, and transferring style in end-to-end TTS. Found-data experiments further show that GSTs can decompose noise and speaker factors into separate tokens, while several extensions remain open.
- 8. Conclusions and Discussions: GSTs learn interpretable embeddings without explicit labels and can control and transfer style in expressive end-to-end TTS.The conclusion presents GSTs as a general technique for uncovering latent variations in data.
- 8. Conclusions and Discussions: Experiments on unlabeled noisy found data show that GSTs decompose noise and speaker factors into separate style tokens.This extends the method beyond speaking-style modeling to latent variation discovery.
- 8. Conclusions and Discussions: Open directions include improving GST learning and using GST weights as targets predicted from text; this work applies GST only to Tacotron.The authors also suggest applying GSTs to other end-to-end TTS models and domains.