Source-linked AI summary
Learning latent representations for style control and transfer in end-to-end speech synthesis
Ya-Jie Zhang, Shifeng Pan, Lei He, Zhen-Hua Ling
TL;DR
The paper addresses unsupervised learning and control of speaking style in end-to-end speech synthesis. It adds a VAE-based style representation that supports latent manipulation and reference-based transfer, using training techniques to avoid KL collapse. The model demonstrates disentangled and compositional style representations and outperforms GST in ABX style-transfer tests, especially for non-parallel transfer.
Problem
The paper seeks continuous, controllable representations of speaker state and speaking style for expressive end-to-end speech synthesis.
Method
A VAE recognition network infers style representations from reference audio, which are combined with Tacotron 2 text states to synthesize speech; KL annealing and periodic KL updates prevent collapse.
Results
The learned latent space supports disentanglement, interpolation, and combination, while the proposed model outperforms GST in ABX tests for parallel and non-parallel style transfer.
Takeaways & Limitations
Latent style representations can provide direct style control and reference-based transfer in an end-to-end TTS framework.
Takeaways & Limitations
The study uses a single-speaker dataset, and future work targets more disentangled, interpretable representations and multispeaker style transfer.
Abstract
from arXiv · showhide
In this paper, we introduce the Variational Autoencoder (VAE) to an end-to-end speech synthesis model, to learn the latent representation of speaking styles in an unsupervised manner. The style representation learned through VAE shows good properties such as disentangling, scaling, and combination, which makes it easy for style control. Style transfer can be achieved in this framework by first inferring style representation through the recognition network of VAE, then feeding it into TTS network to guide the style in synthesizing speech. To avoid Kullback-Leibler (KL) divergence collapse in training, several techniques are adopted. Finally, the proposed model shows good performance of style control and outperforms Global Style Token (GST) model in ABX preference tests on style transfer.
2 Microsoft China
The paper focuses on unsupervised learning, variational autoencoders, style transfer, and speech synthesis.
- The paper combines unsupervised learning with variational autoencoders for style transfer in speech synthesis.
1. INTRODUCTION
The paper targets expressive, controllable speech synthesis by introducing VAE-based style representations into end-to-end TTS. These representations support manipulation, reference-based transfer, and style variation, with the approach distinguished from related work by its control objective and end-to-end setting.
- End-to-end TTS has achieved high voice quality, while expressive speech synthesis remains an active research interest.
- VAE models complicated distributions unsupervised and offer disentangled factors, smooth interpolation, and continuous latent sampling.
- The paper introduces VAE into Tacotron2 to learn continuous latent representations of speaker state for speaking-style control.
- Reference-audio variational inference enables style transfer by inferring a style representation and using it to guide synthesized speech.
- Direct latent manipulation controls style, while sampling the prior can generate varied speaking styles for data augmentation.
- Unlike Akuzawa et al., this work emphasizes style control through latent manipulation or reference inference in an end-to-end TTS model.
2. MODEL
The model combines a VAE recognition network with Tacotron 2 to encode reference-audio style and condition text-to-speech generation. Its training objective includes reconstruction, KL, and stop-token terms, with KL annealing and delayed KL updates used to prevent collapse.
- 2.1. Variational Autoencoder: The VAE approximates an intractable posterior with a recognition model and optimizes a variational lower bound.
- 2.1. Variational Autoencoder: The prior is Gaussian, while encoder networks learn posterior mean and variance; reconstruction and KL terms form the training losses.
- 2.1. Variational Autoencoder: Reparameterization expresses sampling as z = µ + σ ⊙ϵ, allowing stochastic latent variables to be trained by backpropagation.
- 2.2. Proposed Model Architecture: The proposed architecture encodes reference audio into a fixed-length style vector and combines it with text-encoder states in Tacotron 2.
- 2.2. Proposed Model Architecture: Reference audio is processed by a recurrent encoder and fully connected layers that produce latent means and standard deviations before reparameterized sampling.
- 2.2. Proposed Model Architecture: The TTS loss conditions reconstruction on latent variable z and text t, adds stop-token loss, and uses mel-spectrogram L2 loss.
- 2.3. Resolve KL collapse problem: KL collapse is addressed by gradually increasing the KL weight and applying KL loss only once every K steps.
3. EXPERIMENTS AND ANALYSIS
Experiments evaluate latent-style control and reference-based transfer using a 32-dimensional VAE representation on audiobook speech. Manipulating and combining latent dimensions controls style attributes, while transfer produces reference-like prosody and outperforms GST in ABX tests.
- Experimental setup: The experiments use 105 hours of audiobook recordings from one English speaker, with 58,453 training utterances, 200 test utterances, and GST as the baseline.The proposed model uses 32-dimensional latent variables.
- Experimental setup: Parallel transfer preserves the reference audio’s text, whereas non-parallel transfer uses different target text and reference audio.Both transfer settings are generated and evaluated.
- Latent representation analysis: Interpolating between latent representations gradually decreases both pitch and speaking rate in generated speech.The two endpoint representations correspond to high-rate/high-pitch and low-rate/low-pitch speech.
- Latent representation analysis: Adjusting individual latent dimensions independently controls pitch height, local pitch variation, or speaking rate, supporting disentangled latent factors.Spectrogram manipulations show that changing one dimension alters only one style attribute while other dimensions remain fixed.
- Latent representation analysis: Combining pitch-height and pitch-variation dimensions produces audio inheriting characteristics from both attributes, demonstrating additive latent control.The combination is evaluated by summing the corresponding latent representations.
- Style transfer evaluation: The proposed model outperforms GST on parallel and non-parallel style transfer in ABX tests, with p-value < 10^-5.Transferred spectrograms show similarities to references in pitch height, pause time, speaking rate, and pitch variation.
4. CONCLUSION
The model learns continuous latent speaking-style representations for unsupervised style control and achieves strong style-transfer performance, outperforming GST in ABX testing. Future work targets more disentangled, interpretable representations and extends style transfer beyond single-speaker settings.
- The VAE learns continuous latent speaking-style representations that can control speaking style in synthesized speech.
- The learned latent space supports disentangled factors, enabling flexible representation of speaking style.
- The proposed model outperforms the GST model in ABX style-transfer tests, especially for non-parallel transfer.
- Future work will pursue more disentangled and interpretable latent representations.
- The scope of style-transfer research is intended to expand from single-speaker to multispeaker settings.