Source-linked AI summary
Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder
Kei Akuzawa, Yusuke Iwasawa, Yutaka Matsuo
TL;DR
Autoregressive speech synthesis has difficulty modeling global characteristics such as speaker identity and speaking style, leaving expression control without labels an open problem. The paper combines VoiceLoop with a VAE to incorporate those characteristics into synthesis; experiments report improved quality and controllable speech expressions.
Problem
Autoregressive speech synthesis struggles to model global speech characteristics and lacks a structured way to control expressions, while expression labels can be costly and unreliable.
Source: §1 · p. 1Method
VAE-Loop combines VoiceLoop with a VAE that represents global characteristics as a tractable distribution and provides hints to the speech generator.
Source: §1 · p. 1Results
The VAE helps VoiceLoop attain lower test errors and higher MOS without labels, and its latent variables control speaker individualities and speaking styles and interpolate between them.
Source: §1 · p. 1Takeaways & Limitations
The experiments support incorporating global characteristics explicitly into speech generation to synthesize higher-quality speech and control expressions without labels.
Source: §5 · p. 4The study considers only synthesizing speech with a particular expression, not predicting expressive information from text.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Recent advances in neural autoregressive models have improve the performance of speech synthesis (SS). However, as they lack the ability to model global characteristics of speech (such as speaker individualities or speaking styles), particularly when these characteristics have not been labeled, making neural autoregressive SS systems more expressive is still an open issue. In this paper, we propose to combine VoiceLoop, an autoregressive SS model, with Variational Autoencoder (VAE). This approach, unlike traditional autoregressive SS systems, uses VAE to model the global characteristics explicitly, enabling the expressiveness of the synthesized speech to be controlled in an unsupervised manner. Experiments using the VCTK and Blizzard2012 datasets show the VAE helps VoiceLoop to generate higher quality speech and to control the expressions in its synthesized speech by incorporating global characteristics into the speech generating process.
1. Introduction
The paper targets expressive speech synthesis without expression labels, motivated by the cost and imperfect reliability of annotation. It combines VoiceLoop with a VAE to model global speech characteristics and reports improved VoiceLoop quality and controllable expressions.
1. Introduction
- Natural speech varies with speaker individualities, emotions, and speaking styles; incorporating expressiveness has been reported to make synthesis more pleasant to hear and interact with.The introduction also notes that autoregressive seq2seq systems can be trained on text-audio pairs with minimal human annotation.Source: §1 · p. 1
- Unsupervised expressive speech synthesis avoids costly labels for readily available speech and unreliable annotations, whose emotion categories can contain different strengths.Prior work reports that labels help model complex audio data [7].Source: §1 · p. 1
- Autoregressive models generate data point by point, making sentence-level characteristics such as gender or emotion difficult to model and expression difficult to control.The paper links this difficulty to reduced synthesized-speech quality.Source: §1 · p. 1
- VAE-Loop combines autoregressive VoiceLoop with a VAE that represents global speech characteristics as a tractable distribution and provides hints to the speech generator.The proposed method is described as end-to-end and usable in seq2seq speech synthesis.Source: §1 · p. 1
- The VAE helps VoiceLoop achieve lower test errors and higher MOS without labels, while latent variables control speaker individualities and speaking styles and support interpolation.These results are reported for the proposed approach across the experiments.Source: §1 · p. 1
2. Related Work
Related work distinguishes methods for unsupervised expression control from this paper’s distribution-based VAE approach. The paper also situates its method among text-conditioned speech synthesis and VAE-autoregressive models outside speech synthesis.
2. Related Work
- VoiceLoop was selected as the seq2seq speech-synthesis base because prior work reported natural, intelligible speech and handling of prosody.VoiceLoop was also reported to generate high-quality speech from YouTube data containing varied speaking styles.Source: §2 · p. 1
- Prior unsupervised expressive-speech work includes non-neural emotion representations from clustering and PCA, as well as seq2seq models that learn and control speaking styles [19].The VAE method instead learns expressions as a tractable distribution that may support interpolation and semi-supervised learning.Source: §2 · pp. 1–2
- This paper considers only synthesizing speech with a particular expression, not predicting expressive information from text [2, 18].Source: §2 · p. 2
- Earlier VAE speech studies include speaker-label-conditioned voice conversion, whereas this work conditions speech synthesis on text and tests expression learning and control.The most relevant cited voice-conversion studies are.Source: §2 · p. 2
- Outside speech synthesis, a VAE-combined recurrent language model generated sentences with consistent global style or topic, while related work addressed ignored latent variables.This paper adopts the KL cost annealing approach used in to alleviate latent-variable neglect.Source: §2 · p. 2
3. Models
The section builds VAE-Loop by treating VoiceLoop as a conditional VAE decoder, with a global latent variable intended to represent expressions not predictable from text alone. It then describes how speech is generated, trained, and regularized.
3.1. Conditional Variational Autoencoder
- The conditional VAE uses acoustic features x and phonemes c, with an encoder qφ(z|x) and decoder pθ(x|z,c).Its prior and approximate posterior are modeled as Gaussian distributions.Source: §3.1 · p. 2
3.2. VoiceLoop
- VoiceLoop predicts each audio feature from previous audio features and phonemes, using a shifting buffer and speaker embedding in its update.The update function depends on attention-mediated context, the previous buffer, the latest output, and speaker embedding.Source: §3.2 · p. 2
3.3. Proposed model: VAE-Loop
- VAE-Loop conditions VoiceLoop's speech generation on a global latent variable, explicitly incorporating global characteristics into the autoregressive process.VoiceLoop is treated as the conditional VAE decoder.Source: §3.3 · p. 2
3.3.1. Modeling various expressions using VAE
- The latent variable is expected to encode expression information that is useful for estimating audio but absent from the text.The model changes VoiceLoop's probability distribution to condition generation on z.Source: §3.3.1 · p. 2
3.3.2. Generating speech using the global latent variable
- Because z does not vary with time step, it conditions speech generation globally, unlike the time-varying buffer and context.Figure 1 illustrates the VAE-Loop speech-generating process.Source: §3.3.2 · p. 2, Figure 1 · p. 3
3.3.3. Training and inference
- During training, z is sampled from the encoder, while inference samples z from the prior; the objective combines a regularizer and reconstruction error.Reconstruction error is estimated using mean squared error between estimated and true audio features.Source: §3.3.3 · p. 3
3.3.4. KL cost annealing
- KL cost annealing weights the KLD term so its regularization can increase over training.The method starts λ at 0 and increases it linearly to 1.Source: §3.3.4 · p. 3
4. Experiments
Experiments evaluated VAE-Loop on VCTK and Blizzard2012, finding gains in test error and subjective quality under some inference settings, as well as latent-variable control of speech characteristics.
4.3. Effect of latent variables on the test error
- On VCTK, VAE-Loop had lower test error than VoiceLoop without speaker labels; KL-cost annealing paired higher KLD with lower test error.Table 1 varies annealing epochs and latent dimension.Source: Table 1 · p. 3, §4.3 · p. 4
4.1. Datasets
- VCTK supplied about five hours from 21 North American speakers; Blizzard2012 supplied about 10 hours from one male audiobook reader, using utterances under five seconds.Source: §4.1 · p. 3
4.2. Experimental setup
- Semiteacher forcing modified the prior-step input to stabilize training and inference; the VAE encoder used five stride-2 convolutional layers followed by pooling and fully connected layers.Training used Adam for 150 epochs, with learning rates of 1e-4 on VCTK and 5e-5 on Blizzard2012.Source: §4.2 · p. 3
4.4. Mean opinion score tests
- VAE-Loop achieved higher MOS than unlabeled VoiceLoop on VCTK, matching labeled VoiceLoop; on Blizzard2012, suppressing prior variance raised MOS above the baseline.The MOS study used more than 15 US-based raters per dataset and a 1–5 naturalness scale.Source: Table 2 · p. 4, §4.4 · p. 4
4.5. Controlling speech expressions using latent variables
- Different latent values changed F0 characteristics on VCTK, while Blizzard2012 latent variables characterized pitch fluctuations; interpolated values yielded averaged F0 trajectories.The VCTK figures use the same latent values across two texts; the z endpoints correspond to high- and low-pitched voices.Source: Figure 2 · p. 4, Figure 3 · p. 4, §4.5 · p. 4
5. Conclusions
The conclusion presents VAE-Loop as a way to add unsupervised global expression modeling to VoiceLoop. Experiments support higher speech quality than unlabeled VoiceLoop and control over speech expressions.
5. Conclusions
- Combining VoiceLoop with a VAE incorporates global speech expressions explicitly into the autoregressive speech-generating process without labels.Source: §5 · p. 4
- Experiments indicate the method can generate higher-quality speech than VoiceLoop without labels and control synthesized speech expressions.Source: §5 · p. 4
- Future work will extend the approach to semi-supervised learning with limited labels and infer latent variables from text.Source: §5 · p. 4