Source-linked AI summary
Hierarchical Generative Modeling for Controllable Speech Synthesis
Wei-Ning Hsu, Yu Zhang, Ron J. Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen, Jonathan Shen, Patrick Nguyen, Ruoming Pang
TL;DR
The paper addresses controllable TTS when important speech attributes are rarely annotated and vary together in real data. It introduces a hierarchical VAE with mixture-based latent spaces for interpretable, disentangled control, and finds that the resulting system can independently control speaker, noise, and style while synthesizing clean speech for noisy speakers.
Problem
Speech attributes such as style, prosody, noise, and recording conditions are difficult to annotate and often vary simultaneously, limiting independent control from a reference.
Method
GMVAE-Tacotron uses hierarchical Gaussian-mixture latent variables, with separate representations for unlabeled attributes and observed speaker-related attributes.
Results
The model independently controls speaker, noise, and style, and synthesizes high-quality clean speech for target speakers whose training data is noisy.
Takeaways & Limitations
Disentangled latent representations support controllable TTS and systematic sampling on large-scale data containing rich, unannotated speech variation.
Takeaways & Limitations
Speaker-similarity comparisons are confounded by acoustic-condition similarity and, for one baseline, unequal training-set sizes.
Abstract
from arXiv · showhide
This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model based on the variational autoencoder (VAE) framework, with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, we train a high-quality controllable TTS model on real found data, which is capable of inferring speaker and style attributes from a noisy utterance and use it to synthesize clean speech with controllable speaking style.
1 INTRODUCTION
The paper targets controllable synthesis of speech attributes that are difficult to annotate and often vary simultaneously in real data. It introduces hierarchical mixture-based latent representations to disentangle, interpret, sample, and independently control these attributes.
- Motivation: Speaking style, prosody, recording channel, and noise levels are difficult-to-annotate attributes that conditional auto-encoding can infer from reference speech.Such systems can synthesize speech resembling a reference’s prosody or noise conditions despite differing text or speaker identity.
- Motivation: Simultaneous variation of prosody, speaker, and noise conditions makes copying all attributes from one reference inadequate for independent control.Disentangled representations would allow factors such as prosody and noise condition to be selected separately and support random-attribute synthesis for data augmentation.
- Approach: The model extends Tacotron 2 with separate labeled and unlabeled latent spaces, each using Gaussian mixture priors within a variational autoencoding framework.The resulting representations support disentanglement, interpretable clusters, and systematic sampling from learned priors.
- Contributions: The proposed system is evaluated on four datasets using subjective, objective, and qualitative studies, with independent control of speaker, noise, and style despite unannotated variation.The contribution list frames this as a high-quality controllable TTS system trained on real found data with varied recording conditions, speakers, prosody, and style.
2 MODEL
GMVAE-Tacotron models speech with hierarchical latent variables: a categorical latent attribute class selects a mixture component, and a continuous representation specifies its configuration. VAE inference and Tacotron-based neural modules learn these representations from speech.
- Conditional generative model: A conventional Tacotron-like system conditions on text and optional observed labels but integrates out unlabeled attributes, producing unpredictable latent speech characteristics.The resulting conditional distribution has higher variance because factors such as prosody are not represented explicitly.
- Conditional generative model: The latent attribute class y_l is categorical, while the latent attribute representation z_l is continuous; generation samples y_l, then z_l, then speech conditioned on text and observed attributes.Their joint model factorizes as p(X, y_l, z_l | Y_t, y_o) = p(X | Y_t, y_o, z_l) p(z_l | y_l) p(y_l).
- Hierarchical latent prior: A uniform categorical prior and class-conditional diagonal Gaussians make the marginal latent prior a Gaussian mixture model with equal mixture weights.The mixture structure is intended to capture complex unseen attributes and natural clusters such as clean and noisy speech.
- Variational inference and training: The model approximates posterior inference with q(y_l | X)q(z_l | X) and trains by maximizing an evidence lower bound containing reconstruction and KL terms.The categorical posterior reuses the continuous posterior through an expectation involving p(y_l | z_l), while q(z_l | X) is estimated by Monte Carlo sampling.
- Observed attribute space: A second continuous latent space z_o models within-class observed-attribute variation, enabling one-shot inference for unseen classes such as speakers.Each observed class forms a Gaussian mixture component, and speech generation conditions on text, z_l, and a sample z_o.
- Disentanglement: Initializing p(z_o | y_o) with smaller variances than p(z_l | y_l) encourages observed and latent attributes to disentangle.The design assigns label-correlated variation to the tighter observed-attribute mixture components.
- Architecture: The implementation uses Tacotron 2 as a synthesizer plus recurrent observed and latent encoders that infer fixed-dimensional posterior parameters from mel-spectrograms.Figure 2 identifies the synthesizer, latent encoder, and observed encoder as the model’s three modules; sampling is shown with dashed lines.
3 RELATED WORK
The paper builds on reference-embedding and VAE-based TTS methods but uses mixture distributions to add interpretable latent clusters and disentangled conditional control. Its extension also models speaker attributes continuously for unseen-speaker transfer.
- Reference-based TTS: Prior TTS systems use reference embeddings to capture prosody or noise, with GST restricting embeddings to weighted learned vectors and related work further imposing one-hot weights.These approaches motivate explicit latent modeling for attributes not specified by text or speaker labels.
- VAE and mixture priors: Unlike an isotropic-Gaussian VAE that provides sampling without interpretability, GMVAE-Tacotron uses a mixture distribution to discover latent attribute clusters automatically.The mixture formulation is presented as a way to represent interpretable modes in latent attributes.
- Speaker modeling: The speaker extension adds a second mixture distribution, learning disentangled speaker and latent-attribute representations that can approximate previously unseen speakers.This generalizes prior speaker-embedding approaches by allowing continuous within-class speaker representations.
- Relation to mixture VAEs: Compared with prior mixture VAEs for images and topics, the proposed model adds flexibility for jointly modeling latent and observed attributes in conditional generation.It also relates to variable-level disentanglement methods that define different priors for different latent variables.
4 EXPERIMENTS
Experiments across varied speech conditions show that GMVAE-Tacotron discovers interpretable, disentangled latent attributes and supports controlled synthesis, including denoising and style variation. It also achieves strong synthesis quality, while speaker-similarity evaluation remains sensitive to acoustic-condition mismatch.
- Experimental setup: Across four datasets spanning speaker, channel, noise, prosody, and style variation, the model used hierarchical latent variables to evaluate controllable speech synthesis.The experiments used K = 10 latent attribute classes and 16-dimensional latent representations, with Tacotron 2 as the baseline.
- Multi-speaker English corpus: Mixture components grouped utterances by speaker, gender, and accent, achieving 92.9% assignment consistency with respect to speaker.Several components also separated groups of speakers sharing characteristics such as distinct F0 ranges.
- Multi-speaker English corpus: Individual latent dimensions controlled F0, speaking rate, accent, and starting-silence duration, demonstrating disentangled attribute representations.The model was probed by varying one dimension while keeping other factors fixed.
- Control of the background noise level: The 13th latent dimension isolated noise-level control: varying it changed SNR while other dimensions left SNR nearly constant.The analysis averaged estimated SNR over 200 utterances from two speakers, using clean or noisy component means for the other dimensions.
- Synthesizing clean speech for noisy speakers: The proposed model achieved the highest MOS and SNR when synthesizing clean speech for held-out noisy speakers.The comparison included a baseline, GST, and an isotropic-Gaussian VAE; the VAE retained moderate noise, while GST sometimes produced unnatural prosody.
- Single-speaker audiobook corpus: Prior sampling produced natural speech with wide variation in speaking rate, rhythm, and F0, while latent dimensions enabled independent style control.GST required heuristic or fitted sampling distributions and careful tuning because its weight simplex was not fully exploited during training.
- Clean synthesis for speakers with noisy training data: Denoised latent representations substantially improved SNR over the baseline and produced better quality than the original noisy audio.Using a clean-component mean or denoising arbitrary inferred latent features were both effective approaches.
- Speaker similarity: Speaker-similarity ratings were unreliable under noise and channel variation, and comparison with the larger d-vector system was not fair because training-data sizes differed.The authors identify unbiased speaker-similarity evaluation and combining d-vector transfer with GMVAE controllability as future directions.
5 CONCLUSION
GMVAE-Tacotron learns interpretable, disentangled latent representations for fine-grained attribute control and systematic sampling. The paper also extends the model for continuous speaker attributes and one-shot inference from unseen references.
- GMVAE-Tacotron enables fine-grained control of latent attributes through an interpretable and disentangled latent representation.It also provides a systematic sampling scheme for these attributes.
- The model was extensively evaluated across tasks involving varied signal conditions and independently controlled many latent attributes.It also clustered latent attributes without supervision.
- The model synthesized high-quality clean speech for target speakers despite low-quality source data, according to subjective and objective tests.These results support training controllable TTS systems on large-scale data with rich styles.
- The training objective uses Monte Carlo estimates of the variational lower bound with reparameterized latent samples.The estimator is differentiable with respect to the model parameters and posterior parameters.
A.3 DIFFERENTIABLE TRAINING OBJECTIVE WITH OBSERVED ATTRIBUTE REPRESENTATION
With an observed attribute representation, the model extends its ELBO to include observed and latent encodings. Reparameterized Monte Carlo estimates are used to train this objective.
- The observed-attribute ELBO models speech likelihood using text, observed representation, and latent representation.Its joint structure includes p(zo | yo), p(zl | yl), and p(yl).
- The objective penalizes divergence between q(zo | X) and p(zo | yo), while weighting latent-space divergence by q(yl | X).
- Training replaces continuous latent variables with reparameterized Monte Carlo samples and uses the resulting estimator.N′ and N′′ denote the sample counts used for estimation.
B NEURAL NETWORK ARCHITECTURE DETAILS
The synthesizer is an attention-based sequence-to-sequence network closely following Tacotron 2, with auxiliary encoders supplying attribute conditioning. The decoder consumes latent or observed representations at every step before waveform reconstruction by a vocoder.
- The synthesizer generates mel spectrograms from text and auxiliary-encoder conditioning signals using an attention-based sequence-to-sequence network.Its text encoder uses convolutional layers followed by a bidirectional LSTM.
- The decoder concatenates latent and observed attribute representations with its original input at each autoregressive step.The resulting vector is passed through the decoder network.
- A separately trained WaveRNN vocoder converts generated mel spectrograms into time-domain waveforms.WaveRNN replaces WaveNet for more efficient inference.
- Reference-speech encoders predict posterior means and log variances for latent and observed variables from mel spectrograms.Their disentanglement is encouraged through different KL-divergence objectives.
- Each encoder uses convolutional layers, bidirectional LSTMs, mean pooling, and a linear projection to produce posterior parameters.
C DETAILED EXPERIMENTAL SETUP
Experiments use multiple datasets and fixed latent configurations, with Adam-based ELBO training and MOS evaluation. The setup reports dimensionality trade-offs and prior hyperparameters, while noting unresolved capacity limits at lower dimensions.
- The network uses Adam with initial learning rate 10^-3, exponential decay, Xavier initialization, and batch size 256.
- Prior hyperparameters are specified separately for multi-speaker, noisy multi-speaker, audiobook, and crowdsourced datasets.The minimum conditional standard deviation was lowered from e^-1 to e^-2 for numerical stability and sufficient variation.
- Increasing zl dimensionality from 16 to 32 improves reconstruction quality but makes individual dimensions harder to interpret.Reducing dimensionality too much may limit latent-attribute modeling capacity, but this lower bound was not carefully explored.
- Sixteen-dimensional zl was empirically appropriate across the four datasets, although the number of dummy dimensions varied with dataset variation.
- The latent graphical model contains a discrete attribute class yl and a continuous attribute representation zl.
D.1 POSTERIOR COLLAPSE OF LATENT ATTRIBUTE REPRESENTATION
The latent attribute representation zl remains informative in GMVAE-Tacotron, avoiding posterior collapse because speech reconstruction benefits substantially from it. The categorical latent class yl is less prone to degenerate clusters because the Gaussian-mixture formulation trades off latent expressiveness against KL divergence.
- Continuous latent representation: The KL-divergence of zl never drops to zero, indicating that the decoder consistently uses the continuous latent representation.This occurs without KL annealing or other training tricks.
- Continuous latent representation: Speech reconstruction improves significantly when using zl, and this improvement outweighs the associated increase in KL divergence.The authors attribute the absence of collapse to the complexity of speech sequence distributions compared with text.
- Categorical latent class: The paper reports no degenerate clusters under the listed training hyperparameters.The discrete latent variable indexes mixture components in the latent attribute representation space.
- Categorical latent class: Unlike Dil-GMVAE, this model reduces to a single Gaussian when all components match, creating a trade-off between marginal expressiveness and the KL divergence on yl.That trade-off discourages degenerate components that all represent the same distribution.
- Categorical latent class: For yl, posterior collapse corresponds to all mixture components having the same conditional mean and variance.A near-uniform q(yl|X) pushes every p(zl|yl) toward the same distribution, promoting collapse.
E.2 QUANTITATIVE ANALYSIS OF THE US FEMALE COMPONENTS
The US female mixture components capture distinct speaker-related variation rather than collapsing into one shared representation. Their differences are reflected in F0 ranges and can be assessed using classifiers trained on zl.
- F0 distributions: Each of the three US female components models a different F0 range when decoding the same 25 text sequences.The comparison uses 20 latent encodings from each component and 500 synthesized utterances per component.
- F0 distributions: The component comparison isolates variation in synthesized F0 while holding the text sequence fixed.The shared text emphasizes distinctions among the US female components.
- Latent-feature classification: Linear discriminant analysis classifiers trained on zl predict speaker identity, gender, and accent.The evaluation uses 2,098 training utterances and 234 evaluation utterances.
F.1 RANDOM SAMPLES FROM NOISY AND CLEAN COMPONENTS
Sampling different mixture components produces a categorical separation between clean and noisy speech. Within the noise-related dimension, clean components occupy a narrower and lower-variance range than noisy components.
- Component-level noise control: Samples from the clean component are clean, whereas samples from the noisy component contain obvious background noise.Samples within each row share a speaker, and samples within each column share the latent attribute representation zl.
- Dimension-level noise control: Varying dimension 13 changes the synthesized background noise level while holding the other latent dimensions fixed.Rows use seeds from either noisy or clean components, with speaker conditioning shared across selected rows.
- Component-level noise control: The first four mixture components model noisy speech, while the remaining six model clean speech.The two groups are clearly separated along dimension 13, which controls background noise level.
G.1 QUANTITATIVE EVALUATION ON DISENTANGLED CONTROL
Quantitative and qualitative evaluations show that GMVAE-Tacotron supports primarily independent control of pitch, speaking rate, noise, and style. Style-transfer performance improves with a larger latent representation, while higher capacity may reduce interpretability and generalization.
- Disentangled attribute control: F0 varies substantially when the pitch dimension changes, while speech remains constant; duration varies substantially when the speaking-rate dimension changes.Speaking-rate manipulation also causes a smaller inverse pitch effect consistent with natural speaking behavior.
- Parallel style transfer: The D = 16 model outperforms the baseline but remains inferior to the 16-token GST model on parallel style-transfer metrics.The GST model has effectively 60 degrees of freedom through four-head attention.
- Parallel style transfer: Increasing the latent dimension to D = 32 greatly reduces the gap to GST while retaining fewer latent degrees of freedom and parameters.The evaluation uses MCD13 and FFE, for which lower values are better.
- Non-parallel style transfer: The model supports non-parallel style transfer by generating speech whose text differs significantly from the reference utterance.Reference style is transferred through the latent representation inferred from the reference.
H ADDITIONAL RESULTS ON THE CROWD-SOURCED AUDIOBOOK CORPUS
Figure 16 demonstrates independent control of style-, recording-channel-, and noise-related attributes using synthesized mel-spectrograms and F0 tracks. Each row varies one dimension while holding the others fixed for the same text input.
- Figure 16 varies style, recording-channel, and noise-condition attributes in synthesized mel-spectrograms and F0 tracks.
- The same text input is used for every sample, with one dimension varied per row while the remaining dimensions stay fixed.
- The top row controls F0 by traversing dimension zero without changing speaker identity.