Source-linked AI summary
Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis
Guangzhi Sun, Yu Zhang, Ron J. Weiss, Yuan Cao, Heiga Zen, Yonghui Wu
TL;DR
Prosody representations can be difficult to interpret when phone-level latent dimensions are entangled and phone-only modeling misses word-level coherence. The paper adds hierarchical conditioning across utterance, word, phone, and latent-dimension levels to Tacotron 2; it reports improved interpretability and disentanglement while maintaining similar reconstruction performance.
Problem
Phone-level latent dimensions can be entangled, while phone-level representations alone miss within-word coherence and other utterance attributes.
Source: §1 · p. 1Method
Results
The model improves interpretability through disentanglement; scheduled training yields energy-duration-F0 ordering, while word-level F0 sampling sounds more natural than phone-level sampling.
Source: §5.2 · p. 3, §6 · p. 4Takeaways & Limitations
Word-level prosody adjustment retains the prosody distribution within a word, whereas phone-level adjustment required manual assignment to keep it natural.
Source: §5.2 · p. 3The multilevel VAE structure assumes that different latent dimensions have independent effects on prosody attributes.
Source: §4 · p. 2
Abstract
from arXiv · showhide
This paper proposes a hierarchical, fine-grained and interpretable latent variable model for prosody based on the Tacotron 2 text-to-speech model. It achieves multi-resolution modeling of prosody by conditioning finer level representations on coarser level ones. Additionally, it imposes hierarchical conditioning across all latent dimensions using a conditional variational auto-encoder (VAE) with an auto-regressive structure. Evaluation of reconstruction performance illustrates that the new structure does not degrade the model while allowing better interpretability. Interpretations of prosody attributes are provided together with the comparison between word-level and phone-level prosody representations. Moreover, both qualitative and quantitative evaluations are used to demonstrate the improvement in the disentanglement of the latent dimensions.
1. INTRODUCTION
The paper targets interpretable control of prosody at utterance, word, and phone scales, where phone-level latent dimensions can be entangled and phone-only representations miss within-word coherence. It proposes hierarchical conditioning and scheduled training to organize these representations.
1. INTRODUCTION
- Earlier fine-grained prosody work aligned representations to phones and reported localized control with more variability and higher robustness to speaker perturbations.Many earlier approaches instead used utterance-level representations that capture salient utterance features [11].Source: §1 · p. 1
- The goal is disentangled control of prosody attributes at multiple levels, including fine-scale features such as a vowel’s F0.Phone-level dimensions may be entangled, and phone-level representations alone do not capture within-word coherence, noise, or channel properties.Source: §1 · p. 1
- The model extracts prosody representations at utterance, word, and phone levels, with finer features conditioned on coarser ones.Word-level features are intended to capture prosody within each word, while phone-level features capture information associated with each phone.Source: §1 · p. 1
- A conditional VAE conditions each latent dimension on preceding dimensions, and scheduled training yields a consistent ordering of phone-level prosody attributes.The paper evaluates disentanglement on three significant attributes.Source: §1 · p. 1
2. PRIOR WORK
Prior work developed utterance-level and phone-aligned prosody representations, while this paper situates its multilevel approach within hierarchical modeling and disentanglement research. The related approaches differ in how they extract hierarchy and what their decomposition is intended to achieve.
2. PRIOR WORK
- The proposed multilevel model extends fine-grained VAE work and is related to hierarchical VQVAE [18] and multilevel information extraction.Unlike image-oriented downsampling in [18], this model uses the hierarchical structure of spoken language.Source: §2 · p. 1
- Prior disentanglement methods often learn independently distributed variables without associating them with actual latent factors.Other approaches use labeled features [27] [28] or adversarial training.Source: §2 · p. 1
- A prior result showed unsupervised disentanglement is impossible without inductive bias, while the most similar hierarchical decomposition aimed to learn statistically independent variables.The two cited works address distinct issues: identifiability in and statistical independence in.Source: §2 · p. 1
3. MULTILEVEL PROSODY MODELING STRUCTURE
The multilevel structure extends phone-aligned fine-grained prosody modeling with word-level features and integrates the hierarchy into Tacotron 2. Word and phone representations are extracted from aligned spectrograms and supplied to the decoder for reconstruction.
3. MULTILEVEL PROSODY MODELING STRUCTURE
- Location-sensitive attention aligns the target spectrogram to phone encodings, after which phone-aligned features are used by the decoder for reconstruction.The fine-grained VAE extracts a sequence of phone-level latent prosody features aligned with the phone sequence.Source: §3 · pp. 1–2
- Word-level latent features are extracted from spectrograms averaged over the phones in each word, using known phone-word alignment.Phone-level features are extracted conditioned on word-level features, and both are concatenated according to the alignment before decoding.Source: §3 · p. 2
- The multilevel evidence lower bound includes latent features associated with words and phones, with the utterance-level feature incorporated by conditioning finer features on it.The phone index maps to its corresponding word index, and the model can add a KL penalty for the utterance-level latent feature.Source: §3 · p. 2
4. INTERPRETABLE CONDITIONAL VAE STRUCTURE
The conditional VAE addresses dependence among latent dimensions by inferring each dimension using information from those extracted earlier. Combined with conditioning across phone, word, and utterance levels, scheduled training yields an interpretable energy-duration-F0 ordering.
4. INTERPRETABLE CONDITIONAL VAE STRUCTURE
- The conditional VAE models dependence between latent dimensions because an earlier dimension’s meaning can inform interpretation of a later one.For example, knowing that z1 represents phone energy provides information that z2 represents attributes other than energy.Source: §4 · p. 2
- Each latent dimension is extracted using the aligned spectrogram and the sum of previously extracted projected latent features.The resulting projected features are passed directly to the decoder, and the current dimension is encouraged to represent prosody not already captured.Source: §4 · p. 2
- The auto-regressive posterior decomposition is applied across latent dimensions and across phone, word, and utterance levels.The earlier multilevel Gaussian structure assumes latent dimensions have independent effects on prosody attributes.Source: §4 · p. 2
- Scheduled training with the conditional VAE yields phone-level prosody features in an energy-duration-F0 order.The first dimension is trained before the second begins; the second can then seek duration information after energy has been represented.Source: §4 · p. 3
5. EXPERIMENTS
Experiments assess reconstruction, multilevel prosody control, and phone-level interpretability. The reported results show similar reconstruction performance for the fine-grained and fully-hierarchical VAEs, alongside improved average disentanglement and more natural word-level independent F0 sampling.
5. EXPERIMENTS
- Experiments use LibriTTS and Blizzard Challenge 2013, measure F0, energy, and duration, and evaluate reconstruction with FFE and MCD13.Phone durations are estimated from decoder attention; energy and F0 are measured over aligned phone frames, excluding 50 samples at both margins to reduce alignment variance.Source: §5 · p. 3
5.1. Reconstruction Performance
- The fine-grained VAE with 3-dimensional latent features and the fully-hierarchical VAE achieve similar reconstruction performance; lower FFE and MCD13 are better.The evaluation also reports progressive improvements from global to 3-dimensional latent representations, including reductions in VDE and F0 error.Source: §5.1 · p. 3, Table 1 · p. 3
5.2. Multilevel Controllability
- Traversing phone-level latent dimensions changes F0, energy, or duration, while word-level energy control can also change phone duration and shifts phone-level curves.The attribute control is linear across a dimension from −1 to +1; word-level effects combine across vowels and consonants.Source: §5.2 · p. 3, Figure 3 · p. 4
- Word-level independent F0 sampling sounds more natural than phone-level sampling because it retains neutral prosody structure within each word.For the MOS comparison, the unsampled level is set to zero; phone-level adjustment requires manual assignment to keep prosody natural.Source: §5.2 · p. 3, Table 2 · p. 3
5.3. Improved phone-level interpretability
- The conditional VAE improves average disentanglement, and scheduled training captures prosody attributes in the order energy, duration, then F0.The reported disentanglement measure sums ratios comparing variation in the controlled attribute with the second-largest attribute deviation; repetitions use different random seeds.Source: §5.3 · p. 4
6. CONCLUSIONS
The conclusion reports improved interpretability through better disentanglement, explains the order in which prosody attributes are extracted, and presents hierarchical control across phone, word, and utterance levels.
6. CONCLUSIONS
- Experimental results show improved disentanglement, explain the order of prosody attributes extracted, and analyze differences between phone- and word-level control.The conclusion identifies these findings as evidence of improved interpretability.Source: §6 · p. 4
- The model provides multilevel control of prosody through a hierarchy spanning phone, word, and utterance levels.Source: §6 · p. 4
- At the phone and word levels, a conditional VAE imposes a hierarchy across all latent dimensions.Source: §6 · p. 4