Source-linked AI summary
Robust and fine-grained prosody control of end-to-end speech synthesis
Younggun Lee, Taesu Kim
TL;DR
The paper addresses limited temporal control and speaker sensitivity in unsupervised prosody transfer. It introduces temporally structured, variable-length prosody embeddings on the speech or text side, plus speaker-wise normalization. The methods control pitch and amplitude at frame and phoneme resolutions, improve MCD quality, and transfer song prosody across speakers.
Problem
Prior prosody embeddings offered limited control at specific moments and were not robust when source and target speakers had different pitch ranges.
Method
The paper uses variable-length prosody embeddings with temporal structure on the speech or text side and normalizes them by speaker-wise prosody means.
Results
The methods changed pitch and amplitude at frame-level and phoneme-level resolutions, improved MCD quality over GST Tacotron, and transferred song prosody to another speaker.
Takeaways & Limitations
Temporally structured and normalized prosody embeddings support fine-grained expressive control and more robust inter-speaker prosody transfer.
Takeaways & Limitations
The paper regularized the prosody embedding network only through bottleneck size and identifies disentanglement as future work.
Abstract
from arXiv · showhide
We propose prosody embeddings for emotional and expressive speech synthesis networks. The proposed methods introduce temporal structures in the embedding networks, thus enabling fine-grained control of the speaking style of the synthesized speech. The temporal structures can be designed either on the speech side or the text side, leading to different control resolutions in time. The prosody embedding networks are plugged into end-to-end speech synthesis networks and trained without any other supervision except for the target speech for synthesizing. It is demonstrated that the prosody embedding networks learned to extract prosodic features. By adjusting the learned prosody features, we could change the pitch and amplitude of the synthesized speech both at the frame level and the phoneme level. We also introduce the temporal normalization of prosody embeddings, which shows better robustness against speaker perturbations during prosody transfer tasks.
1. INTRODUCTION
The paper addresses two limitations of prior prosody embeddings: weak temporal control and non-robust transfer across speakers with different pitch ranges. It proposes variable-length embeddings and normalization to enable sequential control and more robust transfer.
- Limitations of prior work: Fixed-length prosody embeddings lose temporal information, making prosody control at a specific moment difficult.Earlier systems could set global styles such as lively or sad but lacked fine-grained temporal control.
- Limitations of prior work: Prosody transfer becomes unreliable when source and target speakers have substantially different pitch ranges.A higher-pitched female source can transfer excessive pitch to a male target.
- Proposed solution: The proposed approach uses variable-length prosody embeddings matched to reference speech or input text for sequential control.The two embedding types support control on the speech side or text side.
- Proposed solution: Speaker-normalized variable-length embeddings improve transfer robustness while enabling frame-specific control, including in singing voice.The paper reports both frame-level control and transfer between speakers.
2. RELATED WORK
Earlier prosody systems relied on costly annotations or unsupervised fixed-length and style-token representations. These approaches enabled style transfer but offered limited control over specific moments or individual phonemes.
- Supervised prosody modeling: Supervised prosody modeling used annotations such as ToBI, but hand annotation was problematic and expensive.The related work motivates unsupervised alternatives.
- Reference-encoder approaches: Fixed-length reference encoders supported prosody transfer but could not control prosody at a specific time.They also performed poorly when reference speech was shorter than the speech being generated.
- GST Tacotron: GST Tacotron represents speaking styles through attention-weighted style tokens learned without supervision.The style embedding is formed from contributions of multiple learned tokens.
- GST Tacotron: GST-based segment control could roughly alter text-segment style but remained limited for phoneme-wise prosody control.The limitation follows from shared style conditioning across input phonemes.
3. BASELINE MODEL
The baseline is a Tacotron-style end-to-end encoder-decoder conditioned on text, speaker identity, and prosody embeddings. A reference encoder extracts prosody from mel-spectrograms, producing either fixed-length or variable-length representations.
- Base architecture: The model uses a simplified Tacotron encoder-decoder with Post-processing and Griffin-Lim waveform conversion.Phoneme sequences encode normalized text, while speaker identity is mapped to a speaker embedding.
- Base architecture: The encoder-decoder represents text states as e, prosody as p, and decoder states as d.These variables define the baseline conditioning structure.
- Reference prosody encoder: A convolutional reference encoder followed by a unidirectional GRU maps reference mel-spectrograms to prosody embeddings.The final GRU output gives fixed-length p, while all GRU outputs form variable-length p_1:N.
4. PROPOSED METHOD
The proposed method introduces temporally structured prosody embeddings on either the speech or text side, aligning them with decoder or encoder time steps. It also uses positional and normalization mechanisms to support sequential control and speaker-robust transfer.
- Temporal prosody control: Variable-length prosody embeddings condition either the encoder for speech-side control or the decoder for text-side control.The embeddings are aligned and downsampled to the relevant speech or text sequence length.
- Embedding design: CoordConv supplies positional information, while ReLU constrains prosody-embedding values to [0, ∞].The authors report that these modifications improved generation quality.
- Training: Training uses the target-versus-generated spectrogram L1 loss without additional supervision for the reference encoder.The proposed models otherwise follow Tacotron training settings.
- Speech-side control: Speech-side embeddings are length-matched to decoder time steps by exploiting the linear relation between reference spectrogram length and decoder output length.Convolutional stride choices shorten the reference representation to the decoder length.
- Speech-side control: At each decoder step, the prosody vector helps attention compute a text context vector, which is then used with speaker information for decoding.Speaker identity is excluded from the attention module under a conditional-independence assumption.
- Text-side control: Text-side control uses reference attention to align text states with prosody embeddings when their lengths are not directly related.Scaled dot-product attention obtains keys and values from prosody and queries from text states.
- Prosody normalization: Normalization subtracts each speaker’s speaker-wise prosody mean from every temporal embedding value during training and evaluation.This design targets robustness to speaker-dependent prosody offsets.
5. EXPERIMENTS AND RESULTS
The experiments show that variable-length prosody embeddings support frame- and phoneme-level control, outperform GST Tacotron on MCD13, and improve robustness in inter-speaker prosody transfer when normalized.
- Dataset: The final dataset contained 104 hours of English and Korean speech from 136 speakers.It comprised 58 hours of English and 46 hours of Korean data.
- Speech-side control: Speech-side prosody embeddings changed pitch and amplitude at specific frames, with separate dimensions tending to control these features.Pitch was assessed from harmonic spacing, while amplitude was assessed from harmonic intensity in spectrograms.
- Speech-side control: The reference attention module aligned reference speech with text because its attention pattern had an analogous shape to the original attention module.
- Text-side control: Text-side prosody embeddings changed phoneme-level amplitude, pitch, and length, although their dimensions appeared entangled.The authors attribute the entanglement to imposing no disentanglement constraints.
- Comparison with GST Tacotron: The proposed methods outperformed GST Tacotron in MCD13, with speech-side prosody control achieving the lowest MCD.Lower MCD is better, and speech-side control provides the highest temporal resolution.
- Comparison with GST Tacotron: The proposed methods transferred melody from an unseen song by an unseen singer, whereas GST could not reconstruct it.Speech-side control produced a generated song described as almost identical to the original.
- Inter-speaker prosody transfer: Normalization reduced MCD in prosody transfer compared with the non-normalized model and avoided the higher-pitch tendency associated with speaker mismatch.Without normalization, female reference prosody used with a male speaker sometimes failed to generate speech.
6. CONCLUSION AND FUTURE WORK
The paper uses temporally structured prosody embeddings to control expressive speech at frame- and phoneme-level resolutions, while normalization improves prosody transfer across pitch discrepancies. Its main limitation is that bottleneck size is the only regularization applied to the prosody embedding network.
- Conclusion: Temporally structured prosody embeddings control expressive speech at both frame-level and phoneme-level resolution.The methods change synthesized speech pitch and amplitude at both temporal resolutions.
- Conclusion: Normalized prosody embeddings make prosody transfer more robust to pitch discrepancies between reference and generated speakers.The approach also supports transferring a song’s prosody to another speaker for song voice conversion.
- Future work: The prosody embedding network is regularized only through bottleneck size, motivating disentanglement methods as future work.The authors propose factorizing embeddings into more explainable prosodic features and separating them from other speech features.