Source-linked AI summary
Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J. Weiss, Rob Clark, Rif A. Saurous
TL;DR
Text underspecifies prosody, including contextual variation in intonation and stress. The paper adds an acoustic reference encoder to Tacotron, whose learned embedding transfers prosody across speakers and texts, while evaluation also exposes limits in speaker preservation and embedding robustness.
Problem
Text does not fully specify prosodic meaning, motivating methods that represent prosody without requiring explicit annotations.
Method
The paper extends Tacotron with a learned fixed-length prosody embedding extracted from reference acoustic input and combined with transcript and speaker representations.
Results
The learned representation enables prosody transfer between utterances in an almost speaker-independent fashion, including references with different speakers or text.
Takeaways & Limitations
Acoustic prosody embeddings provide a representation for controlling Tacotron and support quantitative and subjective evaluation of prosody transfer.
Takeaways & Limitations
Prosody is transferred in a pitch-absolute manner, and variable-length embeddings are less robust to text and speaker perturbations than fixed-length embeddings.
Abstract
from arXiv · showhide
We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody. We show that conditioning Tacotron on this learned embedding space results in synthesized audio that matches the prosody of the reference signal with fine time detail even when the reference and synthesis speakers are different. Additionally, we show that a reference prosody embedding can be used to synthesize text that is different from that of the reference utterance. We define several quantitative and subjective metrics for evaluating prosody transfer, and report results with accompanying audio samples from single-speaker and 44-speaker Tacotron models on a prosody transfer task.
1. Introduction
Prosody carries meaning beyond text, making speech synthesis underdetermined. The paper learns an acoustic prosody embedding for Tacotron and demonstrates transfer across speakers and utterances.
- Motivation: Prosody includes intonation, stress, rhythm, and speaking style that a simple text input does not specify.These factors must be inferred to produce realistic speech.
- Motivation: Different intonation and stress patterns can convey contextual meaning absent from the transcript.Examples include stressing “mat” or changing pitch on “orange” depending on the conversational context.
- Approach: The paper avoids explicit prosody annotations by learning a latent representation directly from ground-truth speech audio.It defines prosody as variation remaining after accounting for phonetics, speaker identity, and channel effects.
- Implications: The learned embedding can control Tacotron using a different voice from training and may later be predicted from non-acoustic context.Suggested contexts include prosody labels and conversation state.
- Contribution: The proposed fixed-length encoder representation transfers prosody between utterances in an almost speaker-independent fashion.The authors evaluate this transfer with quantitative and qualitative metrics.
2. Related Work
Prior work modeled prosody through clustering, predefined style vectors, annotation schemes, signal-driven features, and prosody transplantation. The paper relates its approach to voice conversion while addressing acoustic prosody control.
- Earlier prosody modeling: Earlier HMM-based systems used clustered training data or predefined style vectors to model speaking style.These approaches included cluster-adaptive training and estimated transformations for style vectors.
- Prosody annotation: Several systems developed annotation schemes or automatic labels for prosodic phenomena, but expert annotation can be difficult and inconsistent.The cited schemes include ToBi, AuToBI, Tilt, INTSINT, and SLAM.
- Acoustic reference control: Few prior works used acoustic reference signals to control TTS prosody, including signal-driven symbolic features and prosody transplantation.Prosody transplantation used an enriched phonetic transcription with desired text.
- Voice conversion: Prosody transfer is related to voice conversion, which synthesizes an utterance in another speaker’s voice from its acoustic signal.The paper also cites a related autoencoder-based approach.
3. Model Architecture
The model extends Tacotron with speaker conditioning and a reference encoder that compresses acoustic input into a prosody embedding. This embedding is concatenated with transcript and speaker representations to condition autoregressive decoding.
- Tacotron base: Tacotron predicts mel spectrograms from grapheme or phoneme sequences, which can be converted to audio with WaveNet or Griffin-Lim.The implementation uses phoneme inputs and GMM attention rather than the original Bahdanau attention.
- Multi-speaker Tacotron: Multi-speaker Tacotron broadcasts each speaker embedding across transcript-encoder timesteps and concatenates it with the transcript representation.No additional changes or loss metrics are necessary for this conditioning scheme.
- Reference encoder: A reference encoder maps a variable-length acoustic reference signal to a fixed-dimensional prosody embedding.The embedding is intended to represent prosody and support meaningful control of output speech.
- Conditioning: The prosody embedding is broadcast-concatenated with transcript and speaker embeddings, producing conditioning inputs that remain fixed across decoder timesteps.The resulting representation is interpreted as an RNN encoder-decoder conditioned on text and speaker identity.
- Training and inference: During training, the target audio supplies the reference signal and Tacotron reconstruction error alone trains the reference encoder.At inference, the encoder can process an utterance whose text and speaker differ from the synthesis inputs, enabling prosody transfer.
- Reference representation: The reference encoder uses convolutional processing followed by recurrent pooling, while the input representation determines which prosodic factors can be learned.Mel spectrograms are used in this paper; pitch tracks or MFCCs can omit energy or pitch information relevant to prosody.
- Embedding length: Fixed-length embeddings create a scaling bottleneck for longer utterances, whereas variable-length embeddings generalize better to very long utterances but are less robust to text and speaker perturbations.The paper therefore focuses on fixed-length embeddings.
4. Experiments and Results
The experiments use single- and multi-speaker English datasets and evaluate prosody transfer with acoustic metrics derived from speech signals. The evaluation includes Mel Cepstral Distortion, pitch error, and voicing error measures.
- Datasets: The single-speaker dataset contains 147 hours of animated and emotive audiobook recordings.
- Datasets: The multi-speaker dataset contains 296 hours of English speech from 44 speakers with varied accents.
- Training: Models are trained for at least 200k steps with Adam, using scheduled learning-rate decay; baselines omit the reference encoder.
- Evaluation metrics: The evaluation adapts general audio-processing metrics because prosody transfer has no generally accepted metrics.
- Evaluation metrics: MCD compares frame-level MFCC differences while excluding the overall-energy coefficient.
- Evaluation metrics: GPE measures voiced frames with pitch deviations above 20%, while VDE measures voicing-decision errors.
F0 Frame Error (FFE) (Chu & Alwan, 2009):
The evaluation combines objective frame-level error measures with an anchored human discrimination test. The subjective test asks raters to compare prosody against a reference while ignoring voice-quality and pronunciation differences.
- Objective metric: FFE counts frames containing either a 20% pitch error or a voicing-decision error.
- Subjective evaluation: The anchored prosody side-by-side presents a reference and two competing samples for a 7-point human comparison.
- Subjective evaluation: Raters evaluate intonation, stress, speaking rate, and pauses while ignoring audio quality or pronunciation differences.
- Interpretation: Table 1 treats lower MCD and FFE as better and higher subjective scores as better than the baseline.
4.3. Same-text Prosody Transfer
Same-text experiments show that reference conditioning transfers detailed timing and pitch patterns while retaining the target speaker across reference-speaker conditions. Quantitative and subjective evaluations support better matching than unconditioned baselines.
- Spectrograms and pitch tracks: Reference-conditioned synthesis matches the reference rhythm, pause characteristics, utterance length, harmonic structure, and onset structure more closely than the unconditioned model.
- Spectrograms and pitch tracks: The prosody embedding model closely follows the reference pitch contours, unlike the unconditioned model.
- Experimental conditions: The experiments compare same-speaker and different-speaker references, including seen and unseen speakers in the multi-speaker setting.
- Quantitative and subjective evaluations: Reference encoders match reference prosody more accurately across all baseline/model pairs, regardless of whether reference and target speakers match.
- Quantitative and subjective evaluations: Different-speaker transfer requires verifying that the output preserves the target speaker rather than copying the reference speaker.
4.4. Templated Prosody Transfer
The paper tests whether fine-grained reference prosody can be transferred when the synthesized text changes. Drastic structural changes produce undesirable transfer, but some transformations preserve intelligibility and desired prosody for templated dialogue.
- Templated prosody transfer: The Figure 4 example compares an unseen-speaker reference with reference-conditioned and unconditioned syntheses of the same utterance.
- Templated prosody transfer: Fine-time-detail prosody embeddings make transfer to radically different utterances difficult to define.
- Templated prosody transfer: Drastic changes to sentence or phrase structure result in undesirable prosody transfer.
- Templated prosody transfer: The paper provides examples where text transformations preserve intelligibility and desired prosody.
4.5. Preservation of Speaker Identity
The prosody-enhanced model transfers reference prosody across speakers, but pitch transfer can entangle prosody with speaker identity. Speaker-identification tests and listening examples reveal both the transfer strength and its limitations.
- Reference-based synthesis matched reference prosody significantly better than the baseline in anchored side-by-side evaluation.This evaluation did not assess whether target speaker identity was preserved.
- The audio examples preserved distinct accents and vocal-tract properties while matching the reference prosody across target speakers.The examples used references from 6 speakers, with each utterance synthesized 6 times for different target speakers.
- The prosody representation encodes pitch absolutely, so cross-gender transfer can make targets sound as though they imitate a deeper or higher voice.The authors interpret this as evidence that prosody and speaker representations are somewhat entangled.
- 61% of prosody-enhanced outputs were identified as the reference speaker, while 21% were identified as the target speaker.The ideal target-speaker identification rate was 100%, although the authors report that samples sounded substantially more like the target speaker.
- Using 13 MFCCs, the model identified outputs as the reference speaker 41% of the time and the target speaker 32% of the time.Because MFCCs contain less pitch content, these results suggest that speaker-dependent pitch content transfers from reference to output.
4.6. Bottleneck Size and Shape
Bottleneck size and activation shape how much reference information reaches the output. Larger bottlenecks improve prosody reproduction, while softmax performs worse than tanh.
- Prosody-encoder models outperform the baseline on both MCD13 and FFE, with performance improving as bottleneck size increases.The comparison is reported in Figure 6.
- Softmax produces worse MCD13 and FFE metrics than tanh because it imposes a more severe bottleneck.The authors suggest exponential suppression of non-maximal components as a possible explanation.
- Increasing bottleneck size allows more information flow from the reference and improves reproduction of the reference.The experiment uses the same speaker as both reference and target, effectively conditionally autoencoding mel spectrograms given text.
- Larger tanh bottlenecks improve audio similarity and make outputs more faithful to reference prosody.The quantitative results agree with the accompanying audio samples.
5. Discussion and Future Work
The paper demonstrates end-to-end prosody transfer while identifying unresolved issues in pitch disentanglement, cross-text transfer, evaluation, and prosody generation.
- Prosody transfer succeeds across speakers, but the representation transfers pitch in an absolute manner.The authors propose pitch-relative encoding to preserve speaker identity more completely.
- Disentangling textual information in the reference signal from prosodic information remains an open question.The paper reports partial success when reference and target transcripts are relatively close.
- The task of controlling prosody with utterances having different transcripts is described as somewhat ill-defined and needing more careful formalization.
- The paper defines objective and subjective prosody-transfer metrics and evaluates its architecture on these benchmarks.The authors identify metrics covering prosodic similarity and disentanglement as an important long-term need.
- Sampling prosody rather than transferring it remains future work, such as learning a prior distribution over the prosody space.
A. Subjective Evaluation Template
The subjective evaluation compares two candidate utterances against a reference, asking raters to judge prosodic similarity while controlling rating instructions and aggregation.
- Raters compare reference A with competing stimuli X and Y on a 7-point prosody-similarity scale mapped to −3 through 3.The endpoints range from X being much closer to Y being much closer, with both equally distant at the midpoint.
- Raters evaluate intonation, stress, speaking rate, and pauses while ignoring audio quality and pronunciation differences.
- Each triplet receives 4 independent ratings, and no rater evaluates more than 6 items in one evaluation.
- Scores are averaged and reported with 95% confidence intervals.