Source-linked AI summary
Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
Rafael Valle, Kevin Shih, Ryan Prenger, Bryan Catanzaro
TL;DR
Text-to-speech systems can reproduce text while leaving users with limited control over the non-textual qualities that convey voice and expression. Flowtron uses an autoregressive flow to map mel-spectrograms into a manipulable latent space. The authors report state-of-the-art-comparable speech quality and demonstrate control of variation and style transfer.
Problem
Text-to-speech systems provide limited user control over non-textual speech qualities, which convey voice and human expressiveness.
Source: §1 · p. 1Method
Results
Flowtron’s MOS is similar to state-of-the-art TTS models, with demonstrations of variation control, interpolation, and style transfer across seen and unseen speakers.
Source: §1 · p. 2, §5 · pp. 8–9Takeaways & Limitations
Flowtron produces expressive speech without labeled data or having seen expressive data during training.
Source: §5 · p. 9At σ^2 = 0, reported variations are attributed to different z values for WaveGlow.
Source: §4.3.2 · p. 6
Abstract
from arXiv · showhide
In this paper we propose Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis with control over speech variation and style transfer. Flowtron borrows insights from IAF and revamps Tacotron in order to provide high-quality and expressive mel-spectrogram synthesis. Flowtron is optimized by maximizing the likelihood of the training data, which makes training simple and stable. Flowtron learns an invertible mapping of data to a latent space that can be manipulated to control many aspects of speech synthesis (pitch, tone, speech rate, cadence, accent). Our mean opinion scores (MOS) show that Flowtron matches state-of-the-art TTS models in terms of speech quality. In addition, we provide results on control of speech variation, interpolation between samples and style transfer between speakers seen and unseen during training. Code and pre-trained models will be made publicly available at https://github.com/NVIDIA/flowtron
1. Introduction
Flowtron addresses the limited user control over speech’s non-textual qualities by mapping mel-spectrograms to a manipulable latent space. The paper reports state-of-the-art-comparable speech quality alongside control over variation and style transfer.
1. Introduction
- Users lack insight into or control over non-textual speech qualities because these qualities are unlabeled and treated as a black box.Prior unsupervised models can recreate training-set patterns, but do not provide users control over those qualities.Source: §1 · p. 1
- Flowtron maps mel-spectrograms to a Gaussian-parameterized latent space, where sampling and variance adjustment provide control over speech characteristics and variation.The paper also describes posterior sampling and Gaussian-mixture components for targeting acoustic qualities.Source: §1 · p. 2
- Flowtron can use posterior evidence or Gaussian-mixture components to target acoustic qualities, including making a monotonous speaker more expressive.The paper contrasts this manipulable latent-space approach with audio embeddings that require prespecified dimensionality and may not capture all non-textual information.Source: §1 · pp. 1–2
- The authors report sharp mel-spectrograms from likelihood maximization without additional Prenet or Postnet layers or compound loss functions.They describe likelihood-based optimization as making training simple and stable.Source: §1 · p. 2
- Flowtron’s mean opinion scores match state-of-the-art TTS models, and the paper reports variation control, sample interpolation, and style transfer across seen and unseen speakers.The authors identify this as evidence that normalizing flows can be used for text-to-speech synthesis.Source: §1 · p. 2
2. Related Work
Prior expressive-speech approaches condition synthesis on hand-engineered audio statistics, learned style embeddings, or both. These choices can constrain what users specify or control.
2. Related Work
- Earlier TTS systems treated non-textual information as a black box and added Prenet, Postnet, and modified-loss components to improve convergence, generalization, and acoustic detail.The cited discussion describes these as requirements in prior systems.Source: §2 · p. 2
- Hand-engineered-statistic approaches condition utterances on measures such as F0, but require choosing statistics and target values before synthesis.The paper cites Nishimura et al. (2016) and Lee et al. (2019) as examples.Source: §2 · p. 2
- Learned prosody or global-style embeddings use unlabeled data, but manipulating these latent variables offers only coarse control over expressive characteristics.Examples cited are Skerry-Ryan et al. (2018) and Wang et al. (2018).Source: §2 · p. 2
- Mellotron combines engineered audio statistics with a latent embedding derived from a reference acoustic representation, while still requiring statistics to be determined before synthesis.Source: §2 · pp. 2–3
3. Flowtron
Flowtron generates mel-spectrogram frames autoregressively through invertible transformations conditioned on text and speaker information. Its affine coupling design supports likelihood training, and inference samples from a Gaussian or Gaussian mixture.
3. Flowtron
- Flowtron models a mel-spectrogram sequence autoregressively, generating each frame conditioned on previous frames through invertible transformations from a latent distribution.The paper considers zero-mean spherical Gaussian and spherical Gaussian-mixture distributions, with transformations that map latent samples to mel-spectrograms.Source: §3 · p. 3
- Autoregressive affine transformations make each Jacobian determinant lower triangular, enabling likelihood training through the change-of-variables formula.Only the affine scale term changes mapping volume and contributes a change-of-variables term to the loss.Source: §3 · p. 3, §3.1 · p. 3
3.1. Affine Coupling Layer
- In affine coupling, a causal network uses preceding frames and text and speaker inputs to produce scale and bias for the succeeding frame, preserving overall invertibility.Reversing input order at every even flow step lets the model learn dependencies in both temporal directions while remaining causal and invertible.Source: §3.1 · p. 3
3.2. Model architecture
- The architecture modifies Tacotron’s text encoder with instance normalization, removes its Prenet and Postnet, and concatenates a speaker embedding with encoder outputs at each token.A dense sigmoid-output layer gates the flow step closest to z; Figure 1 also depicts channel-wise concatenation of text and speaker embeddings and a zero-valued vector concatenated with x over time.Source: §3.2 · p. 3, Figure 1 · p. 4
3.3. Inference
- At inference, Flowtron samples latent values from a spherical Gaussian or Gaussian mixture and reverses input order when needed.Training used σ^2 = 1; the passage says lower standard deviation often sounded better and reports σ^2 = 0.5 as the default inference setting unless specified otherwise.Source: §3.3 · p. 4
4. Experiments
The experiments report MOS comparable to state-of-the-art text-to-mel systems and qualitative capabilities including variation control, interpolation, and style transfer. WaveGlow is used as the waveform decoder.
4. Experiments
- Flowtron’s MOS is reported as comparable to state-of-the-art text-to-mel-spectrogram systems such as Tacotron 2.Source: §4 · p. 4
- Qualitative results demonstrate speech-variation control, interpolation between samples, and style transfer between speakers seen and unseen during training.The passage contrasts these features with Tacotron and Tacotron 2 GST as not possible or not efficient there.Source: §4 · p. 4
- All mel-spectrograms are decoded with a single pre-trained, single-speaker WaveGlow model; the authors suggest it may serve as a universal decoder.The authors note that the results are best appreciated by listening to samples.Source: §4 · p. 4
4.1. Training setup
Flowtron and comparison models were trained on combined LJSpeech and proprietary single-speaker data, with a separate Flowtron training stage on LibriTTS. The setup also specifies text encodings, audio features, optimization settings, and staged transfer between models.
4.1. Training setup
- Training combined LJSpeech with proprietary Sally and Helen datasets, and Flowtron was also trained on LibriTTS train-clean-100.The combined dataset included 20 hours from Sally and 10 from Helen; LibriTTS training used 123 speakers averaging 25 minutes each.Source: §4.1 · p. 4
- Models used normalized text and ARPAbet encodings, without data augmentation, and 22,050-Hz audio represented by 80-bin mel-spectrograms.The STFT used 1024-point FFT and window sizes, with a 256-sample hop.Source: §4.1 · p. 4
- Flowtron used Adam with a 1e-4 learning rate and 1e-6 weight decay; Tacotron 2 and GST used 1e-3 and 1e-5, respectively.Learning rates were annealed when generalization error plateaued; training stopped when it stopped decreasing significantly or began increasing.Source: §4.1 · p. 4
- Training proceeded from a single-flow-step Flowtron on LSH to models with more flow steps, then optionally fine-tuned on LibriTTS with a new speaker embedding.The authors report that learning attention with one flow step and more data was faster than using multiple steps with less data.Source: §4.1 · p. 4
4.2. Mean Opinion Score comparison
The section compares mean opinion scores for real LJSpeech and synthesized speech, using crowd-sourced ratings. The authors report that the results roughly match their qualitative assessment and identify control and latent-space manipulation as Flowtron’s larger advantage.
4.2. Mean Opinion Score comparison
- The comparison includes real LJSpeech data, two-step Flowtron, and Tacotron 2 trained on LSH, while MOS are computed only on LJSpeech.Ground-truth LJSpeech MOS are taken from Prenger et al. (2019).Source: §4.2 · p. 5
- Crowd-sourced raters who passed a hearing test rated utterance pleasantness on a five-point scale.The evaluation used 30 volume-normalized utterances from speakers disjoint from training.Source: §4.2 · p. 5
- MOS results with 95% confidence intervals roughly match the qualitative assessment; Flowtron’s larger advantage is control of speech variation and latent-space manipulation.The table reports scores for various sources, with approximately 250 ratings per source.Source: §4.2 · p. 5, Table 1 · p. 5
4.3. Sampling the prior
Flowtron samples a Gaussian prior whose variance controls speech variation, and the section tests variation in duration and F0 as well as interpolation between samples. The reported results include increased variation with preserved speech quality and interpolation across samples and speakers.
4.3. Sampling the prior
- Flowtron samples z ∼ N(0, σ²), with σ² controlling the amount of speech variation: zero removes variation, while larger values increase it.Source: §4.3 · p. 5
4.3.1. SPEECH VARIATION
- Higher σ² increases duration and F0 variation while retaining high-quality speech; duration varies more than with Tacotron 2, and harmonics and formants remain sharp without compound loss or Prenet/Postnet layers.At σ² = 0, duration and F0 show no variation; increasing σ² increases both.Source: §4.3.1 · pp. 5–6, Figure 3 · p. 6
4.3.2. INTERPOLATION BETWEEN SAMPLES
- Latent-space interpolation preserves correct alignment between samples and blends characteristics of Sally and Helen when interpolating between speakers without speaker embeddings.The same-speaker experiment interpolated between two samples drawn from z ∼ N(0, 0.8) over 100 steps.Source: §4.3.2 · p. 6
4.4. Sampling the posterior
Posterior sampling transfers acoustic traits from evidence samples to a target speaker. Across experiments, Flowtron transfers expressive styles and speaker characteristics, including to speakers unseen during training.
4.4. Sampling the posterior
- For posterior sampling, Flowtron obtains evidence latents from audio and text using the target speaker ID, repeating latents when needed to meet text-length requirements.Tacotron 2 GST instead queries style tokens using existing audio, either from one sample or an average over multiple samples.Source: §4.4 · pp. 6–7
4.4.1. SEEN SPEAKER WITHOUT ALIGNMENTS
- Conditioning on 30 high-F0-variance Helen samples makes monotonic Sally gradually more expressive with Flowtron, while Tacotron 2 GST barely changes her characteristics.Flowtron can sample from the posterior or interpolate between it and a standard Gaussian prior.Source: §4.4.1 · p. 7, §4.4 · p. 7
4.4.2. SEEN SPEAKER WITH ALIGNMENTS
- Transferring a female speaker’s acoustic characteristics and alignment maps to a male speaker makes his lower partials oscillate more like hers.The female source speaker has a distinctive nasal voice and F0 oscillation.Source: §4.4.2 · p. 7
4.4.3. UNSEEN SPEAKER STYLE
- Flowtron transfers Sally’s unseen somber narrative style, including its long pauses, whereas Tacotron 2 GST fails to emulate the style.Sally’s training data consists of news readings, while evaluation uses her interpretation of Born of Darkness.Source: §4.4.3 · p. 7, Figure 5 · p. 7
4.4.4. UNSEEN SPEAKER
- Flowtron transfers surprised speech and Feynman’s prosody and articulation to Sally, while Tacotron 2 GST does not emulate either style.The experiments use a RAVDESS speaker’s surprised style and Richard Feynman’s speech, with sentences absent from both source sets.Source: §4.4.4 · pp. 7–8
4.5. Sampling the Gaussian Mixture
Flowtron Gaussian Mixture models assign components in ways associated with speaker identity and gender. Translating individual component dimensions modulates specific speech attributes, with evidence of disentanglement.
4.5.1. VISUALIZING ASSIGNMENTS
- In Flowtron GM-A, Helen’s data is assigned almost exclusively to component 7, unlike LJS and Sally, which are mostly assigned across components.Assignments are averaged over time and samples.Source: §4.5.1 · p. 8, Figure 6 · p. 8
- In Flowtron GM-B, component 7 receives more probability for male speakers and component 6 more for female speakers.This model uses one flow step without speaker embeddings and predicts the mixture mean and covariance.Source: §4.5.1 · p. 8, Figure 7 · p. 8
4.5.2. TRANSLATING DIMENSIONS
- Translating mixture dimensions modulates pitch or word duration without changing the other attribute, providing evidence of disentangled speech attributes.Pitch-height dimensions change pitch contours while preserving duration; the first-word-length dimension does not change that word’s pitch.Source: §4.5.2 · p. 8
5. Discussion
Flowtron combines autoregressive flows with a manipulable latent space for text-to-mel synthesis. The paper reports speech quality comparable to state-of-the-art models alongside controllable variation and style transfer.
5. Discussion
- Flowtron achieves MOS comparable to state-of-the-art text-to-speech models while allowing speech variation and style transfer.The model is optimized by maximizing likelihood and learns a latent space for non-textual information without extra cost or a compound loss term.Source: §5 · p. 8
- Flowtron’s latent space supports controllable variation, transfer of styles across seen and unseen speakers and different text, and making monotonic speech more expressive.The reported samples retain high speech quality despite increased variation.Source: §5 · pp. 8–9
- The authors report expressive speech without labeled or expressive training data and identify normalizing flows as a new approach to text-to-mel synthesis.They describe potential applications in human-computer interaction and the arts, where realism and expressivity matter.Source: §5 · p. 9