Source-linked AI summary
FastPitch: Parallel Text-to-speech with Pitch Prediction
Adrian Łańcucki
TL;DR
FastPitch addresses quality limitations in fully parallel TTS by conditioning a FastSpeech-based feed-forward model on fundamental-frequency contours. It predicts pitch during inference, enabling prosody control while retaining fast synthesis and state-of-the-art-quality speech. The model achieves 912× real-time mel-spectrogram synthesis on an NVIDIA A100 and 63× real-time complete audio synthesis with WaveGlow.
Problem
Fully parallel feed-forward TTS needs better quality and prosody control while retaining its speed advantages.
Method
FastPitch predicts one pitch value per input symbol, adds its pitch embedding to FastSpeech-style hidden representations, and generates mel-spectrograms with feed-forward Transformer stacks.
Results
912× real-time factor is achieved for mel-spectrogram synthesis on an NVIDIA A100, while complete audio synthesis with WaveGlow reaches 63× real-time.
Takeaways & Limitations
Predicted pitch contours can be modified during inference to control F0, expressiveness, and pitch variance while preserving perceived speaker identity.
Abstract
from arXiv · showhide
We present FastPitch, a fully-parallel text-to-speech model based on FastSpeech, conditioned on fundamental frequency contours. The model predicts pitch contours during inference. By altering these predictions, the generated speech can be more expressive, better match the semantic of the utterance, and in the end more engaging to the listener. Uniformly increasing or decreasing pitch with FastPitch generates speech that resembles the voluntary modulation of voice. Conditioning on frequency contours improves the overall quality of synthesized speech, making it comparable to state-of-the-art. It does not introduce an overhead, and FastPitch retains the favorable, fully-parallel Transformer architecture, with over 900x real-time factor for mel-spectrogram synthesis of a typical utterance.
1. INTRODUCTION
FastPitch is a fully parallel FastSpeech-based TTS model that conditions synthesis on predicted pitch contours. This conditioning improves speech quality and enables controllable, expressive pitch variation without sacrificing parallel inference.
- FastPitch is a feed-forward model based on FastSpeech that improves the quality of synthesized speech.
- Conditioning on fundamental frequency estimated for every input symbol matches state-of-the-art autoregressive TTS models.
- Explicit pitch-contour modeling addresses quality shortcomings of the plain feed-forward Transformer architecture.The authors associate those shortcomings with collapsing different pronunciations of identical phonetic units when textual input lacks sufficient linguistic information.
- 60× faster than real-time mel-spectrogram synthesis is achieved with WaveGlow, without kernel-level optimizations.
- Modifying predicted pitch during inference increases or decreases F0 and raises expressiveness and pitch variance.Constant F0 offsets produce naturally sounding low- and high-pitched variations while preserving perceived speaker identity.
2. MODEL DESCRIPTION
FastPitch uses two feed-forward Transformer stacks to predict durations and per-symbol pitch, inject pitch into token representations, and generate mel-spectrograms. Its training pipeline derives durations and pitch from auxiliary analyses, then uses predicted values at inference.
- The architecture contains token-resolution and frame-resolution feed-forward Transformer stacks, with the first producing hidden representation h.
- Duration and average pitch are predicted for every input character from the first stack’s hidden representation.
- Projected pitch is added to hidden representation h, then the resulting g is discretely upsampled for output mel-spectrogram generation.
- Predicted durations and pitch replace ground-truth values during inference, while training optimizes mean-squared error against ground-truth modalities.
- 2.1. Duration of Input Symbols: Different Tacotron 2 alignment models produce FastPitch systems with similar speech quality despite differing character durations.The longest durations tend to occur in similar locations but may be assigned to different characters.
- 2.2. Pitch of Input Symbols: Pitch is obtained by autocorrelation-based periodicity detection, averaged over each input symbol, standardized, and set to 0 when no voiced estimate exists.Unvoiced values are excluded, and one F0 value is produced for each frame before symbol-level averaging.
- 2.2. Pitch of Input Symbols: One pitch value per input symbol was preferred over three values per symbol, whose model was judged inferior.
3. RELATED WORK
FastPitch builds on parallel TTS and explicit prosody conditioning, particularly fundamental frequency. Its per-input-symbol pitch representation differs from higher-resolution approaches and is intended to support prediction and interactive control.
- FastPitch predicts one pitch value per input symbol, whereas FastSpeech 2 predicts frame-level contours discretized to 256 frequency values and also uses energy.
- Parallel TTS models accelerate mel-spectrogram or waveform synthesis while conditioning can incorporate linguistic features and fundamental frequency.
- Parallel TTS research also includes self-aligning models, duration-based adaptations, dynamic-programming alignment, and convolutional GAN-TTS.
- Earlier prosody modeling encoded log-duration, start and end log-pitch, and log-energy in four-value vectors using a separate pitch-extraction tool.
4. EXPERIMENTS
The experiments evaluate FastPitch across data, model, quality, multi-speaker, pitch-control, and inference-speed settings. Results include interactive pitch manipulation and highly parallel mel-spectrogram generation.
- Experimental setup: FastPitch was trained on LJSpeech 1.1, approximately 24 hours of single-speaker speech at 22,050 Hz, with manually corrected transcriptions.The authors retained the same train/dev/test split used for WaveGlow to avoid evaluation leakage.
- Evaluation: The evaluation compared FastPitch with Tacotron 2 using crowdsourced Mean Opinion Scores from at least 250 ratings per model.The study used the first 30 development samples and 60 unique Turkers who passed a hearing test.
- Model variants: Glicko-2 rankings compared FastPitch variants with 1, 2, or 4 attention heads, 6 or 10 layers, and one or three pitch values per input token.The comparisons were based on randomly selected pairs scored by more than 50 participants.
- Multiple speakers: FastPitch was extended to multiple speakers by adding a global speaker embedding to the input tokens and compared with multi-speaker Tacotron 2 and Flowtron.All models used the same training data, with multi-speaker Tacotron 2 providing FastPitch’s training alignments.
- Pitch conditioning: Modifying predicted pitch during inference controls perceived speech qualities, including increased or decreased F0, greater expressiveness, and pitch variance.Figure 5 illustrates a 50 Hz frequency shift using paired shifted spectrograms and absolute differences from the unshifted spectrograms.
- Inference performance: 912× RTF was measured for mel-spectrogram generation on an NVIDIA A100 with batch size 1, while complete audio synthesis with WaveGlow reached 63× RTF.On an Intel Xeon Gold 6240 CPU, the measured RTF was 108×.
5. CONCLUSIONS
FastPitch is a parallel FastSpeech-based TTS model that conditions synthesis on prosodic information. It rapidly generates high-fidelity mel-spectrograms while providing substantial prosody control and potential for interactive, multi-speaker use.
- FastPitch is a parallel text-to-speech model based on FastSpeech that rapidly synthesizes high-fidelity mel-scale spectrograms.
- Conditioning on prosodic information improves convergence and synthesized-speech quality in the feed-forward model.
- FastPitch provides high prosody control without introducing computational overhead.
- The model supports interactive prosody adjustment and has potential for multi-speaker scenarios because it is fast and highly expressive.