Source-linked AI summary
Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens
Rafael Valle, Jason Li, Ryan Prenger, Bryan Catanzaro
TL;DR
Expressive speech synthesis needs control over characteristics absent from text, while emotive and singing data can be limited. Mellotron addresses this by conditioning a multispeaker model on explicit rhythm and continuous pitch from audio or music scores alongside learned style variables. It reports lower F0 Frame Error and generates speech styles, including singing, that are absent from training data.
Problem
Expressive speech synthesis requires control over pitch, rhythm, and emphasis, but some approaches require limited-access emotive or singing voice data.
Method
Mellotron combines explicit text, speaker, pitch, and rhythm variables with learned global style tokens and alignment representations, conditioning synthesis on audio- or score-derived contours.
Results
Mellotron is superior to other models with respect to F0 Frame Error and generates styles ranging from read speech to singing although these styles are absent from training data.
Takeaways & Limitations
Mellotron enables combining target-speaker identity with source text, rhythm, pitch, and latent style characteristics for expressive speech and singing synthesis.
Takeaways & Limitations
Audio experiments depend on the quality of supplied rhythm and pitch contours, and pitch outside a speaker’s vocal range is clipped to a highest or lowest value.
Abstract
from arXiv · showhide
Mellotron is a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data. By explicitly conditioning on rhythm and continuous pitch contours from an audio signal or music score, Mellotron is able to generate speech in a variety of styles ranging from read speech to expressive speech, from slow drawls to rap and from monotonous voice to singing voice. Unlike other methods, we train Mellotron using only read speech data without alignments between text and audio. We evaluate our models using the LJSpeech and LibriTTS datasets. We provide F0 Frame Errors and synthesized samples that include style transfer from other speakers, singers and styles not seen during training, procedural manipulation of rhythm and pitch and choir synthesis.
1. INTRODUCTION
Mellotron targets expressive speech synthesis by adding fine-grained pitch and rhythm control to text, without requiring emotive or singing data. It extends this control to singing voice synthesis using read speech training data.
- Mellotron addresses expressive speech synthesis by conditioning on text together with pitch, rhythm, and emphasis.
- Unlike coarse latent-variable control, Mellotron enables fine-grained manipulation through automatically extracted pitch and rhythm from audio or music scores.
- Mellotron extends expressive speech synthesis to singing voice synthesis without singing data or manually aligned pitch and text.
- Training requires only read speech and transcriptions, while inference can alter speaker style, emotion, and singing through source pitch and rhythm.
2. METHOD
Mellotron factorizes speech into explicit variables and learned latent variables, combining controllable pitch, rhythm, text, and speaker identity with learned style and alignment representations. This factorization supports transferring source characteristics to a target speaker.
- Mellotron combines explicit signal variables with learned latent variables representing audio characteristics that are difficult to formalize.
- A mel-spectrogram is factorized into text, speaker identity, pitch contour with voicing decisions, global style tokens, and rhythm.
- The formulation represents M = [T, S, P, R, Z], where T, S, P, R, and Z denote text, speaker, pitch, rhythm, and global style tokens.
- The model learns rhythm as an alignment map between text and mel-spectrogram frames, enabling rhythm control during inference without extracted training alignments.
- At inference, source text, pitch, and rhythm are combined with sampled global style tokens and a target speaker identity.
3. IMPLEMENTATION
The implementation extends Tacotron 2 GST with speaker embeddings and pitch contours, trains on text-audio-speaker pairs without manual alignments, and accepts audio-derived or score-derived rhythm and pitch at inference.
- Mellotron extends Tacotron 2 GST with speaker embeddings and pitch contours concatenated into the encoder and decoder pathways.
- Training requires text-audio pairs with speaker IDs and uses automatically extracted Yin pitch contours without manually aligned text, pitch, and mel-spectrogram data.
- Inference supplies text, rhythm, pitch, a global style token, and speaker identity from an audio signal or musical score.
- Text is obtained by speech recognition or transcription, then cleaned and converted from graphemes to phonemes.
- Rhythm comes from forced alignment or Mellotron alignment maps, which may be manually fine-tuned or refined through additional training.
- Pitch is extracted with Yin or Melodia, with Melodia used qualitatively because it is more precise regarding false voiced decisions.
- Music-score inputs contain pitch, note duration, and syllables, with durations converted into frames while pitch remains continuous.
4. EXPERIMENTS
Experiments use read-speech datasets and include style transfer, rhythm manipulation, and choir synthesis from music scores. The evaluations cover sources and styles beyond those present in training data.
- Models are trained on LJSpeech, Sally, and a subset of LibriTTS, all consisting of read speech.
- Experiments include style transfer from seen and unseen speakers, singers, procedural rhythm manipulation, and choir synthesis from music scores.
4.1. Training Setup
Mellotron experiments use speech from LJS, Sally, and LibriTTS, with staged training and standard spectrogram-processing and optimization choices.
- Training uses LJS, Sally, and LibriTTS train-clean-100, covering over 100 speakers with about 25 minutes per speaker on average.Speakers with under 5 minutes of data and files longer than 10 seconds were filtered out, without data augmentation.
- Mel-spectrograms use 80 bins at 22050 Hz, with a 1024-point FFT, 256-sample hop, and 1024-sample window.
- Optimization uses ADAM with a 1e-3 starting learning rate, annealed when the loss plateaus, on one NVIDIA DGX-1 with 8 GPUs.
- WaveGlow decoding uses one model trained on Sally, and the authors report that it can serve as a universal decoder.
- Mellotron is first trained on LJS and Sally to learn attention alignments, then fine-tuned for LibriTTS with a new speaker embedding.LibriTTS fine-tuning starts at 5e-4 and anneals as the loss plateaus.
4.2. Quantitative Results
The quantitative evaluation compares Mellotron with E2E-Prosody using pitch and voicing metrics, emphasizing pitch conditioning’s effect on synthesis accuracy and prosody transfer.
- The evaluation compares Mellotron and E2E-Prosody using Gross Pitch Error, Voicing Decision Error, and F0 Frame Error.The metrics are computed with the Yin algorithm, and rhythm conditioning gives reference and predicted audio the same length without padding.
- Conditioning on pitch drastically reduces the error between the source and synthesized voice.Low pitch error is particularly important for preserving melody identity in singing voice synthesis.
- Lower FFE provides evidence of more precise style transfer to the target speaker.
4.3. Style transfer from Audio Signal
Mellotron transfers rhythm, pitch, and speaker style from audio while enabling procedural speech-rate changes and hybrid voice-style synthesis.
- 4.3. Style transfer from Audio Signal: Audio-based synthesis is constrained by the quality of supplied rhythm and pitch contours.Forced-alignment rhythm required repeated fine-tuning, and pitch outside the target speaker’s range was scaled to that range.
- 4.3.1. Rhythm Transfer: Mellotron transfers rhythm and associated text to a target speaker, enabling procedural control over each phoneme’s duration.The pitch contour can be set to zeros for rhythm-only transfer, while processed rhythm can produce effects such as acceleration or slowing.
- 4.3.1. Rhythm Transfer: Figure 1 compares the source alignment with the processed alignment used for rhythm manipulation.
- 4.3.2. Rhythm and Pitch Transfer: Conditioning on rhythm and pitch transfers source-speaker characteristics while retaining the target speaker’s voice.In a solemn-speech example, Mellotron preserves the source’s pauses and speech rate, whereas the original Tacotron 2 does not convey the same solemnity.
4.4. Singing Voice Synthesis
Mellotron generalizes rhythm and pitch conditioning to unseen speakers and musical styles, producing singing from audio inputs and music scores.
- 4.4.1. Singing Voice from Audio Signal: Mellotron synthesizes singing voice from unseen speakers across rap, pop, Hindustani, and western European classical music.
- 4.4.1. Singing Voice from Audio Signal: For the Sweet Dreams example, Mellotron’s pitch contour is closer to the source than E2E-Prosody’s.The experiment transfers text, rhythm, and scaled pitch to Sally.
- 4.4.1. Singing Voice from Audio Signal: Figure 2 compares source, Mellotron, and E2E-Prosody pitch contours.
- 4.4.2. Style transfer from Music Score: Music-score inputs provide rhythm and pitch contours that are correct by design.The paper demonstrates choir, multi-part, and single-voice synthesis, with precise pitch and rhythm except when pitches exceed the speaker’s vocal range.
5. CONCLUSION
Mellotron provides direct rhythm- and pitch-conditioned style control, and its results support synthesis across speech and singing styles absent from training. Its pitch matching is strongest within a speaker’s vocal range, while future work targets audio quality, vocal range, and emotive storytelling data.
- Mellotron enables direct control of style by conditioning on rhythm and pitch from an audio signal or music score.
- Mellotron outperforms other models on F0 Frame Error and generates styles from read speech and expressive speech to rap and singing without those styles in training data.
- Within a speaker’s vocal range, synthesized rhythm and pitch contours are extremely similar to the source audio or music score.Outside that range, Mellotron defaults to the lowest or highest tone.
- Future work will compare audio- versus score-conditioned contours, study and extend vocal range, and train on animated emotive storytelling data.