Better speech synthesis through scaling

1.1 Text-to-speech

• TTS research has prioritized efficient, high-sampling-rate deployment, while limited transcribed data and scaling challenges constrain traditional encoder-decoder approaches.Source: §1.1 · p. 1

1.1.1 Neural MEL Inverters

• MELs compress waveform data 256x while retaining most information; neural MEL inverters decode them, with Univnet used as TorToise's final stage.Source: §1.1.1 · p. 1

1.2 Image generation

• Unlike TTS's latency focus, image generation prioritizes high-quality outputs regardless of sampling time.Source: §1.2 · pp. 1–2

1.2.1 DALL-E

• DALL-E (Ramesh et al., 2021) applies autoregressive decoding to text-image generation, but full-sequence attention scales O(N^2), and the author attributes much sample blur to its learned VQVAE decoder.Source: §1.2.1 · p. 2

1.2.2 DDPMs

• DDPMs are presented as producing crisp, coherent, diverse images, but fixed output shapes block text-audio alignment and iterative sampling adds latency.Source: §1.2.2 · p. 2

1.2.3 Re-ranking

• DALL-E reranks k autoregressive samples by quality, using CLIP (Radford et al., 2021) as its text-image discriminator.Source: §1.2.3 · p. 2

2.1 Joining Autoregressive Decoders and DDPMs

• Autoregressive models convert text to speech tokens, which a DDPM decodes into a high-quality speech representation; the combination draws on their complementary strengths for continuous data.The paper notes that both model types have demonstrated scaling with additional compute and data.Source: §2.1 · pp. 2–3

2.2 Applying Autoregression+DDPMs to TTS

• The system trains an autoregressive text-conditioned speech-token predictor, a contrastive output ranker, and a DDPM that converts speech tokens into spectrograms.Figure 1 depicts text and reference audio flowing through decoding and filtering networks to produce speech.Source: §2.2 · p. 3, Figure 1 · p. 3

2.2.1 Conditioning Input

• Same-speaker audio clips are converted to MEL spectrograms and encoded separately for the autoregressive generator and DDPM as a speech conditioning input.The encoders’ outputs are averaged across layers and clips before use; the conditioning is intended to capture vocal tone and prosody and reduce the possible outputs for given text.Source: §2.2.1 · pp. 3–4

2.2.2 The “TorToise Trick”

• After training the DDPM on discrete speech codes, TorToise fine-tunes it on autoregressive latents, which the author describes as more semantically rich and more efficient for downstream diffusion.The author calls this fine-tuning one of the largest contributors to output quality among the training-process changes.Source: §2.2.2 · p. 4

2.3 CLVP

• CLVP scores text and discretized speech-token pairs to rerank autoregressive outputs without invoking the diffusion model for each candidate.It is trained contrastively on paired speech and text and serves as TorToise’s scoring model for candidate selection.Source: §2.3 · p. 4

• Training used eight NVIDIA RTX-3090 GPUs over one year.The paper refers readers to Appendix B for training specifics.Source: §3 · p. 4

• Inference generates many autoregressive candidates, scores them with CLVP, selects the top k, decodes them to MEL spectrograms with the DDPM, and converts them to waveforms with a vocoder.Source: §4 · p. 4
• Autoregressive decoding uses nucleus sampling with P=.8, repetition penalty=2, and softmax temperature=.8.Source: §4 · p. 4
• The author chose DDIM as the DDPM sampling algorithm, describing the configuration as the best quality–inference-speed balance found at design time.Source: §4 · p. 4

• LibriTTS and HiFiTTS provide 896 hours of transcribed speech, supplemented by an extended 49,000-hour audiobook and podcast dataset scraped from the internet.The official LibriTTS test split was used for validation.Source: §5 · p. 5

• The evaluation challenge is that many state-of-the-art TTS systems are closed source, leaving few samples for comparison.Source: §6 · p. 5
• The suite uses CLVP to measure distance between real and generated samples, and an open-source wav2vec model to characterize speech intelligibility.Source: §6 · p. 5
• Comparisons between TorToise samples and samples from other papers are available separately.Source: §6 · p. 5

• TorToise is reported to outperform previous TTS models in realism despite using architectures not specifically designed for audio.Source: §7 · p. 5
• The conclusion names generalist transformer architectures, a large high-quality dataset, and large-scale, high-batch training as the approach’s three components.Source: §7 · p. 5
• The author’s takeaway is that adhering to these three points produces strong results and may apply to any digitized modality.Source: §7 · p. 5

• The extended dataset draws on audiobooks and podcasts scraped from the web, with disk-space limits constraining the amount collected.Source: A Extended Dataset Collection · p. 6
• Clips between 5 and 20 seconds were retained after splitting at 500 ms silences and filtering for noise, music, poor quality, multiple speakers, and reverb.Source: A Extended Dataset Collection · p. 6
• The cleaning pipeline yielded 49,000 hours of audio, which was transcribed using wav2vec2-large fine-tuned for punctuation on LibriTTS and HiFiTTS.Source: A Extended Dataset Collection · p. 6

B.1 VQVAE

• The VQVAE compresses MEL spectrograms with a small residual convolutional network and produces a codebook of 8192 tokens.Source: §B.1 · p. 6
• Larger batch sizes decreased VQVAE reconstruction losses; training inputs were limited to 40960 PCM readings, or 2 seconds of audio.Source: §B.1 · p. 6
• Figure 2 plots VQVAE MSE loss against training steps, with both axes on a log-log scale.Source: Figure 2 · p. 7

B.2 Autoregressive Prior

• The autoregressive decoder uses a GPT-2 architecture with dense self-attention, following the DALLE-1 training instructions; its prompt combines speech conditioning, text, and MEL tokens.Source: §B.2 · p. 8
• A separate encoder converts a related speaker’s MEL clip into a conditioning vector; the maximum conditioning input is 132,300 samples, or 6 seconds.Source: §B.2 · p. 8
• The model supports sequences of up to 402 text tokens and 604 MEL tokens; training progressed from clips under 6 seconds to clips up to 27 seconds, followed by fine-tuning on LibriTTS and HiFiTTS.Source: §B.2 · pp. 8–9

B.3 CLVP

• The CLIP-style selection pattern scores generated candidates against a prompt, as in the original DALL-E pipeline.DALL-E passed generated images through CLIP and selected the image judged closest to the input text.Source: §B.3 · p. 10
• CLVP uses separate CLIP-like encoders for text and MEL tokens to produce distance metrics for text/speech pairs.Both encoders use fixed positional embeddings; tokens from each are dropped out at 15%.Source: §B.3 · p. 10
• Figure 4 plots late CLVP cross-entropy loss against sample count on log-log axes; early training curves were lost.Source: Figure 4 · p. 10

B.4 Diffusion Decoder

• The diffusion decoder combines residual convolutions and dense self-attention, without upsampling or downsampling, and receives timestep, speech-conditioning, and autoregressive activations.Source: §B.4 · p. 11
• Training on autoregressive activations produced the largest reported quality gain, while full attention improved Frechet distance and MEL decoding with a vocoder outperformed PCM decoding.PCM training required very small context windows and took an inordinate amount of time; training on autoregressive activations was expensive because each step required forwarding through the autoregressive network.Source: §B.4 · p. 11
• Classifier-free guidance is applied to both speech conditioning and autoregressive activations, with both dropped and replaced by a learned embedding 15% of the time.Source: §B.4 · p. 11
• Diffusion training clips audio to 10 seconds and conditioning inputs to 5 seconds; 24kHz MEL output supports the pretrained Univnet vocoder rather than performance gains.Source: §B.4 · p. 11

C Future Work

• The author recommends smaller VQVAE codebook embeddings, relative positional encodings, larger CLVP batches, and longer CLVP audio clips.Fixed positional encodings limit generated speech length, and CLVP saw only 13-second clips, which likely explains weaker reranking on longer samples. The author reports drastic performance improvements from constricting VQVAE codebook dimensions and notes that contrastive models benefit from very large batches.Source: C Future Work · p. 12
• Future diffusion changes include adding feed-forward blocks and aligning the stack at 24kHz or retraining Univnet at 22kHz.The author calls omission of feed-forward blocks a poor design decision.Source: C Future Work · p. 12
• The author recommends more data and longer training, noting the curves indicated training was far from overfitting, despite limited GPUs and mistakes during experimentation.Source: C Future Work · p. 12

D Special Thanks

• The project is credited to the open-source community, with several contributors and organizations thanked for work instrumental to building TorToiSe.Source: D Special Thanks · p. 12