BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data

1 Introduction

• The paper situates large-scale TTS within broader progress in NLP, computer vision, and speech modeling using Transformer-based approaches.Prior TTS systems include causal language models, non-causal modules, and diffusion decoders.Source: §1 · pp. 1–2
• Earlier leading neural TTS systems used only a few hundred hours and typically struggled to render ambiguous and complex text expressively.Historically, achieving greater expressiveness often relied on labeled data for specific speech phenomena.Source: §1 · p. 2
• BASE TTS uses a decoder-only autoregressive Transformer to model text tokens followed by discrete speech representations called speechcodes.Its cross-entropy objective is described as capturing complex expressive-speech distributions.Source: §1 · p. 2
• Speaker-disentangled speechcodes assign segmental content, prosody, and duration to the autoregressive predictor, while a separate decoder reconstructs speaker identity and recording conditions.The convolution-based decoder reduces whole-system synthesis time by over 70% versus the baseline diffusion decoder.Source: §1 · p. 2
• BASE TTS is presented as the largest TTS model to date, with 1 billion parameters and 100K hours of public-domain speech data.Source: §1 · p. 2
• BASE TTS performs better than publicly available large-scale TTS baselines in subjective evaluations.Source: §1 · p. 3
• Scaling BASE TTS variants improves quality monotonically on a new test set for prosody on textually complex inputs.The test set measures text understanding and rendering in large-scale TTS models.Source: §1 · p. 3
• The proposed speech representations are built on WavLM SSL, target phonemic and prosodic information, and decode to high-quality waveforms at 400 bits/s.They outperform the baseline quantization method, and the decoder is described as simple, fast, and streamable.Source: §1 · p. 3

2.2 Discrete speech representations

• WavLM-based speechcodes use speaker-disentangling losses and BPE compression to make compact representations for autoregressive modeling.The paper also compares them with VQ-VAE codes; both representations are compressed to 325 bits/s and 400 bits/s, respectively, versus 6k bits/s for popular audio codecs.Source: §2.2 · p. 4

2.2.1 Autoencoder-based speech tokens

• The VQ-VAE baseline reconstructs mel-spectrograms through a discrete bottleneck, with a global reference encoder to partially separate speaker information.Informal listening found that these codes still contained speaker information, motivating improved disentanglement.Source: §2.2.1 · p. 4

2.2.2 WavLM-based speechcodes

• The WavLM tokenizer quantizes content encodings while separately extracting speaker embeddings, using reconstruction and speaker-related losses to encourage disentanglement.The authors hypothesize that this encourages separation of content and speaker information.Source: §2.2.2 · p. 4

2.2.3 Byte-pair encoding on speechcodes

• BPE reduces the average WavLM speechcode sequence length by around 40% to mitigate Transformer sequence-length memory costs and complexity.The method iteratively merges frequent pairs of speechcodes into new tokens until reaching a predetermined vocabulary size.Source: §2.2.3 · p. 5

2.1 Overview

• SpeechGPT models text and speechcodes autoregressively, conditioned on text and reference speech, and a separate decoder turns predicted representations into waveforms.The decoder uses the Transformer’s last hidden state in practice, which improves segmental and acoustic quality but couples the decoder to a specific SpeechGPT version.Source: §2.1 · p. 3, §2.3 · p. 5, §2.4 · p. 6

2.3 Autoregressive speech modeling (SpeechGPT)

• SpeechGPT is a GPT-2-architecture autoregressive model that predicts speechcodes from text and a reference utterance encoded as a fixed-size embedding.It also predicts text tokens, with a lower loss weight than the speech prediction objective, to retain textual information for prosody.Source: §2.3 · p. 5

2.4 Waveform generation

• The proposed speechcode decoder predicts waveforms end-to-end, replacing the diffusion spectrogram decoder and separate vocoder baseline.In practice, it takes SpeechGPT’s last hidden state as input; this improves segmental and acoustic quality but makes the decoder dependent on a specific SpeechGPT version and forces sequential construction.Source: §2.4 · pp. 5–6, Figure 3 · p. 6

3 Experimental setup

• The experiments compare tokenizer and decoder architectures, assess three data and model scales for emergent abilities, and evaluate naturalness, intelligibility, speaker similarity, and open-source TTS comparisons.The architecture experiments compare autoencoder- and WavLM-based speechcodes, then diffusion-based and speechcode decoders.Source: §3 · p. 6

3.1 Dataset

• The dataset contains 100K hours of public-domain speech, over 90% English, mostly recorded outside studio conditions without additional denoising.ASR and voice activity detection produce segments up to 40 seconds; partial text restoration replaces about half of the transcripts and increases rare punctuation representation.Source: §3.1 · pp. 6–7

3.2 Training and hyperparameters

• Training proceeds through tokenizer training, SpeechGPT training on the full dataset, and speechcode decoder training.The tokenizer variants use WavLM codes compressed with BPE or VQ-VAE codes; SpeechGPT is trained with cross-entropy on text and speechcodes, and the decoder uses a modified GAN scheme.Source: §3.2 · p. 7

3.3 Evaluation methodology

• Evaluation combines native-listener MUSHRA ratings, expert judgments on seven categories of challenging text, and automated WER and speaker-similarity measures.MUSHRA results report means with 95% confidence intervals, and pairwise differences are tested for significance.Source: §3.3 · p. 8

4.1 VQ-VAE speechcode vs. WavLM speechcodes

• English MUSHRA scores were statistically tied (VQ-VAE: 74.8 vs WavLM: 74.7), while WavLM significantly led for Spanish (VQ-VAE: 73.3 vs WavLM: 74.7).The authors use WavLM-based speechcodes in later experiments because they perform at least as well or better.Source: §4.1 · p. 8, Table 3 · p. 10

Compound Nouns

• Table 3 compares mean MUSHRA scores for VQ-VAE and WavLM speechcodes across 10 US English and Spanish speakers.The caption specifies that statistically significant winners are highlighted.Source: Table 3 · p. 10

4.2 Diffusion-based decoder vs. speechcode decoder

• The speechcode decoder preserved quality while outperforming diffusion decoding for four of six voices; differences for the other two were not statistically significant.The authors prefer it for faster inference and suggest that the diffusion decoder can be dropped.Source: Table 4 · p. 10, §4.2 · p. 10

4.3 Emergent abilities - Data and model size ablation

• BASE-small never averaged 1.75 in any expert-rated category, whereas BASE-medium scored at least 1.75 in every category; further gains were smaller and compound-noun performance saturated.Emotions and paralinguistics remained most challenging, with BASE-large around 2.0 and prosody still imperfect to experts.Source: Figure 4 · p. 11, §4.3 · p. 11
• Naturalness improved significantly from BASE-small to BASE-medium, while the BASE-medium-to-large difference was significant only for Male speaker A.The evaluation used two US English speakers.Source: §4.3 · p. 10, Table 5 · p. 11

4.4 BASE TTS vs. industry baselines

• BASE TTS significantly outscored industry baselines on MUSHRA naturalness for two US English speakers and led overall on naturalness, WER, and speaker similarity.Tortoise followed relatively closely on naturalness and WER; Bark and YourTTS performed much worse on those measures, while Bark was relatively close in speaker similarity.Source: §4.4 · pp. 11–12, Table 6 · p. 12, Table 7 · p. 12

4.5 Synthesis efficiency gain due to speechcode decoder

• For 1-billion-parameter SpeechGPT, speechcode decoding cut average synthesis time from 69.1 seconds to 17.8 seconds and enabled first-byte latency as low as 100 ms.The benchmark generated 1,000 utterances of around 20 seconds on a V100 GPU at batch size 1; the decoder supports incremental streaming.Source: §4.5 · p. 12

Text-to-speech as language modeling

• Prior language-model TTS includes TortoiseTTS [21] and VALL-E [17], with later systems extending the approach to translation, recognition, and speech-editing tasks.The paper reports that BASE TTS uses 100K hours and shows improved naturalness over Tortoise-TTS, with complex prosody rendered without explicit emotion labels.Source: §5 · pp. 12–13

Discrete speech representations

• BASE TTS combines self-supervised semantic information with speaker-disentangled acoustic tokens, then applies byte-pair encoding to reduce memory needs and permit longer training sequences.Earlier discrete acoustic encoders can ignore semantics and lack disentanglement; other approaches add self-supervised embedding-prediction losses.Source: §5 · p. 13

LTTS simplifies modeling

• Traditional TTS separates text processing, spectrogram generation, and vocoding, while BASE TTS and other large TTS models aim to simplify this pipeline.The paper says errors in an earlier stage propagate forward; BASE TTS requires audio and texts obtainable by speech recognition, without phoneme extraction.Source: §5 · p. 13

Contextual and emotional TTS

• Contextual and emotional TTS systems often rely on dedicated predictors or annotated recordings, whereas BASE TTS evaluates contextual text understanding across styles without supervised training or annotation.The paper notes that existing systems can miss strong textual cues for dramatic changes such as shouting or whispering.Source: §5 · p. 13

BASE TTS has data efficiency built-in

• The paper positions BASE TTS among large TTS models setting a new standard in data efficiency through larger datasets, larger models, and a dedicated speaker encoder.Prior low-resource approaches include voice conversion, data generation, modeling techniques, and recording-script optimization.Source: §5 · p. 13

• BASE TTS reports new state-of-the-art TTS results against Tortoise, Bark, and YourTTS.Source: §6 · p. 14
• LTTS models built with 10K hours of data and 400 million parameters start to exhibit an advanced grasp of text enabling contextually appropriate prosody.Source: §6 · p. 14
• BASE TTS uses novel SSL-based speechcodes and a streamable speechcode decoder as an alternative to diffusion.Source: §6 · p. 14
• The model is described as the largest of its kind known to the authors in both parameters and training data, using around 100K hours of public-domain speech.Source: §6 · p. 14
• BASE TTS occasionally produces hallucinations or cutoffs, and the paper reports results for only one speechcode configuration.The authors identify misalignment between audio and ASR-generated text as worsening hallucinations and cutoffs, and leave broader speechcode study for future work.Source: §6 · p. 14

• The authors chose not to open-source BASE TTS as a precaution against potential misuse of its voice-mimicking capability.Source: §7 · p. 14
• Using synthetic voices for people who have lost speech is presented as a possible application, subject to informed consent and rigorous data privacy reviews.Source: §7 · p. 14
• The authors note that speech-data composition affects expression of linguistic, ethnic, dialectal, and gender minorities, and advocate research on impacts, bias, and inclusivity.Source: §7 · p. 14

10. Where has Jason Abhisheki moved all the flowers to?

• Two example questions raise water-supply and Brexit issues.They ask about Jordan’s water supply and whether ministers will find answers to the Brexit question in time.Source: §10 · p. 21

14. Here’s the full story for today, would you like to learn more?

• A sample question frames an interview with Channel 4 as a catastrophe.It addresses the chairman and asks whether the anticipated interview has turned into a catastrophe.Source: §14 · p. 22

16. Johnny boy, don’t go around acting tough if you can’t back it up, right?

• One example challenges someone over a policy and alleged deference to leadership.It asks whether the person supports the Latex usage policy or is “sucking up to leadership.”Source: §16 · p. 22

19. Madam, have you tried turning it off and on again?

• Another example asks which of two devices someone used to take a picture.The alternatives are a hand-held camera and an “android phone.”Source: §19 · p. 22

A.2 Emotions

• The emotion examples depict states including excitement, anger, grief, fear, pride, and determination.They use narrated scenes and dialogue, including an acceptance-letter reaction, a reunion, a feared accident, and a detective’s resolve.Source: §A.2 · p. 22

A.3 Compound Nouns

• The compound-noun examples place compounds in varied everyday and topical sentences.Examples include “school bus,” “breakfast sandwich,” “bird feeder,” and “town hall,” alongside other compounds.Source: §A.3 · p. 23

A.4 Syntactic Complexity

• The syntactic-complexity examples range from ambiguous sentences to long clauses with nested modifiers.Examples include “Time flies like an arrow; fruit flies like a banana” and sentences containing multiple embedded clauses.Source: §A.4 · pp. 23–24

A.5 Foreign Words

• The foreign-word examples incorporate terms from several languages and cultural contexts.They include French expressions such as “joie de vivre,” German terms such as “Zeitgeist,” and Japanese terms such as “nigiri” and “umami.”Source: §A.5 · pp. 24–25

A.6 Punctuations

• The passages show punctuation in context: titles with & and %, nested parentheses, a five-dot ellipsis and em dash, degree and section symbols, credential placeholders, and numbered instructions.Source: §A.6 · pp. 25–26

A.7 Paralinguistics

• The examples convey speech through interjections like “Ouch!”, whispering, hesitant fillers such as “um”, stuttering, laughter (“haha”), and dialectal pronunciation (“ain’t meanin’ well”).These cues appear in dialogue as expressions of reaction or delivery.Source: §A.7 · p. 26

14. David whispered to Emily as the lights dimmed in the theater, "Shh, it’s starting."

• A dialogue example uses hesitation and exclamations to convey nervousness, trepidation, and excitement during a steep climb.The speakers’ changing reactions culminate in Emily’s enthusiasm for the view.Source: §14 · p. 27

17. Ta-da! well? What do you think? This is the best right?

• Three short examples pair exclamations with apology and impatience, amazement, or relief after a game or journey.The situations include Jack apologizing to his father, Chris reacting to a final play, and Tom expressing relief after reaching a seat.Source: §17 · p. 27