Source-linked AI summary
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, Neil Zeghidour
TL;DR
SPEAR-TTS addresses the need for large amounts of parallel speech data in TTS by combining hierarchical discrete speech representations with pretraining, backtranslation, and example prompting. With 15 minutes of parallel data, it achieves competitive intelligibility, diverse voices, zero-shot speaker adaptation from 3 seconds, and subjective quality comparable to natural speech.
Problem
TTS typically requires hundreds of hours of transcribed utterances, limiting availability for low-resource languages despite abundant audio-only speech data.
Method
SPEAR-TTS composes text-to-semantic-token and semantic-to-acoustic-token seq2seq tasks, using audio-only data for speaking and 3-second example prompts for unseen-speaker control without speaker labels.
Results
15 minutes of parallel data yields intelligibility comparable to an adapted FastSpeech2-LR trained on 24 hours, while speech naturalness matches natural speech (MOS 4.96 vs. 4.92).
Takeaways & Limitations
SPEAR-TTS supports high-fidelity, voice-diverse, multi-speaker synthesis with minimal parallel supervision and zero-shot adaptation to unseen speakers.
Takeaways & Limitations
SPEAR-TTS relies on relatively large Transformer models that require substantial computing resources for training and inference.
Abstract
from arXiv · showhide
We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can be trained with minimal supervision. By combining two types of discrete speech representations, we cast TTS as a composition of two sequence-to-sequence tasks: from text to high-level semantic tokens (akin to "reading") and from semantic tokens to low-level acoustic tokens ("speaking"). Decoupling these two tasks enables training of the "speaking" module using abundant audio-only data, and unlocks the highly efficient combination of pretraining and backtranslation to reduce the need for parallel data when training the "reading" component. To control the speaker identity, we adopt example prompting, which allows SPEAR-TTS to generalize to unseen speakers using only a short sample of 3 seconds, without any explicit speaker representation or speaker-id labels. Our experiments demonstrate that SPEAR-TTS achieves a character error rate that is competitive with state-of-the-art methods using only 15 minutes of parallel data, while matching ground-truth speech in terms of naturalness and acoustic quality, as measured in subjective tests.
1 Introduction
SPEAR-TTS addresses the need for large amounts of transcribed speech by separating TTS into “reading” and “speaking” tasks. This enables audio-only training, pretraining, backtranslation, and example prompting for low-supervision multi-speaker synthesis.
- TTS typically requires hundreds of hours of transcribed utterances, while audio-only speech data is relatively abundant, including for low-resource languages.
- SPEAR-TTS maps text to semantic tokens and then semantic tokens to acoustic tokens, composing TTS from two sequence-to-sequence tasks.
- Decoupling “reading” from “speaking” allows the speaking component to use acoustic tokens extracted from massive unlabeled speech data.
- Pretraining and backtranslation reduce the parallel supervision required to train SPEAR-TTS.
- Example prompting conditions speaking on an audio clip of the target voice, simplifying controllable multi-speaker TTS when parallel data is single-speaker.
- With 15 minutes of single-speaker parallel data, SPEAR-TTS achieves CER 1.92% on LibriSpeech test-clean and MOS 4.96 versus 4.92 for ground-truth speech.
3 SPEAR-TTS Overview
SPEAR-TTS uses semantic tokens as an intermediate representation between text and acoustic tokens. Its first stage is trained with pretraining and backtranslation strategies that exploit speech-only data while reducing parallel-data requirements.
- S1 translates tokenized text into semantic tokens, while S2 translates semantic tokens into acoustic tokens decoded into speech.
- Semantic tokens encode mostly phonetic content with limited prosody and speaker information, making text-to-semantic mapping easier than direct text-to-acoustic mapping.
- S2 can be trained from semantic-acoustic token pairs extracted from audio-only data, separating speaking supervision from parallel text-audio supervision.
- S1 pretraining uses a denoising task in which corrupted semantic-token sequences are mapped back to their uncorrupted sequences.
- Backtranslation finetunes a decoder to map semantic tokens to transcripts, transcribes speech-only data, and uses the resulting synthetic pairs to train forward text-to-semantic mapping.
- Acoustic token sequences are at least 6× longer than semantic-token sequences in the described setup.
5 S2: Controlling the Generation Process
SPEAR-TTS controls generated voice and speaking conditions through example prompting in S2. The model uses prompt semantic and acoustic tokens to generate target acoustic tokens while preserving prompted characteristics.
- S2 maps semantic tokens to acoustic tokens and can generate randomly varying voice, tempo, and recording conditions observed in training data.
- Prompted generation concatenates prompt semantic tokens, target semantic tokens, and prompt acoustic tokens before generating target acoustic tokens.
- Prompt acoustic tokens preserve the prompt speaker’s voice and speaking conditions while target semantic tokens determine the generated content.
- During training, two non-overlapping speech windows provide prompt and target sequences, and the transcript of the prompt is not needed.
- Separator tokens at segment boundaries prevent boundary artifacts during prompted generation.
- Sampling multiple outputs and selecting with a no-reference audio-quality metric provides an inference-time way to reduce background noise.
6 Experimental Setup
The experiments evaluate fidelity, voice diversity, voice preservation, and perceived quality using speech-only and low-supervision datasets. They compare SPEAR-TTS with low-resource FastSpeech2 and VALL-E baselines.
- Datasets: LibriLight supplies around 60,000 hours of English audiobooks from more than 7,000 speakers for self-supervised speech representations.
- Datasets: S1 is trained in the low-resource regime on LJSpeech, a single-speaker dataset containing 24 hours of parallel data.
- Training data: Pretraining uses semantic tokens extracted from LibriLight, while backtranslation uses 551 hours of unlabeled LibriTTS speech.
- Metrics: CER measures faithfulness to input transcripts, voice diversity uses entropy over classified speakers, and voice preservation measures prompt-to-generation speaker-label agreement.
- Metrics: MOS rates audio quality and naturalness from Poor (1) to Excellent (5).
- Baselines: FastSpeech2-LR is the main low-supervision baseline, while VALL-E provides a comparison for zero-shot voice adaptation using a 3 second prompt.
7 Hyperparameters & Training details
SPEAR-TTS uses discrete semantic and acoustic tokens with Transformer-based sequence modeling, pretraining, backtranslation, and sampling strategies. The training setup specifies tokenization, optimization, model configurations, and inference controls.
- Token representations: Semantic tokens use a 512-entry codebook at 25 tokens per second, while acoustic tokens use three 1024-entry codebooks at 150 tokens per second.Semantic tokens are derived from the seventh w2v-BERT layer; acoustic tokens come from SoundStream and are interleaved across quantization levels.
- Pretraining: Pretraining uses a denoising task with 0.6 token-deletion probability, 0.5 dropout, batch size 256, and 1M updates on T5-Large.The resulting checkpoint is reused in the reported experiments.
- Fine-tuning and ablations: Fine-tuning searches dropout rates and the number of trainable layers, while ablations compare pretrained and from-scratch S1 training configurations.The fine-tuning search considers dropout values of 0.1, 0.3, and 0.5; the from-scratch ablation varies T5 model size.
- Inference: Beam search is used for S1 and temperature sampling for S2, combining transcript faithfulness with more diverse and natural speech.S1 uses beam size 10, while S2 selects temperature 0.75 on the LJSpeech validation set.
- Inference: A quality-based selection procedure samples three candidate utterances and retains the one with the highest no-reference audio-quality score.The setting balances synthesized-speech quality against computational cost.
8 Experiments
The experiments evaluate SPEAR-TTS for intelligibility, voice diversity, and zero-shot voice preservation under limited parallel supervision. Results show strong transcript fidelity, diverse voices, and effective reproduction of unseen speakers from short prompts.
- Experimental setup: SPEAR-TTS is evaluated across transcript faithfulness, voice diversity, speaker-voice control, and comparisons with existing TTS systems.Training conditions include from-scratch, pretrained, and backtranslation-based settings.
- 8.1 Intelligibility and Supervision Efficiency: 2.21% CER is achieved with 15 minutes of parallel data when pretraining and backtranslation are combined.Backtranslation alone reaches 2.88% CER with the same amount of parallel data, while pretraining maintains CER ≤4% with 2 hours.
- 8.2 Voice diversity: SPEAR-TTS generates diverse voices, including speakers absent from the single-speaker LJSpeech parallel training data.The experiment tests whether synthesized voice diversity changes with the number of speakers in the parallel data.
- 8.2 Voice diversity: SPEAR-TTS voice variability remains stable across training sets containing 1 to 247 speakers, ranging from 6.16 to 6.28 bits.Its measured variability is higher than FastSpeech2-LR, reported as 6.11 versus 0.66.
- 8.3 Prompted generation: 92.4% top-1 speaker accuracy is obtained when reproducing unseen target voices from 3-second prompts, with 0.41 bits of voice variability.Prompted generation also reaches 1.92% CER, compared with 2.21% without prompting.
- 8.3 Prompted generation: SPEAR-TTS reaches 0.56 cosine speaker similarity versus YourTTS at 0.34 and nearly matches VALL-E at 0.58 despite using 240,000× less parallel data.The comparison uses speaker-verification embeddings computed from prompts and generated speech.
9 Subjective Evaluation
Subjective tests compare SPEAR-TTS with baselines and natural speech using human ratings of audio quality and naturalness. With only 15 minutes of parallel data, SPEAR-TTS achieves MOS values comparable to ground-truth speech and favorable results against VALL-E on shared examples.
- Evaluation protocol: Human raters score audio quality and speech naturalness on a 1-to-5 scale using Mean Opinion Scores.Each audio sample is evaluated by 20 raters, and scores are averaged for each treatment.
- Evaluation protocol: The subjective evaluation uses 20 held-out audiobook sentences totaling 133 seconds and compares SPEAR-TTS with FastSpeech2-LR models trained on 15 minutes, 1 hour, and 24 hours.The evaluation sentences were taken from an audiobook chapter released after the datasets used for training.
- Results: SPEAR-TTS achieves MOS 4.96 versus 4.92 for ground-truth speech, despite training with only 15 minutes of parallel data.The paper reports considerably higher quality than the baseline systems in this minimal-supervision setting.
- Results: On 24 shared examples, SPEAR-TTS achieves MOS 4.75 and is reported as more natural and higher quality than VALL-E.The comparison uses the same transcripts, prompts, and subjective-testing protocol.
10 Related Work
SPEAR-TTS builds on discrete speech modeling and related semi-supervised and prompted-generation systems, differing through minimal supervision and transcript-free voice prompting.
- Generative spoken-language modeling established discretized speech as a foundation for unconstrained generation, continuation, emotion transfer, prosody, and dialog applications.
- Low- and semi-supervised TTS: Guided-TTS and Guided-TTS 2 exploit audio-only data, but rely on stronger supervision through a phoneme classifier or pretrained speaker verification system.
- Prompted Audio Generation: SPEAR-TTS instead uses parameter-less audio prompting without speaker labels, conditioning speech generation on self-supervised audio tokens.
- Prompted Audio Generation: SPEAR-TTS extends AudioLM-style voice prompting through prompt-aware training of its speaking model.
- Prompted Audio Generation: VALL-E also supports prompt-based voice and emotion conditioning, but requires prompt transcripts and 60,000 hours of ASR-transcribed audio.
11 Conclusions & Future work
SPEAR-TTS combines hierarchical discrete speech representations with audio-only data, pretraining, and backtranslation to reduce parallel-data requirements while retaining intelligibility, voice diversity, and zero-shot voice control.
- SPEAR-TTS uses semantic and acoustic tokens to decompose TTS into “reading” and “speaking” sequence-to-sequence tasks.
- Audio-only data trains the speaking model, supports pretraining, and supplies synthetic parallel data for backtranslation.
- 92.4% speaker-maintenance accuracy and 0.56 cosine similarity demonstrate zero-shot voice preservation from a 3-second example of an unseen speaker.
- MOS 4.96 versus 2.11 beats a strong single-voice baseline with 96× less parallel data, while MOS 4.92 matches natural speech.
- The authors identify applying the findings to truly low-resource languages and further reducing supervision as future directions.
12 Limitations
The study’s scope and data impose practical boundaries, while voice mimicry introduces misuse risks that require safeguards.
- The initial investigation uses English and well-studied datasets rather than a truly low-resource language.
- LibriLight provides diverse audio but is limited to 16 kHz and lower average quality than curated datasets, requiring additional high-sampling-rate synthesis.
- Large Transformer models require substantial computing resources for SPEAR-TTS training and inference.
- Voice mimicry may enable biometric bypass or impersonation, although synthesized speech was detected with 82.5% accuracy on a balanced dataset.
A Bandwidth extension: from 16 to 24 kHz
SPEAR-TTS uses bandwidth extension to combine the acoustic diversity of 16 kHz LibriLight with the quality of 24 kHz speech. The extension maps 16 kHz SoundStream tokens to 24 kHz tokens.
- Bandwidth extension: SPEAR-TTS can generate 24 kHz speech while benefiting from the diverse speakers and conditions in LibriLight.LibriLight contains only 16 kHz audio, motivating the bandwidth-extension stage for applications preferring 24 kHz output.
- Bandwidth extension: A sequence-to-sequence bandwidth-extension model maps 16 kHz SoundStream tokens to 24 kHz SoundStream tokens.The 24 kHz codec uses four residual vector-quantizer layers, a codebook size of 1024 per layer, and a 50 Hz embedding rate.
- Bandwidth extension: The approach combines a large, low-resolution dataset’s acoustic diversity with a smaller, high-resolution dataset’s quality.The bandwidth-extension training data is orders of magnitude smaller than the data used for S2.
- Bandwidth extension: Selecting higher-quality samples increases estimated quality and lowers CER, with ns = 3 chosen as a complexity-quality trade-off.For each input, multiple samples are generated and the one with the highest MOS estimate is selected.
E Detecting synthesized speech
The study tests whether SPEAR-TTS speech can be distinguished from natural speech using a classifier trained on AudioLM outputs. On the evaluated samples, the classifier achieves 82.5% accuracy.
- Detection setup: The evaluation uses an AudioLM-trained classifier to distinguish generated speech from natural speech.The classifier is trained on SoundStream-compressed LibriSpeech examples versus AudioLM-generated speech.
- Detection result: 82.5% accuracy was achieved when discriminating SPEAR-TTS-generated speech from natural speech on LibriSpeech dev-clean samples.Each example was paired with a SPEAR-TTS synthesis and a ground-truth re-synthesis via acoustic tokens.
- Evaluation caveat: The comparison excludes uncompressed ground-truth audio because coding artifacts would make the discrimination task trivial.The evaluation instead compares generated speech with ground-truth audio re-synthesized through acoustic tokens.