Source-linked AI summary
Simple and Effective Unsupervised Speech Synthesis
Alexander H. Liu, Cheng-I Jeff Lai, Wei-Ning Hsu, Michael Auli, Alexei Baevski, James Glass
TL;DR
High-quality neural TTS generally requires paired speech and text, a resource that limits its availability across languages. This paper builds TTS from unpaired audio and text using unsupervised ASR-generated pseudo-labels and a lexicon. On LJSpeech, unsupervised TTS approached supervised naturalness, though its word error rate remained higher.
Problem
High-quality neural TTS requires sizable speech-text paired datasets, limiting its availability across languages.
Source: §1 · p. 1Method
The system uses unpaired audio and text plus a lexicon, generating speech pseudo-labels with unsupervised ASR before training TTS on them.
Source: §1 · p. 1Results
On LJSpeech, unsupervised TTS scored MOS 3.91 versus supervised 3.94, but had 21.7% WER, 2.5% above supervised and 3.7% above real audio.
Source: §5.1 · pp. 3–4Takeaways & Limitations
The system matched supervised TTS performance on an English dataset without human annotation, according to the authors’ preliminary study.
Source: §6 · p. 4Multi-speaker unsupervised TTS remains challenging because speaker variance and annotation errors must be modeled together; the authors note this problem may be harder than in supervised TTS.
Source: §5.2 · p. 4
Abstract
from arXiv · showhide
We introduce the first unsupervised speech synthesis system based on a simple, yet effective recipe. The framework leverages recent work in unsupervised speech recognition as well as existing neural-based speech synthesis. Using only unlabeled speech audio and unlabeled text as well as a lexicon, our method enables speech synthesis without the need for a human-labeled corpus. Experiments demonstrate the unsupervised system can synthesize speech similar to a supervised counterpart in terms of naturalness and intelligibility measured by human evaluation.
1. Introduction
High-quality neural TTS typically requires speech-text pairs, limiting availability across languages. This study proposes a two-step unsupervised approach and reports speech quality on par with supervised TTS despite imperfect transcriptions.
1. Introduction
- High-quality neural TTS requires sizable speech-text paired datasets, leaving systems unavailable for most languages.The paper identifies this resource requirement as the limitation its approach aims to address.Source: §1 · p. 1
- The method first uses unsupervised ASR to create pseudo-labels, then trains TTS on machine-annotated speech instead of human-labeled pairs.It uses unpaired audio, unpaired text, and a grapheme-to-phoneme lexicon.Source: §1 · p. 1
- Experiments report that unsupervised TTS can synthesize intelligible, natural speech and perform on par with supervised TTS despite imperfect transcriptions.The authors evaluate this with objective and subjective tests.Source: §1 · p. 1
2. Background
The background describes neural TTS and vocoders, semi-supervised efforts to use unpaired data, and unsupervised recognition as a route to speech transcriptions without paired data.
2.1. Supervised Speech Synthesis
- Neural encoder-decoder TTS maps phone sequences to Mel spectrograms, which a separately trained neural vocoder converts into waveforms.High-quality systems still require considerable human-annotated speech.Source: §2.1 · p. 1
2.2. Semi-supervised Speech Synthesis
- Semi-supervised TTS uses unpaired data for pretraining, text representation, or augmentation, but remains bounded by the amount of paired training data.This work therefore uses no paired training data.Source: §2.2 · p. 1
2.3. Unsupervised Speech Recognition
- Wav2vec-U learns speech recognition without paired data through adversarial training, while wav2vec-U 2.0 removes special audio preprocessing and segment-boundary determination.The framework uses wav2vec-U 2.0 as its first stage to generate pseudo-labels for TTS.Source: §2.3 · pp. 1–2, Figure 1 · p. 2
3. Unsupervised TTS
The method assumes unpaired speech and text with no matching sentences or domain mismatch, plus a pronunciation lexicon. It generates pseudo-labels with unsupervised ASR, then trains TTS on those labels.
3.1. Problem formulation
- The task assumes unpaired speech and text with no exact sentence matches or domain mismatch, plus a lexicon mapping text words to phones.The pipeline is designed to build TTS from these resources.Source: §3.1 · p. 2
3.2. Pseudo labeling speech via unsupervised ASR
- Wav2vec-U 2.0 is trained on unpaired audio and text, then decodes the speech corpus into pseudo-labeled phone sequences.These pseudo-labels provide the annotations for the next stage.Source: §3.2 · p. 2
3.3. Unsupervised text-to-speech with pseudo label
- A sequence-to-sequence TTS learns to recover Mel spectrograms from imperfect phone annotations; a separate vocoder converts synthesized spectrograms into waveforms.At test time, the TTS receives the phone sequence for the desired sentence; vocoder training needs no text.Source: §3.3 · p. 2
4. Experimental Setup
The experiments use single-speaker LJSpeech and multi-speaker LibriTTS, train unsupervised recognition and TTS models, and assess naturalness and intelligibility through subjective and objective measures.
4.1. Datasets
- LJSpeech supplies single-speaker training audio, while LibriTTS supplies multi-speaker audio; text is unpaired, and the English lexicon yields 39 phones.Text sentences matching the speech corpora are excluded; LibriTTS uses train-clean audio and six target speakers.Source: §4.1 · pp. 2–3
4.2. Unsupervised ASR model
- The ASR recognizer is trained with wav2vec-U 2.0 on speech representations and unpaired text, using adversarial learning to predict phone sequences.Decoding uses a phone-based 6-gram language model and beam size 15; reported phone error rates are 6.97% on LJSpeech and 7.78% on LibriTTS.Source: §4.2 · p. 3
4.3. TTS model
- Transformer-TTS maps phones to Mel spectrograms, with speaker embeddings and Global Style Tokens for LibriTTS; separate pretrained vocoders produce waveforms.The setup uses Parallel WaveGAN for LJSpeech and HiFi-GAN for LibriTTS.Source: §4.3 · p. 3
4.4. Evaluation
- Naturalness is measured with MOS, intelligibility with WER on LJSpeech, and pairwise tests compare unsupervised and supervised synthesis for naturalness and intelligibility.The evaluation also includes natural speech and supervised TTS as comparison settings.Source: §4.4 · p. 3
5. Results
On LJSpeech, unsupervised speech was close to supervised speech in naturalness, though Google-ASR WER was higher; listeners showed no obvious naturalness preference and favored unsupervised speech for intelligibility. On LibriTTS, human evaluation still found considerably realistic speech, while the larger WER gap left multi-speaker modeling challenging.
5.1. Single Speaker TTS on LJSpeech
- 3.91 MOS for unsupervised TTS was 0.03 below supervised TTS’s 3.94 on LJSpeech, versus 4.05 for real speech.Source: §5.1 · p. 3, Table 1 · p. 4
- 21.7% WER for unsupervised TTS was 2.5% and 3.7% higher than the supervised topline and real audio, respectively.Source: Table 2 · p. 4, §5.1 · p. 4
- Listeners showed no obvious naturalness preference and preferred unsupervised speech for intelligibility in A/B tests, though test text lacked ground truth.Source: §5.1 · p. 4
5.2. Multi-speaker TTS on LibriTTS
- Despite a larger supervised–unsupervised gap than on LJSpeech, human evaluations found considerably realistic speech in the LibriTTS multi-speaker setup.Source: Table 4 · p. 4, §5.2 · p. 4
- 7.78% pseudo-label phone error, only slightly above LJSpeech’s 6.97%, accompanied a larger WER gap, leaving speaker variance and annotation errors as joint challenges.Source: §5.2 · p. 4
6. Conclusions
The paper presents a preliminary framework for completely unsupervised speech synthesis, combining unsupervised speech recognition with neural TTS. On an English dataset, it reports matching supervised systems without human annotation and identifies multi-speaker modeling, lexicon reduction, and low-resource-language experiments as future directions.
6. Conclusions
- The preliminary system matched supervised TTS performance on an English dataset without human annotation.The framework combines unsupervised speech recognition with the established neural speech-synthesis paradigm.Source: §6 · p. 4
- The framework relies on unsupervised speech recognition and the relatively mature neural-based speech-synthesis paradigm.Source: §6 · p. 4
- Future directions include better multi-speaker modeling, reducing resource requirements such as the lexicon, and testing low-resource languages.Source: §6 · p. 4