Source-linked AI summary
Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations
Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Yu Zhang, Wei Han, Ankur Bapna, Michiel Bacchiani
TL;DR
Web speech could expand TTS training data, but studio-quality recording is expensive and TTS quality depends on training-sample quality. Miipher combines speech and transcript representations in a restoration model, and experiments report robust restoration and TTS quality on par with training on a TTS-designed dataset.
Problem
Studio recording is expensive, while TTS generation quality is directly affected by training-sample quality, making Web-collected speech difficult to use for TTS training.
Source: §1 · p. 1Method
Miipher uses parametric resynthesis with w2v-BERT speech features and transcript-derived PnG-BERT linguistic conditioning to restore in-the-wild speech.
Source: §1 · p. 1, §2.1 · p. 1, §2.2 · p. 2Results
Miipher is reported to be robust against various audio degradations, and TTS trained on restored Web speech has quality on par with training on a TTS-designed dataset.
Source: §1 · p. 1Takeaways & Limitations
The authors conclude that Miipher can increase the value of in-the-wild speech samples by improving their quality for speech-generation training.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Speech restoration (SR) is a task of converting degraded speech signals into high-quality ones. In this study, we propose a robust SR model called Miipher, and apply Miipher to a new SR application: increasing the amount of high-quality training data for speech generation by converting speech samples collected from the Web to studio-quality. To make our SR model robust against various degradation, we use (i) a speech representation extracted from w2v-BERT for the input feature, and (ii) a text representation extracted from transcripts via PnG-BERT as a linguistic conditioning feature. Experiments show that Miipher (i) is robust against various audio degradation and (ii) enable us to train a high-quality text-to-speech (TTS) model from restored speech samples collected from the Web. Audio samples are available at our demo page: google.github.io/df-conformer/miipher/
1. INTRODUCTION
Miipher targets robust restoration of in-the-wild speech, where degradation can undermine both restoration and the quality of TTS training data. It combines speech and transcript representations, and experiments report robustness to varied degradation and high-quality TTS trained from restored Web speech.
1. INTRODUCTION
- The study uses restoration of Web-collected speech to increase high-quality training data for speech generation, where studio recording is expensive.TTS generation quality is directly affected by the quality of its training samples.Source: §1 · p. 1
- The central application challenge is restoration robustness, because failed restorations can introduce artifacts that harm subsequent speech-generation training.The authors aim to reduce the number of failed restorations as much as possible.Source: §1 · p. 1
- Miipher focuses on phoneme masking and deletion, including missing phonemes caused by codecs or down-sampling that can introduce unrecoverable errors.Masking by noise or reverberation can make speech difficult to distinguish from noise without additional information.Source: §1 · p. 1
- For its input, Miipher uses speech representations from w2v-BERT, trained on degraded speech, instead of conventional log-mel spectrograms.The authors expect the representation’s ASR gains to help make restoration robust against speech degradation.Source: §1 · p. 1
- Miipher uses transcript representations from PnG-BERT as linguistic conditioning for the deleted-phoneme speech-inpainting problem.PnG-BERT is a text self-supervised model for TTS use.Source: §1 · p. 1
- Miipher is reported to be robust against various audio degradations and to enable high-quality TTS training from restored Web speech.The resulting TTS quality was reported as on par with a model trained on a dataset designed for TTS.Source: §1 · p. 1
2. SPEECH RESTORATION MODEL
Miipher follows a parametric resynthesis pipeline: a feature cleaner predicts clean speech representations from degraded input, then a neural vocoder synthesizes the waveform. The design conditions cleaning on transcript and speaker information and adapts WaveFit to w2v-BERT features.
2.1. Model overview
- The model restores 24 kHz speech through parametric resynthesis, cleaning degraded features before a neural vocoder synthesizes the waveform.It uses w2v-BERT input features and transcript-derived PnG-BERT linguistic conditioning.Source: §2.1 · pp. 1–2, Figure 1 · p. 2
2.2. Feature cleaner
- The feature cleaner predicts clean w2v-BERT features using degraded features, transcript-derived linguistic features, and a speaker embedding.It aligns feature dimensions, combines speaker information with text features through FiLM, and uses cross-attention and DF-Conformer blocks to refine the input features.Source: §2.2 · p. 2
- Feature cleaning and Post-Net refinement are iterated twice with shared parameters, and the loss combines feature reconstruction and spectral-convergence-like terms.The loss is calculated before and after Post-Net and across iterations; iteration count is embedded and mixed into text and speaker embeddings.Source: §2.2 · p. 2
2.3. Neural vocoder
- WaveFit synthesizes waveforms from cleaned w2v-BERT features, which are upsampled fourfold to match the vocoder’s feature-rate needs.The method uses white noise as the initial waveform and replaces WaveFit’s gain normalization because the feature representation does not provide the original signal’s maximum amplitude.Source: §2.3 · p. 2
2.4. Model training
- Training uses 2,680 hours of noisy and studio-quality speech pairs, with simulated noise, reverberation, and codec artifacts in the noisy data.The setup also specifies frozen w2v-BERT, PnG-BERT, and speaker encoders, and a four-block DF-Conformer feature cleaner.Source: §2.4 · p. 3
3. EXPERIMENTS
On the synthesized test set, full w2v-BERT Miipher achieved the best MOS among the SR comparison models and preserved text and speaker characteristics. The experiments compared speech and conditioning features against clean recordings.
3.1. Experimental setup
- Testing used 534 studio-recorded US-English utterances contaminated with noise, reverberation, and codec artifacts, with MOS, WER, and speaker similarity evaluated.The MOS study used 89 subjects; WER was interpreted by trend because the ASR model used a different text normalizer.Source: §3.1 · p. 3, §3.2 · pp. 3–4
3.2. Results
- The full w2v-BERT model achieved the best MOS among SR comparison models and was 0.15 points below studio-recorded clean speech.Table 2 reports results on the synthesized test dataset with 95% confidence intervals.Source: Table 2 · p. 3, §3.2 · p. 4
- The proposed method's WER was comparable to clean speech, while W without transcript conditioning was 2.9 points behind it.The authors attribute the text-preservation effect to PnG-BERT transcript features.Source: §3.2 · p. 4
- SPK for W without speaker embedding was significantly lower than for the proposed method, supporting the embedding's role in retaining speaker characteristics.SPK is cosine similarity between speaker embeddings from clean and restored samples.Source: §3.2 · p. 4
4. TTS MODEL TRAINING WITH NON-TTS DATASET
The study restored Common Voice and LibriVox speech for TTS training and found strong results, including MOS close to restored LJspeech for LibriVox. It also reports a Common Voice TTS MOS of 3.63.
4. TTS MODEL TRAINING WITH NON-TTS DATASET
- The TTS system combined UNAT with a WaveRNN vocoder, trained for 100k and 400k steps respectively without fine-tuning.Evaluation synthesized LJspeech test sentences; the LibriVox model used four selected speakers.Source: §4 · p. 4
- The restored training data comprised 685 hours from Common Voice and 13,270 hours from LibriVox, with LibriVox covering 4,000 speakers.The Common Voice TTS model was single-speaker; the LibriVox model was multi-speaker.Source: §4 · p. 4
- TTS trained on restored LibriVox achieved MOS almost equal to restored LJspeech, while Common Voice TTS achieved MOS 3.63.The authors conclude that restoring large Web speech datasets enables high-quality TTS training.Source: §4 · p. 4
- The same WaveRNN vocoder could not be trained on original Common Voice because training failed to converge due to noise in the target speech.For reference, human-spoken and TTS-generated LJspeech MOS were 4.33 and 4.36, respectively.Source: §4 · p. 4
5. CONCLUSIONS
The conclusion presents Miipher as a parametric-resynthesis model for restoring in-the-wild speech using w2v-BERT speech features and PnG-BERT transcript features. The authors report robustness to degradation and high-quality TTS training from restored Web speech, while identifying several future directions.
5. CONCLUSIONS
- Miipher uses w2v-BERT speech representations and PnG-BERT transcript representations in a parametric-resynthesis model for in-the-wild speech restoration.The paper focused on phoneme masking and deletion as difficult degradations.Source: §5 · p. 4
- The authors report robustness to varied audio degradation and high-quality TTS models trained on restored public Web speech.They conclude that restoration can increase the value of in-the-wild speech for speech-generation training.Source: §5 · p. 4
- Future work includes using ASR-predicted transcripts for samples without transcriptions and developing multilingual Miipher for low-resource-language TTS.The authors also identify SSL representations from 24 kHz signals as future work.Source: §5 · p. 4