Source-linked AI summary
A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion
Benjamin van Niekerk, Marc-André Carbonneau, Julian Zaïdi, Mathew Baas, Hugo Seuté, Herman Kamper
TL;DR
Unsupervised voice conversion must preserve linguistic content while removing speaker information, yet discrete units can cause content loss and mispronunciations. The paper proposes soft units that predict distributions over discrete units, finding improved intelligibility and naturalness and better transfer to unseen languages.
Problem
Unsupervised voice conversion avoids labels and parallel speech, but systems still have a quality and intelligibility gap, while discrete units can discard linguistic content and cause mispronunciations.
Method
The paper trains soft content encoders to predict distributions over discrete speech units and compares them with discrete units across CPC and HuBERT voice-conversion systems.
Results
Soft units improve intelligibility and naturalness, with WER improving by around 50% relative and PER by over 20% versus discrete units.
Takeaways & Limitations
Soft units retain linguistic content while still discarding speaker information and transfer better to unseen languages in cross-lingual voice conversion.
Abstract
from arXiv · showhide
The goal of voice conversion is to transform source speech into a target voice, keeping the content unchanged. In this paper, we focus on self-supervised representation learning for voice conversion. Specifically, we compare discrete and soft speech units as input features. We find that discrete representations effectively remove speaker information but discard some linguistic content - leading to mispronunciations. As a solution, we propose soft speech units. To learn soft units, we predict a distribution over discrete speech units. By modeling uncertainty, soft units capture more content information, improving the intelligibility and naturalness of converted speech. Samples available at https://ubisoft-laforge.github.io/speech/soft-vc/. Code available at https://github.com/bshall/soft-vc/.
1. INTRODUCTION
Voice conversion aims to preserve linguistic content while changing speaker identity, but unsupervised systems still lag supervised systems in quality and intelligibility. The paper proposes soft speech units to retain more content than discrete units while removing speaker information.
- Unsupervised voice conversion avoids costly labels and parallel speech but still has a quality and intelligibility gap relative to supervised systems.
- Discretization removes speaker information but can discard linguistic content, increasing mispronunciations such as changing fin to thin.
- Soft speech units predict distributions over discrete units to model assignment uncertainty and retain more content information.
- The study compares discrete and soft units using CPC and HuBERT for any-to-one and cross-lingual voice conversion.
- The paper reports improved intelligibility and naturalness, including better transfer to unseen languages, with soft units.
2. VOICE CONVERSION SYSTEMS
The voice conversion system extracts discrete or soft speech units, converts them into a target-speaker spectrogram, and synthesizes audio. Soft units are learned by predicting distributions over discrete units rather than using raw continuous features or hard assignments.
- The system contains a content encoder, acoustic model, and vocoder that respectively extract units, predict a target spectrogram, and generate waveform audio.
- Content Encoders: The discrete content encoder extracts audio features, clusters them with k-means, and outputs a sequence of discrete speech units.
- Content Encoders: Raw continuous features can retain substantial speaker information, so the soft encoder instead predicts a distribution over discrete units.
- Content Encoders: Soft units provide a middle ground: discretization removes speaker information, while distributions reduce content loss caused by hard assignments.
- Content Encoders: During training, a backbone and linear layer produce soft units supervised by discrete-unit labels; at test time, the units feed the acoustic model.
- The acoustic model maps either unit type to a target-speaker spectrogram, and the vocoder converts that spectrogram into audio.
3. EXPERIMENTAL SETUP
Experiments evaluate discrete and soft units for any-to-one conversion across intra- and cross-lingual settings, using CPC and HuBERT systems, baselines, and intelligibility, speaker-similarity, and naturalness measures.
- The study uses LJSpeech as the target speaker and tests LibriSpeech intra-lingual conversion plus French and Afrikaans cross-lingual conversion.
- CPC and HuBERT serve as feature extractors or soft-encoder backbones in the compared systems.
- Discrete units are formed by k-means clustering intermediate CPC-big or HuBERT-base representations, while soft encoders are fine-tuned to predict those units.
- The acoustic model predicts target-speaker spectrograms and HiFi-GAN converts them into waveforms.
- AutoVC and a cascaded ASR-TTS system provide comparison baselines.
- Evaluation Metrics: Intelligibility is measured with WER and PER, speaker similarity with x-vector cosine-similarity EER, and naturalness with five-point MOS ratings.
- Evaluation Metrics: Phoneme-level PER is additionally broken down for HuBERT-Discrete and HuBERT-Soft.
4. EXPERIMENTAL RESULTS
Soft speech units improve intelligibility and naturalness over discrete units while largely preserving speaker similarity. They also transfer better to unseen languages, though with a larger cross-lingual speaker-similarity drop.
- Intelligibility: Soft units reduce WER by around 50% relative and PER by over 20% compared with discrete units.Both Hubert-Soft and CPC-Soft approach ground-truth intelligibility and outperform the cascaded ASR-TTS baseline.
- Intelligibility: Soft-unit improvements are concentrated mainly in consonants, especially /Ù/, /Z/, /k/, and /g/.The vowel /U/ is the main exception.
- Speaker Similarity: Discrete units effectively discard source-speaker information, while soft units cause only a small decrease in speaker similarity.Raw features perform notably worse, placing soft units between discrete and continuous features.
- Naturalness: Soft units significantly improve MOS naturalness over discrete units across both CPC- and HuBERT-based models.HuBERT-Soft performs best and approaches ground-truth naturalness scores.
- Cross-Lingual Voice Conversion: In cross-lingual conversion, soft units transfer better to unseen languages but produce a larger drop in speaker similarity.The authors suspect retained source accent information contributes to this trade-off.
5. CONCLUSION
The paper proposes soft speech units for unsupervised voice conversion. They represent linguistic content accurately while discarding speaker information, improving intelligibility and naturalness in objective and subjective evaluations.
- Soft speech units provide a middle ground between discrete and continuous features for unsupervised voice conversion.They accurately represent linguistic content while still discarding speaker information.
- Objective and subjective evaluations show that soft units improve intelligibility and naturalness.