Source-linked AI summary
FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion
Jingyi li, Weiping tu, Li xiao
TL;DR
Voice conversion needs clean source-content and target-speaker representations, but existing approaches can leak speaker information, require annotated data, or suffer conversion–vocoder mismatch. FreeVC combines VITS-based reconstruction with WavLM bottleneck extraction and spectrogram-resize augmentation without text annotation. Experiments report superiority over baselines and greater robustness, while future work targets unseen-speaker similarity.
Problem
Existing voice-conversion approaches may leak speaker information from content representations, require extensive annotation, or degrade reconstruction through conversion–vocoder feature mismatch.
Method
FreeVC adapts VITS for text-free one-shot conversion, extracting content through a WavLM bottleneck and using spectrogram-resize augmentation to improve disentanglement.
Results
FreeVC outperforms all baseline models in speech naturalness, speaker similarity, WER, CER, and F0 variation consistency across the reported evaluation scenarios.
Takeaways & Limitations
The proposed method preserves linguistic content and prosody while remaining robust to low-quality source speech, including against text-based baselines.
Abstract
from arXiv · showhide
Voice conversion (VC) can be achieved by first extracting source content information and target speaker information, and then reconstructing waveform with these information. However, current approaches normally either extract dirty content information with speaker information leaked in, or demand a large amount of annotated data for training. Besides, the quality of reconstructed waveform can be degraded by the mismatch between conversion model and vocoder. In this paper, we adopt the end-to-end framework of VITS for high-quality waveform reconstruction, and propose strategies for clean content information extraction without text annotation. We disentangle content information by imposing an information bottleneck to WavLM features, and propose the spectrogram-resize based data augmentation to improve the purity of extracted content information. Experimental results show that the proposed method outperforms the latest VC models trained with annotated data and has greater robustness.
1. INTRODUCTION
FreeVC targets text-free one-shot voice conversion by combining cleaner content extraction with VITS-based waveform reconstruction. It addresses annotation demands, speaker-information leakage, and conversion–vocoder feature mismatch.
- One-shot voice conversion changes source speaker identity while preserving linguistic content, using one target-speaker utterance as reference.
- Typical systems disentangle source content and target speaker information before reconstructing converted speech, so disentanglement and reconstruction both affect quality.
- Text-based VC uses ASR-derived phonetic representations or TTS linguistic knowledge but requires extensive annotated training data.
- Two-stage systems can degrade waveform quality because conversion-model acoustic features differ from the real-speech features used to train the separate vocoder.
- FreeVC is a text-free one-shot VC system that adopts VITS while learning content representations without text annotation.
- WavLM features and a bottleneck extractor provide content information, while spectrogram-resize augmentation distorts speaker information without changing content information.
2. METHODS
FreeVC adapts VITS into a text-free one-shot VC pipeline, using WavLM bottleneck features for content and speaker encoding for target identity. Spectrogram-resize augmentation and combined CVAE/GAN losses support disentanglement and waveform reconstruction.
- Architecture: FreeVC inherits VITS’s CVAE-plus-GAN backbone but replaces text input to the prior encoder with raw waveform input and adds a speaker encoder for one-shot conversion.
- Content extraction: The prior encoder combines WavLM, a bottleneck extractor, and a normalizing flow to model content information.
- Content extraction: WavLM produces 1024-dimensional SSL features, which the bottleneck compresses to d dimensions to discard content-irrelevant information.
- Speaker encoding: FreeVC supports pretrained and jointly trained non-pretrained speaker encoders; the latter is intended to model missing speaker information when content representations are clean.
- SR-based data augmentation: Because narrow bottlenecks lose content and wide bottlenecks retain speaker information, SR augmentation distorts source speaker information to improve clean-content learning.
- SR-based data augmentation: Vertical spectrogram resizing changes mel-spectrogram frequency structure before neural-vocoder waveform reconstruction, using interpolation followed by padding or cutting.
- Training loss: Training combines CVAE reconstruction and KL losses with GAN adversarial and feature-matching losses; the KL term reduces feature mismatch.
- Inference: During inference, FreeVC extracts content through WavLM and the bottleneck extractor as in training, rather than relying on source-speaker embedding quality.
3. EXPERIMENTS
Experiments evaluate FreeVC against text-free and text-based baselines using subjective and objective measures across seen and unseen speakers. FreeVC models outperform baselines, preserve linguistic content and prosody, and remain robust to degraded source speech.
- Experimental Setup: Evaluations use VCTK for training, VCTK and LibriTTS for testing, and seen-to-seen, unseen-to-seen, and unseen-to-unseen scenarios.Subjective tests measure 5-scale MOS and SMOS; objective tests measure WER, CER, and F0-PCC.
- Speech Naturalness and Speaker Similarity: FreeVC models outperform all baseline models in every scenario for both speech naturalness and speaker similarity.The comparison uses MOS and SMOS results with 95% confidence intervals.
- Robustness: Unlike baselines, FreeVC models are barely affected by low recording quality or unclear pronunciation, demonstrating robust content extraction.The baselines show quality degradation under these source-speech conditions.
- Ablation and Speaker Encoder: FreeVC without SR performs worse than FreeVC, while FreeVC-s is similar to FreeVC; pretrained speaker encoders help more for unseen targets.FreeVC outperforms FreeVC-s in the unseen-to-unseen scenario.
- Speech Intelligence and F0 Variation Consistency: FreeVC models achieve lower WER and CER than all baselines, including text-based systems, indicating strong preservation of source linguistic content.For these metrics, lower values are better.
- Speech Intelligence and F0 Variation Consistency: FreeVC produces higher F0-PCC consistency with source speech, while SR-based augmentation slightly improves speech intelligence and F0 variation consistency.F0-PCC ranges from -1 to 1, with higher values better.
4. CONCLUSION
The paper concludes that FreeVC is a text-free one-shot voice conversion system combining VITS reconstruction with bottleneck-based WavLM content extraction and SR-based augmentation. Experiments demonstrate the superiority of the proposed methods.
- Conclusion: FreeVC combines VITS-based waveform reconstruction, bottleneck extraction from WavLM features, and SR-based augmentation for text-free one-shot voice conversion.The SR-based augmentation is intended to improve the model’s disentanglement ability.
- Conclusion: Experimental results demonstrate the superiority of the proposed methods.