Source-linked AI summary
VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion
Disong Wang, Liqun Deng, Yu Ting Yeung, Xiao Chen, Xunying Liu, Helen Meng
TL;DR
One-shot voice conversion requires disentangling speaker, content, and pitch information while using only one target utterance, but existing training often permits content leakage into speaker representations. VQMIVC combines vector-quantized content encoding with mutual-information minimization, and experiments report effective disentanglement with improved content and intonation preservation, speaker characterization, naturalness, and speaker similarity.
Problem
Existing one-shot voice-conversion methods generally ignore dependencies among speech representations, allowing content information to leak into speaker representations and degrade conversion performance.
Method
VQMIVC combines vector quantization and contrastive predictive coding for content encoding with mutual-information loss to reduce dependencies among content, speaker, and pitch representations.
Results
Experiments report effective disentanglement that preserves source linguistic content and intonation variations while capturing target speaker characteristics, with higher speech naturalness and speaker similarity than current one-shot systems.
Takeaways & Limitations
The proposed representation learning approach produces high-quality one-shot voice conversion by separating content, speaker, and pitch information in an unsupervised manner.
Abstract
from arXiv · showhide
One-shot voice conversion (VC), which performs conversion across arbitrary speakers with only a single target-speaker utterance for reference, can be effectively achieved by speech representation disentanglement. Existing work generally ignores the correlation between different speech representations during training, which causes leakage of content information into the speaker representation and thus degrades VC performance. To alleviate this issue, we employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training, to achieve proper disentanglement of content, speaker and pitch representations, by reducing their inter-dependencies in an unsupervised manner. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations for retaining source linguistic content and intonation variations, while capturing target speaker characteristics. In doing so, the proposed approach achieves higher speech naturalness and speaker similarity than current state-of-the-art one-shot VC systems. Our code, pre-trained models and demo are available at https://github.com/Wendison/VQMIVC.
1. Introduction
The paper targets one-shot voice conversion across arbitrary speakers using disentangled speech representations. It addresses content leakage into speaker representations by combining vector-quantized content encoding with mutual-information-based dependency reduction.
- One-shot voice conversion changes speaker identity across arbitrary speakers using only one target-speaker utterance as reference.
- Existing disentanglement methods decompose speech into speaker and content representations but lack direct measures and correlation constraints during training.
- The proposed VQMIVC decomposes speech into content, speaker, and pitch factors and combines VQCPC, reconstruction, and mutual-information losses.
- During inference, one-shot conversion replaces the source speaker representation with a target representation extracted from one target utterance.
2. Related work
Prior one-shot voice-conversion work separates speaker information from spoken content, but information leakage remains a challenge without explicit dependency constraints. VQMIVC builds on vector quantization and mutual information to address this issue in fully unsupervised training.
- One-shot voice conversion must handle arbitrary speakers, including unseen speakers, using only one target-speaker utterance for reference.
- Prior speech-representation-disentanglement approaches include information-constraining bottlenecks, instance normalization, and vector quantization.
- Without explicit constraints between speech representations, information leakage tends to occur and degrades voice-conversion performance.
- The paper uses mutual information as a regularizer to constrain dependency between variables and adopts VQCPC to extract content representations.
3. Proposed approach
VQMIVC disentangles content, speaker, and pitch representations through a four-module voice-conversion system, while mutual-information minimization reduces their inter-dependencies without supervision.
- System architecture: The system extracts content, speaker, and pitch representations, then decodes them into mel-spectrograms for voice conversion.Its four modules are a content encoder, speaker encoder, pitch extractor, and decoder.
- System architecture: VQCPC quantizes frame-level acoustic features into discrete content representations and uses contrastive predictive coding to capture local structures.The quantizer applies a trainable codebook, while the CPC objective distinguishes future positive samples from negatives.
- System architecture: The speaker encoder produces a fixed-dimensional vector representing global speaker characteristics, while normalized log-F0 represents utterance-level intonation variations.Speaker-independent pitch processing encourages the speaker encoder to provide speaker information such as vocal range.
- MI minimization: Mutual-information loss sums estimated dependencies between content, speaker, and pitch representations and is weighted within the overall VC objective.vCLUB provides an upper bound for mutual information, and its variational networks and VC network are optimized alternately.
- MI minimization: Training uses VC loss combining VQ, CPC, reconstruction, and MI terms, without text transcriptions or speaker labels.The MI weight controls how strongly the loss promotes disentanglement.
- One-shot conversion: During one-shot conversion, source content and pitch are combined with speaker information extracted from one target utterance before decoding.The decoder generates converted mel-spectrograms from these three representations.
4. Experiments
Experiments on unseen VCTK speakers evaluate disentanglement, content preservation, pitch consistency, naturalness, and speaker similarity. Mutual-information minimization reduces representation leakage and supports stronger one-shot voice-conversion performance.
- Experimental setup: Experiments use VCTK with 90 training and 20 unseen testing speakers for one-shot voice conversion.Audio is represented with 16 kHz, 80-dimensional mel-spectrograms and F0 features.
- Speech representation disentanglement performance: Increasing λ_MI decreases mutual information among content, speaker, and pitch representations.The MI weight controls the disentanglement capacity of the MI loss.
- Speech representation disentanglement performance: When λ_MI=0, generated speech shows the largest ΔC and ΔW, while larger λ_MI reduces both measures of content leakage.The evaluation mixes content and pitch from one utterance with the speaker representation of another utterance from the same speaker.
- Speech representation disentanglement performance: Increasing λ_MI makes the quantized content representation contain less speaker and pitch information, while the speaker representation retains abundant speaker information.The selected setting for subsequent experiments is λ_MI=1e-2.
- Content preservation and F0 variation consistency: VQMIVC achieves the lowest CER and WER among compared methods and the largest reported F0-PCC of 0.781.These results indicate preservation of source linguistic content and consistency of converted intonation variations.
- Speech naturalness and speaker similarity: MI minimization improves speech naturalness and speaker similarity relative to the w/o MI system in subjective tests.The evaluation uses 15 subjects and 5-point MOS scores with 95% confidence intervals.
5. Conclusions
The proposed unsupervised approach combines VQCPC and mutual information minimization to disentangle content, speaker, and pitch representations for one-shot voice conversion. Experiments show improved preservation of linguistic content and intonation, target-speaker characteristics, and converted-voice quality.
- The model combines VQCPC and mutual information minimization for unsupervised disentanglement of content, speaker, and pitch representations.It jointly uses reconstruction, VQCPC, and MI losses during training.
- VQCPC learns accurate content representations that preserve the source utterance’s linguistic content.
- The speaker representation captures desired target-speaker characteristics, while the pitch representation retains source intonation variations.
- The resulting system produces high-quality converted voice through reduced information leakage between speech representations.