Source-linked AI summary
Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, Hsin-Min Wang
TL;DR
Non-parallel voice conversion is difficult because conventional systems often require aligned speech, especially when speakers use different texts or languages. The paper proposes VAW-GAN, which combines VAE-based latent inference with W-GAN speech synthesis and directly optimizes a non-parallel VC objective. Experiments report improved conversion quality, including significantly better naturalness than the VAE baseline and more realistic spectral shapes.
Problem
Voice conversion from non-parallel or different-language speech lacks the paired utterances required by many traditional systems.
Method
VAW-GAN combines VAE inference with W-GAN synthesis and directly optimizes a non-parallel voice-conversion loss.
Results
The framework produces more realistic spectral shapes and significantly outperforms the VAE baseline in naturalness.
Takeaways & Limitations
The experiments demonstrate that VAW-GAN can build a voice-conversion system from unaligned data with improved conversion quality.
Takeaways & Limitations
Objective evaluations were not reported because they were inconsistent with subjective evaluations, and speaker similarity was not improved under the shared speaker model.
Abstract
from arXiv · showhide
Building a voice conversion (VC) system from non-parallel speech corpora is challenging but highly valuable in real application scenarios. In most situations, the source and the target speakers do not repeat the same texts or they may even speak different languages. In this case, one possible, although indirect, solution is to build a generative model for speech. Generative models focus on explaining the observations with latent variables instead of learning a pairwise transformation function, thereby bypassing the requirement of speech frame alignment. In this paper, we propose a non-parallel VC framework with a variational autoencoding Wasserstein generative adversarial network (VAW-GAN) that explicitly considers a VC objective when building the speech model. Experimental results corroborate the capability of our framework for building a VC system from unaligned data, and demonstrate improved conversion quality.
1. Introduction
The paper targets voice conversion without parallel speech by replacing frame-aligned transformation learning with a unified generative model. It introduces VAW-GAN, combining VAE inference with W-GAN synthesis, and reports more realistic spectra.
- Motivation: Traditional voice conversion struggles with different-language or non-parallel speech because it commonly requires paired utterances with identical texts.The paper seeks to bridge parallel and non-parallel VC systems.
- Framework: The framework uses a unified generative speech model that avoids explicit frame alignment and phone or frame clustering.Its focus shifts to inferring a latent variable and synthesizing speech matching the target distribution.
- Framework: VAW-GAN assigns inference to a variational autoencoder and speech synthesis to a Wasserstein generative adversarial network.This implementation explicitly incorporates the VC objective into the generative speech model.
- Results: The proposed W-GAN is reported to synthesize more realistic spectra for non-parallel voice conversion.The introduction identifies realistic spectral synthesis as a demonstrated capability.
2. Non-parallel voice conversion via deep generative models
The framework models speech with latent phonetic-content and speaker variables, enabling conversion without frame alignment. It progressively combines C-VAE, GAN, and Wasserstein objectives to directly optimize non-parallel voice conversion.
- Model formulation: Voice conversion is decomposed into a speaker-independent encoder that infers latent content and a speaker-dependent decoder that reconstructs or converts spectral frames.The decoder mixes latent content z with speaker representation y; replacing y enables conversion.
- Model formulation: The model explains observations with speaker representation y and phonetic content vector z, although the phonetic interpretation is limited to natural, noise-free, non-emotional speech.The paper explicitly restricts the term phonetic content to its experimental setting.
- C-VAE: C-VAE training simplifies the observation and latent distributions to Gaussian forms and uses sampling to make optimization tractable.The synthesis covariance is set to identity, the prior is standard normal, and expectations are approximated by sampling.
- GAN enhancement: The Gaussian, dimensionally uncorrelated observation assumption can produce muffled converted voices, motivating adversarial speech modeling.The paper attributes the synthesis inaccuracy to the simplified observation model.
- GAN and W-GAN: A GAN adds a generator and discriminator to improve spectral realism, while VAW-GAN replaces the Jensen-Shannon objective with a Wasserstein objective that directly targets non-parallel VC.The proposed model combines VAE and GAN components while explicitly incorporating a VC loss.
- Wasserstein VC objective: The Wasserstein formulation removes alignment requirements by comparing batches of real target frames with synthetic frames converted from source speech.Training uses real target frames, converted source frames, and a discriminator; encoder, synthesizer, and discriminator are optimized in alternating order.
3. Experiments
Experiments evaluated unaligned frame-by-frame VC with VAW-GAN and a VAE baseline on inter-gender and intra-gender conversions. Subjective tests found significantly better naturalness for VAW-GAN, alongside more structured spectra, while some objective measures were not reported.
- The system used the VCC 2016 dataset without frame alignment, training on 3 speakers with approximately 10 minutes per speaker.
- Conversion was performed frame by frame by inferring phonetic content from the source, selecting a target speaker representation, and synthesizing the output frame.
- The VAW-GAN was compared against a convolutional C-VAE baseline using pairwise five-point MOS tests with 10 listeners.
- VAW-GAN significantly outperformed the VAE baseline in naturalness, with paired t-tests yielding p-value ≪0.01.
- Objective evaluations such as mean melcepstral coefficients were not reported because they were inconsistent with subjective evaluations, and speaker similarity was unreported.
- VAW-GAN spectra showed richer frequency-axis variability and more observable peaks and troughs, reflecting clearer voices and enhanced intelligibility.
4. Discussions
VAW-GAN produces more structured and variable spectra than the VAE baseline, although its speaker modeling does not improve speaker similarity. The discussion attributes this limitation to using one global speaker variable.
- 4.1. W-GAN improved spectrum modeling: VAW-GAN spectra show more structured envelopes, with observable peaks and troughs that reflect clearer speech.More detailed high-frequency spectral shapes are associated with less muffled synthetic voices.
- 4.2. W-GAN as a variance modeling alternative: The Wasserstein objective minimizes the distance between converted-spectrum and true-data distributions.
- 4.2. W-GAN as a variance modeling alternative: VAW-GAN output spectra have larger variance than VAE outputs, indicating less severe centralization around the mean.The authors note that VAW-GAN’s global variance may still not match the data’s variance.
- 4.3. Imperfect speaker modeling in VAW-GAN: Speaker similarity does not improve because VAE and VAW-GAN optimize the same probabilistic graphical model and speaker model.The authors suggest that one global speaker variable may be insufficient and identify alternative graphical models as future work.
5. Related work
Related non-parallel VC systems commonly use explicit alignment through frame-, segment-, or cluster-based schemes. Prior work also uses i-vectors or ASR-derived senone probabilities, whereas the discussed models differ in their alignment choices.
- 5. Related work: Many non-parallel VC systems use frame-, segment-, or cluster-based alignment schemes.ASR can provide phonetic labels for each frame before explicit alignment or model adaptation.
- 5. Related work: Prior work represented speakers with i-vectors but used explicit alignment during training.
- 5. Related work: Another prior system represented phonetic space with ASR-derived senone probabilities and generated voices through a TTS module.
6. Conclusions
The paper presents VAW-GAN as a voice conversion framework for non-parallel data. It directly incorporates a non-parallel VC criterion into training and reports more realistic spectral shapes and significantly improved performance over the baseline.
- 6. Conclusions: VAW-GAN directly incorporates a non-parallel VC criterion into the objective function.
- 6. Conclusions: The framework improves converted outputs by producing more realistic spectral shapes.
- 6. Conclusions: Experimental results demonstrate significantly improved performance over the baseline system.