Source-linked AI summary
Voice Conversion from Non-parallel Corpora Using Variational Auto-encoder
Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, Hsin-Min Wang
TL;DR
Spectral conversion commonly requires parallel corpora or explicit alignments, which restricts training when such data are unavailable. This paper modifies a variational auto-encoder with speaker-independent encoding and speaker-conditioned decoding, and reports comparable objective and subjective performance to aligned-data baselines, including 2.76 MOS versus 2.75 MOS for ENMF-3000.
Problem
Many spectral-conversion methods require parallel corpora, phonetic alignments, or explicit frame correspondence, limiting training with non-parallel data.
Method
A VAE-based framework encodes speaker-independent phonetic representations and decodes speaker-dependent spectra using speaker identity.
Results
2.76 MOS for VAE-pair versus 2.75 MOS for ENMF-3000, while objective and target-similarity evaluations were comparable.
Takeaways & Limitations
The evaluations validate spectral conversion from unaligned data with performance comparable to systems trained using aligned data.
Abstract
from arXiv · showhide
We propose a flexible framework for spectral conversion (SC) that facilitates training with unaligned corpora. Many SC frameworks require parallel corpora, phonetic alignments, or explicit frame-wise correspondence for learning conversion functions or for synthesizing a target spectrum with the aid of alignments. However, these requirements gravely limit the scope of practical applications of SC due to scarcity or even unavailability of parallel corpora. We propose an SC framework based on variational auto-encoder which enables us to exploit non-parallel corpora. The framework comprises an encoder that learns speaker-independent phonetic representations and a decoder that learns to reconstruct the designated speaker. It removes the requirement of parallel corpora or phonetic alignments to train a spectral conversion system. We report objective and subjective evaluations to validate our proposed method and compare it to SC methods that have access to aligned corpora.
I. INTRODUCTION
Spectral conversion traditionally depends on aligned source-target frames or phonetic alignments, limiting use of non-parallel corpora. The proposed framework directly uses speaker identity and self-reconstruction to avoid these requirements while achieving baseline-comparable performance.
- I. INTRODUCTION: Most spectral-conversion methods require aligned source-target frames or phonetic-state alignments, often obtained with dynamic time warping.DTW fails when parallel corpora are unavailable.
- I. INTRODUCTION: Frame-wise alignment pairs source and target frames with similar phonetic contents, making alignment nearly sufficient for spectral conversion.Frame-to-model alignment instead maps frames to phonetic models before conversion.
- I. INTRODUCTION: The proposed framework exploits speaker identity directly and decomposes conversion into encoding and decoding without explicitly aligning source and target frames.This reformulates conversion as controlled self-reconstruction.
- I. INTRODUCTION: Parallel corpora are no longer necessary, and experiments found performance comparable to baseline systems.The introduction presents this as evidence for applicability to general spectral-conversion tasks.
II. THE PROPOSED METHOD
The method models speech frames through speaker-independent latent representations and speaker-conditioned decoding. By combining these factors, it reconstructs or converts spectra without frame alignment, under explicit factorization assumptions.
- II. THE PROPOSED METHOD: The framework hypothesizes that speech frames factor into speaker identity and phonetic content, analogous to identity and variation in handwritten digits.This analogy motivates the VAE formulation.
- A. Auto-encoder Reformulation for SC from Unaligned Data: The encoder fφ maps each observed frame xn to a speaker-independent latent variable zn interpreted as a phonetic representation.The representation may contain information beyond phonetic traits.
- A. Auto-encoder Reformulation for SC from Unaligned Data: The decoder concatenates speaker representation yn with zn to reconstruct a speaker-dependent frame ˆxn.Changing yn designates the reconstructed speaker.
- A. Auto-encoder Reformulation for SC from Unaligned Data: Because encoding and decoding operate frame-wise, the formulation synthesizes spectra without requiring alignment.The method accepts a frame and speaker representation, then combines phonetic and speaker factors.
- A. Auto-encoder Reformulation for SC from Unaligned Data: The framework assumes speaker and phonetic representations can be decoupled and that the decoder can blend them to synthesize a spectral frame.These assumptions define the method’s viability boundary.
B. Architecture
The architecture modifies a variational auto-encoder for spectral conversion from unaligned data. It optimizes a variational lower bound because direct joint-log-probability optimization is intractable.
- B. Architecture: The authors modify a variational auto-encoder to address spectral conversion from unaligned data.They choose VAE over a vanilla auto-encoder for latent-space interpretability and stronger regularization.
- B. Architecture: Spectral conversion is treated as a VAE generative process that maximizes the joint log-probability of individual frames.The formulation models frames probabilistically rather than relying on paired training examples.
- B. Architecture: The VAE objective decomposes into a Kullback-Leibler divergence term and a variational lower bound on marginal probability.The lower bound is further rewritten for optimization.
- B. Architecture: Direct optimization is intractable, so training optimizes the variational lower bound with respect to encoder parameters φ and decoder parameters θ.The procedure separately estimates the expectation and derives the KLD term.
1) Estimating the Expectation Term:
The expectation term is estimated by sampling latent representations, using re-parameterization to make sampling differentiable, and simplifying to one sample per frame.
- Sampling methods estimate the expectation term in the variational lower bound.
- The re-parameterization trick generates latent samples from a standard normal variable through a deterministic, data-driven function.
- 1 sample per frame is used in the final approximation by setting L to 1.
2) Modeling the Latent Space:
The latent space is regularized by encouraging its distribution to remain close to an isotropic standard normal prior, whose KLD has a closed-form evaluation.
- The KLD regularizes the latent variable so its distribution does not differ excessively from the chosen prior.
- The latent variable zn uses an isotropic standard normal prior.
- The Gaussian latent-variable choice permits the KLD term to be evaluated in closed form.
- D denotes the dimension of the latent space.
3) Modeling the Visible Space:
The visible log-spectrum is modeled with a Gaussian distribution having diagonal variance, yielding a closed-form log-probability and a variational objective optimized by SGD.
- The log-spectrum features are assumed to follow a Gaussian distribution with a diagonal variance matrix.
- The decoder uses feed-forward neural networks to parameterize the visible-space distribution.
- D denotes the dimension of the visible feature space.
- Training maximizes the variational lower bound by iteratively optimizing the model parameters.
- Stochastic gradient descent is used for optimization.
4) Conducting Conversion:
Conversion specifies the desired target speaker representation, encodes the input frame into a latent representation, and decodes the pair without sampling.
- The desired target is selected by specifying its speaker representation yn.
- The encoder transforms the input frame into a latent representation, which the decoder combines with yn to reconstruct the converted frame.
- Sampling is not needed during conversion.
C. Training Procedures
Training treats source and target frames as one unified set, pairing each spectral frame with its speaker identity. The VAE encodes speaker-independent phonetic information and reconstructs each input using the designated speaker representation.
- C. Training Procedures: Training uses point-wise spectral-frame and speaker-identity pairs rather than conventional pair-wise source-target examples.The source and target sets are unified, with each frame labeled by its speaker identity.
- C. Training Procedures: The auto-encoder reconstructs each input spectral frame from its frame-speaker pair.Each training example is (x_n, y_n), where the speaker identity is known for the frame.
- C. Training Procedures: The encoder maps incoming frames to a speaker-independent latent phonetic representation.Frames from both source and target speakers train this speaker-independent encoding.
- C. Training Procedures: The decoder samples the latent code and reconstructs the input with the speaker representation.This combines the latent variable z_n with y_n during reconstruction.
- C. Training Procedures: Visible- and latent-variable costs are jointly optimized while network parameters are iteratively updated.Training terminates when the generation probability reaches its maximum.
A. Experimental Settings
The experiments evaluate spectral conversion on a Voice Conversion Challenge 2016 English corpus using selected source and target speakers. Spectra are converted with the proposed method or aligned-data baselines, while F0 and aperiodicity follow common processing procedures.
- A. Experimental Settings: The corpus contains 5 male and 5 female speakers, with 150 training and 12 evaluation utterances per speaker.The evaluation set was aligned, and five speakers served as targets while five served as sources.
- A. Experimental Settings: Experiments use sources SF1 and SM1, targets TF2 and TM3, and intra-gender and cross-gender conversion conditions.One VAE variant was trained on disjoint non-parallel subsets.
- A. Experimental Settings: STRAIGHT extraction produced 513-dimensional aperiodicity and spectral parameters using a 1024-point FFT, 5 ms frame shift, and 25 ms frame length.The study omitted contextual and dynamic features.
- A. Experimental Settings: The proposed method applies logarithms to normalized spectra, whereas baselines use linear nonnegative spectra.All systems transform log-F0 identically, keep aperiodicity unchanged, and resynthesize with STRAIGHT after energy compensation.
- A. Experimental Settings: Baseline ENMF-512 and ENMF-3000 use 512 or 3000 randomly selected parallel source-target frame pairs.Parallel training data are aligned with DTW using 24-ordered Mel-cepstral coefficients.
- A. Experimental Settings: ENMF baselines require no training and perform online conversion through self-reconstruction activations applied to a target dictionary.The activation matrix provides weights for combining dictionary bases.
1) Configurations and Hyper-parameters:
The study compares three VAE configurations and aligned ENMF baselines under objective and subjective evaluation. The non-parallel VAE condition uses mutually exclusive source and target utterances, while subjective tests focus on VAE-pair against ENMF-3000.
- 1) Configurations and Hyper-parameters:: The VAE uses two 512-node hidden layers, a 64-dimensional latent space, mini-batches of 128, and ADAM optimization.ReLU supplies hidden-layer nonlinearity, while the output layers are linear.
- 1) Configurations and Hyper-parameters:: The visible log-spectrum follows a Gaussian model with identity variance, and latent variables use an isotropic standard normal prior.Variance modeling was ignored because it did not affect the system’s generative process.
- 1) Configurations and Hyper-parameters:: VAE-pair uses one source and one target, VAE-multi uses four speakers, and VAE-disj uses non-parallel data.VAE-disj trains on the first 75 source utterances and the other 75 target utterances.
- 1) Configurations and Hyper-parameters:: VAE-pair and VAE-multi use parallel but unaligned data, whereas VAE-disj uses non-parallel data with roughly half the training-set size.The source and target sentence subsets for VAE-disj are mutually exclusive.
- 1) Configurations and Hyper-parameters:: Unaligned-data proposed methods perform on par with aligned-frame baselines on mean Mel-cepstral distortion.The authors note that MCD is not a representative perceptual indicator and therefore add subjective evaluation.
- 1) Configurations and Hyper-parameters:: Subjective evaluation compares VAE-pair with ENMF-3000 using ten listeners, MOS ratings, and ABX tests across inter- and intra-gender conversion.ENMF-3000 was selected because it produced higher-quality synthetic speech than ENMF-512.
F. Training from Non-parallel Corpora
The non-parallel VAE-disj condition matches VAE-pair objectively despite having less training data, supporting conversion from non-parallel corpora while exposing a training-size concern. VAE-multi similarly approaches VAE-pair and suggests a path toward many-to-many conversion.
- F. Training from Non-parallel Corpora: VAE-disj performs around the same as VAE-pair in objective evaluations despite its harsher non-parallel training condition.The authors caution that VAE-disj had roughly half as much training data, so model capability may not have been fully exploited.
- F. Training from Non-parallel Corpora: VAE-multi performs close to VAE-pair while consolidating 12 speaker-conversion permutations into one system.Its broader conversion ability is described as evocative of many-to-many voice conversion.
- G. Toward Many-to-Many Voice Conversion: A many-to-many extension could replace the one-hot speaker representation with a speaker-recognition encoder.The proposed framework is described as conceptually able to accommodate many-to-many tasks.
- G. Toward Many-to-Many Voice Conversion: With a speaker representation inferred from limited target speech, the decoder could blend speaker and phonetic representations to synthesize target-dependent spectra.The authors present this as a likely route to many-to-many conversion.
- IV. CONCLUSIONS: The VAE-based framework utilizes unaligned data and achieves performance comparable to baselines with aligned data in objective and subjective evaluations.Future work includes performance improvement, many-to-many conversion, and generalization to more tasks.