Source-linked AI summary
Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, Jiansheng Wei
TL;DR
One-shot many-to-many voice conversion must copy an unseen target voice from one reference utterance while preserving linguistic content. The paper combines an average-voice encoder with a conditioned diffusion decoder and develops a likelihood-maximizing reverse-SDE solver. The resulting system supports high-quality conversion with as few as 6 reverse-diffusion iterations.
Problem
One-shot many-to-many voice conversion requires adapting to unseen source and target speakers from only one target reference utterance while preserving linguistic content.
Method
The paper uses an average-voice encoder, a target-conditioned diffusion decoder, and a fixed-step reverse-SDE solver that maximizes forward-diffusion sample-path likelihood.
Results
The proposed model achieves good speaker similarity and speech naturalness on out-of-domain unseen speakers, with high-quality conversion using as few as 6 reverse-diffusion iterations.
Takeaways & Limitations
The sampling scheme enables scalable voice conversion with substantially faster synthesis while retaining high-quality results in the reported evaluations.
Takeaways & Limitations
The current experiments set Var(X0|Xt) to zero, leaving estimation of this conditional variance for future research.
Abstract
from arXiv · showhide
Voice conversion is a common speech synthesis task which can be solved in different ways depending on a particular real-world scenario. The most challenging one often referred to as one-shot many-to-many voice conversion consists in copying the target voice from only one reference utterance in the most general case when both source and target speakers do not belong to the training dataset. We present a scalable high-quality solution based on diffusion probabilistic modeling and demonstrate its superior quality compared to state-of-the-art one-shot voice conversion approaches. Moreover, focusing on real-time applications, we investigate general principles which can make diffusion models faster while keeping synthesis quality at a high level. As a result, we develop a novel Stochastic Differential Equations solver suitable for various diffusion model types and generative tasks as shown through empirical studies and justify it by theoretical analysis.
1 INTRODUCTION
The paper targets one-shot many-to-many voice conversion, which must copy an unseen target voice from one reference utterance while preserving source linguistic content. It proposes a diffusion-based model and a faster inference scheme to address quality and speed challenges.
- Motivation: One-shot many-to-many VC copies an unseen target speaker’s voice from one reference utterance while preserving the source utterance’s linguistic content.The task must generalize to source and target speakers absent from training data.
- Prior approaches: Conventional one-shot VC models use autoencoder latent spaces for linguistic content and decoder conditioning for target voice identity.Prior systems condition on speaker embeddings and may add pitch, loudness, or phonetic features.
- Proposed model: The proposed encoder predicts an “average voice” representation by averaging phoneme-level mel features across a large multi-speaker dataset.This design addresses disentanglement between linguistic content and speaker identity.
- Proposed model: The diffusion decoder offers strong speech-generation quality but ordinarily requires hundreds of iterative inference steps.The paper therefore develops a sampling scheme that reduces iterations without model retraining.
- Paper scope: The paper evaluates voice conversion quality and the proposed sampling scheme after presenting the model and its theoretical connection to likelihood maximization.The paper is organized around model design, sampling theory, experiments, and conclusions.
2 VOICE CONVERSION DIFFUSION MODEL
The VC system encodes speech as an average-voice mel representation and uses a conditioned diffusion decoder to generate target-speaker mel-spectrograms. Its training and inference follow forward and reverse SDEs, with speaker information supplied through the target diffusion trajectory.
- Encoder: The encoder converts input mel-spectrograms into average phoneme-level mel features aggregated across LibriTTS.Training uses phoneme alignment, dataset-wide phoneme aggregation, and mean squared error.
- Diffusion model: The encoder parameterizes the forward-diffusion terminal distribution, while the decoder parameterizes the reverse diffusion.The model uses Itô-calculus stochastic processes rather than discrete-time Markov chains.
- Diffusion model: The diffusion framework defines forward and reverse processes with SDEs over t ∈[0, 1], using a noise schedule, score function, Wiener processes, and average-voice vector.The forward process admits an explicit solution, supporting efficient noisy-sample generation during training.
- Diffusion model: A linear noise schedule makes the forward process’s terminal law close to N(¯X, I), which serves as the diffusion prior.The condition is that the integrated noise schedule is close to zero in the stated exponential expression.
- Inference: During inference, the decoder samples from the prior and solves the reverse SDE backward, conditioning on the average-voice representation and target-speaker information.Target-speaker information is computed from the target mel-spectrogram or its forward-diffusion trajectory.
- Inference: Speaker conditioning can use only a speaker embedding, a noisy target mel-spectrogram, or the whole target forward-diffusion trajectory.These variants are named d-only, wodyn, and whole, respectively.
- Decoder: The decoder uses a wider U-Net architecture, and its speaker-conditioning network produces a 128-dimensional vector concatenated as additional channels.The architecture has four times more channels than the referenced Grad-TTS decoder.
- Related work: Compared with earlier diffusion VC systems, the proposed model addresses one-shot many-to-many conversion and targets substantially fewer reverse-diffusion steps.The cited prior systems use no less than 100 reverse diffusion steps, whereas this work reports good quality with as few as 6.
3 MAXIMUM LIKELIHOOD SDE SOLVER
The paper develops a fixed-step reverse-SDE solver that maximizes the likelihood of discretized forward-diffusion paths. Its optimality has exact special cases, extends across diffusion types, and matters most during low-step or final-step inference.
- Maximum-likelihood solver: The proposed fixed-step first-order reverse-SDE solver maximizes the log-likelihood of forward-diffusion sample paths.It differs from Euler–Maruyama by corrections that become significant with few sampling iterations.
- Scope and assumptions: The analysis is derived for VP diffusion, extends to MR-VP, sub-VP, and VE models, and can become applicable under strong conditioning that makes the conditional data distribution nearly deterministic.The VC diffusion model is transformed into MR-VP by a constant shift.
- Theorem 1: Theorem 1 characterizes optimal solver parameters within fixed-step updates combining the current state, score function, and isotropic Gaussian noise.The solver class includes Euler–Maruyama as a special case.
- Theorem 1: Under constant or diagonal-isotropic Gaussian data distributions, the generative model is exact for the theorem’s optimal parameters.The theorem assumes the solver starts from a random variable distributed as Law(X1).
- Practical implications: The improved sampling scheme requires no additional computational cost apart from a data-dependent variance term, no retraining, and no extensive noise-schedule search.The optimality claim concerns likelihood for discrete paths, while continuous-path optimality is separately guaranteed for the optimal model.
- Inference behavior: The optimal solver significantly differs from Euler–Maruyama mainly when the iteration count is small or during final inference steps where t has the same order as h.For fixed t and h approaching zero, its leading noise scale approaches the Euler–Maruyama scale.
4 EXPERIMENTS
Experiments evaluate the proposed diffusion voice-conversion model on VCTK and LibriTTS, including unseen speakers, and test maximum-likelihood sampling across speech and image-generation settings. Results indicate strong subjective quality and useful acceleration with few reverse-diffusion iterations, while the sampling method often improves FID.
- Experimental setup: Models were trained on VCTK and LibriTTS, with held-out speakers used to assess generalization.VCTK contained 109 speakers with 9 held out; LibriTTS contained approximately 1100 speakers with 10 held out.
- Experimental setup: The evaluation compared Diff-VCTK and Diff-LibriTTS with four one-shot many-to-many VC baselines using subjective speech naturalness and speaker-similarity tests.The baselines were AGAIN-VC, FragmentVC, VQMIVC, and BNE-PPG-VC.
- Voice-conversion results: Diff-VCTK significantly outperformed the VCTK-trained baselines in naturalness and speaker similarity, including when using 6 reverse diffusion iterations.For VCTK speakers, speaker-similarity MOS was 3.6 with ML-30 and 3.5 with five times fewer iterations.
- Voice-conversion results: Diff-LibriTTS-ML-30 achieved MOS 4.0 for speech naturalness and 3.4 for speaker similarity on the whole test set, outperforming BNE-PPG-VC.The whole test set included unseen LibriTTS and internal speakers; six-step methods reached RTF around 0.1 on GPU, versus around 0.5 for ML-30.
- Sampling efficiency: ML-N enabled as few as N = 6 iterations with acceptable MOS degradation of 0.2 in naturalness and 0.1 in speaker similarity, unlike competing methods.The competing sampling methods produced much larger quality degradation at the same low iteration count.
- Sampling efficiency: On CIFAR-10, maximum-likelihood sampling was compared with Euler-Maruyama and probability-flow schemes, and it performed best in terms of FID in most cases.The method was also evaluated across diffusion-model types and generative tasks; tuning τ could further improve performance, although likelihood and FID do not perfectly correlate.
5 CONCLUSION
The paper presents a one-shot many-to-many voice conversion model combining an average-voice encoder with a diffusion-based decoder, and develops a likelihood-oriented sampling scheme for fast synthesis.
- The model achieves good speaker similarity and speech naturalness on out-of-domain unseen speakers.Subjective human evaluation supports its competitive performance as a scalable voice-conversion solution.
- The proposed sampling scheme modifies the Euler–Maruyama SDE solver to maximize the likelihood of discrete forward-diffusion sample paths.
- The voice-conversion model produces high-quality conversions with as few as 6 reverse diffusion steps.
- Experiments on image generation show that the proposed SDE solver benefits all known diffusion model types.
A FORWARD VP SDE SOLUTION
The forward VP SDE solution uses stochastic integration properties to analyze the diffusion process. Its deterministic integrand yields a Gaussian martingale whose variance can be calculated through Itô’s isometry.
- Taking the differential of the linear forward process avoids a second-order derivative term in Itô’s formula.
- Integrating the differential expression from s to t produces an Itô integral.
- The deterministic L2[0, 1] integrand makes the Itô integral normally distributed and a zero-mean martingale.Itô’s isometry provides its variance.
- The derivation first obtains asymptotics for γ.
C PROOF OF THE THEOREM 1
The theorem proof derives the reverse transition density and optimizes its parameters by maximizing the likelihood of forward-diffusion sample paths. Exactness is then established under specific initial-data distributions.
- The proof derives the score-function representation needed for the theorem under mild assumptions on the data density.
- The forward conditional law is Gaussian, but the unconditional reverse-time transition density is not generally normal.
- The solver parameters are selected by maximizing the log-likelihood of discrete forward-diffusion sample paths, with each timestep optimized separately.
- The likelihood objective reduces to minimizing an expected quadratic expression involving conditional expectations and the variance of X0 given the later state.
- Under constant or diagonal isotropic Gaussian initial data, the resulting generative model is exact when initialized from the forward terminal law.
D REVERSE MR-VP SDE SOLVER
The reverse MR-VP and sub-VP solvers extend the likelihood-optimization argument across diffusion variants. For MR-VP, the optimized parameters coincide with those for VP because the processes differ only by a constant shift.
- D REVERSE MR-VP SDE SOLVER: MR-VP DPM is defined through forward and reverse diffusions, with Gaussian conditional distributions derived for the intermediate states.
- D REVERSE MR-VP SDE SOLVER: The considered reverse SDE solvers use independent standard-normal noise at timesteps t = 1, 1 − h, …, h.
- D REVERSE MR-VP SDE SOLVER: The likelihood-optimal MR-VP solver parameters are the same as the optimal VP-DPM parameters.The paper attributes this equality to the constant shift separating MR-VP DPM and VP-DPM.
- D REVERSE MR-VP SDE SOLVER: The same derivation is applied to sub-VP and VE diffusion probabilistic models to obtain corresponding likelihood-optimal reverse SDE solvers.
G TOY EXAMPLES
Toy examples compare Maximum Likelihood and Euler-Maruyama SDE solvers across step counts and score-matching accuracy. The proposed solver converges accurately in fewer steps and can exactly reconstruct constant data distributions under optimal scores.
- Experimental setup: The experiments compare Maximum Likelihood and Euler-Maruyama solvers using toy distributions with one or two points across N ∈{1, 2, 5, 10, 100, 1000} steps.Table 5 reports Mean Square Error, with MSE < 0.001 marked conv and MSE > 1.0 marked div.
- General trends: Both methods improve with larger N and more accurate score-matching networks, while converging to similar performance at large step counts.The score-matching accuracy is represented by smaller ε.
- Solver comparison: The proposed Maximum Likelihood solver reaches the data distribution with good accuracy in fewer steps than Euler-Maruyama.This advantage is most relevant when diffusion sampling uses a small number of iterations.
- Solver comparison: With ε = 0.0, the optimal Maximum Likelihood solver exactly reconstructs a constant data distribution regardless of N.This result follows the stated Theorem 1 case for constant data distributions and optimal score matching.
- Two-point distribution: At ε = 0.0 and N = 10, Euler-Maruyama generates 54% i-samples, whereas Maximum Likelihood generates 50%, matching the two-point probabilities better.The frequencies were calculated from 500k independent runs.
H SPEAKER CONDITIONING NETWORK
The speaker conditioning network processes target mel-spectrogram information, time embeddings, and convolutional features to produce a 128-dimensional conditioning vector. Its convolutional configuration varies with the input type.
- Architecture: The network uses a tanh(softplus(x)) non-linearity and processes time embeddings through six convolutional blocks.The convolution channel configurations differ between wodyn and whole input types.
- Architecture: After the sixth block, a final convolution and average pooling over time and frequency produce a 128-dimensional vector c.All preceding convolutions use kernel 3, stride 1, and zero padding 1; the final convolution uses kernel 1, stride 0, and zero padding 0.
- Speaker conditioning: For target-speaker conditioning, the network concatenates speaker embedding d, convolutional features c, and time embedding t′ before two linear modules produce a 128-dimensional output.For d-only input, only d and t′ are concatenated.
I TRAINING HYPERPARAMETERS AND OTHER DETAILS
The models use separate encoder and decoder training settings, 22.05kHz operation, mel-spectrogram inputs, and implementation-specific preprocessing. Decoder training uses short random speech segments to fit GPU memory.
- Data preparation: The datasets were downsampled to 22.05kHz, and decoders were trained on random speech segments of approximately 1.5 seconds.VCTK recordings were silence-trimmed, and reconstruction and conditioning segments were separately sampled from the same training utterance.
- Spectrogram configuration: The voice-conversion models use 80-mel-feature spectrograms at 22.05kHz, computed with 1024 frequency bins, a Hann window of length 1024, and hop size 256.These settings define the spectrogram representation used by the models.
- Post-processing: Diff-LibriTTS models apply mel-domain spectral subtraction with β = 0.02 to reduce background noise produced by some models.The noise spectrum is estimated from automatically detected silence fragments in the source mel-spectrogram.
J DETAILS OF AMT TESTS
The AMT evaluations standardize recordings and ask workers to rate either speech naturalness or speaker similarity on separate five-point scales. The two tests target different perceptual properties.
- Naturalness tests: Naturalness tests use loudness-normalized recordings downsampled to 16kHz and rate cleanliness and human-likeness on a five-point Likert scale.Workers assessed overall synthesized-speech quality while wearing headphones in a quiet environment.
- Speaker similarity tests: Speaker-similarity tests ask assessors to compare synthesized samples with target speech on a separate five-point scale.Assessors were instructed to ignore overall quality factors such as background noise or incorrect pronunciation.