MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion

• MeanVoiceFlow’s content encoder remains a primary computational bottleneck, taking approximately ten times as long as a single flow step.Source: §1 · p. 1
• MeanVoiceFlow2 jointly optimizes a flow-based conversion module and a trainable content encoder through MeanVoiceFlow conversion distillation and real-data reconstruction.Diffusion-GAN training with sample mixing and teacher-guided conditioning augmentation are also used to improve realism and content–speaker disentanglement.Source: §1 · p. 1
• Unlike FasterVoiceGrad, MeanVoiceFlow2 uses a Mean Flow formulation and requires no additional pretrained modules beyond a teacher trained on the same data.Source: §1 · p. 1
• MeanVoiceFlow2 achieved higher perceptual quality and significantly faster inference than MeanVoiceFlow while maintaining comparable speaker similarity in zero-shot VC.Source: §1 · p. 1

• MeanVoiceFlow is a one-step zero-shot voice-conversion model based on Mean Flow, conditioning on speaker and content embeddings extracted from target and source speech, respectively.Source: §2 · p. 1
• Its linear flow path uses a diffused source sample, with α sampled from a logit-normal distribution during training and fixed during inference.Using the diffused source sample instead of pure noise gives a unified formulation for unconditional generation and source-conditioned conversion.Source: §2 · p. 1
• The model represents average velocity over [r, t] with a neural network, corresponding to average displacement across that interval.Source: §2 · p. 2
• One-step conversion uses the average-velocity model, with the target speaker embedding obtained by randomly shuffling source-speaker embeddings within a training mini-batch.Source: §2 · p. 2

3. Proposal: MeanVoiceFlow2

• MeanVoiceFlow2 jointly trains an efficient, trainable content encoder and student average-velocity network under guidance from the fixed teacher.This design targets the content-encoder bottleneck in MeanVoiceFlow.Source: §3 · p. 2

3.1. Joint conversion distillation and reconstruction

• Conversion distillation aligns student conversion behavior with the teacher, while real-data reconstruction adds consistency with real speech without paired source–target data.The conversion loss uses an adaptively weighted distance; reconstruction supervises the student using a target derived from real source data.Source: §3.1 · p. 2

3.2. Diffusion-GAN training with sample mixing

• Diffusion-GAN training addresses averaging from pairwise distance losses, while sample mixing smooths the discriminator boundary and emphasizes subtle teacher–student differences.Teacher-generated samples are treated as real examples; the adversarial procedure is also applied to reconstructed samples.Source: §3.2 · pp. 2–3

3.3. Teacher-guided conditioning augmentation

• Teacher-guided conditioning augmentation adds a forward path using teacher-generated speech under augmented speaker conditions to encourage speaker-invariant content representations.The augmented speaker embedding is produced by randomly shuffling source-speaker embeddings within a minibatch.Source: §3.3 · p. 3

4.1. Experimental setup

• The evaluation used zero-shot unseen-to-unseen conversion, with ten speakers and ten sentences held out, and assessed objective metrics on 8,100 speaker–sentence pairs.Experiments used VCTK and LibriTTS; additional subjective evaluations and speed assessments were conducted for the key comparisons.Source: §4.1 · p. 3

4.2. Component analysis

• Removing reconstruction degraded UT, DNSP, and CER, while removing conversion degraded speaker similarity and perceptual quality despite low CER.The full model achieved the best overall performance across most metrics; the ablation was conducted without conditioning augmentation.Source: §4.2 · p. 3, Table 1 · p. 4
• Adversarial training improved DNSP, DNS, CER, and SECS while maintaining UT; diffusion improved UT and CER, and sample mixing improved DNSP and DNS.The full configuration performed best overall and was competitive or superior to waveform- and vocoder-projected-feature-discriminator alternatives without a pretrained neural vocoder.Source: §4.2 · p. 4, Table 2 · p. 4
• Conditioning augmentation improved all objective metrics over its ablation, whereas direct feature distillation degraded DNSP and CER and appeared to overconstrain the student encoder.The authors characterize conditioning augmentation as more effective implicit regularization than explicit feature-level distillation.Source: §4.2 · p. 4, Table 3 · p. 4

4.3. Comparison with previous models

• Approximately 9× lower RTF than MeanVoiceFlow accompanied higher perceptual quality and comparable intelligibility and speaker similarity; versus FasterVoiceGrad, performance was better or comparable across metrics.The FasterVoiceGrad comparison showed significant improvements in nMOS and DNSP, without relying on a pretrained neural vocoder for training.Source: §4.3 · p. 4, Table 4 · p. 4

4.4. Evaluation on LibriTTS

• On LibriTTS, MeanVoiceFlow2 performed better or comparably across metrics, with significant UT and DNSP improvements and approximately 9× lower RTF than MeanVoiceFlow.The authors report this evaluation as a robustness check across datasets.Source: §4.4 · p. 4, Table 5 · p. 4

5. Conclusion

• MeanVoiceFlow2 combines conversion distillation, real-data reconstruction, diffusion-GAN training with sample mixing, and conditioning augmentation, and achieves approximately 9× faster inference than MeanVoiceFlow.It reports higher perceptual quality while maintaining comparable speaker similarity; extending the framework to accent and real-time VC is future work.Source: §5 · p. 4

6. Generative AI Use Disclosure

• The authors used generative AI tools to edit and polish the manuscript language and verified the final manuscript.Source: §6 · p. 5