Source-linked AI summary
One-Step Voice Conversion by Learning kNN Transport in WavLM Space
Anton Selitskiy, David Millard
TL;DR
Voice conversion needs a reliable way to map source to target speech representations, particularly when target-speaker data are limited. This paper learns kNN-derived transport in WavLM space with a conditional flow-matching network, and reports improved intelligibility and estimated quality over discrete matching baselines, including in one-step conversion.
Problem
Learning reliable transport between source and target embedding distributions remains challenging, particularly with limited target-speaker data.
Source: §1 · p. 1Method
A conditional flow-matching network learns kNN-derived transport directly in WavLM space, with target-speaker conditioning and three Gaussian probability paths.
Source: §1.3 · p. 2Results
The flow-matching variants achieve substantially lower WER and higher estimated speech quality than discrete kNN, kDOT, and MKL baselines.
Source: §3.4 · p. 4Takeaways & Limitations
The discussion reports that speaker similarity and FAD capture complementary aspects of conversion quality and should be interpreted alongside intelligibility and perceptual quality.
Source: §4 · p. 4The source–kNN target pair is only an approximate coupling, so its variance schedule is not uniquely determined by the endpoints.
Source: §2.1.2 · p. 2
Abstract
from arXiv · showhide
Voice conversion (VC) systems fall into two families: non-parametric embedding-space methods, which need no trained model but degrade on short target utterances, and spectrogram-based neural architectures, which achieve strong quality via multi-module pipelines with tens of millions of parameters. We propose kNN-FM-VC, a single conditional flow-matching network that learns to approximate the kNN-VC mapping between WavLM embedding distributions of source and target speakers, replacing explicit pointwise kNN matching with a neural regressor trained on kNN-generated pairs. The model is conditioned on the target speaker via cross-attention and FiLM, and trained under three Gaussian conditional paths (Schr\"odinger bridge, straight line, and constant-variance Gaussian tube), enabling few-step sampling. Unlike Phoneme Hallucinator, which uses an upsampling stage followed by kNN matching, our 13M-parameter model performs conversion with a single learned network and supports one-step inference. On LibriSpeech, the one-step Gaussian Bridge achieves lower WER and higher estimated speech quality than FreeVC and Phoneme Hallucinator. Relative to kNN and kDOT, it substantially reduces WER.
1. INTRODUCTION
Voice conversion must learn reliable transport between source and target embedding distributions, especially with limited target-speaker data. The paper proposes conditional flow matching in WavLM space to learn kNN-derived transport without explicit inference-time matching.
1. INTRODUCTION
- A central challenge is learning a reliable transformation between source and target embedding distributions when target-speaker data are limited.Self-supervised speech representations provide a latent space that separates conversion from waveform generation and permits direct manipulation of high-level speech features.Source: §1 · p. 1
1.1. Spectrogram-based Approaches
- Spectrogram-based voice conversion systems use speaker or F0 conditioning, with examples including FreeVC and MeanVC.MeanVC is excluded because it is trained on Mandarin speech.Source: §1.1 · p. 1
1.2. Embedding-space, Non-parametric Approaches
- Embedding-space methods such as kNN-VC and kDOT-VC convert pretrained speech features, but degrade when short target utterances make the target embedding set sparse.MKL-VC improves speech clarity in this setting but weakens speaker preservation; Phoneme Hallucinator expands the target set before kNN matching.Source: §1.2 · pp. 1–2
1.3. Proposed Method
- The proposed conditional flow-matching network learns kNN-based transport directly in WavLM space, using cross-attention and FiLM for target-speaker conditioning.The roughly 13M-parameter model removes pointwise nearest-neighbor search and a separate upsampling stage, and supports single-step inference.Source: §1.3 · p. 2
2.1. Flow-Matching
The method learns a time-dependent vector field that transports source WavLM embeddings toward kNN-derived target embeddings. At inference, integrating the field produces a converted representation that is decoded to speech.
2.1. Flow-Matching
- A Lagrangian flow describes samples moving along trajectories under a dynamical system, with the evolving distribution satisfying a continuity equation.The trajectory takes a sample x at time 0 to sample y at time 1.Source: §2.1 · p. 2
2.1.1. Conditional Probability Paths
- Conditional flow matching trains a neural vector field against per-example conditional vector fields, with an objective whose gradients match those of the intractable marginal flow-matching objective.The joint distribution q(x0, x1) is the source-target distribution.Source: §2.1.1 · p. 2
2.1.2. Application to VC
- The network is conditioned on target-speaker information through frame-level cross-attention and global speaker-and-time FiLM conditioning.The training path connects the source embedding and kNN target with a Gaussian bridge; because their pair is only an approximate coupling, its variance schedule is not uniquely fixed by the endpoints.Source: §2.1.2 · p. 2
2.1.3. Inference
- At inference, integration starts from source embeddings and ends at a converted WavLM representation, which HiFi-GAN decodes to a waveform.The target utterance supplies the conditioning sequence, and the number of integration steps is selectable.Source: §2.1.3 · p. 2
2.2. Variants of FM objectives
The three FM objectives differ in their variance schedule and the vector field used for regression. They compare a Schrödinger bridge, a straight-line drift with bridge noise, and a constant-variance Gaussian tube.
2.2.1. Schr¨odinger Bridge
- The Schrödinger Bridge uses a Brownian-bridge variance that vanishes at both endpoints; its field approaches the straight-line field as σ approaches zero and approximates the bridge for σ > 0.The approximation is simulation-free and connects the empirical source and pseudo-target distributions.Source: §2.2.1 · p. 3
2.2.2. Gaussian Bridge with Straight Line
- The Gaussian Bridge with Straight Line retains Brownian-bridge noise but regresses onto the coupled endpoint displacement, omitting the exact Gaussian-path score correction.With approximately optimal-transport endpoint pairs, displacement fields can have reduced ambiguity and straighter marginal trajectories.Source: §2.2.2 · p. 3
2.2.3. Gaussian Tube
- The Gaussian Tube fixes variance at the Schrödinger-bridge midpoint value, leaving nonzero variance at both endpoints.The authors report a trade-off between path straightness and robustness to approximate transport pairs; they describe the construction as new to VC/FM.Source: §2.2.3 · p. 3
3. EXPERIMENTAL SETUP AND RESULTS
On LibriSpeech, the flow-matching variants outperform discrete matching baselines on WER and estimated speech quality, while comparisons with learned baselines and among paths reveal metric trade-offs. The experiments also examine training, evaluation, and integration-step settings.
3.1. Training and Evaluation Data
- The network is trained on LibriSpeech train-clean-100 and evaluated on its corresponding test set.Audio uses 1024-dimensional features from layer 6 of a frozen WavLM Large encoder.Source: §3.1 · p. 3, §3.2 · p. 3
3.2. Implementation Details
- Training pairs average the k = 4 cosine-nearest target frames for each source frame, with target-speaker conditioning from a separate 200-frame utterance crop.The network has 13.26M parameters, four-head cross-attention, and four FiLM residual blocks; training also uses velocity MSE and a speaker-consistency loss weighted by 0.1.Source: §3.2 · p. 4
3.3. Metrics
- Evaluation reports WER, UTMOSv2, FAD, and speaker similarity (SIM), with SIM based on a WavLM-Large/ECAPA verifier.Same-speaker pairs yield SIM scores of approximately 0.7.Source: §3.3 · p. 4
3.4. Comparison with Baselines
- The FM variants achieve substantially lower WER and higher estimated speech quality than kNN, kDOT, and MKL; versus FreeVC and Phoneme Hallucinator, they improve WER and quality but trade off speaker similarity.Against Phoneme Hallucinator, relative FAD separation is 14–24% versus 8.5%; kNN and kDOT have the highest SIM despite worse WER and UTMOSv2.Source: §3.4 · p. 4
3.5. Comparison of FM Variants
- At 50 integration steps, the three probability paths show no significant WER differences; Schrödinger Bridge has slightly higher SIM and the largest relative FAD separation, but no path consistently dominates.The paths trade off speaker similarity, FAD, and efficiency.Source: §3.5 · p. 4
3.6. Ablation: Sensitivity to Integration Steps
- Across 50, 5, and 1 integration steps, WER remains similar while SIM is highest at one step and relative FAD separation grows with more steps.One-step inference matches WER and UTMOSv2 at a fraction of the cost, while additional steps improve separation from the source domain.Source: §3.6 · p. 4
4. DISCUSSION
Learning transport in WavLM space primarily improves intelligibility and quality over discrete matching, while path choice and integration steps trade off SIM, FAD, and efficiency.
- Learning transport in WavLM space primarily improves intelligibility and quality over discrete matching, while path choice and NFE trade off SIM, FAD, and efficiency.At one step, FM variants retain more source-speaker character than some baselines, yet target-speaker SIM matches or exceeds FreeVC and MKL; all variants attain the highest relative FAD separation overall.Source: §4 · p. 4
5. CONCLUSION
The paper introduces conditional flow matching to learn kNN-derived transport in WavLM space, with target-speaker conditioning and three Gaussian probability paths. The method matches or exceeds substantially larger baselines in intelligibility and quality using a smaller single-step model.
- The proposed conditional flow-matching framework learns kNN-derived transport in WavLM space with target-speaker conditioning under three Gaussian probability paths.It matches or exceeds substantially larger baselines in intelligibility and quality while using a smaller, single-step model.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study uses publicly available LibriSpeech data and reports no new data collection or human-participant interaction or intervention.
- The study used the publicly available LibriSpeech corpus under its CC BY 4.0 license.Source: §6 · p. 5
- No new data were collected, and the study involved no interaction or intervention with human participants.Source: §6 · p. 5
- The authors report that no additional ethical approval was required.Source: §6 · p. 5