PaperlayerView exact revision on arXiv

Source-linked AI summary

Voice Conversion With Just Nearest Neighbors

Matthew Baas, Benjamin van Niekerk, Herman Kamper

arXiv:2305.18975v1eess.AScs.CLcs.SD

TL;DR

Any-to-any voice conversion seeks to change a source utterance to a target voice using only a few target examples, while recent methods can be complex and costly. kNN-VC matches source and reference self-supervised features and vocodes the converted sequence; evaluations report similar intelligibility and naturalness with improved speaker similarity over recent models.

  • Problem

    Any-to-any voice conversion aims to change a source speaker to a target voice using a few target examples, but recent methods have increased complexity and cost.

    Source: §1 · p. 1

  • Method

    kNN-VC extracts self-supervised features from source and reference speech, replaces each source frame with a nearest reference feature, and vocodes the converted sequence.

    Source: §1 · p. 1, §3.2 · p. 2

  • Results

    kNN-VC achieves similar intelligibility and naturalness scores while improving speaker similarity against recent high-performing voice-conversion models.

    Source: §6 · p. 4

  • Takeaways & Limitations

    The authors present kNN-VC as a robust, easy-to-implement baseline and provide qualitative examples for cross-lingual, whispered, and human-to-animal conversion.

    Source: §6 · p. 4

Abstract

from arXiv · show

Any-to-any voice conversion aims to transform source speech into a target voice with just a few examples of the target speaker as a reference. Recent methods produce convincing conversions, but at the cost of increased complexity -- making results difficult to reproduce and build on. Instead, we keep it simple. We propose k-nearest neighbors voice conversion (kNN-VC): a straightforward yet effective method for any-to-any conversion. First, we extract self-supervised representations of the source and reference speech. To convert to the target speaker, we replace each frame of the source representation with its nearest neighbor in the reference. Finally, a pretrained vocoder synthesizes audio from the converted representation. Objective and subjective evaluations show that kNN-VC improves speaker similarity with similar intelligibility scores to existing methods. Code, samples, trained models: https://bshall.github.io/knn-vc

1. Introduction

Any-to-any voice conversion methods can be costly and complex, raising the question of whether that complexity is needed. The paper introduces kNN-VC and reports competitive intelligibility and speaker similarity against recent systems.

1. Introduction

  • Recent any-to-any methods improve naturalness and speaker similarity, but increased complexity makes training and inference costly and improvements harder to evaluate and build on.The paper asks whether complexity is necessary for high-quality voice conversion.Source: §1 · p. 1
  • kNN-VC converts any-to-any speech by applying k-nearest neighbors to self-supervised source and reference features, without training an explicit conversion model.Each source frame is replaced with a nearest neighbor from the reference; a neural vocoder then produces speech.Source: §1 · p. 1
  • kNN-VC matches or improves intelligibility and speaker similarity in subjective and objective evaluations against several state-of-the-art systems.The paper also examines how target-data size and vocoder training decisions affect conversion quality.Source: §1 · p. 1
  • The authors argue that complexity is not necessary for any-to-any voice conversion because nearest neighbors on self-supervised features can produce convincing results.This is the paper’s main contribution.Source: §1 · p. 1

2. Related work

Related work describes challenges in separating speaker identity from content and trade-offs in text-based approaches. It also connects kNN-VC to concatenative unit selection, using learned speech features rather than human-defined units.

2. Related work

  • Disentangling speaker identity from content is challenging, and recent systems use complex bottlenecks, normalization, or augmentation while still struggling with speaker similarity.Text annotations can provide an intermediate bottleneck, but transcription is costly and text can remove useful prosody; this paper therefore focuses on text-free methods.Source: §2 · p. 1
  • Concatenative conversion selects target-speech units matching source content, but unit selection generally requires parallel data or time-aligned transcriptions.Using target speech units supports speaker similarity, while the data requirements constrain the approach.Source: §2 · p. 1
  • kNN-VC can be viewed as unit selection using self-supervised speech features instead of human-defined units, which indicate shared phonetic content when similar.This motivates asking whether improved speech representations make complex conversion methods necessary.Source: §2 · p. 1

3. kNN-VC

kNN-VC uses an encoder-converter-vocoder pipeline: self-supervised features represent source and target speech, nearest-neighbor matching supplies converted features, and a vocoder generates audio. Prematched training adapts the vocoder to inference-like inputs.

3. kNN-VC

  • The method follows an encoder-converter-vocoder structure that extracts speech representations, maps source frames to reference features, and generates an audio waveform.The Figure 1 implementation uses pretrained WavLM features and HiFi-GAN.Source: §3 · p. 1, Figure 1 · p. 2

3.1. Encoder

  • The encoder forms a query sequence from the source and a matching set by pooling target-speaker feature vectors; it is not fine-tuned for kNN-VC.Self-supervised representations are selected for their ability to place instances of the same phone closer together than different phones.Source: §3.1 · p. 2

3.2. k-nearest neighbors matching

  • For each query frame, kNN regression replaces it with the average of its k nearest matching-set vectors, preserving source content while drawing converted features from target speech.The method is non-parametric and requires no training.Source: §3.2 · p. 2

3.3. Vocoder

  • At inference, the vocoder receives kNN outputs selected across times and phonetic contexts, whose adjacent-frame inconsistencies cause artifacts that reduce intelligibility.The vocoder converts the resulting features into an audio waveform.Source: §3.3 · p. 2

3.4. Prematched vocoder training

  • Prematched training reconstructs each training utterance’s features from other same-speaker utterances, then trains the vocoder to predict the original waveform from those features.This trains on data resembling inference inputs and improves intelligibility and speaker similarity.Source: §3.4 · p. 2

4. Experimental setup

The experiments compare kNN-VC with recent systems and examine target-data size and vocoder training on unseen speakers in LibriSpeech. The setup specifies the implementation, baselines, and subjective and objective evaluation procedures.

4. Experimental setup

  • Experiments use LibriSpeech development and test-clean sets with 40 unseen speakers each, about eight minutes of 16 kHz English speech per speaker.The study compares systems and tests target-data size and prematched vocoder training.Source: §4 · p. 2

4.1. kNN-VC implementation

  • The implementation uses WavLM-Large layer 6 features, k=4 with uniform weighting and cosine distance, and HiFi-GAN trained on pure or prematched features.Preliminary experiments found kNN-VC robust around k=4; with at least 10 minutes of reference audio, k around 20 may improve quality. With eight minutes of reference audio, inference is faster than real-time on a consumer 8GB VRAM GPU.Source: §4.1 · pp. 2–3

4.2. Baselines

  • The baselines are VQMIVC, FreeVC, and YourTTS, evaluated using publicly available pretrained models conditioned on averaged target-speaker embeddings.YourTTS is used in text-free voice-conversion mode.Source: §4.2 · p. 3

4.3. Evaluation metrics

  • Objective intelligibility is measured with Whisper-base word and character error rates on 7,800 conversions from 200 LibriSpeech test-clean utterances to 39 other speakers.Lower error rates indicate better intelligibility.Source: §4.3 · p. 3
  • Objective speaker similarity uses equal error rate from speaker-verification x-vector cosine scores, with higher EER indicating greater similarity to genuine target-speaker speech.The evaluation set consists of unseen LibriSpeech test-clean speakers.Source: §4.3 · p. 3
  • Subjective tests use mean opinion scores for naturalness and 1-to-4 similarity ratings, with ground-truth examples and filtered listener ratings.The reported totals are 2,144 naturalness ratings from 61 listeners and 1,636 similarity ratings from 208 listeners.Source: §4.3 · p. 3

5. Results

In the main comparison, kNN-VC matched the best baseline, FreeVC, on naturalness and intelligibility while improving speaker similarity. Ablations found benefits from prematched vocoder training, declining quality with less reference speech, and little additional gain beyond about five minutes.

5. Results

  • The comparative experiment used about eight minutes of target speech per speaker, while the ablation varied target-data size and prematched vocoder training.kNN-VC used all target data as its matching set; baselines averaged speaker embeddings from target utterances.Source: §5 · p. 3

5.1. Voice conversion

  • kNN-VC achieved similar naturalness (MOS) and intelligibility (W/CER) to the best baseline, FreeVC, while significantly improving speaker similarity (EER and SIM).Table 1 reports these results on LibriSpeech test-clean; subjective MOS and SIM values include 95% confidence intervals.Source: §5.1 · pp. 3–4, Table 1 · p. 3

5.2. Ablation: Prematching and amount of reference data

  • Prematched vocoder training improved speaker similarity (higher EER) and intelligibility (lower WER) across target-data amounts.Prematching maps each training utterance’s WavLM features to nearest vectors from other utterances by the same speaker.Source: §5.2 · p. 4
  • As target-speaker reference speech decreases, intelligibility and speaker similarity deteriorate, with very similar performance from five minutes and the roughly eight-minute maximum.The authors attribute the leveling off to sufficient matching-set coverage of source phones and biphones.Source: §5.2 · p. 4
  • Without prematched vocoder training, kNN-VC remained competitive in intelligibility and superior in speaker similarity, but complex baselines performed better with less than 30 seconds of target data.The former comparison uses Figure 2 against Table 1 scores.Source: §5.2 · p. 4

6. Conclusions

The conclusions present kNN-VC as evidence that simpler methods using self-supervised speech representations can achieve similar intelligibility and naturalness while improving speaker similarity. Prematched vocoder training improved performance, and the authors identify an easy-to-implement baseline and broader conversion examples as future-facing uses.

6. Conclusions

  • Compared with recent high-performing models in an any-to-any setting, kNN-VC achieved similar intelligibility and naturalness while improving speaker similarity.The authors also report moderate intelligibility and speaker similarity with as little as five seconds of target-speaker audio.Source: §6 · p. 4
  • Prematched feature training improved performance, and the authors propose open-source vanilla kNN-VC as a robust, easy-to-implement voice-conversion baseline.They include qualitative cross-lingual, whispered, and human-to-animal examples as areas for further exploration.Source: §6 · p. 4
Loading 2305.18975v1…