Source-linked AI summary
Neural Voice Cloning with a Few Samples
Sercan O. Arik, Jitong Chen, Kainan Peng, Wei Ping, Yanqi Zhou
TL;DR
Voice cloning needs to reproduce an unseen speaker from only a few audio samples. The paper compares fine-tuning a multi-speaker model with directly inferring a speaker embedding, finding good naturalness and similarity for both while speaker encoding uses fewer resources.
Problem
Voice cloning for personalized speech interfaces requires generating speech for unseen speakers from a few audio samples.
Method
The paper studies speaker adaptation by fine-tuning a multi-speaker generative model and speaker encoding by inferring a new speaker embedding from cloning audios.
Results
Both approaches achieve good cloning quality with few audios, while speaker adaptation can achieve better naturalness and similarity and speaker encoding requires significantly less cloning time and memory.
Takeaways & Limitations
Speaker encoding is favorable for low-resource deployment because it reduces cloning time or required memory.
Takeaways & Limitations
Whole-model speaker adaptation is difficult to optimize with small cloning datasets and requires early stopping to avoid overfitting.
Abstract
from arXiv · showhide
Voice cloning is a highly desired feature for personalized speech interfaces. Neural network based speech synthesis has been shown to generate high quality speech for a large number of speakers. In this paper, we introduce a neural voice cloning system that takes a few audio samples as input. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model with a few cloning samples. Speaker encoding is based on training a separate model to directly infer a new speaker embedding from cloning audios and to be used with a multi-speaker generative model. In terms of naturalness of the speech and its similarity to original speaker, both approaches can achieve good performance, even with very few cloning audios. While speaker adaptation can achieve better naturalness and similarity, the cloning time or required memory for the speaker encoding approach is significantly less, making it favorable for low-resource deployment.
1 Introduction
The paper targets voice cloning for unseen speakers from a few audio samples. It introduces speaker adaptation, speaker encoding, automated evaluation, and embedding-based voice morphing.
- Motivation: Multi-speaker speech synthesis conditions generated speech on text and speaker identity but traditionally supports only speakers observed during training.Text controls linguistic content, while speaker identity captures characteristics such as pitch, speech rate, and accent.
- Approach: The paper investigates voice cloning in sequence-to-sequence neural speech synthesis systems.
- Approach: Speaker adaptation fine-tunes a pre-trained multi-speaker model for an unseen speaker using a few samples.
- Approach: Speaker encoding estimates an unseen speaker’s embedding and requires significantly less cloning time and computational resources.
- Evaluation: The paper proposes automated voice-cloning evaluation using neural speaker classification and speaker verification.
- Applications: Embedding manipulations demonstrate voice morphing for gender and accent transformation.
2 Related Work
The work builds on neural speech synthesis, few-shot generative modeling, speaker-dependent speech processing, and voice conversion. Its baseline is Deep Voice 3, while its cloning task must generalize to unseen texts unlike voice conversion.
- Neural speech synthesis: Neural speech synthesis includes sequence-to-sequence models with attention, which offer simpler pipelines and can produce more natural speech.
- Neural speech synthesis: Deep Voice 3 is used as the baseline multi-speaker model because of its convolutional architecture, training efficiency, and fast model adaptation.
- Few-shot generative modeling: Few-shot generative modeling studies how generative systems can learn from only a few examples, including conditional image generation and speech-related tasks.
- Speaker-dependent speech processing: Speaker-dependent neural ASR methods include whole-model, partial-model, and speaker-embedding adaptation, paralleling the paper’s voice-cloning approaches.
- Voice conversion: Voice conversion changes an utterance toward a target speaker while preserving linguistic content, whereas voice cloning must generalize to unseen texts.
3 From Multi-Speaker Generative Modeling to Voice Cloning
The system starts with a multi-speaker generative model conditioned on text and speaker embeddings, then clones an unseen speaker through adaptation or direct speaker encoding. It evaluates generated speech for naturalness, speaker similarity, and discriminative identity properties.
- Multi-speaker generative modeling: The multi-speaker generative model takes text and speaker identity, using trainable parameters and speaker embeddings optimized against ground-truth audio.Speaker embeddings capture speaker characteristics with low-dimensional vectors and can exhibit discriminative properties such as gender and accent.
- Voice cloning: Voice cloning extracts characteristics from a few audios of an unseen speaker and generates audio for arbitrary text, evaluated by naturalness and speaker similarity.Speaker similarity measures whether generated speech sounds like it was pronounced by the target speaker.
- Speaker adaptation: Speaker adaptation fine-tunes a trained multi-speaker model for an unseen speaker using a few audio-text pairs, either by updating the embedding or the whole model.Whole-model adaptation has more degrees of freedom but requires early stopping because optimization is challenging with little cloning data.
- Speaker encoding: Speaker encoding directly estimates an unseen speaker’s embedding from cloning audios, avoiding fine-tuning during cloning and allowing one model to serve all unseen speakers.The encoder is trained from cloning samples of speakers used by the multi-speaker model and computes an embedding for a target speaker at inference.
- Speaker encoding: The speaker encoder processes mel-spectrograms through spectral and temporal modules, aggregates utterance representations, and uses attention over cloning samples.Its architecture includes fully connected layers, convolutional layers with gated linear units and residual connections, average pooling, and multi-head self-attention.
- Discriminative evaluation: Discriminative evaluation uses speaker classification and speaker verification to assess whether cloned audio identifies or matches the target speaker.Speaker verification can evaluate unseen speakers without training on the target speaker’s cloning audios; equal error rate measures closeness to ground-truth audio.
4 Experiments
Experiments compare speaker adaptation and speaker encoding for few-shot cloning using automated speaker evaluations, human ratings, embedding visualization, and dataset variants. Both approaches benefit from more cloning samples, while their relative performance depends on adaptation method and training data.
- 4.1 Datasets: The experiments train models on 820 hours of LibriSpeech audio from 2,484 speakers and clone voices from 108 VCTK speakers.VCTK audio is downsampled from 48 KHz to 16 KHz for consistency with LibriSpeech.
- 4.2 Model specifications: The speaker encoder converts cloning audios into log-mel spectrograms, applies spectral and temporal processing with attention, and produces 512-dimensional embeddings.The inputs use 80 mel-frequency bands; the encoder uses two 1-D convolutional layers and two-head attention.
- 4.3 Voice cloning performance: Speaker classification accuracy increases with more cloning samples, up to ten samples, while embedding adaptation is less prone to overfitting than whole-model adaptation at low sample counts.Embedding adaptation requires significantly more iterations to converge, producing longer cloning time than whole-model adaptation.
- 4.3 Voice cloning performance: When cloning audio exceeds five samples, whole-model adaptation outperforms other techniques; speaker encoding has lower classification accuracy than embedding adaptation but similar speaker-verification performance.Both speaker adaptation and speaker encoding benefit from more cloning audios.
- 4.3 Voice cloning performance: Human evaluations measure naturalness with 5-scale MOS and speaker similarity with a 4-scale score, with higher cloning-audio counts improving both metrics.Similarity scores for speaker encoding slightly improve with more samples and match speaker embedding adaptation.
- 4.4 Voice morphing via embedding manipulation: Speaker embeddings form a meaningful latent space in which algebraic operations can transform gender and accent characteristics while producing high-quality audio.Examples include transforming a British male embedding toward British female or American male characteristics.
- 4.5 Impact of training dataset: Training on cleaner VCTK data improves whole-model adaptation, whereas embedding-only adaptation underperforms because VCTK provides limited speaker diversity.This comparison trains on 84 VCTK speakers and clones 16 held-out speakers.
5 Conclusions
The paper concludes that speaker adaptation and speaker encoding both support good few-shot voice cloning quality. Speaker encoding trades some performance for faster cloning and a smaller per-speaker footprint, while dataset quality and diversity remain important constraints.
- 5 Conclusions: Both speaker adaptation and speaker encoding achieve good cloning quality with only a few cloning audios.The conclusion evaluates quality through naturalness and similarity.
- 5 Conclusions: Both approaches can achieve naturalness similar to the baseline multi-speaker generative model, and similarity improves with more cloning audios.The conclusion also notes that speaker adaptation can outperform embedding-only adaptation in the performance gap discussed.
- 5 Conclusions: Speaker encoding provides fast cloning and a small per-speaker footprint through compact speaker representations.These properties make the approach relevant to resource-constrained deployment.
- 5 Conclusions: Low-quality speech-recognition training audio and limited speaker diversity constrain the observed results, while better datasets are associated with higher naturalness.The paper suggests future gains from improved multi-speaker models and larger-scale data.
A Detailed speaker encoder architecture
The speaker encoder architecture processes cloning audio through a sequence of spectral, temporal, and attention operations to estimate speaker embeddings. The accompanying test sentences use special whitespace characters defined in prior work.
- A Detailed speaker encoder architecture: The speaker encoder diagram reports dimensions for batch size, cloning-sample count, mel-spectrogram timeframes, frequency channels, mapped channels, and speaker-embedding size.The final multiplication represents an inner product along the cloning-sample dimension.
- A Detailed speaker encoder architecture: The test sentences are used to generate evaluation samples for the voice cloning models.The characters / and % follow the same definition as in Ping et al. [2018].
C Speaker verification model
The speaker verification model classifies whether enrollment and test audios come from the same speaker, using learned encodings and PLDA-based similarity scoring. Its EER evaluations cover different enrollment and cloning-sample settings, with a dataset mismatch caveat.
- The model determines whether enrollment and test audios belong to the same speaker through binary classification.It accepts approximately 1–5 enrollment audios, which are from the same speaker.
- Mel spectrograms pass through convolutional layers, a GRU, time and enrollment-audio mean pooling, and a fully connected layer to produce speaker encodings.
- PLDA scores compare enrollment and test encodings, and a sigmoid converts the score into the probability that they come from the same speaker.The model is trained with cross-entropy loss.
- EER on cloned audios could be better than on ground-truth VCTK audio because the verification model is trained on LibriSpeech.
D Implications of attention
The speaker encoder’s attention mechanism combines cloning audios with non-uniform weights that can depend on audio length. The associated analyses compare attention with simple averaging and examine attention distributions across sample lengths.
- Attention assigns non-uniform coefficients when combining information from different cloning samples.The comparison baseline simply averages the cloning samples.
- Longer cloning audios generally receive higher attention coefficients, consistent with their potentially greater information content.
- The analyses measure embedding-estimation mean absolute error as the number of cloning audios changes, with and without attention.
E Speaker embedding space learned by the encoder
Principal-component analysis shows that embeddings inferred by the speaker encoder organize speakers into meaningful patterns related to gender and accent region. Gender information is especially discriminative despite not being a training label.
- The first two principal components of inferred embeddings reveal highly meaningful discriminative patterns for gender and region of accent.The analysis uses VCTK speakers’ ground-truth labels for these attributes.
- A one-dimensional linear transformation of the learned embeddings achieves very high gender-discrimination accuracy, although gender labels are never used during training.
F Similarity scores
The paper examines distributions of human similarity ratings for different cloning-sample counts. With 10 samples, all models receive a ‘same speaker’ rating in more than 70% of evaluations.
- For 10 cloning samples, every model exceeds 70% of evaluations rated ‘same speaker’ by MTurk users.
- Figure 14 compares the distributions of similarity scores for 1-sample and 10-sample conditions.