Source-linked AI summary
One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
Ju-chieh Chou, Cheng-chieh Yeh, Hung-yi Lee
TL;DR
Prior unsupervised voice conversion methods are limited to speakers seen during training, motivating a more flexible approach. This paper uses instance normalization to disentangle speaker and content representations for one-shot conversion, and reports target-speaker similarity plus meaningful unsupervised speaker embeddings.
Problem
Prior unsupervised voice conversion methods cannot synthesize voices of speakers who were never seen during training.
Method
The model disentangles speaker and linguistic content representations with speaker and content encoders, using instance normalization to remove speaker information from content representations.
Results
The model performs one-shot conversion to unseen speakers with good target-speaker similarity and learns meaningful speaker embeddings without supervision.
Takeaways & Limitations
One-shot unsupervised voice conversion can use a single utterance to convert toward speakers outside the training set while producing meaningful speaker representations.
Takeaways & Limitations
Prior unsupervised domain-mapping approaches remain limited because they cannot synthesize voices of speakers unseen during training.
Abstract
from arXiv · showhide
Recently, voice conversion (VC) without parallel data has been successfully adapted to multi-target scenario in which a single model is trained to convert the input voice to many different speakers. However, such model suffers from the limitation that it can only convert the voice to the speakers in the training data, which narrows down the applicable scenario of VC. In this paper, we proposed a novel one-shot VC approach which is able to perform VC by only an example utterance from source and target speaker respectively, and the source and target speaker do not even need to be seen during training. This is achieved by disentangling speaker and content representations with instance normalization (IN). Objective and subjective evaluation shows that our model is able to generate the voice similar to target speaker. In addition to the performance measurement, we also demonstrate that this model is able to learn meaningful speaker representations without any supervision.
1. Introduction
The paper targets one-shot voice conversion to speakers unseen during training by disentangling speaker identity from linguistic content with instance normalization. It addresses limitations of parallel-data methods and prior unsupervised systems while verifying the approach’s disentanglement and speaker-embedding properties.
- Motivation: Parallel-data voice conversion requires frame-level alignment, and collecting aligned source-target data is difficult and time-consuming.These constraints reduce its flexibility for adapting conversion systems to new domains.
- Motivation: Prior unsupervised domain-mapping methods can convert speaker characteristics but cannot synthesize voices of speakers unseen during training.This limits conversion to speakers represented in the training phase.
- Approach: The proposed model factorizes an utterance into speaker and content representations using speaker and content encoders followed by a decoder.The content encoder is intended to retain linguistic information, while the speaker encoder captures speaker information for synthesis.
- Approach: Instance normalization without affine transformation is used to remove speaker information from the content encoder representation instead of adversarial training.The paper states that this substantially reduces computation and makes training easier.
- Contributions: The model performs one-shot unsupervised voice conversion without supervision, using the proposed disentanglement approach.This contribution is positioned as extending conversion beyond speakers seen during training.
- Contributions: The speaker encoder learns meaningful speaker embeddings as a side effect without an explicitly added objective or constraint.The paper reports this finding for its learned speaker representations.
2. Proposed Approach
The proposed VC model factorizes speech into speaker and content representations using encoders and a decoder, with instance normalization helping remove speaker information from content representations. Adaptive instance normalization supplies speaker information to the decoder.
- The model uses a speaker encoder, content encoder, and decoder to factorize acoustic segments into speaker and content representations.The speaker encoder produces zs, while the content encoder produces zc for reconstruction.
- The VAE objective combines reconstruction loss and KL-divergence loss with weighted coefficients λrec and λkl.Under the unit-variance assumption, the KL term reduces to L2 regularization.
- Instance normalization without affine transformation is added to the content encoder to remove speaker information while preserving content information.The normalization computes each channel’s mean and standard deviation before transforming its elements.
- Instance normalization in the content encoder prevents it from learning domain information, encouraging separate speaker and content representations.This design assigns speaker information to the speaker encoder and content information to the content encoder.
- Adaptive instance normalization provides global speaker information from the speaker encoder to the decoder.The speaker encoder output is linearly transformed into per-channel γc and βc parameters.
3. Implementation Details
The implementation processes mel-scale spectrograms with convolutional encoders and decoder components designed to capture long-term and global information. Training uses ADAM with specified regularization and loss weights on VCTK data.
- Architecture: Conv1d layers process frequency information, ConvBank layers capture long-term information, and average pooling encourages global speaker representations.The decoder uses PixelShuffle1d for upsampling and adaIN to provide global information.
- Architecture: Figure 2 presents the architecture of the encoders and decoder.
- Acoustic features: The model uses mel-scale spectrograms derived from 24kHz audio with 512 mel bins after silence trimming, volume normalization, and STFT processing.The preprocessing uses a 50-millisecond window, 12.5-millisecond hop, and 2048-point STFT.
- Training: Training uses ADAM with a 0.0005 learning rate, batch size 256, dropout rate 0.5, and weight decay 0.0001.The model is trained for 200000 iterations with λrec = 10 and λkl = 0.01.
4. Experiments
Experiments evaluate disentanglement, speaker embeddings, objective voice characteristics, spectrogram transformations, and subjective similarity, including speakers unseen during training. The results support target-speaker similarity and preservation of phonetic content.
- Dataset and setup: The evaluation uses CSTR VCTK data with 20 speakers held out for testing and about 16,000 training utterances after filtering.Training segments have length 128, while the fully convolutional model accepts arbitrary lengths at inference.
- Evaluation of disentanglement: Speaker identity classification on content representations compares content encoders with and without instance normalization.Table 1 defines lower accuracy as less speaker information in the content representation; the supplied passage does not include the accuracy values.
- Speaker embedding visualization: Speaker embeddings separate utterances from different speakers, with classification accuracies of 0.9973 for seen and 0.9998 for unseen speakers.The embeddings were evaluated using both seen and unseen speakers and visualized with t-SNE.
- Global variance: Global-variance evaluation found that generated samples matched target speakers in variance distribution across four male-to-male, male-to-female, female-to-male, and female-to-female conversions.The evaluation used 100 randomly chosen utterances and converted results to calculate variance.
- Spectrogram examples: Spectrogram examples show transformed fundamental frequency while preserving the original phonetic content in both male-to-female and female-to-male conversion.The figure presents source and converted spectrogram heatmaps for female and male speakers.
- Subjective evaluation: Subjective tests used four unseen speaker pairs and one source plus one target utterance per conversion; participants reported similarity to the target speaker.The results used a four-scale human similarity judgment comparing converted speech with source or target utterances.
5. Conclusion
The conclusion presents instance normalization as the basis for one-shot unsupervised voice conversion through factorized representations. Objective and subjective evaluations support similarity to target speakers, while visualizations indicate unsupervised speaker embeddings.
- Conclusion: The proposed approach applies instance normalization to enforce factorized speaker and content representations for one-shot unsupervised voice conversion.The method performs conversion to unseen speakers using one utterance.
- Conclusion: Objective and subjective evaluations showed good results in similarity to target speakers.The conclusion also reports disentanglement experiments and visualizations of the learned representations.
- Conclusion: The speaker encoder learned a meaningful embedding space without supervision.This finding is presented as an additional result alongside conversion performance.