Source-linked AI summary
High-quality nonparallel voice conversion based on cycle-consistent adversarial network
Fuming Fang, Junichi Yamagishi, Isao Echizen, Jaime Lorenzo-Trueba
TL;DR
Nonparallel voice conversion is difficult to train and typically performs worse than parallel conversion, although it can use speech with different linguistic content. This paper applies CycleGAN to nonparallel voice conversion, and its inter-gender evaluation reports significantly better speech quality and speaker similarity than two parallel baselines.
Problem
Nonparallel VC typically performs worse because source and target training speech has different linguistic content, making similar acoustic features harder to match.
Source: §1 · p. 1Method
The method uses CycleGAN’s adversarial and cycle-consistency losses to learn speech mappings from unpaired source and target data.
Source: §1 · p. 1, §3 · pp. 2–3Results
Takeaways & Limitations
The inter-gender evaluation shows that this CycleGAN-based nonparallel method can outperform the compared state-of-the-art parallel VC methods.
Source: §8 · p. 4CycleGAN does not strictly guarantee that linguistic information remains unchanged from input to output, so converted speech can map to an unexpected phone distribution.
Source: §7 · p. 4
Abstract
from arXiv · showhide
Although voice conversion (VC) algorithms have achieved remarkable success along with the development of machine learning, superior performance is still difficult to achieve when using nonparallel data. In this paper, we propose using a cycle-consistent adversarial network (CycleGAN) for nonparallel data-based VC training. A CycleGAN is a generative adversarial network (GAN) originally developed for unpaired image-to-image translation. A subjective evaluation of inter-gender conversion demonstrated that the proposed method significantly outperformed a method based on the Merlin open source neural network speech synthesis system (a parallel VC system adapted for our setup) and a GAN-based parallel VC system. This is the first research to show that the performance of a nonparallel VC method can exceed that of state-of-the-art parallel VC methods.
1. INTRODUCTION
Voice conversion changes a source speaker’s speech to sound like a target speaker while preserving linguistic content, but nonparallel training is harder and typically performs worse than parallel training. The paper proposes CycleGAN-based nonparallel conversion and reports its effectiveness in subjective experiments.
1. INTRODUCTION
- Voice conversion modifies source-speaker speech to sound like the target speaker while keeping linguistic information unchanged.The paper also notes applications in personalized TTS, speaking aids, dubbing, language learning, singing conversion, and games.Source: §1 · p. 1
- Nonparallel VC typically performs worse because different linguistic content makes source and target acoustic features harder to match, despite the greater flexibility of using varied utterances.Such data can be collected from sources such as YouTube, and parallel data may be impossible when speakers use different languages or accents.Source: §1 · p. 1
- CycleGAN offers a way to learn from unpaired data by combining cycle-consistency constraints with adversarial loss that makes generated and real target distributions indistinguishable.The paper describes CycleGAN as originally developed for unpaired image-to-image translation.Source: §1 · p. 1
- The proposed CycleGAN-based method uses discriminators to distinguish source from target speech while generators seek to confuse them and maintain source linguistic information.Subjective experiments demonstrated the method’s effectiveness.Source: §1 · p. 1
2. RELATED WORK
Related VC methods either align similar source and target features for mapping or replace speaker identity components. The proposed method avoids parallel data and alignment, using assumed linguistic invariance through the network instead of separating linguistic information from speaker identity.
2.1. Related parallel VC methods
- Parallel VC methods pair similar source and target features for mapping, but imperfect alignment can introduce errors; the proposed method requires neither parallel data nor alignment.Prior approaches include GMM, feed-forward and bidirectional neural networks, and GAN-based training criteria.Source: §2.1 · p. 2
2.2. Related nonparallel VC methods
- Nonparallel feature-pair methods search for similar source and target features, while individuality-replacement methods assume speech can be divided into linguistic and speaker-identity components.Examples include transcription- or HMM-based matching, iterative nearest-neighbor search, adaptive models, and C-VAE/W-GAN conversion.Source: §2.2 · p. 2
- Unlike the related methods, the proposed approach does not separate linguistic information from the source speaker, instead assuming part remains invariant through the network.The paper describes its method as more straightforward than previous nonparallel approaches and identifies Hsu et al.’s GAN-based method as the closest comparison.Source: §2.2 · p. 2
3. CYCLE-CONSISTENT ADVERSARIAL NETWORK
CycleGAN learns bidirectional mappings between unpaired source and target samples by combining adversarial and cycle-consistency losses. The adversarial objectives train generators and discriminators in competition, while cycle consistency reconstructs each input after it passes through both generators.
3. CYCLE-CONSISTENT ADVERSARIAL NETWORK
- Two generators map between distributions X and Y, while discriminators distinguish real samples from generated samples in each distribution.G maps X to Y and F maps Y to X; DY distinguishes Y from G(X), and DX distinguishes X from F(Y).Source: §3 · p. 2
- Cycle-consistency loss requires an input to retain its original form after passing through both generators, enabling mappings to be learned from unpaired samples.The model learns both X → Y → X and Y → X → Y cycles simultaneously.Source: §3 · p. 2
- Generators minimize the adversarial objectives while discriminators maximize them; the full objective combines adversarial and cycle-consistent losses.The relative importance of the two losses is controlled by λ.Source: §3 · p. 3
- Least-squares loss is used in practice because it is more stable than negative log likelihood during backpropagation.The paper rewrites the adversarial loss as LLSGAN for this reason.Source: §3 · p. 3
4. NONPARALLEL VC BASED ON CYCLEGAN
The system extracts speech features, converts selected components, and synthesizes speech. CycleGAN converts lower-order mel-cepstral coefficients, while other components are transformed or copied as described.
4. NONPARALLEL VC BASED ON CYCLEGAN
- The system extracts, converts, and synthesizes speech features, applying CycleGAN to lower-order mel-cepstral coefficients that reflect linguistic information and speaker identity.Higher-order coefficients and aperiodicity are copied; log F0 is linearly transformed to match the target speaker’s mean and standard deviation.Source: §4 · p. 3
- Higher-order mel-cepstral coefficients are copied because they are assumed to carry little speaker information, while aperiodicity is copied because it has no significant effect on speaker characteristics.The mel-cepstrum is split into higher-order fine-structure and lower-order spectral-envelope components.Source: §4 · p. 3
- Source and target feature distributions for CycleGAN training consist only of lower-order mel-cepstral coefficients.Training samples are frame-level collections of coefficients extracted from each speaker’s speech data.Source: §4 · p. 3
5. EXPERIMENTAL SETUP
The study compares CycleGAN-based nonparallel conversion with two parallel baselines in inter-gender conversion using subjective ratings of speech quality and speaker similarity. The setup uses separate parallel and nonparallel training data and a crowdsourced listening evaluation.
5. EXPERIMENTAL SETUP
- The proposed nonparallel system was compared with Merlin-based and GAN-based parallel baselines on female-to-male and male-to-female conversion.Significance was tested using an unpaired two-tailed t-test with a 95% confidence interval and Holm-Bonferroni compensation.Source: §5 · p. 3
5.1. Database and speech feature
- Training used 200 utterances per speaker for each method; the proposed method used different subsets for male and female speakers, while baseline training used matching subsets.Subset I, comprising 50 utterances, was used for testing, and the database transcriptions were not used.Source: §5.1 · p. 3
- The 49-dimensional mel-cepstrum was split into 25 lower-order and 24 higher-order coefficients; parallel data were aligned with DTW, but nonparallel data were not matched.Static, first-derivative, and second-derivative lower-order features formed 75-dimensional vectors for conversion-model learning.Source: §5.1 · p. 3
5.2. Network structure, training and conversion setup
- The GAN and CycleGAN used six-layer feed-forward networks, 128-frame mini-batches, and 400 training epochs; CycleGAN used λ = 10.Merlin used 60 epochs; maximum-likelihood parameter generation and post-filtering were used to generate smooth speech parameters.Source: §5.2 · pp. 3–4
5.3. Subjective evaluation setup
- A total of 110 evaluators rated 300 converted utterances for speech quality and speaker similarity on a 1-to-5 Likert MOS scale.Each utterance received 24 evaluations, producing 7,200 data points through a crowdsourced web interface.Source: §5.3 · p. 4
6. EXPERIMENTAL RESULTS
The proposed CycleGAN-based nonparallel method significantly outperformed the parallel baselines on average speech quality and speaker similarity. However, it did not improve male-to-female conversion over the GAN-based method.
6. EXPERIMENTAL RESULTS
- The CycleGAN-based nonparallel method significantly outperformed the parallel baselines in average speech quality and speaker similarity.The authors conclude that nonparallel VC with CycleGAN can exceed state-of-the-art parallel VC performance.Source: §6 · p. 4
- Male-to-female conversion showed no improvement over the GAN-based method, and its speaker-similarity scores were lower than female-to-male scores for all methods.The authors suggest F0 mismatch and using only the first 25 mel-cepstrum dimensions as possible reasons.Source: §6 · p. 4
7. DISCUSSION
The discussion gives possible reasons for CycleGAN’s advantage over parallel baselines and notes that converted speech does not always preserve linguistic information strictly.
7. DISCUSSION
- The authors suggest that avoiding DTW and heteronym-related matching errors may explain CycleGAN’s advantage over parallel baselines.Heteronyms in the training data could introduce matching errors for parallel systems but would not affect the nonparallel method.Source: §7 · p. 4
- Another possible reason is that CycleGAN can train on any frame pairs, whereas the standard parallel method uses only DTW-aligned pairs.Source: §7 · p. 4
- CycleGAN does not strictly guarantee that converted speech preserves the source’s linguistic information; phones or even silence and voice may change.The paper attributes this to linguistic information not being explicitly constrained between the input and output.Source: §7 · p. 4
8. CONCLUSION AND FUTURE WORK
The proposed CycleGAN-based nonparallel method outperformed two parallel methods in an inter-gender conversion experiment. The authors identify stricter linguistic-content preservation and further evaluation as future work.
8. CONCLUSION AND FUTURE WORK
- In inter-gender conversion, CycleGAN significantly outperformed the Merlin-based and GAN-based parallel methods on speech quality and speaker similarity.Source: §8 · p. 4
- Future work includes developing a method that strictly constrains linguistic information to remain invariant in CycleGAN.Source: §8 · p. 4
- The authors also plan to improve speech quality and speaker similarity and compare their method with others using the Voice Conversion Challenge dataset.Source: §8 · p. 4