Source-linked AI summary
Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
Takuhiro Kaneko, Hirokazu Kameoka
TL;DR
The paper targets voice conversion without temporally aligned parallel data, extra data, modules, or alignment procedures. It proposes CycleGAN-VC, combining cycle-consistent adversarial learning with gated CNNs and identity-mapping loss. On VCC 2016, converted features were close to the target on global variance and modulation spectra, and speech quality was comparable to a GMM baseline trained with parallel and twice the data.
Problem
Many voice-conversion methods require temporally aligned parallel source and target speech, which is painstaking to collect and may suffer quality-degrading misalignment.
Method
CycleGAN-VC learns forward and inverse mappings from unpaired speech with adversarial and cycle-consistency losses, using gated CNNs and identity-mapping loss.
Results
Converted features were reasonably good in global variance and modulation spectra, while converted-speech quality was comparable to a GMM method trained with parallel and twice the data.
Takeaways & Limitations
CycleGAN-VC provides a sequence-based parallel-data-free mapping without extra data, modules, or time-alignment procedures.
Abstract
from arXiv · showhide
We propose a parallel-data-free voice-conversion (VC) method that can learn a mapping from source to target speech without relying on parallel data. The proposed method is general purpose, high quality, and parallel-data free and works without any extra data, modules, or alignment procedure. It also avoids over-smoothing, which occurs in many conventional statistical model-based VC methods. Our method, called CycleGAN-VC, uses a cycle-consistent adversarial network (CycleGAN) with gated convolutional neural networks (CNNs) and an identity-mapping loss. A CycleGAN learns forward and inverse mappings simultaneously using adversarial and cycle-consistency losses. This makes it possible to find an optimal pseudo pair from unpaired data. Furthermore, the adversarial loss contributes to reducing over-smoothing of the converted feature sequence. We configure a CycleGAN with gated CNNs and train it with an identity-mapping loss. This allows the mapping function to capture sequential and hierarchical structures while preserving linguistic information. We evaluated our method on a parallel-data-free VC task. An objective evaluation showed that the converted feature sequence was near natural in terms of global variance and modulation spectra. A subjective evaluation showed that the quality of the converted speech was comparable to that obtained with a Gaussian mixture model-based method under advantageous conditions with parallel and twice the amount of data.
1. INTRODUCTION
The paper addresses the difficulty of training voice-conversion mappings from aligned parallel speech by proposing CycleGAN-VC, a parallel-data-free method without extra data, modules, or alignment. Evaluations found feature sequences near the target on structural measures and speech quality comparable to a GMM baseline trained with more advantageous data.
- Motivation: Parallel voice-conversion training data are difficult to collect, align automatically, and correct when misalignment degrades speech quality.Perfectly aligned data simplify mapping estimation, but real applications may require pre-screening and manual correction.
- Contribution: The proposed method requires no extra data, such as transcripts or reference speech, and no extra modules, such as an ASR module.It is designed for parallel-data-free voice conversion without an alignment procedure.
- Method: CycleGAN-VC learns source-to-target and inverse mappings from unpaired speech using adversarial and cycle-consistency losses.The CycleGAN framework makes it possible to find an optimal pseudo pair from unpaired data.
- Method: Gated CNNs and identity-mapping loss configure CycleGAN-VC to capture speech’s sequential and hierarchical structure while preserving linguistic information.The method uses gated CNNs and does not require an alignment procedure.
- Results: Objective evaluation found converted features reasonably good in global variance and modulation spectra, while subjective quality was comparable to a GMM method trained with parallel and twice the data.The comparison used a more advantageous training condition for the GMM-based method.
2. RELATED WORK
Prior parallel-data-free voice-conversion methods use ASR-based frame pairing, adaptation, or pre-constructed speaker spaces, each with stated requirements or scope limitations.
- ASR-based methods: ASR-based approaches find corresponding frames but require substantial transcript data and may struggle to capture nonverbal information.Their applicability depends on sufficiently robust and accurate ASR performance.
- Alternative methods: Other parallel-data-free approaches use adaptation techniques or incorporate a pre-constructed speaker space.
3. PARALLEL-DATA-FREE VC USING CYCLEGAN
CycleGAN-VC learns source-to-target and inverse mappings from unpaired speech using adversarial and cycle-consistency objectives. Gated CNNs and identity-mapping loss target speech structure and linguistic-information preservation without alignment.
- CycleGAN: CycleGAN learns forward and inverse mappings simultaneously with adversarial and cycle-consistency losses.These losses enable finding pseudo pairs with shared contextual information from unpaired data.
- CycleGAN: Adversarial loss makes converted data resemble the target-data distribution, while the discriminator distinguishes converted from target data.The generator minimizes this loss and the discriminator maximizes it.
- CycleGAN: Cycle-consistency adds an inverse-mapping constraint because adversarial loss alone does not guarantee preservation of the input's contextual information.The additional terms encourage the two mappings to find source-target pairs with the same contextual information.
- CycleGAN-VC: Gated CNNs represent speech's sequential and hierarchical structures while allowing parallelization over sequential data.Their gated mechanism selectively propagates information based on previous-layer states.
- CycleGAN-VC: Identity-mapping loss encourages mappings that preserve composition between input and output, supporting linguistic-information preservation without extra modules.A weighted λidLid term is added with trade-off parameter λid.
- CycleGAN-VC: The generator is fully convolutional and can accept input sequences of arbitrary length T.The architecture uses convolutional processing to capture relationships while preserving temporal structure.
4. EXPERIMENTS
Experiments evaluated CycleGAN-VC on parallel-data-free voice conversion using objective MCEP measures and subjective tests against a GMM baseline. The method produced target-like feature structure and comparable or better speech-quality outcomes despite using non-parallel data and half as much training data.
- Experimental conditions: The experiments used VCC 2016 speakers SF1, SM1, TF2, and TM3, with 162 training and 54 evaluation sentences per speaker.The evaluation covered female-to-female, female-to-male, male-to-female, and male-to-male conversion pairs.
- Experimental conditions: CycleGAN-VC was compared with GMM-VC using GV, modulation spectra, MCEP trajectories, naturalness, and speaker similarity.GMM-VC used parallel training data, totaling twice the amount used by CycleGAN-VC.
- Objective evaluation: The proposed method’s MCEP trajectories had a global structure similar to GMM-VC with GV while preserving comparable local structure.The reported trajectory comparison focused on the SF1–TM3 conversion pair.
- Objective evaluation: CycleGAN-VC with GLU produced MCEP sequences closest to the target in terms of GV and MS.The evaluation used GV and MS because mel-cepstral distortion tends to prefer over-smoothed outputs under its Gaussian assumption.
- Subjective evaluation: The proposed method significantly outperformed the baseline in naturalness MOS.MOS was reported with 95% confidence intervals.
- Subjective evaluation: Speaker similarity was slightly worse than the baseline for SF1–TM3 but better for SF1–TF2, making overall performance comparable.The comparison used source and target speaker similarity judgments.
5. DISCUSSION AND CONCLUSIONS
The paper concludes that CycleGAN-VC enables sequence-based voice conversion without parallel data, extra data, extra modules, or time alignment. Its outputs approach target structure objectively and match the baseline’s converted-speech quality overall, but a gap from original speech remains.
- Discussion and conclusions: CycleGAN-VC learns sequence-based voice-conversion mappings without extra data, modules, or time-alignment procedures.The method combines CycleGAN, gated CNNs, and an identity-mapping loss.
- Discussion and conclusions: MCEP sequences from CycleGAN-VC were close to the target in terms of GV and MS.These objective indicators were used to assess structural similarity of converted features.
- Discussion and conclusions: Converted-speech quality was comparable to GMM-VC trained with parallel data and twice as much data.The conclusion characterizes this comparison as occurring under advantageous conditions for the baseline.
- Discussion and conclusions: A margin remained between original and converted speech, motivating future work on other features and vocoder-free voice conversion.The paper also identifies other voice-conversion applications as future directions.