Source-linked AI summary
CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, Nobukatsu Hojo
TL;DR
Non-parallel voice conversion avoids parallel utterances but remains difficult because of unfavorable training conditions and a persistent gap between converted and real target speech. CycleGAN-VC2 improves CycleGAN-VC with two-step adversarial losses, a 2-1-2D CNN generator, and a PatchGAN discriminator. Objective and subjective evaluations show closer target features and better naturalness and similarity across all evaluated speaker pairs.
Problem
Non-parallel voice conversion avoids parallel data, extra modules, and time alignment, but challenging training conditions leave a gap between converted and real target speech.
Method
CycleGAN-VC2 improves CycleGAN-VC with two-step adversarial losses, a 2-1-2D CNN generator, and a PatchGAN discriminator.
Results
CycleGAN-VC2 brings converted acoustic features closer to targets in global and local structures and outperforms CycleGAN-VC in naturalness and similarity for every speaker pair.
Takeaways & Limitations
The proposed techniques improve both objective and subjective measures for non-parallel voice conversion across intra-gender and inter-gender speaker pairs.
Abstract
from arXiv · showhide
Non-parallel voice conversion (VC) is a technique for learning the mapping from source to target speech without relying on parallel data. This is an important task, but it has been challenging due to the disadvantages of the training conditions. Recently, CycleGAN-VC has provided a breakthrough and performed comparably to a parallel VC method without relying on any extra data, modules, or time alignment procedures. However, there is still a large gap between the real target and converted speech, and bridging this gap remains a challenge. To reduce this gap, we propose CycleGAN-VC2, which is an improved version of CycleGAN-VC incorporating three new techniques: an improved objective (two-step adversarial losses), improved generator (2-1-2D CNN), and improved discriminator (PatchGAN). We evaluated our method on a non-parallel VC task and analyzed the effect of each technique in detail. An objective evaluation showed that these techniques help bring the converted feature sequence closer to the target in terms of both global and local structures, which we assess by using Mel-cepstral distortion and modulation spectra distance, respectively. A subjective evaluation showed that CycleGAN-VC2 outperforms CycleGAN-VC in terms of naturalness and similarity for every speaker pair, including intra-gender and inter-gender pairs.
1. INTRODUCTION
The paper addresses non-parallel voice conversion, which avoids costly parallel utterances and alignment procedures but remains challenging. It proposes CycleGAN-VC2, whose objective, generator, and discriminator improvements reduce the gap to target speech and improve evaluations across speaker pairs.
- Motivation: Non-parallel VC avoids parallel utterances, transcriptions, and time alignment procedures, but its training conditions make it challenging.Parallel data can be laborious or impractical to collect, and alignment may fail or require manual correction.
- Motivation: Extra modules or data can help non-parallel VC training but impose preparation costs that limit application.Examples include automatic speech recognition modules and parallel utterance pairs among reference speakers.
- Prior work: CycleGAN-VC learns a sequence-based mapping with gated CNNs and identity-mapping loss, avoiding extra data, modules, and time alignment procedures.It extends CycleGAN to voice conversion while addressing the time-dependency limitations of frame-by-frame methods.
- Proposed method: CycleGAN-VC2 introduces two-step adversarial losses, a 2-1-2D CNN generator, and a PatchGAN discriminator to reduce the remaining target-conversion gap.The paper analyzes the effect of each technique on the VCC 2018 Spoke non-parallel VC task.
- Results: Objective evaluation finds closer global and local acoustic-feature structures, while subjective evaluation favors CycleGAN-VC2 in naturalness and similarity for every speaker pair.Global and local structures are assessed with Mel-cepstral distortion and modulation spectra distance, respectively; evaluations include intra-gender and inter-gender pairs.
2. CONVENTIONAL CYCLEGAN-VC
Conventional CycleGAN-VC learns non-parallel mappings between source and target acoustic-feature sequences using adversarial, cycle-consistency, and identity-mapping losses. Its generator uses a 1D CNN for temporal-feature relationships, while its discriminator uses a 2D CNN with whole-input realness classification.
- Objective: CycleGAN-VC learns a mapping from source acoustic-feature sequences to target sequences without parallel data.The model uses mappings in both directions between source domain X and target domain Y.
- Objective: Adversarial loss trains the discriminator to distinguish real target features from converted features while the generator attempts to deceive it.The discriminator maximizes the loss, whereas the generator minimizes it.
- Objective: Cycle-consistency loss regularizes forward and inverse mappings so circular conversion can find an optimal pseudo pair of source and target features.It addresses the fact that adversarial loss alone does not guarantee linguistic consistency between input and output.
- Objective: Identity-mapping loss further encourages preservation of the input features.It is included alongside adversarial and cycle-consistency losses in the conventional objective.
- Objective: The conventional full objective uses adversarial loss once for each cycle, defining the one-step adversarial-loss formulation.λ_cyc and λ_id control the trade-offs for cycle-consistency and identity-mapping losses.
- Architecture: The generator uses a 1D CNN to capture feature-direction relationships while preserving temporal structure, whereas the discriminator uses a 2D CNN with whole-input realness classification.The discriminator architecture is called FullGAN.
3. CYCLEGAN-VC2
CycleGAN-VC2 improves CycleGAN-VC with two-step adversarial losses, a hybrid 2-1-2D CNN generator, and PatchGAN discriminator designs.
- Improved Objective: Two-Step Adversarial Losses: Two-step adversarial losses add discriminators for circularly converted features and apply adversarial losses twice within each cycle.The additional losses are added to the original objective to mitigate over-smoothing from the L1 cycle-consistency loss.
- Improved Generator: 2-1-2D CNN: The generator design combines 1D and 2D convolutional layers to balance dynamical-change modeling with local structure preservation.The paper motivates 1D CNNs for feature-direction relationships and 2D CNNs for locally restricted conversion.
- Improved Discriminator: PatchGAN: PatchGAN replaces FullGAN and determines realness from local patches rather than through a final fully connected layer using the overall input structure.The change is motivated by the parameter demands and training difficulty associated with wide-range discriminator receptive fields.
4. EXPERIMENTS
Experiments evaluated CycleGAN-VC2 on non-parallel VCC 2018 conversions using objective measures of global and local structure and subjective tests of naturalness and similarity.
- Experimental Conditions: The evaluation used four speaker-pair directions spanning intra-gender and inter-gender conversions on the VCC 2018 non-parallel task.The selected speakers were SF, SM, TF, and TM, with two sources crossed with two targets.
- Experimental Conditions: The generator and discriminator architectures were implemented with the proposed improvements and trained using randomly cropped 128-frame segments.Training used normalized MCEPs, LSGAN stabilization, and Adam optimization.
- Objective Evaluation: MCD measured global structural differences, whereas MSD measured local differences between target and converted MCEP sequences; smaller values indicate greater similarity.MSD is the root mean square error between logarithmic modulation spectra averaged across MCEP dimensions and modulation frequencies.
- Objective Evaluation: The full CycleGAN-VC2 model achieved state-of-the-art MCD and MSD performance for every speaker pair, with each proposed component contributing to the improvements.Two-step adversarial losses particularly improved MSD, while generator and discriminator changes improved both metrics.
- Subjective Evaluation: Listening tests showed that CycleGAN-VC2 surpassed CycleGAN-VC in naturalness and speaker similarity for every evaluated speaker pair.Naturalness was assessed with MOS, and similarity with an XAB preference test involving ten listeners.
5. CONCLUSIONS
The paper proposes CycleGAN-VC2 for non-parallel voice conversion by improving CycleGAN-VC's objective, generator, and discriminator. Experiments show gains over CycleGAN-VC in objective and subjective measures across every speaker pair.
- 5. CONCLUSIONS: CycleGAN-VC2 combines two-step adversarial losses, a 2-1-2D CNN generator, and a PatchGAN discriminator.These are the three proposed techniques for improving CycleGAN-VC.
- 5. CONCLUSIONS: CycleGAN-VC2 outperformed CycleGAN-VC in both objective and subjective measures for every speaker pair.The conclusion reports improvements across the evaluated non-parallel voice-conversion settings.