Source-linked AI summary
StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversion
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, Nobukatsu Hojo
TL;DR
Non-parallel multi-domain voice conversion must learn mappings among domains without parallel supervision, while StarGAN-VC still leaves a gap between real and converted speech. StarGAN-VC2 revises its conditional objectives and architecture, and evaluations show improvements over StarGAN-VC in objective measures, naturalness, and speaker similarity.
Problem
Non-parallel multi-domain VC is difficult because it learns multiple domain mappings without parallel data, and StarGAN-VC still has a gap between real and converted speech.
Method
StarGAN-VC2 combines a source-and-target conditional adversarial loss with a modulation-based conditional method for domain-dependent acoustic-feature modulation.
Results
StarGAN-VC2 outperforms StarGAN-VC in objective and subjective evaluations, including global and local structure measures, naturalness, and speaker similarity.
Takeaways & Limitations
The proposed model provides a general multi-domain VC model whose demonstrated benefits extend beyond the evaluated multi-speaker VC setting in scope.
Takeaways & Limitations
The source-and-target conditional generator requires the source code during inference, unlike conventional StarGAN-VC.
Abstract
from arXiv · showhide
Non-parallel multi-domain voice conversion (VC) is a technique for learning mappings among multiple domains without relying on parallel data. This is important but challenging owing to the requirement of learning multiple mappings and the non-availability of explicit supervision. Recently, StarGAN-VC has garnered attention owing to its ability to solve this problem only using a single generator. However, there is still a gap between real and converted speech. To bridge this gap, we rethink conditional methods of StarGAN-VC, which are key components for achieving non-parallel multi-domain VC in a single model, and propose an improved variant called StarGAN-VC2. Particularly, we rethink conditional methods in two aspects: training objectives and network architectures. For the former, we propose a source-and-target conditional adversarial loss that allows all source domain data to be convertible to the target domain data. For the latter, we introduce a modulation-based conditional method that can transform the modulation of the acoustic feature in a domain-specific manner. We evaluated our methods on non-parallel multi-speaker VC. An objective evaluation demonstrates that our proposed methods improve speech quality in terms of both global and local structure measures. Furthermore, a subjective evaluation shows that StarGAN-VC2 outperforms StarGAN-VC in terms of naturalness and speaker similarity. The converted speech samples are provided at http://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/stargan-vc2/index.html.
1. Introduction
Voice conversion preserves linguistic information while changing nonlinguistic information, but non-parallel multi-domain conversion remains challenging because parallel data are costly and one-to-one methods require multiple generators. StarGAN-VC addresses this with one conditional generator, while StarGAN-VC2 revises its conditional methods and improves objective and subjective performance.
- Voice conversion changes nonlinguistic information while preserving linguistic information, supporting applications such as speaking aids, style conversion, and pronunciation conversion.
- Parallel VC often requires time-consuming parallel utterances and alignment procedures that can fail or require manual correction.
- Non-parallel multi-domain VC is challenging because one-to-one methods require multiple generators to learn mappings among multiple domains.
- StarGAN-VC extends CycleGAN-VC with domain codes and a single generator, while StarGAN-VC2 rethinks its training objectives and network architectures.
- Objective evaluation found improvements in global and local structure measures, and subjective evaluation found StarGAN-VC2 better than StarGAN-VC in naturalness and speaker similarity.
2. Conventional StarGAN-VC
Conventional StarGAN-VC uses one conditional generator with domain codes and combines adversarial, classification, cycle-consistency, and identity-mapping objectives. Its generator injects domain information by concatenating expanded one-hot codes channel-wise with feature maps.
- StarGAN-VC learns a single generator that maps acoustic features across multiple domains using a target domain code.
- Its training objectives combine adversarial, classification, cycle-consistency, and identity-mapping losses.
- The adversarial loss makes converted features indistinguishable from real target features under target-domain conditioning.
- The classification loss trains a classifier on real features and optimizes the generator to produce features classified as the target domain.
- Cycle consistency preserves input composition, while identity mapping adds a further constraint on input preservation.
- The generator injects domain information by expanding a one-hot domain code to feature-map size and concatenating it channel-wise before convolution.
3. StarGAN-VC2
StarGAN-VC2 rethinks conditional methods in both training objectives and network architectures to improve non-parallel multi-domain voice conversion. It introduces source-and-target conditioning and modulation-based conditioning to address limitations in domain discrimination and feature modulation.
- Training objectives: The classification loss can favor generated data far from real-data decision boundaries, preventing coverage of the whole real-data distribution.
- Training objectives: Target conditional adversarial loss brings generated data toward target-domain real data but must handle unevenly difficult source-target conversions.
- Training objectives: Source-and-target conditional adversarial loss conditions both generator and discriminator on source and target domains, encouraging source-wise and target-wise alignment.This resolves the stated unfair training condition and allows all source-domain data to be converted into the target domain.
- Training objectives: The source-and-target conditional generator requires the source code during inference, unlike conventional StarGAN-VC.The paper notes that speaker recognition can provide this code as a preprocessing step.
- Network architectures: Channel-wise conditioning cannot multiplicatively modulate features, motivating conditional instance normalization for domain-dependent feature modulation.CIN uses domain-specific scale and bias parameters; with source-and-target conditioning, these parameters depend on both source and target domains.
4. Experiments
Experiments evaluate StarGAN-VC2 on four-speaker non-parallel VC using MCEP conversion, objective metrics, and listening tests. The proposed conditional loss and modulation-based generator are compared with alternative conditional methods and StarGAN-VC.
- Experimental conditions: The VCC 2018 subset contains four speakers, requiring one model to learn 12 source-and-target mappings.The selected speakers cover inter-gender and intra-gender conversions.
- Experimental conditions: The experiments apply StarGAN-VC2 to MCEP conversion while using conventional processing for log F0, aperiodicity, and waveform synthesis.No postfilter or powerful vocoder was used to isolate the proposed methods' effects.
- Experimental conditions: The fully convolutional generator accepts an arbitrary-length acoustic sequence during inference.The architecture uses convolutional layers and includes IN, GLU, pixel shuffler, and global sum pooling components.
- Objective evaluation: The source-and-target conditional adversarial loss outperforms other objective losses on both MCD and MSD.MCD measures global structural differences, whereas MSD measures local structural differences.
- Objective evaluation: The modulation-based conditional method outperforms channel-wise conditioning on MSD, indicating improved local structure.The network comparison fixes the training objective as Lst-adv.
- Subjective evaluation: Listening tests compare StarGAN-VC2 with StarGAN-VC using MOS for naturalness and preference scores for speaker similarity.Twelve English speakers participated, with analysis-synthesized speech serving as a naturalness reference at MOS 4.2.
5. Conclusions
The paper proposes StarGAN-VC2 by rethinking StarGAN-VC's conditional methods in training objectives and network architectures. On non-parallel multi-speaker VC, it outperforms StarGAN-VC in objective and subjective measures, while adaptation to other multi-domain tasks remains future work.
- Contributions: StarGAN-VC2 combines a source-and-target conditional adversarial loss with a modulation-based conditional method.These components address conditional training objectives and generator network architecture, respectively.
- Results: On non-parallel multi-speaker VC, StarGAN-VC2 outperforms StarGAN-VC in both objective and subjective measures.The conclusion reports this result as an empirical finding.
- Scope: StarGAN-VC2 is presented as a general multi-domain VC model, with multi-emotion and multipronunciation VC identified as future directions.The paper does not report those adaptations as completed experiments.