StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion

• Most voice conversion methods require parallel utterances for high-quality natural conversion, restricting where the technique can be applied.Source: §1 · p. 1
• Non-parallel VC methods include autoencoder, GAN, and TTS approaches, with constraints, discriminator-related distortions, or unavailable text labels limiting their respective settings.Source: §1 · p. 1
• The proposed unsupervised non-parallel StarGAN v2 framework significantly outperforms AUTO-VC in naturalness and speaker similarity, approaching VTN [9] in VCC2020 [11].Source: §1 · p. 1
• The approach generalizes from monolingual, limited-speaker training to any-to-many, cross-language, and singing conversion, and converts plain reading into emotional or falsetto speech.Source: §1 · p. 1
• The framework applies StarGAN v2 to voice conversion and introduces an adversarial source-classifier loss for speaker identity similarity and perceptual losses using ASR and F0 networks.Source: §1 · p. 1

2.1. StarGANv2-VC

• The generator converts a source mel-spectrogram using a target style code and F0 features from a pre-trained JDC network.The style code comes from either a mapping network or a reference-conditioned style encoder; the generator uses convolutional F0 features.Source: §2.1 · p. 1
• The style encoder extracts a domain-specific style code from a reference mel-spectrogram, while the mapping network can generate one from a random latent code.The reference input is not needed when using the mapping-network scheme.Source: §2.1 · pp. 1–2, Figure 1 · p. 2
• An additional adversarial source classifier loss provides feedback about source-speaker features that remain after conversion.The classifier is designed to capture domain-specific features such as a speaker’s pronunciations that the real/fake classifier may miss.Source: §2.1 · p. 2, §2.2 · p. 2

2.2. Training Objectives

• Style reconstruction and diversification losses encourage generated samples to retain their style code and vary with different style codes.Diversification also maximizes differences between F0 features from samples generated with different style codes.Source: §2.2 · p. 2
• F0 and speech consistency losses compare normalized F0 curves and ASR-network features between input and converted samples.The F0 loss addresses differences in average F0 between male and female speakers; the speech consistency loss is intended to preserve linguistic content.Source: §2.2 · p. 3
• Norm consistency preserves speech and silence intervals, while cycle consistency is used to preserve other input features.The full generator objective combines these terms with adversarial, style, diversification, F0, and speech consistency losses.Source: §2.2 · p. 3

3.1. Datasets

• VCTK supplies 20 speakers for baseline comparison, while JVS and ESD support stylistic-speech experiments and LibriSpeech trains the ASR and F0 models.The datasets are split 80%/10%/10% for training, validation, and testing.Source: §3.1 · p. 3

3.2. Training details

• Training runs for 150 epochs on two-second segments, and the source classifier joins after 50 epochs.AUTO-VC is trained for 1M steps; the ASR model reaches an 8.53% character error rate.Source: §3.2 · p. 3

3.3. Evaluations

• Evaluation combines listener ratings of naturalness and speaker similarity with objective pMOS, speaker-classification accuracy, and CER measures.The study uses 40 conversion pairs and 43 accepted raters; ablations use objective evaluation only because subjective evaluations are expensive and time-consuming.Source: §3.3 · pp. 3–4

• StarGANv2-VC significantly exceeds AUTO-VC in rated naturalness across all four conversion types and yields greater perceived speaker similarity to ground truth.The naturalness comparison is significant at p < 0.001 for both the randomization test and t-test.Source: §4 · p. 4
• Speaker-recognition accuracy and predicted MOS are higher, while CER is lower, for StarGANv2-VC conversions than for AUTO-VC.Table 2 reports objective metrics, including mean and standard deviation of pMOS, evaluated on 1,000 random source-target pairs.Source: §4 · p. 4, Table 2 · p. 4
• Removing λasr harms linguistic-content conversion, removing λadvcls lowers speaker-recognition accuracy, and removing λf0 yields unnatural intonation.Removing λnorm introduces noise during silent intervals; standard pMOS, CLS, and CER metrics may not reflect the F0 effect.Source: §4 · p. 4
• Audio demonstrations show generalization to a variety of voice-conversion tasks.Source: §4 · p. 4

• The model achieves similar MOS to the highest-MOS TTS-based VCC2020 system without text labels during training or autoregressive vocoders.Source: §5 · p. 4
• A style encoder converts plain reading speech into stylistic speech, including emotional acting and falsetto.Source: §5 · p. 4
• The framework generalizes to any-to-many, cross-lingual, and singing conversion without explicit training for those tasks.Source: §5 · p. 4
• With Parallel WaveGAN, conversion runs hundreds of times faster than real time on Tesla P100; future work targets quality improvements for generalization schemes and a real-time system.Source: §5 · p. 4