Source-linked AI summary
StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, Nobukatsu Hojo
TL;DR
Voice conversion must change speaker or other attributes while preserving linguistic content, yet conventional systems often depend on parallel aligned data and some non-parallel approaches produce oversmoothed speech or require test-time source attributes. StarGAN-VC uses StarGAN for non-parallel many-to-many conversion with a single attribute-conditioned generator. The paper reports higher sound quality and speaker similarity than a VAE-GAN baseline in subjective speaker-identity conversion tests, alongside fast generation and low data requirements.
Problem
Voice conversion methods often require aligned parallel data, while non-parallel alternatives can have limited quality, oversmoothed outputs, or require the input attribute at test time.
Method
StarGAN-VC uses a single encoder-decoder generator conditioned on an auxiliary attribute input to learn non-parallel many-to-many mappings.
Results
Subjective tests found higher sound quality and speaker similarity than a baseline based on a VAE-GAN approach.
Takeaways & Limitations
StarGAN-VC supports non-parallel many-to-many conversion, real-time implementations, and reasonably realistic speech from several minutes of training examples.
Takeaways & Limitations
VAE-based voice conversion can suffer from decoder oversmoothing that produces poor-quality buzzy-sounding speech.
Abstract
from arXiv · showhide
This paper proposes a method that allows non-parallel many-to-many voice conversion (VC) by using a variant of a generative adversarial network (GAN) called StarGAN. Our method, which we call StarGAN-VC, is noteworthy in that it (1) requires no parallel utterances, transcriptions, or time alignment procedures for speech generator training, (2) simultaneously learns many-to-many mappings across different attribute domains using a single generator network, (3) is able to generate converted speech signals quickly enough to allow real-time implementations and (4) requires only several minutes of training examples to generate reasonably realistic-sounding speech. Subjective evaluation experiments on a non-parallel many-to-many speaker identity conversion task revealed that the proposed method obtained higher sound quality and speaker similarity than a state-of-the-art method based on variational autoencoding GANs.
1. INTRODUCTION
Voice conversion seeks to preserve linguistic content while changing para/nonlinguistic information, but conventional methods often require aligned parallel data. StarGAN-VC addresses this with non-parallel many-to-many conversion using a single generator, while targeting realistic speech and practical deployment.
- Motivation: Conventional voice conversion methods often require accurately aligned parallel source and target speech, which can be difficult or impossible to collect.Large acoustic gaps can make time alignment difficult and may require manual correction.
- Motivation: Non-parallel methods avoid parallel-data requirements, but their audio quality and conversion effect are usually limited compared with parallel-data methods.The limitation is attributed to disadvantages in the training condition.
- Related methods: CVAE-based conversion can work with unaligned corpora, but VAE decoder oversmoothing commonly produces poor-quality buzzy-sounding speech.This oversmoothing is identified as a key weakness for voice conversion applications.
- Related methods: CycleGAN-VC enables non-parallel conversion but learns only one-to-one mappings, limiting its suitability for many-to-many voice conversion.Its training combines adversarial and cycle-consistency losses.
- Proposed method: StarGAN-VC uses one encoder-decoder generator with an auxiliary attribute input to learn many-to-many mappings across domains without parallel utterances, transcriptions, or time alignment.The method combines advantages associated with CVAE-VC and CycleGAN-VC.
- Proposed method: Unlike CVAE-VC and CycleGAN-VC, StarGAN-VC does not require the input speech attribute at test time.Its adversarial training encourages realistic outputs while preserving linguistic information across attribute mappings.
- Practical considerations: StarGAN-VC is designed for fast generation and several minutes of training examples, contrasting with VQ-VAE methods whose WaveNet generation and data requirements can hinder real-time use.The paper also reports that the paper proceeds from CycleGAN-VC review to StarGAN-VC and experiments.
2. CYCLEGAN VOICE CONVERSION
CycleGAN-VC learns mappings between two acoustic-feature domains with adversarial discriminators, while cycle consistency and identity losses regularize the mappings. These objectives aim to produce realistic converted speech while preserving linguistic information and unchanged-domain inputs.
- Formulation: CycleGAN-VC learns a mapping G from acoustic-feature sequences in domain X to domain Y and an inverse mapping F from Y to X.The inputs are feature sequences with feature dimension Q and potentially different sequence lengths.
- Adversarial training: Discriminators D_X and D_Y predict whether inputs are real acoustic features from domains X and Y, respectively.Their adversarial losses assess whether converted features resemble real speech features in the target domains.
- Adversarial training: The generators G and F attempt to fool the corresponding discriminators so converted-feature distributions match the empirical distributions of the target domains.The discriminators instead minimize their losses to distinguish converted from real feature sequences.
- Regularization: Adversarial losses alone do not guarantee linguistic preservation because infinitely many mappings can induce the same output distributions.Cycle consistency is introduced to regularize this underdetermined mapping problem.
- Regularization: The cycle-consistency loss encourages F(G(x)) ≃ x and G(F(y)) ≃ y, while the identity mapping loss keeps inputs unchanged when they already belong to the target domain.The full objective weights cycle-consistency and identity losses relative to adversarial losses using λ_cyc and λ_id.
3. STARGAN VOICE CONVERSION
StarGAN-VC extends non-parallel voice conversion to many-to-many mappings with a single generator controlled by target attributes. Its training combines adversarial, domain-classification, cycle-consistency, and identity-mapping objectives, while fully convolutional networks process variable-length acoustic sequences.
- StarGAN-VC uses StarGAN to learn non-parallel many-to-many voice-conversion mappings with a single generator network.The generator output attributes are controlled by an auxiliary target attribute input.
- Training objectives: The generator produces acoustic features intended to be realistic and to belong to the requested target attribute, guided by discriminator D and classifier C.D estimates whether features are real, while C predicts their attribute classes.
- Training objectives: Adversarial and domain-classification losses train generated features to appear real and be classified as the target attribute.The discriminator distinguishes generated from real speech features, and the classifier evaluates attribute membership.
- Training objectives: Cycle consistency encourages the generator to preserve linguistic information by making conversion toward another attribute and back approximate the original input.An identity-mapping loss additionally keeps inputs unchanged when they already have the target attribute.
- Conversion process: Conversion reconstructs a time-domain signal from generated features or spectral gain functions using the input spectral envelope and a vocoder.The acoustic features are mel-cepstral coefficients computed from spectral envelopes obtained with WORLD.
- Network architectures: The generator uses gated convolutional layers that take acoustic feature sequences and output sequences of the same length, capturing time dependencies.The fully convolutional architecture accepts entire sequences of arbitrary length without fully connected layers.
4. SUBJECTIVE EVALUATION
The study evaluates StarGAN-VC on non-parallel many-to-many speaker-identity conversion using a subset of the VCC 2018 dataset. It compares converted-speech quality and target-speaker similarity against a VAEGAN-based approach through subjective tests.
- Subjective experiments assess StarGAN-VC on a non-parallel many-to-many speaker-identity conversion task.The evaluation uses recordings from selected female and male US English speakers.
- The experiment uses four VCC 2018 speakers, yielding twelve source-to-target speaker combinations.The target attribute is represented by a four-dimensional one-hot vector.
- AB and ABX tests compare converted-speech sound quality and similarity to the target speaker against a VAEGAN-based approach.The comparison method was implemented with design choices necessitated by missing network and hyperparameter details.
5. CONCLUSION
StarGAN-VC enables non-parallel many-to-many voice conversion with a single generator and practical training and inference requirements. Subjective tests found higher sound quality and speaker similarity than a VAE-GAN baseline.
- StarGAN-VC uses a single generator network to learn many-to-many mappings across voice attribute domains.
- It requires no parallel utterances, transcriptions, or time alignment procedures for speech generator training.
- Converted speech can be generated quickly enough for real-time implementations.
- Only several minutes of training examples are needed to generate reasonably realistic-sounding speech.
- Subjective evaluations reported higher sound quality and speaker similarity than a baseline method based on a VAE-GAN approach.