PaperlayerView exact revision on arXiv

Source-linked AI summary

Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks

Yuki Saito, Shinnosuke Takamichi, Hiroshi Saruwatari

arXiv:1709.08041v1cs.SDcs.LGeess.AS

TL;DR

Synthetic speech quality is reduced by over-smoothed generated parameters relative to natural speech. The paper trains acoustic models with GAN-based adversarial objectives that reduce distribution differences, and reports improved quality in TTS and VC, with W-GAN performing best among tested variants.

  • Problem

    Generated speech parameters are often over-smoothed, and synthetic speech quality remains lower than natural speech.

  • Method

    GAN-based training combines conventional generation loss with adversarial loss while a discriminator distinguishes natural from generated speech parameters.

  • Results

    The method improved synthetic speech quality in both TTS and VC regardless of hyper-parameter settings, and W-GAN improved quality most among the investigated GANs.

  • Takeaways & Limitations

    The method compensates distribution differences in generated speech parameters, including global variance and parameter correlations.

Abstract

from arXiv · show

A method for statistical parametric speech synthesis incorporating generative adversarial networks (GANs) is proposed. Although powerful deep neural networks (DNNs) techniques can be applied to artificially synthesize speech waveform, the synthetic speech quality is low compared with that of natural speech. One of the issues causing the quality degradation is an over-smoothing effect often observed in the generated speech parameters. A GAN introduced in this paper consists of two neural networks: a discriminator to distinguish natural and generated samples, and a generator to deceive the discriminator. In the proposed framework incorporating the GANs, the discriminator is trained to distinguish natural and generated speech parameters, while the acoustic models are trained to minimize the weighted sum of the conventional minimum generation loss and an adversarial loss for deceiving the discriminator. Since the objective of the GANs is to minimize the divergence (i.e., distribution difference) between the natural and generated speech parameters, the proposed method effectively alleviates the over-smoothing effect on the generated speech parameters. We evaluated the effectiveness for text-to-speech and voice conversion, and found that the proposed method can generate more natural spectral parameters and $F_0$ than conventional minimum generation error training algorithm regardless its hyper-parameter settings. Furthermore, we investigated the effect of the divergence of various GANs, and found that a Wasserstein GAN minimizing the Earth-Mover's distance works the best in terms of improving synthetic speech quality.

I. INTRODUCTION

Statistical parametric speech synthesis uses acoustic models for TTS and VC, but generated speech parameters are often over-smoothed and less natural than natural speech. The paper proposes GAN-based training to reduce distribution differences and evaluates it across TTS, VC, and GAN variants.

  • DNN acoustic models represent relationships between input features and acoustic features for both text-to-speech and voice conversion.
  • Generated speech parameters tend to be over-smoothed, leaving synthetic speech quality below natural speech in both TTS and VC.
  • Reducing the distribution difference between natural and generated speech parameters is identified as a route to improving synthetic speech quality.
  • The proposed GAN framework trains acoustic models with a weighted sum of conventional minimum generation error and adversarial losses.The discriminator distinguishes natural from generated samples, while the generator is trained to make generated parameters appear natural.
  • The method generated more natural spectral parameters and F0 than conventional MGE training in DNN-based TTS and VC, regardless of adversarial-loss weight settings.
  • Among the investigated GAN divergences, the Wasserstein GAN minimizing Earth-Mover’s distance produced the greatest synthetic speech-quality improvement.

2) Acoustic model training:

The conventional acoustic-model pipeline predicts speech and duration sequences from input features, using MSE or MGE objectives and dynamic-feature constraints. DNN-based VC uses aligned source and target features with the same MGE objective as DNN-based TTS.

  • 1) DNNs as Acoustic Models:: DNNs predict natural static-dynamic speech-feature sequences from input features, producing generated sequences for acoustic-model training.
  • 2) Acoustic model training:: The standard training criterion is mean squared error between natural and generated static-dynamic speech-feature sequences.
  • 2) Acoustic model training:: Minimum generation error training accounts for the static-dynamic constraint when defining the loss between natural and generated speech parameters.
  • 2) Acoustic model training:: The generated speech sequence is obtained by applying the speech-parameter prediction transform to the DNN output, and model parameters are updated by backpropagation.
  • 2) Acoustic model training:: Phoneme duration models predict duration sequences without dynamic-feature calculation and are trained by minimizing mean squared error.
  • 2) Acoustic model training:: For voice conversion, dynamic time warping aligns source and target features before acoustic models are trained with the same MGE objective used in TTS.

III. DNN-BASED SPSS INCORPORATING GAN

The proposed SPSS framework alternates discriminator and acoustic-model updates, combining conventional generation error with adversarial training to align generated and natural speech-parameter distributions. It also supports feature-based discrimination and language-dependent duration representations.

  • GAN training: GAN training alternates discriminator updates using natural and generated speech parameters with generator updates that deceive the discriminator.The discriminator and generator are iteratively optimized through minibatch stochastic gradient descent.
  • Acoustic-model training: The proposed acoustic-model loss combines conventional MGE with adversarial loss, whose weight is controlled by ωD.ωD = 0 yields conventional MGE training, while ωD = 1 assigns equal weights after scale normalization.
  • Acoustic-model training: Adversarial training makes generated speech-parameter distributions closer to natural distributions while retaining generation-error minimization.The adversarial loss trains the discriminator to recognize generated speech parameters as natural.
  • Feature-based discrimination: A feature function can be inserted before discrimination to compute more distinguishable anti-spoofing features from speech parameters.When φ(ŷ) = Wŷ, its transpose W^⊤ is used for acoustic-model backpropagation.
  • Duration representation: For duration modeling, the method can operate at a language-dependent isochrony level rather than directly on phoneme durations.The figures illustrate mora-level calculation for Japanese and matrix-based isochrony calculation for syllable-timed languages such as Chinese.

C. Application to F0 and Duration Generation

The method extends GAN-based training beyond spectral parameters to continuous F0 and duration generation. Duration training uses language-dependent isochrony representations, while alternative GAN divergences provide different distribution-matching objectives.

  • F0 generation: The method extends GAN training to continuous F0 by giving the discriminator each frame’s joint spectral-parameter vector and continuous F0 value.Continuous F0 is used instead of the F0 sequence for simpler implementation.
  • Duration generation: Duration generation is modified to match natural distributions at the target language’s isochrony level rather than only at the phoneme level.For Japanese, mora duration is calculated from corresponding phoneme durations; training combines phoneme-duration MSE with isochrony-level adversarial loss.
  • Alternative GAN divergences: The original GAN minimizes an approximated JS divergence, while the proposed framework also considers f-GAN, W-GAN, and LS-GAN objectives.The alternatives are introduced as different divergence-minimization formulations for natural and generated speech parameters.
  • Alternative GAN divergences: The f-GAN framework represents distribution differences with f-divergence, encompassing divergences such as KL and JS.The density functions p and q represent natural and generated data, and f is convex with f(1) = 0.

2) Wasserstein GAN (W-GAN) [23]:

W-GAN is introduced as an alternative divergence objective for stabilizing GAN training. It minimizes the Earth-Mover’s distance between natural and generated distributions under a Lipschitz discriminator assumption.

  • W-GAN objective: W-GAN minimizes the Earth-Mover’s distance, also called Wasserstein-1, to stabilize the unstable training of the original GAN.The distance is defined using joint distributions whose marginals are the natural and generated speech-parameter distributions.
  • W-GAN objective: The W-GAN discriminator loss is derived using Kantorovich–Rubinstein duality.The discriminator and adversarial losses are specified separately for the W-GAN formulation.
  • W-GAN constraint: The discriminator is assumed to be K-Lipschitz, enforced by clamping its weight parameters to a fixed interval after each update.An example interval is [−0.01, 0.01].

3) Least Squares GAN (LS-GAN) [24]:

LS-GAN replaces sigmoid cross-entropy with a mean-squared-error objective to address gradient vanishing. The paper uses label settings selected because the standard divergence conditions degraded synthetic speech quality.

  • LS-GAN formulation: LS-GAN formulates the discriminator objective as mean squared error to avoid the gradient-vanishing problem associated with sigmoid cross-entropy.The discriminator loss and generator adversarial loss are defined within this least-squares formulation.
  • LS-GAN formulation: The labels a, b, and c designate generated-as-generated, natural-as-natural, and generated-as-natural discriminator targets, respectively.These labels determine the divergence associated with the LS-GAN objective.
  • LS-GAN formulation: The paper uses a = 0, b = 1, and c = 1 because the alternative conditions b − c = 1 and b − a = 2 degraded synthetic speech quality.Under those alternative conditions, the minimized divergence is a Pearson X^2 divergence between specified mixtures of natural and generated distributions.

E. Discussions

The proposed adversarial training makes generated speech-parameter distributions more like natural speech, affecting spread, variance, and correlations beyond conventional MGE. Its flexible feature design also supports joint and multi-resolution distribution compensation.

  • Method connections: The framework can use analytically or automatically derived discriminator features and accommodate joint-feature, multi-resolution, and hierarchical duration modeling.The method is fully supervised and can use DNN architectures such as LSTM.
  • Distributional effects: The proposed algorithm produces mel-cepstral coefficients with a distribution spread closer to natural speech than conventional MGE.The conventional algorithm’s parameters are narrowly distributed, whereas the proposed parameters are as widely distributed as natural parameters.
  • Distributional effects: The proposed algorithm has a greater effect on the distribution of higher-order mel-cepstral coefficients.
  • Distributional effects: Generated global variances are closer to natural global variances with the proposed algorithm than with conventional MGE.The discussion relates distribution compensation to minimizing moment differences.
  • Distributional effects: The proposed algorithm reduces correlations among generated mel-cepstral coefficients relative to MGE, while compensating distributional differences.MIC values closer to 1 indicate stronger nonlinear correlation.
  • Other acoustic parameters: For F0 and duration parameters, the proposed method generally makes statistics closer to natural speech, but mora-duration compensation tends to reduce mean differences more than variance differences.

IV. EXPERIMENTAL EVALUATION

The experiments evaluate the proposed algorithm in DNN-based TTS and VC using spectral parameters, F0, and duration generation, with controlled speech data and model configurations.

  • Evaluation scope: The evaluation covers spectral-parameter, F0, and duration generation in DNN-based TTS, plus spectral-parameter conversion in DNN-based VC.
  • Data: The dataset contains 503 sentences from one male speaker, split into 450 training sentences and 53 evaluation sentences.Speech was sampled at 16 kHz with a 5 ms shift.
  • Model configuration: Spectral models predict 75-dimensional static-dynamic mel-cepstral sequences from 274-dimensional linguistic features, while discriminators use 25-dimensional static mel-cepstral vectors.
  • Model configuration: For joint spectral and F0 discrimination, the discriminator uses a 26-dimensional vector combining static mel-cepstral coefficients with continuous log F0.
  • Training procedure: Acoustic models were initialized with frame-wise MSE training, followed by 25 iterations of conventional MGE training; discriminators were initialized for five iterations.
  • Metrics: Evaluation uses parameter-generation loss and spoofing rate, where spoofing rate is the fraction of synthetic parameters classified as natural by the discriminator.

1) Objective Evaluation with Hyper-parameter Settings:

Objective and subjective TTS evaluations show that adversarial training can trade higher generation loss for greater discriminator spoofing and improved speech quality across tested settings and F0 generation.

  • Objective evaluation: As ωD increases, generation loss generally rises, while spoofing rate increases sharply from 0.0 to 0.2 and then changes little.
  • Objective evaluation: The method can worsen generation loss while reducing the distribution difference between natural and generated speech parameters through adversarial training.
  • Convergence: Training losses for training data decrease almost monotonically, while evaluation losses vary after several iterations before converging.
  • Subjective evaluation: The proposed algorithm outperforms conventional MGE in both tested hyper-parameter settings, with ωD = 1.0 slightly better than ωD = 0.3.
  • F0 evaluation: The Proposed (sp+F0) method receives a much higher preference score than Proposed (sp) and MGE, supporting effectiveness for both spectral parameters and F0.

5) Subjective Evaluation of Duration Generation:

Duration-generation preference scores show no significant differences, whereas the method outperforms conventional GV compensation and identity features outperform static-and-delta features in speech-quality tests.

  • Duration generation: Duration tests compare conventional MSE with the proposed method applied to phoneme or mora duration using preference AB evaluation.
  • Duration generation: Duration-generation preference scores show no significant differences between the tested methods.A discriminator using durations had lower accuracy than the corresponding discriminator using other speech features.
  • Duration generation: The results suggest that distribution compensation does not work well for duration generation, so the proposed algorithm was not applied to subsequent duration generation.
  • GV compensation: The proposed method achieves a higher speech-quality preference score than conventional MGE-GV post-filtering.
  • Anti-spoofing features: Using static-and-delta features in anti-spoofing yields a much lower speech-quality score than using the identity feature.Although static-and-delta features distinguish natural and synthetic speech effectively, they do not improve speech quality.

8) Subjective Evaluation Using Complicated Model Architecture:

Evaluations with more complicated LSTM architectures confirmed that the proposed method improves speech quality over conventional MGE training. Across GAN divergences and voice-conversion tests, W-GAN performed best overall, while the proposed method also improved speaker individuality.

  • Subjective Evaluation Using Complicated Model Architecture:: The proposed method outperformed conventional MGE training when both acoustic models and discriminator used two-layer unidirectional LSTMs.The acoustic models used 256 memory cells and the discriminator used 128.
  • Effect of Divergence of GAN:: MOS evaluations found improvements for all tested divergences except KL-GAN and JS-GAN.RKL-GAN improved quality where KL-GAN did not, while an approximated JS-based GAN worked better than JS-GAN.
  • Effect of Divergence of GAN:: W-GAN achieved the highest synthetic-speech quality, with MOS significantly above LS-GAN, JS-GAN, and KL-GAN.The comparison used randomized presentation of synthetic speech to 55 listeners.
  • Voice Conversion Evaluation:: In DNN-based voice conversion, the proposed algorithm improved converted-speech quality relative to MGE training and also improved speaker individuality.The authors associate the individuality improvement with compensating global variances of generated speech parameters.
  • Conclusion:: The conclusion reports significant speech-quality improvements in both TTS and VC regardless of the adversarial-loss hyper-parameter settings.The method also compensated correlations among generated speech parameters, in addition to global variance.
Loading 1709.08041v1…