PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, Nima Mesgarani

arXiv:2306.07691v2eess.AScs.AIcs.CLcs.LGcs.SD

TL;DR

Human-level TTS still faces challenges in expressive, robust, and zero-shot synthesis. StyleTTS 2 combines text-dependent style diffusion with SLM-based adversarial training, surpassing human recordings on LJSpeech and matching them on VCTK.

  • Problem

    Robust and accessible human-level TTS remains limited by expressive diversity, out-of-distribution robustness, and massive-data requirements for high-performing zero-shot systems.

  • Method

    StyleTTS 2 samples text-dependent speech styles via diffusion and uses differentiable duration modeling with large pre-trained speech language model discriminators for end-to-end training.

  • Results

    StyleTTS 2 exceeds human recordings on LJSpeech, matches human performance on VCTK, and outperforms previous publicly available models for zero-shot speaker adaptation.

  • Takeaways & Limitations

    StyleTTS 2 demonstrates human-level TTS across single- and multispeaker datasets while generating expressive, diverse speech without reference audio.

  • Takeaways & Limitations

    Zero-shot speaker adaptation can enable voice misuse and deception, including fraud, harassment, and impersonation.

Abstract

from arXiv · show

In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https://styletts2.github.io/.

1 Introduction

StyleTTS 2 models speech style as a latent random variable sampled by diffusion, enabling realistic reference-free synthesis. Evaluations report human-level or better naturalness, including statistically significant gains on LJSpeech and versus NaturalSpeech.

  • Motivation: Robust and accessible human-level TTS remains challenging despite recent progress toward human-level performance.The introduction identifies remaining room for improvement in TTS synthesis.
  • Approach: StyleTTS 2 samples speech styles as latent random variables with a probabilistic diffusion model, without requiring reference audio.Because it samples a style vector rather than the entire speech as a latent variable, the model can efficiently synthesize realistic speech.
  • Evaluation: +0.28 CMOS (p < 0.05) shows StyleTTS 2 surpassing human recordings on LJSpeech, as judged by native English speakers.The result is statistically significant on the LJSpeech benchmark dataset.
  • Evaluation: +1.07 CMOS (p ≪0.01) shows StyleTTS 2 outperforming NaturalSpeech on the reported comparison.The paper describes this result as advancing the state of the art.

2 Related Work

Related work explores diffusion-based speech synthesis, large speech language models for TTS and speaker adaptation, and approaches toward human-level TTS. These methods offer improved control or quality but retain efficiency or optimization challenges.

  • Diffusion Models for Speech Synthesis: Diffusion models support diverse speech sampling and fine-grained control across mel-based TTS, vocoding, and end-to-end speech generation, but require iterative sampling.Their efficiency is limited compared with non-iterative GAN-based models.
  • Text-to-Speech with Large Speech Language Models: Large self-supervised speech language models improve TTS quality and speaker adaptation, commonly providing continuous or quantized representations for speech reconstruction.However, SLM features are not directly optimized for speech synthesis.
  • Human-Level Text-to-Speech: Recent human-level TTS systems use BERT pre-training, end-to-end training, and differentiable duration modeling; VITS achieves MOS comparable to human recordings on LJSpeech and VCTK.PnG-BERT obtains human-level results on a proprietary dataset.

3 Methods

StyleTTS 2 replaces reference-dependent deterministic style generation with conditional latent style diffusion and combines direct waveform synthesis, end-to-end optimization, differentiable duration modeling, and adversarial training with large speech language models. Its method also introduces separate acoustic and prosodic style encoders to stabilize joint training and supports speaker-conditioned style sampling for multispeaker synthesis.

  • End-to-end waveform synthesis: StyleTTS 2 jointly optimizes all TTS components with direct waveform synthesis, avoiding fixed pre-trained vocoders during inference.The decoder is modified to generate waveforms directly from style, aligned phoneme representations, pitch, and energy.
  • Latent style diffusion: The model samples generalized speech style as a conditional latent variable using an EDM-based diffusion process instead of deterministic style generation.Generalized style covers speech characteristics beyond phonetic content, including prosody, lexical stress, formant transitions, and speaking rate.
  • Training stabilization: Separate acoustic and prosodic style encoders are introduced because joint training can produce diverging gradients when one style encoder represents both information types.Acoustic-module pre-training may accelerate TTS-prediction training before joint optimization, but it is not described as essential.
  • Latent style diffusion: In multispeaker synthesis, style diffusion conditions on a target-speaker embedding extracted from reference audio and injects it into the denoiser through adaptive layer normalization.The speaker embedding is c = E(xref), where xref is the target speaker’s reference audio.
  • SLM adversarial training: A 12-layer WavLM discriminator pre-trained on 94k hours of data transfers speech-language-model representations to generation through adversarial training.SLM representations capture acoustic and semantic information and can mimic human perception in synthesized-speech evaluation.
  • Differentiable duration modeling: Differentiable duration modeling enables gradient flow from waveform generation through phoneme upsampling, while Gaussian and attention-based alternatives can be unstable or insufficiently flexible.The proposed approach addresses nondifferentiable duration upsampling and limitations from fixed Gaussian widths or unstable trainable parameters under adversarial training.

4 Experiments

Section 4 evaluates StyleTTS 2 on LJSpeech, VCTK, and LibriTTS using naturalness and, for multispeaker models, similarity ratings from native English speakers, with comparisons against strong public baselines and controlled evaluation procedures.

  • Datasets: Experiments covered single-speaker LJSpeech, multispeaker VCTK, and LibriTTS for zero-shot speaker adaptation.LJSpeech contained 13,100 clips totaling roughly 24 hours; VCTK contained nearly 44,000 clips from 109 native speakers with varied accents.
  • Evaluation metrics: Evaluations used MOS-N for naturalness and MOS-S for reference similarity in multispeaker models, rated by native U.S. English speakers on Amazon Mechanical Turk.Evaluators reported normal hearing and provided informed consent under institutional review board oversight.
  • Baselines and controls: Baselines included VITS, StyleTTS, and JETS for LJSpeech, and VITS, YourTTS, and StyleTTS for LibriTTS.Most baselines used official author checkpoints; audio was resampled to 22.5 kHz for LJSpeech and VCTK and 16 kHz for LibriTTS for fairness.
  • Evaluation protocol: Each speech set was rated by 5 to 10 evaluators on a randomized, label-hidden 1–5 scale with 0.5-point increments, complemented by CMOS significance tests.CMOS ratings compared paired samples on a -6 to 6 scale with increments of 1.

5 Results

StyleTTS 2 achieves strong naturalness and expressiveness, outperforming prior systems on LJSpeech and maintaining quality for out-of-distribution text. Style diffusion separates emotional styles, while ablations confirm the importance of text-dependent diffusion, differentiable upsampling, and SLM discrimination.

  • LJSpeech results: StyleTTS 2 outperforms NaturalSpeech with a CMOS of +1.07 and is preferred over ground truth with a CMOS of +0.28 on LJSpeech.The ground-truth preference may reflect fragmented audiobook passages that disrupt narrative continuity.
  • LJSpeech results: 3.83 MOS surpasses all previous LJSpeech models, and StyleTTS 2 shows no quality degradation for out-of-distribution texts.Other models degraded on OOD texts, while StyleTTS 2 significantly outperformed them in MOS.
  • Limitations: MOS-S differences between StyleTTS and StyleTTS 2 are not statistically significant, indicating remaining room for improvement in speaker similarity.This finding is reported as a future direction despite the model’s broader results.
  • Style diffusion: Style diffusion produces distinct emotional style clusters from text sentiment, enabling expressive speech across five emotions without explicit emotion labels during training.The visualization includes styles from both seen and unseen speakers.
  • Ablation study: −0.46 CMOS results from replacing style-diffusion vectors with random encodings, while removing the differentiable upsampler or SLM discriminator yields −0.21 or −0.32 CMOS.These ablations highlight the contributions of text-dependent style diffusion, differentiable upsampling, and SLM discrimination to natural speech synthesis.

6 Conclusions and Limitations … A.2 Speech Expressiveness

StyleTTS 2 achieves human-level TTS quality while showing promise for zero-shot speaker adaptation, but its evaluations and potential misuse impose important limitations. Additional analyses find that the model generates expressive emotional speech with broader distributional coverage than comparison systems.

  • 6 Conclusions and Limitations: StyleTTS 2 exceeds ground truth on LJSpeech, matches it on VCTK, and shows promising zero-shot speaker adaptation with limited training data.The paper contrasts this performance with large-scale models such as Vall-E.
  • 6 Conclusions and Limitations: Zero-shot speaker adaptation could enable voice imitation for misinformation, fraud, harassment, impersonation, or political influence, motivating a required user code of conduct.The authors state that the code of conduct will be clearly displayed.
  • 6 Conclusions and Limitations: Human preference for StyleTTS 2 over LJSpeech ground truth may depend on isolated audiobook clips and differences between human speech variability and synthesized speech.These factors could skew evaluations toward synthesized speech.
  • A.1 Feedback Analysis from Survey Participants: Survey participants often found LJSpeech sample differences negligible or difficult to discern, strengthening the conclusion that StyleTTS 2 reached human-like synthesis quality.Participants also described the reference reader as professional and expressive.
  • A.1 Feedback Analysis from Survey Participants: Participants noted that excessive expressiveness could sound unnatural or shift perceived emotion and context, despite sometimes making speech feel more natural.Some comments characterized overly dramatic delivery as resembling an actor auditioning for a role.
  • A.1 Feedback Analysis from Survey Participants: Differences between StyleTTS 2 and original audio mainly involved subtle intonation and word-emphasis nuances, while StyleTTS 2 scored statistically higher in MOS and CMOS.The authors attribute this result partly to disrupted narrative continuity in isolated LJSpeech audiobook clips.
  • A.2 Speech Expressiveness: StyleTTS 2 synthesized diverse, expressive speech across 500 GPT-4-generated text samples spanning five emotions, evaluated using mean F0 and energy curves.The analysis maps emotion-associated style vectors with t-SNE.
  • A.2 Speech Expressiveness: StyleTTS 2 produced emotion-specific F0 and energy patterns, covered the ground-truth distributions, and showed better mode coverage than VITS while outperforming deterministic JETS in distributional range.The authors associate its slightly better right-tail energy coverage with diffusion-based modeling rather than VITS’s variational autoencoder.

A.3 Ablation Study · Appendix B Style Diffusion · B.1 EDM Formulation

The ablation study identifies style diffusion as the most important component for natural, ground-truth-like speech, while other components affect specific objective or subjective measures. Appendix B formulates the style diffusion sampler with EDM and specifies its empirically selected noise schedule for fast, high-quality sampling.

  • A.3 Ablation Study: The ablation evaluates MCD, MCD-SL, F0 RMSE, DUR MAD, WER, and subjective CMOS on LJSpeech.MCD-SL measures both alignment length and quality, and the CMOS results are copied from Table 5 as a reference.
  • A.3 Ablation Study: Style diffusion is the most important factor for producing speech close to ground truth and natural human speech.Randomly encoded style vectors considerably affect all metrics, including subjective CMOS, because style influences pauses, emotions, speaking rates, and sound quality.
  • A.3 Ablation Study: Removing large-SLM adversarial training slightly worsens MCD-SL and F0 RMSE but leaves WER unchanged.It yields the lowest duration error, suggesting minor in-distribution underfitting, while subjective evaluations show a significant difference for OOD texts.
  • A.3 Ablation Study: Removing the differentiable upsampler increases F0 RMSE while leaving MCD, MCD-SL, DUR MAD, and WER unaffected.Removing the prosodic style encoder affects all metric scores, whereas omitting OOD texts affects only F0 RMSE objectively.
  • Appendix B Style Diffusion: EDM formulates diffusion-model sampling as an ordinary differential equation and rewrites it in terms of the noise level σ.The noise schedule and derivative are σ(t) and σ̇(t), while the score function is ∇x log p(x; σ(t)).
  • B.1 EDM Formulation: EDM trains a denoiser with a denoising score matching objective using noisy samples, a trainable network Fθ, and σ-dependent scaling functions.The formulation includes skip, input, output, and noise-conditioning transformations, and defines the score function from the denoiser.
  • B.1 EDM Formulation: σmin = 0.0001, σmax = 3, and ρ = 9 define the sampling schedule used for fast sampling with small step sizes and high-quality speech.The schedule shortens steps near σmin; prior work observed optimal performance when σmax ≫ σdata and ρ ∈[5, 10].

B.2 Effects of Diffusion Steps … C.1 DSP-Based Formulation

StyleTTS 2 finds that around 16 diffusion steps provide a strong balance of sample quality, diversity, and speed, while interpolated style vectors improve long-form consistency. The model also separates speech content and style for style transfer and formulates differentiable duration upsampling through DSP principles.

  • B.2 Effects of Diffusion Steps: Training randomly sampled 3–5 diffusion steps because satisfactory-quality samples were possible with as few as three steps, saving time and GPU RAM.Experiments evaluated diffusion steps from 4 to 128, while two-step results were disregarded because of the EDM noise scheduler.
  • B.2 Effects of Diffusion Steps: Around 16 diffusion steps achieved optimal sample quality and diversity with acceptable computational speed.Quality differences across step counts were negligible, while diversity increased and plateaued around 16 steps.
  • B.2 Effects of Diffusion Steps: A 30% RTF increase over 4 steps still left the model twice as fast as VITS for real-time applications.The comparison supports selecting around 16 steps despite the additional computation.
  • B.3 Consistent Long-Form Generation: For long-form synthesis, each sentence receives a style vector interpolated with the previous sentence’s style to reduce inconsistent speaking styles.The algorithm splits a paragraph into sentences, samples the current style, applies αs_current + (1 − α)s_previous, synthesizes speech, and concatenates the outputs.
  • B.4 Style Transfer: Style diffusion separates speech content from style, allowing a style vector derived from emotionally or stylistically specific text to condition any input text.Style examples include emotions, speaking rates, and recording environments.
  • C.1 DSP-Based Formulation: The differentiable duration upsampler is formulated in DSP terms using convolutional shifting by Kronecker deltas and Gaussian-kernel approximation of the Dirac delta.A properly normalized Gaussian kernel converges to the delta function as σ → 0, enabling approximate position shifting through suitable σ selection.
  • Appendix C Differentiable Duration Modeling: The duration predictor remains trainable end-to-end with the SLM discriminator even when predicted alignments differ with and without the differentiable duration upsampler.The supplied passage reports that these alignment differences are perceptually indistinguishable in synthesized speech.

C.2 Effects of σ on Training Stability and Sample Quality · Appendix D SLM Discriminators

The analysis studies how σ balances training stability and sample quality in differentiable upsampling. σ = 1.5 minimizes both measures and matches typical phoneme-duration structure.

  • C.2 Effects of σ on Training Stability and Sample Quality: The gradient transmitted from the SLM discriminator to the duration predictor is computed through the chain rule as a product of individual gradient norms.This propagation is identified as crucial for model performance.
  • C.2 Effects of σ on Training Stability and Sample Quality: As σ approaches zero, Nc(x; σ) approximates δc(x), but its gradient norm diverges as O(1/σ^k).Here, k depends on the input dimension.
  • C.2 Effects of σ on Training Stability and Sample Quality: Smaller σ values can worsen numerical stability, especially when Nc(x; σ) multiplies small predictor outputs q ∈[0, 1].Thus, increasingly sharp δ approximations do not necessarily improve training behavior.
  • C.2 Effects of σ on Training Stability and Sample Quality: The study evaluates σ using gradient norms from the SLM discriminator to the duration predictor and MCD between differentiable and non-differentiable upsampling.Training stability uses the maximum gradient over an epoch; sample quality uses mel cepstral distortion.
  • C.2 Effects of σ on Training Stability and Sample Quality: Figure 6 compares MCD and maximum discriminator-to-duration-predictor gradient norm across different σ values.The figure marks σ = 1.5 with a star and evaluates both sample quality and training stability.
  • C.2 Effects of σ on Training Stability and Sample Quality: σ = 1.5 minimizes both MCD and gradient norm across a wide range of σ values, making it the selected upsampling setting.Gaussian kernels with σ = 1.5 span approximately 3 speech frames, aligning with typical phoneme durations of 2 to 3 frames.

D.1 Layer-Wise Analysis … E.1 Proposed Framework for Subjective Evaluation

The analysis shows that StyleTTS 2’s SLM discriminator combines acoustic and semantic cues, while training stability requires gradient scaling. The paper also proposes standardized subjective-evaluation practices emphasizing qualified raters, controlled comparisons, and statistical testing.

  • D.1 Layer-Wise Analysis: In LJSpeech and LibriTTS, initial layers 1–2 and middle layers 6–7 were most important, whereas final layers 10–11 were least influential.Layers 0–2 encode acoustic information, while the final layer primarily supports pretext tasks.
  • D.1 Layer-Wise Analysis: The SLM discriminator fuses acoustic and semantic information to derive paralinguistic attributes, enhancing synthesized speech’s emotional expressiveness and prosody.Acoustic cues include energy, pitch, and SNR; semantic cues include word identity and meaning.
  • D.1 Layer-Wise Analysis: The VCTK discriminator showed no distinct layer preference, likely because its standard-paragraph recordings provide limited contextual or emotional information.This context shortage may explain the model’s smaller improvement over VITS on VCTK than on LJSpeech and LibriTTS.
  • D.2 Training Stability: 200 maximum gradient norm from the SLM discriminator to the duration predictor can destabilize training, compared with generally less than 10 for other modules.The risk is especially concerning for the BERT-based prosodic text encoder, which is sensitive to gradient explosion.
  • D.2 Training Stability: A 0.2 scaling factor is applied when the predictor’s gradient norm exceeds 20, while the duration predictor’s final projection and LSTM gradients receive 0.01 scaling.These measures address instability from both discriminator gradients and sigmoid probability outputs.
  • Appendix E Subjective Evaluation Procedures: The paper presents a detailed and standardized procedure for evaluating human-like text-to-speech systems.The procedure is framed as adherence to recommended subjective-evaluation guidelines.
  • E.1 Proposed Framework for Subjective Evaluation: Evaluations should use verified native speakers, attention checks, explicit objectives, publicly available official implementations, and paired MUSHRA comparisons rather than isolated MOS ratings.Raters failing attention checks should be excluded, and naturalness definitions should be stated explicitly.
  • E.1 Proposed Framework for Subjective Evaluation: CMOS should replace MOS and MUSHRA for nuanced comparisons and significance testing, using at least 20 raters and more than 50 samples against ground truth.The recommendation targets claims of human-level quality.

E.2 Evaluation Details

Evaluation protocols used screened native U.S. English speakers, attention checks, explicit rating definitions, official checkpoints, MUSHRA-based MOS, and controlled CMOS rater counts. CMOS screening eliminated six raters across experiments.

  • Participant screening: Participants were self-identified native U.S. speakers, verified by residential IP addresses, with MTurk filters requiring approval rates above 95% and more than 50 approved HITs.The location filter required UNITED STATES (US), and proxy or VPN addresses were excluded.
  • Attention checks: MOS attention checks excluded participants whose ground-truth average score failed to rank among the top three of five models.CMOS raters were disqualified when scoring signs differed across more than half of 10 samples; six raters were eliminated in all experiments.
  • Rating criteria: Surveys defined naturalness and similarity on 1-to-5 scales, including native-speaker perception, same-speaker judgments, accent, and speaking habits.Participants were instructed to listen beyond sample distortion when identifying voices.
  • Model checkpoints: Official author-provided checkpoints were used for MOS comparisons, including VITS, JETS, StyleTTS with HiFi-GAN, and YourTTS where applicable.The LibriTTS MOS evaluation used the YourTTS Exp 4 checkpoint; no official VITS checkpoint was available for LibriTTS.
  • Evaluation design: MOS evaluations used a MUSHRA-based approach, while CMOS experiments assigned 20 raters per sample after excluding ineligible raters.NaturalSpeech and Vall-E used doubled rater counts because additional samples were unavailable.

Appendix F Detailed Model Architectures · Appendix G Detailed Training Objectives

Appendix F details StyleTTS 2’s expanded architecture, including style and prosody components, denoising, discrimination, and decoding modules. Appendix G specifies acoustic-module pre-training followed by joint training with the pitch extractor fixed.

  • Appendix F Detailed Model Architectures: StyleTTS 2 integrates eight original StyleTTS modules with an added style diffusion denoiser, prosodic style encoder, and prosodic text encoder.This expands the original architecture with dedicated style-diffusion and prosody components.
  • Appendix F Detailed Model Architectures: The acoustic text encoder, text aligner, and pitch extractor retain the original StyleTTS architecture.The acoustic and prosodic style encoders follow the original StyleTTS style encoder.
  • Appendix F Detailed Model Architectures: The prosodic text encoder uses pre-trained PL-BERT, while MPD and MRD remain the adopted discriminators.The decoder combines the original StyleTTS decoder with either iSTFTNe.
  • Appendix F Detailed Model Architectures: The denoiser processes style vector s, noise level σ, prosodic phoneme embeddings hbert, and optionally speaker embedding c for multispeaker synthesis.The speaker embedding c is introduced through adaptive layer normalization.
  • Appendix F Detailed Model Architectures: Adaptive layer normalization computes gain and bias from style vector s using learned projections Lσ and Lµ.The formulation applies these parameters to feature maps x using layer mean and standard deviation.
  • Appendix F Detailed Model Architectures: The denoiser inputs have sizes s and c of 256 × 1, hbert of 768 × N, and σ of 1 × 1.Here, N denotes the input phoneme length of the mel-spectrogram.
  • Appendix F Detailed Model Architectures: The discriminative head is a 3-layer convolutional neural network followed by a linear projection.T denotes the frame length of WavLM’s output feature hslm.
  • Appendix G Detailed Training Objectives: Training first pre-trains the acoustic modules for accelerated training, then performs joint training while keeping the pitch extractor fixed.The two-stage procedure is specified for the acoustic modules’ pre-training and joint-training objectives.

G.1 Acoustic module pre-training · G.2 Joint training

Acoustic pre-training combines reconstruction, alignment, and adversarial objectives to train the acoustic modules. Joint training adds differentiable duration and prosody prediction, reconstructs speech from predicted attributes, and incorporates SLM adversarial learning into the full objective.

  • G.1 Acoustic module pre-training: Acoustic pre-training uses encoded phoneme representations, attention alignment, acoustic style, pitch, and energy to condition mel-spectrogram reconstruction.Raw attention is used half the time, while a monotonic attention version is used for the other 50%.
  • G.1 Acoustic module pre-training: Acoustic pre-training fine-tunes the text aligner with sequence-to-sequence ASR loss and constrains soft attention toward a differentiable monotonic alignment.The monotonic alignment is obtained through dynamic programming.
  • G.1 Acoustic module pre-training: Acoustic pre-training enhances reconstructed waveform quality with HifiGAN-based LSGAN adversarial and feature-matching losses, plus truncated pointwise relativistic loss.The truncation factor τ is set to 0.04.
  • G.1 Acoustic module pre-training: The acoustic pre-training objective combines reconstruction, alignment, adversarial, relativistic, and feature-matching losses, weighted by λs2s and λmono.The full objective explicitly includes Ladv(G; D), Lrel(G; D), and Lfm.
  • G.2 Joint training: Joint training optimizes duration prediction with cross-entropy and L-1 losses, using ground-truth durations from attention and predicted durations for phoneme timing.The predictor conditions on prosodic style and prosodic text embeddings.
  • G.2 Joint training: Joint training reconstructs pitch and energy with Lf0 and Ln, then uses predicted pitch and energy for mel-spectrogram reconstruction and related adversarial losses.Predicted pitch and energy are generated from prosodic text embeddings and prosodic style.
  • G.2 Joint training: The SLM adversarial objective uses LSGAN loss when the discriminator overpowers the generator, with generated speech conditioned on sampled styles and predicted alignment, pitch, and energy.Predicted alignment comes from a differentiable duration upsampler applied to predicted duration.
  • G.2 Joint training: The joint objective combines duration, prosody, reconstruction, alignment, GAN, SLM, and embedding-discriminator losses, with λs2s = 0.2, λmono = 5, λdur = 1, λf0 = 0.1, λn = 1, and λce = 1.The discriminator objective includes Ladv(D; G), Lrel(D; G), and Lslm(D; G).
Loading 2306.07691v2…