PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt

Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, Helen Meng

arXiv:2301.13662v2cs.SDeess.AS

TL;DR

Expressive TTS needs more flexible and interpretable style control than categorical labels or reference speech provide. InstructTTS introduces natural-language prompting with cross-modal style representations, discrete VQ acoustic modeling, and mutual-information minimization. The reported evaluations show improved speech quality, prompt relevance, prosody, and emotion classification relative to the baseline, with inference speed and dataset scale remaining limitations.

  • Problem

    Existing expressive TTS controls styles through predefined labels or reference speech, limiting style diversity or interpretability; natural-language prompting poses semantic and acoustic-modeling challenges.

  • Method

    InstructTTS uses a three-stage prompt-embedding strategy, discrete diffusion over VQ acoustic features, and mutual-information minimization for style-speaker and style-content leakage.

  • Results

    InstructTTS outperforms the baseline in speech quality, prosody, prompt relevance, and emotion classification, while Mel-VQ favors quality and Wave-VQ favors prosody detail.

  • Takeaways & Limitations

    Natural-language prompts can control expressive speech through discrete-latent acoustic modeling and cross-modal style representations.

  • Takeaways & Limitations

    Inference speed is limited by the large diffusion step count, and the authors plan to build a larger dataset.

Abstract

from arXiv · show

Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, e.g., "Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named as InstructTTS, which is novel in the sense of following aspects: (1) We fully take the advantage of self-supervised learning and cross-modal metric learning, and propose a novel three-stage training procedure to obtain a robust sentence embedding model, which can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt.

I. INTRODUCTION

Expressive TTS seeks user-controlled speaking styles, but categorical labels restrict diversity and reference speech provides unintuitive style information. InstructTTS addresses natural-language style prompting with a new corpus and discrete-latent acoustic modeling.

  • Expressive TTS controls emotion, speaking rate, and other speaking styles according to user demands.
  • Categorical style labels restrict synthesis to a few predefined styles, while reference speech yields style information that may not be intuitive or interpretable.
  • Natural-language style prompts create challenges in semantic prompt representation and one-to-many acoustic modeling.
  • InstructTTS introduces natural-language prompting, a three-stage sentence-embedding strategy, discrete VQ acoustic modeling, and discrete diffusion generation.
  • The study targets free-form natural-language descriptions rather than constrained style-factor words and focuses on Mandarin Chinese TTS.

D. Diffusion Probabilistic Models

Discrete diffusion models operate over categorical latent states through a forward corruption process and a learned reverse denoising process. The paper situates this approach within expressive TTS experiments using a Mandarin corpus with natural-language style annotations.

  • The forward process progressively corrupts data across fixed steps, while the reverse process denoises toward the original data distribution.
  • Discrete diffusion constrains each variable to one of K categorical states and represents transitions with a probability matrix.
  • NLSpeech contains 44 hours and 32k Mandarin Chinese utterances from 7 speakers, with five style prompts per utterance.
  • Annotators provide an overall emotion word, an emotion-level word, and a complete natural-language style sentence for each utterance.

IV. PROPOSED METHOD

InstructTTS combines content and style encoders with a SALN adaptor and discrete diffusion decoder. Its style encoder aligns prompt and audio representations while reducing speaker and content information in the extracted style representation.

  • The framework contains content and style encoders, a speaker embedding module, a SALN adaptor, and a discrete diffusion decoder.
  • Cross-modal metric learning trains audio-text retrieval using positive style-prompt/audio pairs and randomly selected negative samples.
  • The content encoder follows FastSpeech2 and predicts duration, pitch, and related variance information through a variance adaptor.
  • The style encoder uses RoBERTa prompt embeddings and a three-stage strategy designed for semantic, smooth, and generalizable prompt representations.
  • The audio encoder is trained to minimize style-speaker and style-content mutual information, limiting speaker and content leakage.

D. Modelling Mel-spectrograms in Discrete Latent Space

The paper models mel-spectrograms in a discrete latent space, using Mel-VQ-VAE to convert acoustic features into quantized tokens and a discrete diffusion decoder to generate them. This addresses the complex correlations and prediction gap encountered when modeling expressive mel-spectrograms directly.

  • Mel-VQ-VAE: Mel-spectrogram modeling is recast as discrete-token prediction through a pre-trained Mel-VQ-VAE and a Mel-VQ-Diffusion decoder.The VQ-VAE encodes mel-spectrograms into codebook entries, while the decoder generates quantization tokens before reconstructing the mel-spectrogram.
  • Mel-VQ-VAE: Direct mel-spectrogram prediction is difficult because frequency bins are highly correlated across time and frequency, especially for expressive speech.Prediction errors between ground-truth and generated mel-spectrograms can also affect synthesis performance.
  • Mel-VQ-VAE: The Mel-VQ-VAE contains a mel encoder, mel decoder, and codebook that quantizes latent features into discrete spectrogram representations.The decoder reconstructs the mel-spectrogram from quantized codes, with adversarial loss used to improve reconstruction.
  • Mel-VQ-Diffusion decoder: Mel-VQ-Diffusion corrupts ground-truth token sequences toward a stationary distribution and trains a Transformer to recover them conditioned on phone, style, and speaker features.The transition process uses masking and uniform resampling, and the decoder is trained to estimate posterior transitions by minimizing a variational lower bound.
  • Mel-VQ-Diffusion decoder: Classifier-free guidance strengthens the connection between conditional information and generated mel-spectrogram tokens during diffusion decoding.A hyperparameter λ controls the degree of posterior constraint, with conditional and unconditional predictions combined during inference.

E. Modelling Waveform in Discrete Latent Space Via Multiple Vector Quantizers

The paper also investigates waveform modeling in discrete latent space with multiple vector quantizers. This direction targets high-quality synthesis while addressing the long token sequences created by neural audio codecs.

  • Multiple vector quantizers: Wave-VQ-Diffusion uses a non-autoregressive discrete diffusion model intended to improve synthesis speed while maintaining high-quality synthesis performance.The approach is presented as an alternative to VALL-E's two-stage system with an autoregressive first stage.
  • Multiple vector quantizers: Neural audio codec models use more codebooks than Mel-VQ-VAE, improving reconstruction performance but producing longer token sequences.For a 10-second, 24-kHz speech sample, eight codebooks and 240-times downsampling produce 8000 tokens.
  • Multiple vector quantizers: Modeling these long codec-token sequences with Transformers is challenging because Transformer computational complexity depends on sequence length.The resulting sequence length creates practical GPU-memory constraints.

1) Wave-VQ-Diffusion:

Wave-VQ-Diffusion adapts discrete diffusion to multiple neural-audio-codec codebooks. Its architecture and transition process account for differences among codebooks and their progressively decreasing information content.

  • 1) Wave-VQ-Diffusion:: Wave-VQ-Diffusion uses a U-transformer to model multiple codebooks simultaneously.The U-transformer combines U-net downsampling and upsampling blocks with a denoising transformer.
  • 1) Wave-VQ-Diffusion:: Different embedding tables are used for different codebooks because their tokens follow different data distributions.
  • 1) Wave-VQ-Diffusion:: The method introduces an improved mask-and-uniform strategy for the diffusion process.The strategy dynamically allocates different weights to codebooks when defining the transition matrix.
  • 1) Wave-VQ-Diffusion:: The codebooks are concatenated from the first to the last, while the transition design reflects progressively decreasing information across residual layers.The token position and residual-layer index determine the transition-related quantities.

F. The Training and Inference Details

InstructTTS is trained end to end with diffusion, variance, and mutual-information objectives, then generates speech by iteratively denoising sampled discrete tokens. The VQ model is trained using 669 hours of speech data.

  • 1) Training objective: The overall training objective combines diffusion loss, variance reconstruction loss, and mutual-information terms.The variance loss covers duration, pitch, and energy, while mutual-information terms target information relationships involving style, content, and speaker representations.
  • 3) Inference process: During inference, content, style-prompt, and speaker embeddings condition iterative sampling from the stationary distribution before VQ decoding returns the output.The process uses timestep T, repeatedly samples tokens while t > 0, and then applies the VQ-VAE decoder.
  • 2) Training process: Training processes content prompts, style prompts, audio, and speaker identity into content, style, speaker, and acoustic representations before predicting corrupted token sequences.The conditioning representation combines content, acoustic, and speaker features, and the network is updated using the diffusion objective.
  • 2) Training process: The VQ model is trained on 669 hours of speech from an internal Mandarin corpus, VCTK, AISHELL3, and clean LibriTTS splits.The internal dataset contributes 300 hours.

2) Dataset for InstructTTS:

The dataset and InstructTTS implementation combine pretrained acoustic representations with a style encoder, content encoder, and discrete diffusion decoder.

  • Experiments use an internal dataset and pretrain Mel-VQ-VAE and neural audio codec models before end-to-end InstructTTS training.The supplied passages identify the dataset only as NLSpeech and describe the staged pretraining procedure.
  • Mel-VQ-VAE uses temporal and frequency downsampling, 256-dimensional codewords, and a 512-entry codebook.Its stated downsampling factors are 2 along time and 20 along frequency.
  • The audio codec exploration compares residual, group, and group-residual vector quantization models.The models use multiple codebooks to represent audio information.
  • InstructTTS contains style and content encoders plus a discrete diffusion decoder.The content encoder follows FastSpeech2, while the style encoder combines a pretrained prompt encoder with an audio encoder.

C. Baseline Approach

The baseline adapts StyleSpeech to natural-language style control, while evaluation measures both speech quality and prosody similarity.

  • The baseline adapts StyleSpeech and replaces its Mel-Style-Encoder with InstructTTS’s style encoder for a fair comparison.The paper presents this as a baseline because no existing expressive TTS model uses natural-language style prompts.
  • Objective speech-quality evaluation uses MCD, SSIM, and STOI, while prosody similarity uses GPE, VDE, and FFE.These metrics cover spectral similarity, clarity or intelligibility, and pitch-related similarity.
  • MCD measures spectral similarity by comparing reference and synthesized mel-frequency cepstral coefficients across frames.The study sums squared differences over the first 24 MFCCs.
  • Subjective evaluation assesses speech quality and style relevance.The paper uses MOS for quality and RMOS for relevance between synthesized speech and the prompt.

1) Speech quality:

Subjective evaluation compares synthesized speech quality and prompt relevance through listener ratings, preference tests, and emotion-perception analysis.

  • Speech quality: MOS asks listeners to rate naturalness, fidelity, and intelligibility on a five-point scale.Each audio sample is rated by at least 20 testers.
  • Speech quality: RMOS asks listeners to read each natural-language prompt and rate the relevance of the synthesized speech on a five-point scale.The measure directly evaluates correspondence between the prompt and generated speaking style.
  • Style relevance: The AXY test compares baseline and proposed samples for closeness to the natural-language style prompt using a seven-point score and preference choice.Scores range from -3 to 3, and preference rates compare baseline X with method Y.
  • Emotion perception: Emotion-perception evaluation uses happy, sad, and angry samples and an emotion classification model to assess generated emotional speech.The classification model is pretrained on an internal emotion dataset with wav2vec2 features and linear layers.

1) The analysis of objective metrics:

InstructTTS outperforms the baseline across reported objective, subjective, prosody-preference, and emotion-classification evaluations, with different decoders trading speech quality against prosody detail.

  • The analysis of objective metrics: InstructTTS achieves better speech quality and prosody objective metrics than the baseline.The reported metrics are MCD, SSIM, STOI, GPE, VDE, and FFE.
  • The analysis of objective metrics: Mel-VQ-Diffusion provides better speech quality, whereas Wave-VQ-Diffusion better preserves prosody details.The paper attributes this trade-off to frequency downsampling in Mel-VQ-VAE versus direct time-domain modeling in Wave-VQ-Diffusion.
  • Subjective evaluation: InstructTTS (mel) obtains the best MOS, while InstructTTS (wave) obtains the best RMOS and both outperform the baseline on RMOS.These findings come from crowd-sourced subjective evaluation.
  • Subjective evaluation: Raters show higher preference for both InstructTTS variants than the baseline in naturalness of prosody.The comparison uses the AXY preference test.
  • Emotion perception evaluation: InstructTTS achieves better emotion-classification performance than the baseline, with InstructTTS (Wave) performing best.The paper notes consistency with the FFE results.
  • The impact of mutual information minimization training: Mutual information minimization improves both speech quality and pitch similarity, especially pitch similarity.The ablation is presented as evidence for the feature-disentangling strategy.

3) The effectiveness of classifier-free guidance:

Classifier-free guidance improves performance by strengthening the connection between conditional information and predicted results. The broader experiments also examine diffusion strategies, codec choice, and sampling variation in InstructTTS.

  • The effectiveness of classifier-free guidance:: Classifier-free guidance brings better performance by strengthening the connection between conditional information and predicted results.The authors state that this encourages the model to better utilize conditional information.
  • The effectiveness of improved diffusion strategy:: The improved mask and uniform diffusion strategies bring better performance, validating the easy-first-generation principle.
  • Exploring the influence of audio codec for speech synthesis.:: GRVQ achieves the best reconstruction performance, while generated-model performance is not proportional to the number of codebooks.The authors conjecture that more codebooks burden generation models and that the NLSpeech dataset may be insufficiently large.
  • Sampling variation:: InstructTTS sampling can inject variation into generated speech even when text, speaker, and natural-language prompt remain fixed.The authors demonstrate this by generating ten samples and computing their F0 contours.
  • Overall approach:: The proposed method models expressive TTS in discrete latent space and casts speech synthesis as a language-modeling task.The paper explores both mel-spectrogram modeling with Mel-VQ-VAE and waveform modeling with a neural audio codec.
Loading 2301.13662v2…