PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoregressive Speech Synthesis without Vector Quantization

Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei

arXiv:2407.08551v2cs.CLcs.SDeess.AS

TL;DR

Codec-based speech language models use quantized codes that can sacrifice fidelity and pose sampling robustness challenges. MELLE autoregressively generates continuous mel-spectrograms with regression-based objectives and variational sampling. It performs on par with VALL-E 2 on objective metrics and surpasses it on subjective metrics.

  • Problem

    Codec language models use quantized audio codes that can lose fidelity, while random sampling can produce silence or persistent noise and multi-stage decoding adds compute and storage demands.

    Source: §1 · pp. 1–2

  • Method

    MELLE autoregressively predicts continuous mel-spectrogram frames in a single stage, using regression and spectrogram flux losses with variational latent sampling.

    Source: §1 · p. 2

  • Results

    MELLE is on par with VALL-E 2 in objective metrics and surpasses it in subjective metrics; on continuation inference, its WER is 1.47% vs. 1.61% for ground-truth speech.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MELLE offers a streamlined single-stage alternative to codec language models and achieves subjective speech quality comparable to human performance.

    Source: §6 · p. 9

    Evaluation uses the English-only LibriSpeech test set; multilingual evaluation is left for future work.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS). MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition, bypassing the need for vector quantization, which is typically designed for audio compression and sacrifices fidelity compared to continuous representations. Specifically, (i) instead of cross-entropy loss, we apply regression loss with a proposed spectrogram flux loss function to model the probability distribution of the continuous-valued tokens; (ii) we have incorporated variational inference into MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity and model robustness. Experiments demonstrate that, compared to the two-stage codec language model VALL-E and its variants, the single-stage MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes, achieves superior performance across multiple metrics, and, most importantly, offers a more streamlined paradigm. The demos of our work are provided at https://aka.ms/melle.

1 Introduction

Autoregressive codec language models adapt next-token prediction to speech, but discrete codes pose fidelity, robustness, and efficiency challenges. MELLE replaces them with continuous mel-spectrogram prediction and reports strong zero-shot TTS results.

1 Introduction

  • Inspired by next-token prediction in language models, codec language models use discrete codec codes for autoregressive zero-shot TTS.Source: §1 · p. 1
  • Codec language models trade fidelity for quantized codes, risk extended silence or persistent noise from random code sampling, and often require two-pass decoding with higher compute and storage demands.Source: §1 · pp. 1–2
  • Replacing discrete tokens raises two challenges: choosing a training objective for continuous representations and enabling sampling without top-p discrete-code sampling.Source: §1 · p. 2
  • MELLE is a single-pass zero-shot TTS model that autoregressively predicts continuous mel-spectrogram frames from previous tokens, conditioned on text and audio prompts.Source: Figure 1 · p. 1, §1 · p. 2
  • MELLE replaces cross-entropy with regression and spectrogram flux loss to reduce repetition, and uses variational latent sampling to increase output diversity.Source: §1 · p. 2
  • MELLE is on par with VALL-E 2 in objective metrics, surpasses it in subjective metrics, and outperforms VALL-E variants across naturalness, robustness, similarity, and inference efficiency.On continuation inference, it achieves a 47.9% relative WER reduction versus VALL-E and an 8.1% reduction versus VALL-E 2.Source: §1 · p. 2

2 Related Work

Prior TTS work includes end-to-end mel-spectrogram systems and zero-shot language-model approaches that often use codec codes or multi-stage pipelines. MELLE instead autoregressively models continuous-valued tokens in one stage.

2 Related Work

  • MELLE uses mel-spectrograms as its target representation; alternatives such as VAE latent states are left for future research.Source: §2 · pp. 2–3
  • End-to-end neural TTS predicts mel-spectrograms from text and uses a vocoder; TransformerTTS and FastSpeech represent encoder-decoder and non-autoregressive designs.Source: §2 · pp. 2–3
  • Unlike end-to-end TTS trained on small, clean, single- or few-speaker datasets, MELLE uses large-scale language-model training for zero-shot TTS.Source: §2 · p. 3
  • Zero-shot approaches include codec-code language modeling (VALL-E), speech-attribute disentanglement (Mega-TTS), alignment-based robustness methods (ELLA-V, RALL-E, VALL-E R), and repetition-aware sampling (VALL-E 2).Source: §2 · p. 3
  • Some systems use diffusion or flow-matching as a second stage to reconstruct continuous representations, whereas MELLE is a single-stage continuous-token autoregressive model with variational inference.Source: §2 · p. 3

3 MELLE

MELLE treats TTS as autoregressive prediction of continuous mel-spectrogram frames conditioned on text and prior frames. It combines latent sampling, spectrogram refinement, and a four-part training objective in a single model.

3.1 Problem Formulation

  • MELLE predicts the next mel-spectrogram frame from text and preceding frames, with a reduction factor optionally grouping multiple frames per decoding step.At inference, it uses a speech prompt and its transcript to generate target speech while preserving the prompted speaker’s characteristics.Source: §3.1 · p. 3

3.2 MELLE Architecture

  • A single autoregressive model avoids codec language models’ iterative multi-layer code predictions and an additional non-autoregressive model, while allowing multiple frames per step.The paper presents these simplifications as improving computational and storage efficiency.Source: §3.2 · p. 4

3.2.1 Autoregressive Language Model

  • A Transformer decoder models dependencies between text and acoustic embeddings, then supplies representations to MELLE’s output modules.The text and mel-spectrogram pre-nets project their inputs to the model dimension before concatenation.Source: §3.2.1 · p. 4

3.2.2 Latent Sampling Module

  • The latent sampling module predicts a distribution from the language-model output and samples a latent embedding intended to enhance expressive diversity and robustness.A multilayer perceptron with residual connections maps the sampled latent variable back to mel-spectrogram space.Source: §3.2.2 · pp. 4–5

3.2.3 Stop Prediction Layer and Post-Net

  • A linear classifier predicts when generation should stop, while convolutional post-net blocks add a residual to refine the generated mel-spectrogram.During training the model uses teacher forcing; at inference the post-net processes the generated spectrogram after autoregressive generation ends.Source: §3.2.3 · p. 5

3.3 Training Objective

  • Teacher-forced training combines regression, KL-divergence, spectrogram-flux, and stop-prediction losses to train one end-to-end autoregressive model.The regression loss covers intermediate and final spectrogram predictions; the flux loss penalizes low variability between consecutive frames, discouraging static output associated with repetition or prolonged silence.Source: §3.3 · p. 5
  • During zero-shot inference, MELLE samples each next-frame latent from a distribution conditioned on text, speech-prompt content, and previous frames, then vocodes the refined spectrogram.The paper describes learned, input-specific sampling as an alternative to manually configured sampling parameters in codec language models.Source: §3.3 · p. 6

4 Experimental Setup

Experiments use Libriheavy for the main MELLE model and LibriSpeech for a resource-constrained version, with zero-shot evaluation on LibriSpeech test-clean. Evaluation combines objective and subjective measures across continuation and cross-sentence synthesis.

4.1 Training Datasets

  • MELLE was trained on approximately 50K hours of Libriheavy speech, while MELLE-limited used the 960-hour LibriSpeech corpus.Libriheavy contains speech from 6,736 speakers; LibriSpeech contains speech from 1,251 speakers.Source: §4.1 · p. 6

4.2 Experimental Settings

  • The main model uses 80-dimensional log-magnitude mel-spectrograms extracted at 62.5 Hz from 16 kHz audio, and the limited version uses phoneme text tokens.The main-model text uses BPE with a 4K vocabulary; voice activity detection removes abnormal silences from audio.Source: §4.1 · p. 6, §4.2 · p. 6

4.3 Evaluation Settings

  • Zero-shot evaluation uses 4–10-second LibriSpeech test-clean samples in continuation and cross-sentence settings, with WER, speaker similarity, and human ratings among the measures.Continuation prompts with the first three seconds of an utterance; cross-sentence synthesis uses a reference utterance and its transcript to condition target speech.Source: §4.3 · pp. 6–7

5 Results and Discussion

MELLE outperforms VALL-E and its variants on continuation synthesis and is comparable to VALL-E 2 on cross-sentence synthesis, while also offering a more concise paradigm. Evaluations report favorable listener ratings, improvements from its proposed components, and faster inference with larger reduction factors.

5 Results and Discussion

  • Ground-truth speech reconstructed from mel-spectrograms has better robustness and speaker similarity than speech reconstructed from EnCodec codes, consistent with the fidelity cost of discrete codes.The paper presents this comparison as support for the claim that codec codes sacrifice fidelity relative to continuous mel-spectrograms.Source: §5 · p. 7

5.1 Objective Evaluation

  • MELLE reduces WERH relative to VALL-E by 47.9% on continuation and 64.4% on cross-sentence tasks, and is comparable to VALL-E 2 on cross-sentence synthesis.For continuation, it also shows better robustness and speaker similarity than VALL-E 2; subjective metrics favor MELLE over VALL-E 2.Source: §5.1 · pp. 7–8

5.2 Subjective Evaluation

  • MELLE’s synthesized speech leads the compared systems on all subjective metrics, with SMOS 4.40 vs. 3.94 for ground truth and CMOS -0.032 vs. 0.The CMOS comparison is reported as not statistically different from human-level quality (p-value > 0.1).Source: §5.2 · p. 8, Table 3 · p. 8

5.3 Ablation Study

  • Latent sampling and spectrogram flux loss improve robustness and speaker similarity, with especially pronounced gains on cross-sentence synthesis.Inference-time latent sampling also yields more robust and natural outputs than disabling it during inference.Source: §5.3 · p. 9
  • Latent sampling provides comparable SIM gains to spectrogram flux loss with less WER improvement, while helping preserve speaker characteristics from the prompt.The paper attributes some spectrogram-flux-loss SIM gains to improved robustness and more accurate semantic-context generation.Source: §5.3 · p. 9

5.4 Efficiency Comparison

  • Increasing reduction factor r accelerates training and inference by approximately r times; MELLE-R2 halves inference time and MELLE-R4 reduces it to one quarter.MELLE is more efficient than VALL-E 2 because it omits NAR inference steps, and these settings still show satisfactory performance.Source: §5.4 · p. 9

6 Conclusion

MELLE is a zero-shot TTS language model that directly predicts continuous-valued mel-spectrogram tokens from text and a speech prompt, without vector quantization. The conclusion reports that its latent sampling and spectrogram flux loss support diverse, robust, natural speech, while a reduction factor can accelerate decoding.

6 Conclusion

  • MELLE directly predicts continuous-valued mel-spectrogram tokens conditioned on text and a speech prompt, eliminating vector quantization.This avoids the two-stage training and inference procedures typical of neural codec language models such as VALL-E.Source: §6 · p. 9
  • Latent sampling and spectrogram flux loss support more diverse and robust predictions, with subjective speech quality described as highly natural and comparable to human performance.Source: §6 · p. 9
  • Setting the reduction factor can further accelerate decoding.Source: §6 · p. 9

Limitations

The paper identifies three limitations: synthesized speech quality depends partly on vocoder capability, evaluation is limited to English LibriSpeech, and the target representation is limited to mel-spectrograms.

Limitations

  • Synthesized speech quality can be limited by the vocoder used.The authors anticipate improvement from training a more powerful vocoder on a large-scale corpus.Source: Limitations · pp. 9–10
  • Evaluation uses only the English LibriSpeech test set; multilingual evaluation is left for future work.Source: Limitations · p. 10
  • The study uses mel-spectrograms as its only continuous acoustic target representation.Future work will explore alternatives such as VAE latent hidden states.Source: Limitations · p. 10

Broader Impacts and Ethical Statements

The paper describes potential uses and misuse risks for MELLE, while stating that its experiments assume the user agrees to be the target speaker. For real-world use with unseen speakers, it recommends speaker approval and synthesized-speech detection protocols.

Broader Impacts and Ethical Statements

A.1 Derivation of Kullback-Leibler (KL) Divergence Loss

The derivation assumes independent dimensions of a multivariate Gaussian latent variable and uses that assumption to analytically compute KL divergence across time steps.

A.1 Derivation of Kullback-Leibler (KL) Divergence Loss

  • The derivation assumes z_t follows a multivariate Gaussian distribution with independent dimensions.Source: §A.1 · p. 13
  • Under that assumption and equation (4), the KL divergence loss is analytically computed over T time steps.Source: §A.1 · p. 13
  • The symbol d denotes the dimensionality of the feature space.Source: §A.1 · p. 13

A.2 Mel-Spectrogram Extraction Protocol

The appendix specifies the mel-spectrogram target and extraction settings, training configuration, subjective-assessment protocol, and an ablation examining latent sampling and spectrogram flux loss.

A.2 Mel-Spectrogram Extraction Protocol

  • MELLE uses log-magnitude mel-spectrograms from resampled 16 kHz audio, extracted with a 1024-point STFT, 256 hop, 80 mel bins, and an 80–7600 Hz range.The resulting mel-filter output is converted with a base-10 logarithm.Source: §A.2 · p. 13

A.3 Training Details

  • Training uses 16 V100 GPUs, 400K update steps, and a peak learning rate of 5e-4 after 32K warmup updates, followed by linear decay.The passage also specifies loss weights and a KL-weight schedule.Source: §A.3 · p. 13

A.4 Detailed Subjective Assessment Criteria

  • The subjective assessment uses 40 test samples, at least 10 contributors per utterance, and MOS, SMOS, and CMOS ratings.MOS and SMOS use a 1–5 scale in 0.5-point increments; CMOS ranges from -3 to 3 in unit intervals.Source: §A.4 · pp. 13–14

A.5 Ablation Study with Five-Time Sampling

  • The ablation reports that latent sampling and spectrogram flux loss enhance robustness and speaker similarity in both single-time and five-time sampling setups.Table A1 presents the ablation; its ✦ marks latent sampling enabled during training but disabled during inference.Source: §A.5 · p. 14, Table A1 · p. 14
Loading 2407.08551v2…