A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech

Motivation

• Real-world speech offers diverse prosody and paralinguistic information, while standard TTS corpora mostly contain reading or acted speech recorded in controlled environments.The authors suggest that real-world speech exposure could help TTS handle this diversity.Source: §1 · p. 1

Scope

• The work explores YouTube and podcast speech, using an already transcribed corpus and focusing on TTS rather than transcribing real-world speech with ASR.The authors identify higher prosodic variation and background noise than in controlled reading speech as challenges.Source: §1 · p. 1

Approach

• Mel-spectrogram autoregressive systems fail to form proper text-audio alignment during inference, producing unintelligible speech, while learned discrete codebooks address this issue.The authors attribute the failure to decoding error accumulation and the discrete representations’ greater resilience to input noise; they conjecture one codebook cannot cover spontaneous prosody, motivating multiple codebooks and architectures for multi-code sampling and monotonic alignment.Source: §1 · p. 1

Results

• MQTTS performs better in intelligibility and speaker transferability, achieves slightly higher naturalness and much higher prosody diversity, while non-autoregressive TTS excels in robustness and computation speed.With a clean silence prompt, MQTTS can also achieve a much lower SNR.Source: §1 · p. 1

Autoregressive TTS

• Autoregressive TTS uses sequential mel-spectrogram generation followed by a separately trained neural vocoder, but alignment learning can be sensitive to corpus variance and inference can produce faulty alignments.The paper uses monotonic chunk-wise attention and directly treats attention weights as a transition criterion; it also reports that an external duration predictor degrades MQTTS performance.Source: §2 · p. 2

Non-autoregressive TTS

• Non-autoregressive TTS commonly uses duration prediction and parallel generation, and is associated with better robustness, inference speed, and lower training difficulty than autoregressive models.VITS is selected as a representative system for comparison on real-world data.Source: §2 · p. 2

Quantization-based Generative Models

• Prior quantization-based speech approaches include autoregressive generation from VQVAE codes and unit-based systems using self-supervised representations, while VQTTS maps text to discrete tokens non-autoregressively.MQTTS instead uses multi-code generation and monotonic alignment, and learns codebooks specialized for speech synthesis rather than using self-supervised representations.Source: §2 · p. 2

TTS on real-world speech

• Prior real-world TTS work jointly trains denoising and TTS with external noise data, whereas this work focuses on TTS and uses an audio prompt for noise reduction without additional training.The authors do not design specific architectures for spontaneous-speech features such as filled pauses, leaving models to learn them implicitly.Source: §2 · p. 2

3 MQTTS

• The two-stage MQTTS system first quantizes raw waveforms into discrete codes, then trains a transformer to generate codes conditioned on speaker embeddings and phoneme sequences.The quantizer is fixed during the second stage.Source: §3 · p. 2

3.1 Quantization of Raw Speech

• Each speech frame is represented by four codes from distinct codebooks, whose embeddings are concatenated for waveform reconstruction.The quantizer combines quantization and HiFi-GAN adversarial losses; group normalization is applied to encoder residual-block outputs to address training divergence.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2 Conditional Synthesis with Transformer

• A single cross-attention layer on the decoder’s last layer provides a unique alignment map, while a sequential sub-decoder models conditional distributions across codebook groups for nucleus sampling.The sub-decoder generates each code given prior codes in its group and the main decoder state.Source: §3.2 · pp. 3–4

3.3 Inference

• A repetition token represents consecutive repeated codes, and three codes from an encoded silence clip are prepended to prompt cleaner speech synthesis.The repetition token is decoded back into the original repeated code during inference.Source: §3.2 · p. 4, §3.3 · p. 4
• Inference restricts attention to a moving encoder context window, advancing it when attention to its first state falls below a threshold; smaller windows enforce stricter monotonicity but reduce phonetic context.Window sizes from 3 to 6 worked well in qualitative listening, and alignment progress also serves as the stopping criterion.Source: §3.3 · p. 4

4.1 Datasets

• The training set contains 896 hours of cleaned GigaSpeech podcast and YouTube audio, with 2.4 hours for validation and utterances lasting 5–15 seconds.Human evaluation used 40 utterances with speaker references from different VoxCeleb test-set speakers.Source: §4.1 · p. 4

4.2 Training and Model Configuration

• The experiments compare MQTTS ablations, 40M, 100M, and 200M variants, Transformer TTS, Tacotron 2, and VITS, with training and inference settings specified.Ablations remove monotonic alignment or replace the sub-decoder; the 200M model uses a context window of 3, while the 40M and 100M models use 4.Source: §4.2 · pp. 4–5

4.3 Evaluation Metrics

• Evaluation measures intelligibility with RCER and WER, human-rated quality and naturalness with MOS, prosody with P-FID, speaker transfer with cosine similarity, and reconstruction with MCD.RCER uses the input utterance’s ASR transcription as ground truth; WER is reported for the full TTS system.Source: §4.3 · p. 5

5.1 Quantizer Analysis

• Multiple code groups are needed for real-world speech: a single codebook distorts phone sequences, while increasing its size from 1,024 to 65,536 codes yields little metric change.A single codebook suffices for LJSpeech, and increasing the number of groups from four to eight brought little perceptible quality or intelligibility difference; experiments use four groups of 160 codes.Source: §5.1 · p. 5

5.2 Performance

• Tacotron 2 fails to align during inference, while Transformer TTS produces only partial alignment and unintelligible output despite clear training alignment.A duration predictor resolves Transformer TTS’s alignment issue and improves performance, but a sizable gap remains; its inference alignment is compared in Figure 3.Source: §5.2 · pp. 5–6, Figure 3 · p. 6
• Removing monotonic alignment lowers MOS-N and raises WER, while replacing the Sub-decoder lowers MOS-N and P-FID but affects intelligibility less.The authors associate monotonic alignment with fewer repetition and deletion errors, and the Sub-decoder with improved prosody naturalness and diversity; both ablations remain better than Transformer TTS.Source: §5.2 · p. 6
• MQTTS has lower P-FID than VITS, indicating more diverse samples, while scaling VITS from 40M to 100M lowers P-FID without changing MOS-N.The authors note that VITS often generates shorter intermittent pauses, whereas MQTTS can generate longer pauses that increase misalignment with reference speech and lower MCD.Source: §5.2 · pp. 6–7
• At matched 100M parameters, MQTTS outperforms VITS on every reported metric except MCD; as model capacity grows, MQTTS shows steadily improving MOS-N, while VITS gains diversity.Overfitting was observed for MQTTS at 100M and 200M parameters, with greater severity at 200M; this suggests a larger training corpus is needed for further improvement.Source: §5.2 · p. 7
• MQTTS has lower sampling robustness than non-autoregressive systems, with prolonged vowels, poor-recording prompts, and more deletion errors among observed failure cases.The authors report that these errors gradually diminish as model capacity increases from 40M to 200M.Source: §5.2 · p. 7

5.3 Audio Prompt and SNR

• SNR falls as prompt-noise level σ rises, while σ below 10^-4 increases SNR relative to using no prompt; the other experiments use σ = 10^-5.The comparison uses 1,472 syntheses; the authors also report fuzzy noise across VITS syntheses and conjecture that it reflects unsuccessful modeling of environmental noise.Source: §5.3 · p. 7, Figure 5 · p. 7

• On real-world speech, multiple discrete codes are preferable to mel-spectrograms for autoregressive synthesizers, and suitably modeled MQTTS performs better than a non-autoregressive synthesizer.Source: §6 · p. 7
• A sizable gap remains between the best-performing syntheses and human speech.Source: §6 · p. 7
• The authors identify acquiring more data and leveraging ASR to transcribe real-world speech corpora as one direction for addressing that gap.Source: §6 · p. 7
• They propose silence detection during decoding to prevent phoneme transitions before phonation and mitigate deletion errors.Source: §6 · p. 7
• They plan to compare and incorporate self-supervised discrete speech and prosody representations with their learned codebooks.Source: §6 · p. 7