Source-linked AI summary
MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis
Ziyue Jiang, Yi Ren, Ruiqi Li, Shengpeng Ji, Boyang Zhang, Zhenhui Ye, Chen Zhang, Bai Jionghao, Xiaoda Yang, Jialong Zuo, Yu Zhang, Rui Liu, Xiang Yin, Zhou Zhao
TL;DR
Zero-shot TTS systems face a trade-off between robust speech-text alignment and naturalness. MegaTTS 3 uses sparse alignment with latent diffusion, multi-condition guidance, and PeRFlow acceleration; it reports state-of-the-art speech quality, flexible accent control, and efficient generation, while its current language and function coverage remain limited.
Problem
Existing TTS systems either lack robust explicit speech-text alignment or use predefined alignments that constrain naturalness and depend on duration-model quality.
Method
MegaTTS 3 inserts sparse alignment boundaries to guide a latent diffusion transformer, adds multi-condition classifier-free guidance, and uses piecewise rectified flow for acceleration.
Results
MegaTTS 3 achieves state-of-the-art zero-shot TTS speech quality, flexible accent-intensity control, and high-quality generation with 8 sampling steps.
Takeaways & Limitations
Sparse alignment combines alignment guidance with retained search freedom, while text guidance provides flexible accent control and PeRFlow enables a more efficient pipeline.
Takeaways & Limitations
MegaTTS 3 currently supports English and Chinese, and broader language and instruction-based style coverage remains future work.
Abstract
from arXiv · showhide
While recent zero-shot text-to-speech (TTS) models have significantly improved speech quality and expressiveness, mainstream systems still suffer from issues related to speech-text alignment modeling: 1) models without explicit speech-text alignment modeling exhibit less robustness, especially for hard sentences in practical applications; 2) predefined alignment-based models suffer from naturalness constraints of forced alignments. This paper introduces \textit{MegaTTS 3}, a TTS system featuring an innovative sparse alignment algorithm that guides the latent diffusion transformer (DiT). Specifically, we provide sparse alignment boundaries to MegaTTS 3 to reduce the difficulty of alignment without limiting the search space, thereby achieving high naturalness. Moreover, we employ a multi-condition classifier-free guidance strategy for accent intensity adjustment and adopt the piecewise rectified flow technique to accelerate the generation process. Experiments demonstrate that MegaTTS 3 achieves state-of-the-art zero-shot TTS speech quality and supports highly flexible control over accent intensity. Notably, our system can generate high-quality one-minute speech with only 8 sampling steps. Audio samples are available at https://sditdemo.github.io/sditdemo/.
1 Introduction
Existing zero-shot TTS systems face a trade-off between robust speech-text alignment and natural prosody. MegaTTS 3 addresses this with sparse alignment, flexible conditioning, and accelerated generation.
- Motivation: Implicit-alignment models can lose robustness or intelligibility, especially on difficult sentences, while forced-alignment methods constrain naturalness.Implicit methods also provide limited fine-grained pronunciation-duration control, whereas predefined alignments depend heavily on duration-model quality.
- Approach: MegaTTS 3 inserts sparse phoneme information within forced-alignment regions, then lets the latent diffusion transformer refine fine-grained alignment paths.This combines coarse pronunciation guidance with unconstrained transformer-based refinement.
- Results: The model achieves nearly state-of-the-art speech intelligibility and speaker similarity on LibriSpeech test-clean with only 8 sampling steps.The paper also reports high speech naturalness under this setting.
- Control: Multi-condition classifier-free guidance separately controls speaker timbre and text content, with text guidance also modulating personal accent intensity.This provides flexible control without requiring a separate accent-control mechanism in the stated contribution.
- Efficiency: Piecewise rectified flow reduces inference from 25 to 8 steps with minimal quality degradation.The technique is presented as an efficiency improvement for zero-shot TTS generation.
2 Background
Zero-shot TTS synthesizes unseen voices from speech prompts, while accented TTS targets controllable variation in accent intensity. MegaTTS 3 extends this setting with flexible accent control and sparse alignment guidance.
- Zero-shot TTS: Zero-shot TTS aims to synthesize unseen voices using speech prompts.The surrounding literature includes neural codec language models and other zero-shot synthesis approaches.
- Accented TTS: Accented TTS can support customized services by increasing expressiveness and improving listeners’ comprehension of speech content.The paper positions accented TTS as a potentially valuable but not yet mainstream application.
- Model context: Figure 1 presents WaveVAE and the model overview, including sparse alignment anchors inserted into the latent sequence.The anchors provide coarse alignment information for subsequent transformer processing.
- Accent control: MegaTTS 3 adjusts accent intensity without paired data or accurate accent labels through multi-condition classifier-free guidance.The target range includes speech closer to native English or accented second-language English speakers.
3 Method
MegaTTS 3 combines a WaveVAE latent representation, masked latent diffusion, sparse alignment, PeRFlow acceleration, and multi-condition guidance. These components provide alignment cues while preserving flexible generation and accent control.
- Architecture: WaveVAE encodes speech into a downsampled latent vector and reconstructs the waveform from that representation.After training, one second of speech is encoded into 25 vector frames.
- Architecture: The latent diffusion transformer learns a transport map from Gaussian noise to speech latents using rectified flow.The ODE converts Z0 from the Gaussian distribution π0 to Z1 from the speech-latent distribution π1.
- Architecture: Masked speech modeling predicts target latent vectors from a prompt region and phoneme embeddings while training loss is applied only to the masked region.The prompt proportion is randomly sampled from γ ∼ U(0.1, 0.9).
- Sparse alignment: Sparse alignment combines rough duration-based alignment with transformer attention that constructs fine-grained implicit paths.This is designed to improve intelligibility without imposing the naturalness constraints of fully predefined paths.
- Acceleration: PeRFlow distills the pretrained model over segmented time windows to reduce the number of flow-solver iterations.The method uses a teacher endpoint for each window and trains a student on the resulting segmented trajectory.
- Conditional guidance: Multi-condition CFG separately applies speaker and text guidance, and increasing text guidance changes pronunciation toward the target language’s standard pronunciation.The text guidance scale is therefore used to control accent intensity.
4 Experiments
Experiments evaluate MegaTTS 3 on zero-shot and accented TTS, including intelligibility, speaker similarity, naturalness, accent learning, controllability, and codec effects. Across these evaluations, the model reports strong zero-shot performance, improved prosodic naturalness with sparse alignment, and superior accented-TTS results over CTA-TTS.
- Zero-shot speech synthesis: MegaTTS 3 achieves state-of-the-art SIM-O, SMOS, and WER on LibriSpeech test-clean, comparable to NaturalSpeech 3 and stronger than baselines without explicit alignments.The analysis attributes improved similarity and WER to simplified text-to-speech mapping and robustness.
- Zero-shot speech synthesis: MegaTTS 3 significantly surpasses all baselines in CMOS, while PeRFlow acceleration preserves quality relative to the teacher model and enables fast inference.The same analysis also reports state-of-the-art speaker similarity and intelligibility on LibriSpeech-PC test-clean.
- Prosodic naturalness: MCD 4.42, GPE 0.31, VDE 0.29, and FFE 0.34 are the best reported prosodic-naturalness results for sparse alignment.The sparse-alignment system outperforms NaturalSpeech 3 and the forced-alignment variant across the evaluated metrics.
- Accented TTS: MegaTTS 3 significantly outperforms CTA-TTS on accent similarity MOS, MCD, pitch-distribution statistics, CMOS, and SMOS on L2-ARCTIC.The evaluation measures accent learning while maintaining consistent vocal timbre.
- Accented TTS: MegaTTS 3 significantly surpasses CTA-TTS in accent controllability when listeners classify generated speech as standard or accented.The experiment generates both accent types and compares perceived with intended categories using confusion matrices.
- WaveVAE: WaveVAE uses 25 tokens per second, and MegaTTS 3 outperforms versions using DAC or EnCodec latents in zero-shot TTS performance.The authors link this result to WaveVAE’s compact latent space and high reconstruction quality.
5 Conclusions
MegaTTS 3 combines sparse alignment, PeRFlow acceleration, and multi-condition CFG for efficient, controllable zero-shot TTS. The authors report state-of-the-art speech quality, enhanced prosodic naturalness, and robustness to suboptimal duration prediction.
- MegaTTS 3 uses sparse alignment boundaries to ease alignment learning while retaining naturalness.
- PeRFlow accelerates the generation process, producing a more efficient inference pipeline.
- Multi-condition CFG provides flexible control over accent intensity.
- The sparse alignment strategy improves prosodic naturalness and robustness against a suboptimal duration predictor.
Limitations
The paper identifies limited language and function coverage as current limitations of MegaTTS 3. The authors propose broader multilingual data, adaptation methods, and instruction-based or paralinguistic fine-tuning as future directions.
- Language Coverage: MegaTTS 3 currently supports English and Chinese but has limited coverage of the world’s languages.
- Language Coverage: The authors plan to add multilingual training data and use adaptation techniques such as LoRA for low-resource languages.
- Function Coverage: The system does not yet generate varied speaking styles from text descriptions through instruction-based fine-tuning.
- Function Coverage: Further fine-tuning on paralinguistic corpora is proposed to produce speech closer to natural human style.
A.6 Details in Subjective Evaluations
The subjective evaluation measures audio quality, speaker similarity, and accent similarity through controlled MTurk listening tests. The protocol includes repeated clips and known-quality controls to assess listener attention and consistency.
- Subjective tests evaluate audio quality, speaker similarity, and accent similarity on Amazon Mechanical Turk.
- Each dataset contributes 40 randomly selected test samples, with each audio sample heard by at least 10 listeners.
- Listeners compare pairs of system-generated audio for CMOS evaluation.
- Known-quality samples and repeated clips check evaluator attention and rating consistency.
- Participant compensation totaled about $500.
C Details of PeRFlow Training Procedure
PeRFlow trains an accelerated student solver on shortened segments of the teacher model’s flow trajectories. The procedure reduces the required iterations from 25 to 8 while lowering the teacher’s computation during training.
- PeRFlow trains an accelerated solver using shortened segments of the teacher model’s ODE trajectories.
- At each training step, the procedure samples data and Gaussian noise, then randomly selects a time window.
- The teacher solver predicts each selected segment’s endpoint instead of solving the full trajectory.
- The student model is optimized on the segmented trajectory between the sampled start and predicted endpoint.
- 25 teacher iterations are distilled into 8 student iterations, accelerating the training process.
D Details about Data and Model Scaling Experiments
The scaling experiments vary both training-data size and model parameters while holding the speech-compression VAE unchanged. Performance improves with more data and larger models, with larger models also strengthening paralinguistic capabilities.
- Experimental setup: Only model parameters are increased in the scaling experiment; the speech-compression VAE remains unchanged.The authors leave jointly scaling both models for future work.
- Data scaling: Data scaling from 2kh to 600kh consistently improves both speaker similarity and speech intelligibility.The authors report strong data scalability across the tested training-data sizes.
- Model scaling: 12.1% SIM-O improvement and 9.52% WER reduction result when model size scales from 0.5B to 7.0B parameters.These results validate model scalability for MegaTTS 3.
- Model scaling: Increasing model parameters also enhances the model’s paralinguistic capabilities.The paper points to specific audio examples on its demo page.
F Visualization of Attention Matrices
Attention visualizations examine duration-control examples and layer behavior using 8 sampling steps. The analysis separates transformer functions into feature extraction, speech-text alignment, and latent-feature refinement.
- Visualization setup: The attention matrices are visualized across all layers of the 1.4B MegaTTS 3 model using 8 sampling steps.The visualizations include sentence-level and phoneme-level duration-control cases.
- Attention behavior: Attention matrices remain identical across timesteps within the same layer.The authors interpret this as indicating that each layer maintains a consistent function across timesteps.
- Layer functions: Bottom transformer layers handle text and audio feature extraction.This role is identified from the lower-layer attention patterns.
- Layer functions: Middle transformer layers focus on speech-text alignment.These layers occupy the alignment-focused stage of the reported three-part functional grouping.
- Layer functions: Top transformer layers refine the target latent features.The paper associates this function with the top-layer attention patterns.
H Experiments of Prosodic Naturalness for Zero-Shot TTS
The prosodic-naturalness ablation compares sparse alignment with forced alignment and tests multi-condition CFG and F-LM duration against alternatives. Sparse alignment improves the reported naturalness metrics, while the additional components contribute positively.
- Ablation design: The ablation replaces sparse alignment with forced alignment, multi-condition CFG with standard CFG, and F-LM duration with standard autoregressive duration prediction.These substitutions isolate the contributions of the proposed design choices.
- Ablation design: Sparse alignment brings significant improvements in the prosodic-naturalness ablation.The study evaluates MCD, SSIM, STOI, GPE, VDE, and FFE on the LibriSpeech test-clean subset.
- Results: Multi-condition CFG and F-LM duration each contribute positively to performance.The conclusion is based on the ablation comparisons reported in Table 9.
- Evaluation scope: The longer-sequence comparison uses 10 sentences with more than 50 words and 400 target samples.VoiceCraft and CosyVoice serve as comparison systems.
J Experiments with Hard Sentences
The hard-sentence experiments target difficult textual patterns because LibriSpeech transcriptions are relatively simple. MegaTTS 3 shows stronger robustness against hard transcriptions, while the section also reports ethical safeguards and reproducibility details.
- Evaluation setup: The hard-sentence evaluation uses 100 difficult textual patterns from ELLA-V with randomly sampled 3-second LibriSpeech prompts.F5-TTS and E2-TTS are evaluated using official checkpoints or an official inference API.
- Results: MegaTTS 3 shows stronger robustness against hard transcriptions.This outcome is reported from the results in Table 11.
- Ethics: The paper identifies misuse risks including malicious voice cloning.It proposes deepfake detection, synthetic-audio watermarks, verification methods, and licensing restrictions as mitigations.
- Reproducibility: The authors describe architecture, hyperparameters, evaluation metrics, and mostly public datasets to support reproducibility.The listed metrics include WER, SIM-O, MCD, pitch-distribution moments, alignment error, CMOS, SMOS, and ASMOS.