PaperlayerView exact revision on arXiv

Source-linked AI summary

MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer

Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, Zhizheng Wu

arXiv:2409.00750v3cs.SDcs.AIcs.LGeess.AS

TL;DR

Large-scale zero-shot TTS systems face robustness, speed, alignment, and duration-control limitations across autoregressive and non-autoregressive approaches. MaskGCT addresses these issues with a two-stage masked generative codec transformer that predicts semantic and acoustic tokens non-autoregressively. Experiments using 100K hours of in-the-wild speech report strong quality, similarity, intelligibility, and duration controllability.

  • Problem

    Existing autoregressive and non-autoregressive zero-shot TTS systems have limitations involving robustness, inference speed, alignment supervision, phone-level duration prediction, and duration controllability.

  • Method

    MaskGCT uses two masked generative stages: text predicts semantic tokens, then semantic tokens condition acoustic-token prediction, without alignment supervision or phone-level duration prediction.

  • Results

    MaskGCT outperforms state-of-the-art TTS systems on speech quality, similarity, and intelligibility while achieving human-level similarity, naturalness, and intelligibility across three benchmarks.

  • Takeaways & Limitations

    MaskGCT demonstrates flexible total-duration control and scalability across speech translation, voice conversion, emotion control, and speech content editing.

Abstract

from arXiv · show

The recent large-scale text-to-speech (TTS) systems are usually grouped as autoregressive and non-autoregressive systems. The autoregressive systems implicitly model duration but exhibit certain deficiencies in robustness and lack of duration controllability. Non-autoregressive systems require explicit alignment information between text and speech during training and predict durations for linguistic units (e.g. phone), which may compromise their naturalness. In this paper, we introduce Masked Generative Codec Transformer (MaskGCT), a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. MaskGCT is a two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the mask-and-predict learning paradigm. During training, MaskGCT learns to predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of a specified length in a parallel manner. Experiments with 100K hours of in-the-wild speech demonstrate that MaskGCT outperforms the current state-of-the-art zero-shot TTS systems in terms of quality, similarity, and intelligibility. Audio samples are available at https://maskgct.github.io/. We release our code and model checkpoints at https://github.com/open-mmlab/Amphion/blob/main/models/tts/maskgct.

1 Introduction

MaskGCT is a fully non-autoregressive zero-shot TTS system that uses masked generative transformers to avoid text-speech alignment supervision and phone-level duration prediction. It combines semantic and acoustic token generation and reports strong quality, similarity, intelligibility, flexibility, and controllability.

  • Masked generative transformers use mask-and-predict training and iterative parallel decoding, providing the modeling paradigm underlying MaskGCT.
  • MaskGCT avoids explicit text-speech alignment supervision and phone-level duration prediction through masked generative modeling.
  • Its two-stage pipeline predicts semantic tokens from text, then acoustic tokens conditioned on semantic tokens and prompt acoustic tokens.
  • MaskGCT achieves comparable or superior speech quality, similarity, prosody, and intelligibility across LibriSpeech, SeedTTS test-en, and SeedTTS test-zh.
  • 100K hours of in-the-wild speech support human-level similarity, naturalness, and intelligibility, alongside flexible duration control and broader speech-generation tasks.

2 Related Work

Large-scale TTS research targets zero-shot synthesis by scaling models and data, while using autoregressive, masked generative, and discrete speech-representation approaches. Semantic tokens align more closely with text, whereas acoustic tokens better reconstruct audio, motivating two-stage systems.

  • Modern large-scale TTS systems pursue zero-shot synthesis for unseen speakers using speech prompts and are commonly divided into autoregressive and non-autoregressive models.
  • Masked generative transformers have shown strong results in image, video, and audio generation, with potential comparable or superior to autoregressive and diffusion models.
  • Semantic discrete representations are usually quantized from speech SSL features, while acoustic representations are obtained from waveform-reconstruction speech codecs.
  • Semantic tokens correlate more strongly with text or phonemes, whereas acoustic tokens reconstruct audio more effectively, motivating two-stage TTS models.

3 Method

MaskGCT is a two-stage, fully non-autoregressive TTS framework that applies masked generative modeling to semantic and acoustic token generation without text-speech alignment supervision or phone-level duration prediction.

  • Framework: MaskGCT uses masked generative transformers in both stages, avoiding text-speech alignment supervision and phone-level duration prediction.The framework predicts semantic tokens from text and prompts, then acoustic tokens from semantic representations.
  • Speech Codecs: MaskGCT represents speech with semantic tokens from a speech SSL representation codec and multi-layer acoustic tokens from a speech acoustic codec.The semantic codec uses a VQ-VAE to discretize speech SSL features, while the acoustic codec uses residual vector quantization with 12 layers.
  • Text-to-Semantic Model: The T2S model predicts masked semantic tokens from text tokens and prompt semantic tokens, and can generate a target sequence of any specified length.A separate flow-matching duration model is trained for total-duration prediction conditioned on text and prompt speech duration.
  • Semantic-to-Acoustic Model: The S2A model predicts masked acoustic tokens conditioned on semantic tokens, prompt acoustic tokens, and lower acoustic layers.During inference, it generates acoustic layers from coarse to fine using iterative parallel decoding within each layer.
  • Extensions: The framework extends beyond zero-shot TTS to duration-controllable speech translation, emotion control, speech content editing, and voice conversion.These tasks require simple modifications or assistance from external tools.

4 Experiments and Results

MaskGCT is evaluated against established zero-shot TTS baselines across objective and subjective metrics, duration settings, style imitation, inference steps, and model sizes. It demonstrates strong similarity, robustness, naturalness, tempo flexibility, style imitation, and efficient scaling behavior.

  • Experimental Settings: The evaluation uses LibriSpeech, SeedTTS test-en, and SeedTTS test-zh, with SIM-O, WER, FSD, and subjective quality metrics.Baselines include NaturalSpeech 3, VALL-E, VoiceBox, VoiceCraft, XTTS-v2, and CosyVoice.
  • Comparison with Baselines: MaskGCT achieves human-level similarity across all three test sets, with SIM-O gains of +0.017, -0.002, and +0.027 over human recordings.The corresponding SMOS differences are +0.28, +0.32, and +0.25 across LibriSpeech, SeedTTS test-en, and SeedTTS test-zh.
  • Comparison with Baselines: MaskGCT remains nearly on par with ground truth in robustness and reaches human-level naturalness, with WERs of 2.634, 2.623, and 2.273 and CMOS gains of +0.10, +0.03, and +0.05.These results are reported for LibriSpeech, SeedTTS test-en, and SeedTTS test-zh, respectively.
  • Autoregressive vs. Masked Generative Models: Compared with AR + SoundStorm, MaskGCT improves similarity, robustness, and CMOS by +0.12, +0.08, and +0.37 across the three test sets.MaskGCT also controls total duration and uses 25 to 50 T2S inference steps, whereas AR inference steps increase linearly with speech length.
  • Duration Length Analysis: At duration multipliers of 0.7 to 1.3, WER is lowest at 1.0, remains close to optimal at 0.9 and 1.1, and stays reasonable at 0.7 and 1.3.The results indicate accurate content generation across different speech tempos.
  • Speech Style Imitation: MaskGCT achieves strong accent and emotion imitation, including accent SIM-O of 0.717 and emotion SIM-O of 0.600.It also leads the reported subjective metrics for accent imitation and emotion imitation.
  • Inference-Step Analysis: WER decreases from 10.19 to 2.507 on test-zh and from 8.096 to 2.346 on test-en between 5 and 25 inference steps, with minimal changes beyond 25.The analysis suggests around 25 steps balance SIM and WER effectively.
  • Model Size: The large T2S model outperforms the base model across metrics, but the difference is not significant, suggesting the base setting performs well with 100K hours of data.The authors defer more comprehensive model-size and data-scaling laws to future work.

5 Conclusion

MaskGCT is a large-scale, fully non-autoregressive zero-shot TTS system based on masked generative codec transformers. It avoids text-speech alignment supervision and phone-level duration prediction while supporting controllable duration and broader speech-generation tasks.

  • MaskGCT predicts semantic tokens from text and acoustic tokens conditioned on them, without text-speech alignment supervision or phone-level duration prediction.
  • MaskGCT outperforms state-of-the-art TTS systems in speech quality, similarity, and intelligibility.
  • The system can control total generated-speech duration and scales to speech translation, voice conversion, emotion control, and speech content editing.

A.1 Model Architecture

MaskGCT uses a Llama-style Transformer backbone adapted for bidirectional masked generation. Key architectural hyperparameters are summarized in Table 7.

  • MaskGCT uses a Llama-style Transformer with SwiGLU activation, rotary position encoding, and adaptive RMSNorm conditioned on timestep t.
  • The model replaces causal attention with bidirectional attention for masked token prediction.
  • Table 7 presents the key hyperparameters of the MaskGCT models.

A.2 Inference Steps for the T2S model

MaskGCT’s inference quality improves rapidly with additional steps, with most gains achieved by approximately 25 steps. The system also uses layered acoustic inference and a separate duration predictor.

  • Inference-step analysis: SIM reaches high values with around 10 steps, peaks near 25 steps, and changes minimally thereafter on SeedTTS test-zh and test-en.
  • Inference-step analysis: WER falls from 10.19 to 2.507 for test-zh and from 8.096 to 2.346 for test-en between 5 and 25 inference steps.
  • S2A inference: The S2A model generates acoustic tokens layer by layer from coarse to fine, using more iterations in the initial RVQ layers.
  • Codec configuration: The semantic codec uses a VQ-VAE over W2v-BERT 2.0 hidden features, while the acoustic codec uses RVQ and waveform reconstruction losses.
  • Duration prediction: MaskGCT predicts total target duration with a flow-matching duration predictor by summing phone-level durations.
  • Text tokenization: G2P outperforms BPE in English with a SIM-O of 0.728.
  • Classifier-free guidance: Classifier-free guidance is applied to both T2S and S2A, with default guidance and rescaling weights of 2.5 and 0.75.

D Evaluation Baselines

The evaluation compares MaskGCT with autoregressive, non-autoregressive, diffusion, flow-matching, and token-infilling TTS baselines using published results, official implementations, or checkpoints.

  • VALL-E is an autoregressive system with an additional non-autoregressive model for predicting neural speech-codec tokens.
  • NaturalSpeech 3 is a non-autoregressive system using factorized speech codecs and factorized diffusion models, with human-level naturalness reported on LibriSpeech.
  • VoiceBox is a non-autoregressive, multi-task speech-generation model based on flow matching.
  • XTTS-v2 is an open-source multilingual autoregressive TTS model supporting 16 languages.
  • VoiceCraft is a token-infilling neural codec language model for text editing and text-to-speech.
  • CosyVoice combines an autoregressive first stage with a diffusion second stage and is trained on 170,000 hours of multilingual speech.

E Multilingual Zero-Shot TTS

MaskGCT was evaluated for zero-shot TTS in Japanese, Korean, German, and French, with improvements over baselines except for Japanese WER. The multilingual extension retrained only the text-to-semantic model on expanded data.

  • Multilingual evaluation: MaskGCT significantly improves over XTTS-v2, Emilia-AR, and Emilia-NAR across Japanese, Korean, German, and French, except for WER in Japanese.The evaluation used SIM-O and WER on the test sets for the four additional languages.
  • Multilingual evaluation: The multilingual evaluation expands training data with 2,500 hours of Japanese, 7,400 Korean, 6,900 German, and 8,200 French speech.These data were added to the existing English and Chinese training foundation.
  • Training scope: Only the text-to-semantic model was retrained with expanded multilingual data; tokenizers and semantic-to-acoustic models were not retrained.The authors suggest further improvements may result from retraining all components on the expanded data.

F Duration-Controllable Speech Translation

MaskGCT performs cross-lingual speech translation while preserving semantic, timbre, and prosodic properties, with control over total duration. Pre-translation speech can serve as a prompt for maintaining timbre and prosody.

  • Speech translation: MaskGCT translates speech across languages while preserving the original semantic content, timbre, and prosody.The task targets cross-lingual speech translation, including scenarios such as dubbing.
  • Duration control: MaskGCT can control total speech duration, including keeping duration relatively unchanged for cross-lingual dubbing.Duration control is presented as a capability of the speech translation system.
  • Prompt conditioning: In-context learning uses pre-translation speech as a prompt to maintain the translated output’s timbre and prosody.The prompt provides the source speech context during translation.
  • Evaluation: The evaluation uses 200 randomly selected samples from SeedTTS test-zh and 200 samples from another speech translation set.The passage states the sample counts but truncates the name of the second set.

G Post-Training for Emotion Control

MaskGCT supports emotion control through post-training of its text-to-semantic model with emotion labels. Its speech content editing capability remains insufficiently robust and may require a more suitable training paradigm.

  • Emotion control: MaskGCT adds emotion control by fine-tuning the text-to-semantic model with an emotion label prefixed to the original input sequence.Fine-tuning uses ESD, containing 350 parallel utterances from 20 native English and Mandarin speakers.
  • Emotion control: The emotion-control post-training uses ESD, a dataset of 350 parallel utterances averaging 2.9 seconds from 10 English and 10 Mandarin speakers.The dataset supports fine-tuning after large-scale pretraining.
  • Speech content editing: Speech content editing uses an aligner to locate editing boundaries, masks the selected semantic tokens, and predicts replacements from edited text and unmasked tokens.This editing procedure follows the mask-and-predict mechanism.
  • Speech content editing: The authors observe that the system is not very robust for editing tasks and conjecture that fill-in-mask training may be better suited.The proposed training change is presented as a possible explanation and remedy, not an established result.

J Hard Cases Evaluation

MaskGCT is evaluated on hard TTS cases designed to expose hallucinations, especially in complex or repetitive phrases. The supplied evidence also notes misuse risks associated with high speaker similarity.

  • Hard-case design: The SeedTTS test-hard evaluation targets phrases with repeated words, tongue twisters, and other complex linguistic structures.These cases are described as situations where large-scale TTS models, particularly autoregressive systems, often hallucinate.
  • Hard-case comparison: Table 13 compares MaskGCT with AR + SoundStorm on the SeedTTS test-hard set.The supplied table caption identifies the comparison and evaluation set but provides no numerical outcomes.
  • Broader impact: MaskGCT’s high speaker similarity creates potential misuse risks, including spoofing voice identification and impersonating specific speakers.The authors conducted experiments assuming user consent to target-speaker synthesis.
Loading 2409.00750v3…