Source-linked AI summary
IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, Jingchen Shu
TL;DR
Autoregressive TTS offers naturalness but lacks precise duration control for synchronization-sensitive applications. IndexTTS2 combines duration control with disentangled emotion and speaker modeling, achieving state-of-the-art expressive speech synthesis with controllable duration and emotions.
Problem
Autoregressive TTS models provide naturalness and flexibility, but token-by-token generation complicates precise control of synthesized speech duration.
Method
IndexTTS2 uses autoregressive duration adaptation, prompt-based speaker–emotion disentanglement, GPT-latent fusion, and language-based emotion control.
Results
IndexTTS2 achieves state-of-the-art performance in expressive speech synthesis, leading baselines in emotional rendering while maintaining strong textual accuracy.
Takeaways & Limitations
IndexTTS2 produces zero-shot expressive speech with controllable duration and emotion for applications including animated dubbing and video narration.
Takeaways & Limitations
Some published experiments were reimplemented, producing minor performance variations within acceptable ranges while preserving overall rankings.
Abstract
from arXiv · showhide
Existing autoregressive large-scale text-to-speech (TTS) models have advantages in speech naturalness, but their token-by-token generation mechanism makes it difficult to precisely control the duration of synthesized speech. This becomes a significant limitation in applications requiring strict audio-visual synchronization, such as video dubbing. This paper introduces IndexTTS2, which proposes a novel, general, and autoregressive model-friendly method for speech duration control. The method supports two generation modes: one explicitly specifies the number of generated tokens to precisely control speech duration; the other freely generates speech in an autoregressive manner without specifying the number of tokens, while faithfully reproducing the prosodic features of the input prompt. Furthermore, IndexTTS2 achieves disentanglement between emotional expression and speaker identity, enabling independent control over timbre and emotion. In the zero-shot setting, the model can accurately reconstruct the target timbre (from the timbre prompt) while perfectly reproducing the specified emotional tone (from the style prompt). To enhance speech clarity in highly emotional expressions, we incorporate GPT latent representations and design a novel three-stage training paradigm to improve the stability of the generated speech. Additionally, to lower the barrier for emotional control, we designed a soft instruction mechanism based on text descriptions by fine-tuning Qwen3, effectively guiding the generation of speech with the desired emotional orientation. Finally, experimental results on multiple datasets show that IndexTTS2 outperforms state-of-the-art zero-shot TTS models in terms of word error rate, speaker similarity, and emotional fidelity. Audio samples are available at: https://index-tts.github.io/index-tts2.github.io/
Introduction
IndexTTS2 is introduced as a zero-shot TTS model targeting both precise fixed-duration generation and natural-duration synthesis while improving emotional expressiveness. Its design combines autoregressive semantic generation, multi-stage training, and separate emotion-control components.
- Background: Zero-shot TTS models have advanced through vector quantization, Transformers, and large-scale data, enabling timbre, prosody, and emotion synthesis from minimal audio prompts.These models are presented as more natural and flexible than traditional systems and applicable to AI dubbing.
- Model architecture: IndexTTS2 comprises Text-to-Semantic, Semantic-to-Mel, and Vocoder modules, with an autoregressive transformer generating semantic tokens from text, timbre/style prompts, and an optional token count.The optional speech token count enables duration specification during semantic generation.
- Training and emotion control: A multi-stage training strategy addresses limited high-quality emotional data and enhances expressive capabilities.The introduction also describes a Text-to-Emotion module that distills emotion-distribution prediction into Qwen-3-1.7b using LoRA.
- Duration control: IndexTTS2 combines precise duration control with natural-duration generation in an autoregressive zero-shot TTS model.The proposed duration adaptation scheme is described as scalable to autoregressive large-scale TTS models.
- Emotion and speaker control: The model decouples emotional and speaker-related features from prompts to independently support expressive speech and speaker identity.A feature-fusion strategy is designed to preserve semantic fluency and pronunciation clarity during emotionally rich expressions.
Related Work
Related work spans autoregressive and non-autoregressive zero-shot TTS, with non-autoregressive methods emphasizing duration prediction and emotion-control methods using textual, preset, embedding, diffusion, network, and token-based representations.
- Duration control: Non-autoregressive TTS methods support duration control through diffusion-, transformer-, flow-, or language-model-based duration predictors.MaskGCT uses flow modeling for phoneme-level durations, while F5-TTS estimates durations from text-speech length ratios.
- Generation paradigms: Zero-shot large-scale TTS models use both autoregressive and non-autoregressive generation paradigms.The supplied related work contrasts these paradigms in the context of duration control.
- Emotion control: Emotion control methods employ natural-language descriptions, preset instructions, interpretable style embeddings, diffusion-based style vectors, style networks, or content-style token systems.Examples include ControlSpeech, CosyVoice, EmoSphere++, StyleTTS 2, SC VALL-E, and Vevo.
Proposed Method
IndexTTS2 is a cascaded autoregressive zero-shot TTS system with separate T2S, S2M, and BigVGANv2 modules. Its method combines explicit duration conditioning, disentangled speaker–emotion control, three-stage T2S training, GPT latent enhancement, and natural-language emotion instructions.
- System Architecture: IndexTTS2 cascades Text-to-Semantic, Semantic-to-Mel, and BigVGANv2 vocoder modules, trained separately with tailored strategies for emotional expressiveness.T2S generates semantic tokens from text, style/timbre prompts, and an optional token count; S2M predicts mel-spectrograms from these tokens and the timbre prompt.
- Duration and Emotion Control: The T2S module controls duration through an embedding computed from target semantic token length T, while p = 0 enables free-form autoregressive generation.During inference, p = Wnumh(T) specifies the target token length, and a desired emotion vector e directly manipulates emotional prosody.
- Duration and Emotion Control: Speaker and emotion representations are disentangled by combining speaker features c with emotion embedding e and applying a GRL with a speaker classifier.The emotion embedding is extracted from style prompts by a Conformer-based emotion perceiver conditioner, while c primarily encodes timbral characteristics.
- Three-Stage Training: T2S training uses three stages: foundational training with p randomly zeroed 30% of the time, emotion refinement on 135 hours of emotional speech, and full-dataset robustness fine-tuning.Stage 2 freezes the speaker conditioner, trains the emotion conditioner, and uses GRL-based disentanglement; Stage 3 freezes all feature conditioners.
- Semantic-to-Mel Generation: The S2M module uses flow matching and fuses GPT latent features HGPT with semantic features to improve pronunciation robustness in highly expressive speech.The fused representation is used for S2M training, and ablations report reduced word error rate for highly expressive speech synthesis.
- Natural-Language Emotion Control: Natural-language emotion control distills Deepseek-r1’s seven-dimensional emotion distributions into Qwen-3-1.7b, which replaces the teacher at inference with reduced computational cost.The seven basic emotions are Anger, Happiness, Fear, Disgust, Sadness, Surprise, and Neutral, represented by a fixed emotion embedding set.
Experiments · Experimental Settings
IndexTTS2 was evaluated on multiple benchmarks using objective and subjective measures of intelligibility, speaker similarity, prosody, quality, and emotional fidelity. Experiments compared state-of-the-art zero-shot TTS systems and included ablations of the model’s architectural and training design.
- Datasets:: The model was trained on 55K hours of data, comprising 30K Chinese and 25K English hours, including 135 hours of emotional data from 361 speakers.The emotional data included 29 hours from ESD and the remainder from commercial purchases; most overall data came from Emilia, with additional audiobooks and purchasing data.
- Datasets:: Evaluation used four benchmarks to validate the fundamental capabilities of TTS systems.The supplied passage begins listing SeedTTS test-en as the first benchmark but is truncated before the full benchmark list is provided.
- Evaluation Metrics:: Objective intelligibility was measured by word error rate (WER), using FunASR for Chinese and Whisper for English content.Speaker similarity was computed as cosine similarity between speaker embeddings, while emotion similarity used emotion2vec representations.
- Evaluation Metrics:: Subjective evaluation used 1–5 multidimensional MOS ratings for speaker similarity, prosody, audio quality, and emotional fidelity.The corresponding measures were SMOS, PMOS, QMOS, and EMOS.
- Baseline:: IndexTTS2 was compared with MaskGCT, F5-TTS, CosyVocie2, SparkTTS, and the original IndexTTS in zero-shot TTS experiments.These systems were identified as state-of-the-art zero-shot TTS baselines.
- Baseline:: Two ablation experiments examined IndexTTS2’s architectural design and training methodology, including removal of GPT-derived latent feature enhancement.The GPT latent enhancement removal ablation evaluated the functional contribution of that enhancement.
- Training Hyperparameter Details:: IndexTTS2 was trained for three weeks on 8 NVIDIA A100 80GB GPUs with AdamW and an initial learning rate of 2e-4.The implementation used IndexTTS’s text tokenizer and MaskGCT’s semantic codec.
Experiment Results
IndexTTS2 achieves state-of-the-art or leading results across standard, emotional, and natural-language emotion-control evaluations. Its duration-control experiments show near-zero token-count error while preserving speech quality under scaled timing conditions.
- Standard TTS Evaluation: IndexTTS2 achieves state-of-the-art objective performance across most standard test sets and surpasses all but one baseline in subjective evaluation.Evaluation covered LibriSpeech-test-clean, SeedTTS test-zh/en, and AIShell-1 test, with only marginal underperformance on AIShell-1 in some comparisons.
- Emotional Speech Synthesis: IndexTTS2 obtains the highest scores across all four subjective emotional-evaluation dimensions and leads baseline models in SS and ES, despite higher WER than IndexTTS.Ablation results indicate the full multi-stage training approach balances emotional expressiveness with speech clarity and textual accuracy.
- Natural-Language Emotion Control: IndexTTS2 outperforms CosyVoice2 across timbre similarity, emotion similarity, rhythm, and audio quality in double-blind evaluation of natural-language emotion control.The constructed test set combined manually assigned emotion prompts with target texts used as prompts.
- Duration Control: Token-number error rates remain below 0.02% for original durations and below 0.03% for 0.875×/1.125× scaling, reaching only 0.067% for SeedTTS test-zh at 0.75×.Experiments used original, 0.75×, 0.875×, 1.125×, and 1.25× duration scalings, indicating a negligible gap between generated tokens and target durations.
- Duration-Controlled Speech Quality: Under duration control, IndexTTS2 matches F5-TTS on test-en, surpasses it by 0.5 pp on test-zh, and exceeds MaskGCT by 2 pp on test-zh.Performance declines only marginally under scaled durations, supporting speech-quality preservation during timing control.
Conclusion
IndexTTS2 is a zero-shot speech synthesis system that combines precise duration control, emotional expressiveness, phonetic clarity, and disentangled speaker and emotion features. With specialized training and augmentation, it achieves SOTA-level performance for expressive emotional restoration and supports controllable speech generation for dubbing and narration.
- Conclusion: IndexTTS2 enhances duration modeling, emotional expressiveness, and phonetic clarity through an autoregressive architecture with optimized training.Its design includes unique duration control and decouples emotional and speaker features for emotion-specific speech generation from reference audio.
- Conclusion: An LLM-driven module matches emotion vectors, supporting emotion-specific speech generation from reference audio.The supplied passage identifies this module as part of the system’s mechanism for controlling emotional expression.
- Conclusion: SOTA-level performance in high-expressive emotional restoration results from specialized training and data augmentation strategies.The model remains efficient in zero-shot settings while producing expressive speech with controllable duration and emotions.
- Conclusion: IndexTTS2 advances voice solutions for animated dubbing and video narration while pushing speech synthesis technology forward.These applications follow from its zero-shot expressive speech generation with controllable duration and emotions.