Source-linked AI summary
E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS
Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, Yanqing Liu, Sheng Zhao, Naoyuki Kanda
TL;DR
Zero-shot TTS systems face latency, alignment, tokenizer, and input-format complexities. E2 TTS addresses these with a fully non-autoregressive flow-matching model trained for speech infilling over character sequences with filler tokens. It achieves state-of-the-art capabilities comparable to or better than Voicebox and NaturalSpeech 3, while remaining simple and flexible in its input representation.
Problem
Zero-shot TTS architectures face sequential inference latency, alignment requirements, tokenizer complexity, and additional components for phoneme-based inputs.
Method
E2 TTS uses a fully non-autoregressive flow-matching mel spectrogram generator trained on speech infilling with character sequences and filler tokens.
Results
E2 TTS achieves state-of-the-art zero-shot TTS capabilities comparable to or surpassing Voicebox and NaturalSpeech 3.
Takeaways & Limitations
E2 TTS combines a simple architecture with flexible input representation for zero-shot speech generation.
Takeaways & Limitations
Inference must automatically identify the boundary between audio-prompt and text regions to avoid retaining prompt audio or omitting target text.
Abstract
from arXiv · showhide
This paper introduces Embarrassingly Easy Text-to-Speech (E2 TTS), a fully non-autoregressive zero-shot text-to-speech system that offers human-level naturalness and state-of-the-art speaker similarity and intelligibility. In the E2 TTS framework, the text input is converted into a character sequence with filler tokens. The flow-matching-based mel spectrogram generator is then trained based on the audio infilling task. Unlike many previous works, it does not require additional components (e.g., duration model, grapheme-to-phoneme) or complex techniques (e.g., monotonic alignment search). Despite its simplicity, E2 TTS achieves state-of-the-art zero-shot TTS capabilities that are comparable to or surpass previous works, including Voicebox and NaturalSpeech 3. The simplicity of E2 TTS also allows for flexibility in the input representation. We propose several variants of E2 TTS to improve usability during inference. See https://aka.ms/e2tts/ for demo samples.
1. INTRODUCTION
E2 TTS addresses latency, tokenizer, alignment, and input-format complexities in zero-shot TTS with a simple fully non-autoregressive architecture. It uses character sequences with filler tokens and speech infilling to achieve performance comparable to or better than strong prior systems.
- Autoregressive codec sampling increases inference latency and requires careful tokenizer selection plus additional techniques for long audio sequences.
- Fully non-autoregressive systems enable parallel inference but require alignment between text and substantially longer audio sequences.
- Existing alignment strategies use frame-wise phoneme alignment or MAS, with MAS still requiring a phoneme-duration model during inference.
- Phoneme-based models additionally require grapheme-to-phoneme conversion, while tokenizer and input-format choices can require text normalization.
- E2 TTS uses only a flow-matching mel spectrogram generator and a vocoder, conditioning on character sequences padded with filler tokens and training through speech infilling.
- Despite its simplicity, E2 TTS achieves state-of-the-art zero-shot TTS capabilities comparable to or surpassing Voicebox and NaturalSpeech 3, while supporting flexible input representations.
2. E2 TTS
E2 TTS trains a Transformer-based mel spectrogram generator for audio infilling, using character prompts extended with filler tokens to align text and audio lengths. During inference, audio and text prompts are concatenated with filler tokens, and conditional flow matching generates the target spectrogram.
- 2.1. Training: Training uses an audio sample and transcription to construct a character sequence extended with filler tokens.The filler-token sequence is designed to match the relevant audio representation length.
- 2.1. Training: The spectrogram generator learns masked speech conditioned on unmasked speech and the extended character sequence.A binary temporal mask selects the speech regions modeled by the infilling task.
- 2.1. Training: Conditional flow matching trains the generator to model the speech-infilling distribution.The model uses a vanilla Transformer with U-Net-style skip connections.
- 2.2. Inference: Inference combines an audio prompt and its transcription with a text prompt and a chosen target duration.The target duration is represented internally by the generated frame length.
- 2.2. Inference: The inference character sequence concatenates audio and text characters with repeated filler tokens, using T = T aud + T gen − M aud − M text.This construction makes the extended sequence length equal to the combined audio-prompt and generated-audio frame lengths.
- 2.1. Training: Conditional flow matching transforms an initial distribution into the target speech-feature distribution, with an ODE solver generating log mel-filterbank features at inference.
E2 TTS
E2 TTS uses conditional flow matching and character-based conditioning to simplify zero-shot speech synthesis. Its X1 and X2 extensions adapt inference to missing audio-prompt transcripts and explicit pronunciation control.
- E2 TTS: Conditional flow matching trains a Transformer mel spectrogram generator to model masked speech from unmasked speech and character conditioning.The generator learns a speech-infilling distribution and outputs a vector field for the conditional flow-matching objective.
- E2 TTS: E2 TTS replaces Voicebox’s frame-wise phoneme conditioning with a character sequence containing filler tokens, eliminating separate phoneme conversion, alignment, and duration components.The paper characterizes the mel spectrogram generator as jointly modeling these functions.
- E2 TTS X1: E2 TTS X1 forms the inference sequence without the audio-prompt transcription while using transcription of the masked region.The remaining inference procedure is unchanged from basic E2 TTS.
- E2 TTS X2: E2 TTS X2 enables pronunciation control by replacing selected words with parenthesized phoneme sequences during training and inference.The implementation uses CMU dictionary phonemes with a 15% training replacement probability and retains surrounding punctuation.
- E2 TTS X2: The X2 representation keeps a simple character sequence, distinguishing words from phonemes through parentheses and their contents.Punctuation surrounding replaced words remains available to the model.
3.1. Training data
The models were trained primarily on Libriheavy, a large read-English corpus with case- and punctuation-preserving transcriptions, alongside proprietary data for scalability experiments.
- 3.1. Training data: Libriheavy provides 50,000 hours of read English speech from 6,736 speakers with transcriptions preserving case and punctuation.Training used these case- and punctuated transcriptions without preprocessing.
- 3.1. Training data: An additional 200,000 hours of proprietary training data were used to investigate E2 TTS scalability.
3.2. Model configurations
E2 TTS uses a 335-million-parameter Transformer mel spectrogram generator with U-Net skip connections and a BigVGAN-based vocoder, with optional unsupervised pre-training and a comparison duration model.
- 3.2. Model configurations: The E2 TTS architecture has 24 Transformer layers, 16 attention heads, 1024-dimensional embeddings, and 335 million parameters.It models 100-dimensional log mel-filterbank features at 24 kHz with 10.7-millisecond extraction intervals.
- 3.2. Model configurations: A BigVGAN-based vocoder converts the generated log mel-filterbank features into waveforms.
- 3.2. Model configurations: Some E2 TTS models were initialized with unsupervised pre-training on 200,000 hours of unlabeled anonymized data over 800,000 mini-batch updates.
- 3.2. Model configurations: A regression-based duration model was trained to estimate target durations for a fair comparison with the Voicebox baseline.The duration model uses an 8-layer Transformer and was trained with 75,000 mini-batch updates and 120,000 frames.
3.3. Evaluation data and metrics
Evaluation uses 1,132 LibriSpeech-PC test-clean samples from 39 speakers, three-second audio prompts, objective intelligibility and similarity metrics, and subjective naturalness and speaker-similarity tests.
- 3.3. Evaluation data and metrics: The test set contains 1,132 samples from 39 speakers, filtered to utterances lasting 4–10 seconds.Each audio prompt uses the last three seconds of a randomly sampled speech file from the same speaker.
- 3.3. Evaluation data and metrics: Objective evaluation averages metrics across three random seeds and measures WER for intelligibility and SIM-o for speaker similarity.WER is computed with a HuBERT-large-based ASR system.
- 3.3. Evaluation data and metrics: Table 1 reports objective results for the LibriSpeech-PC test-clean set, with WER expressed as a percentage and dataset abbreviations for Librilight, Libriheavy, and Proprietary.
- 3.3. Evaluation data and metrics: Subjective evaluation includes CMOS naturalness and SMOS speaker-similarity tests on 39 samples assessed by 12 native English evaluators.CMOS uses a 7-point scale comparing ground-truth and generated samples.
3.4. Main results
E2 TTS achieves strong objective and subjective zero-shot TTS results against Voicebox, VALL-E, and NaturalSpeech 3, despite its simple framework. Its variants also extend usability by removing prompt-transcription requirements and supporting pronunciation control.
- 1.9% WER and 0.708 SIM-o were achieved by E2 TTS (P2), surpassing the ground-truth audio on both metrics.
- E2 TTS X1 removes the need for audio-prompt transcription while achieving results nearly identical to basic E2 TTS.
- E2 TTS outperformed strong baselines including VALL-E, NaturalSpeech 3, and Voicebox despite its extremely simple framework.
- All E2 TTS variants achieved better CMOS scores than NaturalSpeech 3 and Voicebox, with (P2) reaching -0.05 CMOS.
- E2 TTS demonstrated zero-shot performance that was superior or comparable to Voicebox and NaturalSpeech 3 across speaker similarity, intelligibility, and naturalness.
- E2 TTS X2 showed only marginal WER degradation when 50% of words were replaced with phoneme sequences, enabling pronunciation specification without retraining.
3.5. Evaluation of E2 TTS extensions
The E2 TTS extensions improve inference usability in two ways: X1 eliminates audio-prompt transcription, while X2 permits pronunciation control through phoneme substitutions.
- Extension 1: E2 TTS X1: E2 TTS X1 achieved results nearly identical to basic E2 TTS, especially with unsupervised pre-training, while removing prompt-transcription requirements.
- Extension 2: E2 TTS X2: E2 TTS X2 was trained with a 15% phoneme replacement rate and evaluated with replacement probabilities from 0% to 50%.
- Extension 2: E2 TTS X2: E2 TTS X2 achieved parity when no words were replaced and only marginal WER degradation when 50% were replaced by phoneme sequences.
- Extension 2: E2 TTS X2: E2 TTS X2 allows new-term pronunciations to be specified during inference without retraining the base model.
- Prompt-length analysis: Prompt-length buckets compare WER and SIM-o for E2 TTS configurations (P1), (P2), and (P3).
- Speech-rate analysis: Speech-rate experiments compare WER and SIM-o for rates from 0.7 to 1.3 across configurations (P1), (P2), and (P3).
3.6. Analysis of the system behavior
E2 TTS converges more slowly than Voicebox but ultimately achieves better WER and SIM-o, while remaining robust to prompt-length and target-duration changes. A practical limitation is the need to identify the prompt/text boundary during inference.
- 3.6.1. Training progress: Voicebox had good WER at 10% training, whereas E2 TTS required significantly more training but achieved better final WER.
- 3.6.1. Training progress: E2 TTS required more training iterations than Voicebox but ultimately achieved better final SIM-o.
- 3.6.2. Impact of audio prompt length: E2 TTS required automatic identification of the prompt/text boundary during inference to avoid retaining prompt audio or omitting generated text.
- 3.6.2. Impact of audio prompt length: WER showed no obvious relationship with prompt length, while SIM-o improved as prompts became longer, including prompts up to 10 seconds.
- 3.6.3. Impact of target duration: Speech rates of 0.7 and 1.3 caused only a moderate WER increase while maintaining high SIM-o.
4. CONCLUSIONS
E2 TTS is a simple fully non-autoregressive zero-shot TTS system using filler-token character sequences and flow-matching audio infilling. It achieves state-of-the-art capabilities comparable to or surpassing Voicebox and NaturalSpeech 3, while supporting flexible input representations.
- E2 TTS converts text into character sequences with filler tokens matching the input and output sequence lengths.
- A flow-matching mel spectrogram generator is trained with an audio infilling task.
- E2 TTS achieves state-of-the-art zero-shot TTS capabilities comparable to or surpassing Voicebox and NaturalSpeech 3.
- Its simple design allows flexibility in input representation and motivates variants intended to improve inference usability.