Source-linked AI summary
Matcha-TTS: A fast TTS architecture with conditional flow matching
Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, Gustav Eje Henter
TL;DR
Diffusion-based TTS can achieve high-quality speech but is slowed by many sequential sampling steps. Matcha-TTS combines a lightweight encoder-decoder with OT-CFM-trained ODE sampling, and it reports strong naturalness, memory, intelligibility, and long-utterance speed relative to baselines.
Problem
Diffusion probabilistic models suffer from slow synthesis because accurate sampling requires numerous sequential steps, each evaluating an entire neural network.
Method
Matcha-TTS is a probabilistic, non-autoregressive TTS acoustic model using a 1D CNN-and-Transformer encoder-decoder and optimal-transport conditional flow matching.
Results
Matcha-TTS provides superior speech naturalness, uses less memory than baselines, is highly intelligible with two synthesis steps, and matches the fastest model on long utterances.
Takeaways & Limitations
The architecture and training method together reduce the speed–quality trade-off while learning to speak and align without external alignments.
Abstract
from arXiv · showhide
We introduce Matcha-TTS, a new encoder-decoder architecture for speedy TTS acoustic modelling, trained using optimal-transport conditional flow matching (OT-CFM). This yields an ODE-based decoder capable of high output quality in fewer synthesis steps than models trained using score matching. Careful design choices additionally ensure each synthesis step is fast to run. The method is probabilistic, non-autoregressive, and learns to speak from scratch without external alignments. Compared to strong pre-trained baseline models, the Matcha-TTS system has the smallest memory footprint, rivals the speed of the fastest models on long utterances, and attains the highest mean opinion score in a listening test. Please see https://shivammehta25.github.io/Matcha-TTS/ for audio examples, code, and pre-trained models.
1. INTRODUCTION
Diffusion models provide powerful speech-generation methods but synthesize slowly because reverse-time sampling requires many sequential neural-network evaluations. Matcha-TTS addresses this bottleneck with a lightweight encoder-decoder architecture and OT-CFM training for fast, high-quality acoustic synthesis.
- Diffusion probabilistic models generate continuous-valued data by transforming a target distribution to a prior and learning a reverse sampling process.Their reverse-time SDE has a corresponding probability-flow ODE that produces samples from the same distribution deterministically.
- Numerous sequential sampling steps make diffusion-based synthesis slow, with each step requiring an entire neural-network evaluation.This synthesis speed has long been the main practical issue with diffusion probabilistic models.
- Matcha-TTS introduces a probabilistic, non-autoregressive TTS acoustic model based on continuous normalising flows.The model is designed for fast sampling from text to acoustic features.
- An encoder-decoder combining 1D CNNs and Transformers reduces memory consumption and evaluation cost, improving synthesis speed.The decoder design is one of the paper’s two main innovations.
- OT-CFM trains ODEs with simpler source-to-target paths, enabling accurate synthesis in fewer steps than diffusion models.The paper reports that the architecture and training method together reduce the trade-off between synthesis speed and quality while learning to speak and align without an external aligner.
2. BACKGROUND
Recent TTS systems use diffusion, normalising flows, and encoder-decoder architectures, but differ in alignment requirements, positional encoding, decoder design, and training approach. Matcha-TTS builds on these developments by combining RoPE-based encoding with OT-CFM and a pure TTS objective.
- Diffusion-based TTS includes waveform-generation, end-to-end, and acoustic-modelling systems, with both non-autoregressive and autoregressive variants.Grad-TTS conceptualised the diffusion process as an SDE, while TorToiSe demonstrated an autoregressive model with quantised latents.
- Modern TTS encoder-decoders commonly use Transformer encoders, but positional-embedding choices trade off long-sequence generalisation, context handling, and memory use.Matcha-TTS uses RoPE in the encoder, reducing RAM use compared to Grad-TTS.
- TTS decoder designs range from dilated 1D convolutions to 2D-convolutional U-Nets, whose extra frequency dimension generally increases memory requirements.The cited discussion notes that speech mel-spectra are not fully translation-invariant along frequency.
- Many TTS systems learn speaking and alignment jointly, while others rely on external alignments; monotonic-alignment mechanisms support fast and effective training.Monotonic alignment search is used by systems including Glow-TTS and VITS, while Grad-TTS uses a related prior-loss mechanism.
- Conditional flow matching learns ODEs through vector-field regression as an alternative to score-matching approaches and distillation-based acceleration.OT-CFM is the flow-matching method used by Matcha-TTS for pure TTS acoustic modelling.
- Voicebox is a substantially different CFM-based system using masking, multiple speech tasks, large proprietary training data, and external alignments.Its English model uses 330M parameters, reported as 18 times larger than Matcha-TTS in the experiments.
3. METHOD
Matcha-TTS uses conditional flow matching to learn an ODE-based decoder, with optimal transport providing simple paths from noise to acoustic frames. Its encoder-decoder design conditions synthesis on text-predicted acoustic means while using a fast, memory-efficient decoder.
- Flow-matching training: Conditional flow matching replaces intractable marginal densities and vector fields with tractable conditional counterparts having closed-form solutions.
- Flow-matching training: OT-CFM matches each acoustic datum x1 to random Gaussian noise x0 and uses a linear, time-invariant vector field depending only on x0 and x1.The target flow uses a small σmin hyperparameter, set to 1e-4 in the experiments.
- Flow-matching training: These optimal-transport properties enable faster training and generation than DPMs while supporting better performance.
- Proposed architecture: The non-autoregressive encoder predicts durations and mean acoustic features µ, which condition the decoder’s vector-field prediction during synthesis.Durations are rounded up and used to duplicate encoder outputs before producing µ.
- Proposed architecture: The decoder combines 1D convolutional residual blocks with Transformer blocks in a U-Net, reducing evaluation time and memory relative to Grad-TTS’s 2D convolutional U-Net.The decoder embeds the flow-matching step and uses no positional embeddings in its Transformer blocks.
4. EXPERIMENTS
The experiments compare Matcha-TTS with three pre-trained TTS baselines on memory, intelligibility, naturalness, and synthesis-speed scaling. Matcha-TTS combines strong quality with low memory use and competitive speed, particularly for longer utterances.
- 4.1. Data and systems: The evaluation compares Matcha-TTS with Grad-TTS, FastSpeech 2, and VITS using pre-trained LJ Speech checkpoints and multiple ODE solver step counts.ODE conditions are labeled MAT-n, GRAD-n, and GCFM-n, where n is the number of neural-network evaluations.
- 4.2. Evaluations, results, and discussion: Matcha-TTS uses less training memory than all baselines and is smaller than the other systems, including VITS after adding its vocoder.The MAT model is approximately the same size as GRAD/GCFM but smaller than all other systems.
- 4.2. Evaluations, results, and discussion: Matcha-TTS is the most intelligible system even with two synthesis steps, is faster than VITS, and is only slightly slower than FastSpeech 2 at its fastest setting.At the same number of function evaluations, MAT is equally fast or faster than GRAD/GCFM.
- 4.2. Evaluations, results, and discussion: Matcha-TTS achieves similar or better naturalness than comparable baselines, with MAT consistently exceeding GRAD at matched NFE and surpassing FastSpeech 2.The reported pairwise tests used α = 0.05; MAT-4 also exceeds GCFM-4, indicating contributions from both architecture and training.
- 4.2. Evaluations, results, and discussion: MAT-2 becomes competitive with FastSpeech 2 on longer utterances, while MAT-4 remains close; the speed gap from MAT to GRAD grows with utterance length.The study generated 180 sentences of different lengths and fitted regression lines to wall-clock synthesis times.
5. CONCLUSIONS AND FUTURE WORK
Matcha-TTS is presented as a fast, probabilistic, non-autoregressive, memory-efficient ODE-based acoustic model that jointly learns speech and alignment. The authors report superior naturalness and speed matching the fastest model on long utterances, while identifying broader data and capability extensions as future work.
- 5. CONCLUSIONS AND FUTURE WORK: Matcha-TTS is a fast, probabilistic, non-autoregressive, memory-efficient ODE-based TTS acoustic model trained with conditional flow matching.The approach jointly learns to speak and align.
- 5. CONCLUSIONS AND FUTURE WORK: Compared with three strong pre-trained baselines, Matcha-TTS provides superior speech naturalness and matches the speed of the fastest model on long utterances.The authors attribute the improvements to both the new architecture and the new training method.
- 5. CONCLUSIONS AND FUTURE WORK: Future work includes multispeaker modelling, probabilistic duration modelling, and applications to challenging diverse data such as spontaneous speech.These directions extend the model beyond the reported setting.