PaperlayerView exact revision on arXiv

Source-linked AI summary

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, Xie Chen

arXiv:2410.06885v3eess.AScs.SD

TL;DR

F5-TTS addresses robustness and convergence problems in simple non-autoregressive text-to-speech systems that avoid explicit phoneme alignment and duration modeling. It combines ConvNeXt-enhanced text modeling with flow matching and inference-time Sway Sampling, achieving robust zero-shot synthesis and an RTF of 0.15. The paper also identifies longer mel sequences and limited fine-grained paralinguistic control as remaining boundaries.

  • Problem

    Non-autoregressive TTS needs robust text–speech alignment, while existing simple alignment-free designs can converge slowly and fail on some zero-shot samples.

  • Method

    F5-TTS uses a fully non-autoregressive DiT flow-matching pipeline with ConvNeXt text modeling, padded character sequences, and inference-time Sway Sampling.

  • Results

    F5-TTS achieves WER 2.42 with 32 NFE and RTF 0.15 with 16 NFE while retaining WER 2.53 and high-quality generation.

  • Takeaways & Limitations

    F5-TTS provides robust, natural zero-shot speech generation with a simple pipeline and an inference strategy that can transfer to flow-matching models without retraining.

  • Takeaways & Limitations

    Mel spectrogram sequences remain much longer than text, and the system lacks fine-grained control of paralinguistic details such as emotion.

Abstract

from arXiv · show

This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT). Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS. However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness. To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech. We further propose an inference-time Sway Sampling strategy, which significantly improves our model's performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining. Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, our F5-TTS exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. We have released all codes and checkpoints to promote community development, at https://SWivid.github.io/F5-TTS/.

1 Introduction

F5-TTS addresses the difficulty of robust text–speech alignment in non-autoregressive TTS without explicit phoneme alignment or duration modeling. It combines a simplified flow-matching pipeline with ConvNeXt V2 and inference-time Sway Sampling to improve robustness, naturalness, and efficiency.

  • Non-autoregressive TTS offers parallel, low-latency synthesis, but requires careful alignment between input text and generated speech.
  • Rigid phoneme-level alignment or duration modeling can hinder naturalness, while implicit length assignment improves prosody and rhythm.
  • E2 TTS removes explicit phoneme duration modeling but exhibits robustness problems in text–speech alignment.
  • F5-TTS uses a Diffusion Transformer with ConvNeXt V2 while omitting phoneme alignment, a duration predictor, a text encoder, and a semantically infused codec model.
  • Inference-time Sway Sampling improves generation naturalness, intelligibility, and speaker similarity and can be applied to flow-matching models without retraining.

2 Preliminaries

Flow Matching trains a neural vector field to transform a simple initial distribution into the data distribution. Conditional and optimal-transport formulations support tractable training, while ODE integration performs inference with an accuracy–computation trade-off.

  • Flow Matching connects a simple distribution such as Gaussian noise to a target data distribution through a probability path.
  • The FM loss regresses the path’s vector field with a neural network across the full flow-step and data range.
  • Conditional Flow Matching uses Gaussian conditional paths and has gradients identical to the original FM objective with respect to model parameters.
  • The conditional flow map interpolates between initial noise and training data, with the path converging to the initial and target distributions at its endpoints.
  • Optimal Transport uses the linear path ψ_t(x) = (1 − t)x + tx_1 for the conditional flow objective.
  • At inference, an ODE solver integrates the learned vector field from sampled noise to generated data; higher NFE improves accuracy but increases computation.

3 Method

F5-TTS trains a non-autoregressive speech-infilling model that aligns character-level text with speech through ConvNeXt-enhanced representations and flow matching. It uses ODE-based generation with inference-only Sway Sampling to improve efficiency and robustness without retraining.

  • 3.1 Pipeline: Training predicts masked speech from surrounding audio and the full transcription using noisy speech, masked speech, and text conditions.
  • 3.1 Pipeline: Text is represented as characters or pinyin and padded with filler tokens to the mel-frame length, forming an extended sequence.
  • 3.1 Pipeline: During inference, the audio prompt provides speaker characteristics while the generated text prompt guides speech content.
  • 3.1 Pipeline: The generation length is estimated from the character-count ratio of generated and reference text, assuming the total character length does not exceed mel length.
  • 3.1 Pipeline: The model starts from sampled noise and uses an ODE solver to integrate the flow under reference mel and extended-text conditions before vocoding the generated mel.
  • 3.2 F5-TTS: ConvNeXt blocks give text its own modeling space before concatenation with speech, avoiding explicit phoneme-level force alignment.
  • 3.2 F5-TTS: Sway Sampling applies non-uniform flow-step sampling only at inference, enabling faster generation with fewer NFE while preserving performance and requiring no retraining.
  • 3.2 F5-TTS: Negative Sway Sampling emphasizes smaller flow steps because early denoising establishes speech–text alignment and benefits from more precise initial integration.

4 Experimental Setup

The experiments use multilingual and Mandarin speech data, substantial training on A100 GPUs, and comparisons against leading autoregressive and non-autoregressive TTS systems. The authors also release a dedicated LibriSpeech-PC subset to improve comparison consistency.

  • Datasets: The base models use approximately 95K hours of English and Chinese speech from the Emilia multilingual dataset.
  • Datasets: Small ablation and architecture-search models use the 945-hour Mandarin WenetSpeech4TTS Premium subset.
  • Datasets: The released LibriSpeech-PC test set contains 1127 samples with 4-to-10-second prompts to facilitate community comparisons.
  • Training: Base models are trained for 1.2M updates over more than one week on 8 NVIDIA A100 80G GPUs.
  • Baselines: Evaluation covers leading autoregressive and non-autoregressive TTS baselines, including Voicebox, NaturalSpeech 3, DiTTo-TTS, Seed-TTSDiT, and reproduced E2 TTS.

5 Experimental Results

F5-TTS shows robust, efficient zero-shot synthesis and stable text-speech alignment, while ablations identify ConvNeXt text refinement and Sway Sampling as important design choices.

  • Main results: WER 2.42 at 32 NFE on LibriSpeech-PC test-clean demonstrates robust zero-shot generation, while 16 NFE reaches RTF 0.15 with WER 2.53.The reproduced E2 TTS has much worse zero-shot WER despite excellent speaker similarity.
  • Main results: F5-TTS achieves WER close to ground truth with comparable SIM scores on Seed-TTS test sets.It also obtains CMOS 0.31 on test-en and 0.21 on test-zh, with SMOS 3.89 and 3.83 respectively.
  • Model architecture ablation: At 800K updates, F5-TTS reaches WER 4.17 and SIM 0.54, compared with E2 TTS at WER 9.63 and SIM 0.53.The small-model comparison uses 155M-parameter models trained on 945 hours of Mandarin data.
  • Model architecture ablation: F5-TTS’s ConvNeXt text refinement supports alignment learning, whereas pure adaLN DiT fails to learn alignment from padded character sequences.The authors report that F5-TTS handles zero-shot generation with stronger robustness.
  • Model architecture ablation: Adding ConvNeXt to the audio branch increases WER by 1.61 while improving SIM by 0.01, and transferring the long-skip or text-refinement designs degrades performance.These results indicate that the architectural components do not transfer directly across the tested backbones or modalities.
  • Sway Sampling ablation: More negative Sway Sampling coefficients improve performance, and leaked audio content can be overridden according to the text prompt when Sway Sampling is used.Uniform flow-step sampling instead produces speech dominated by the leaked utterance; the strategy requires no retraining.

6 Conclusion

F5-TTS is a fully non-autoregressive flow-matching TTS system using a Diffusion Transformer, ConvNeXt text modeling, and test-time Sway Sampling. It reports state-of-the-art zero-shot ability, faster training and inference, and a test-time RTF of 0.15.

  • F5-TTS is a fully non-autoregressive text-to-speech system based on flow matching with a Diffusion Transformer.
  • ConvNeXt models text representations, while Sway Sampling improves speech-generation robustness and inference efficiency at test time.Sway Sampling does not require retraining existing flow-matching models.
  • 0.15 test-time RTF demonstrates competitive inference efficiency with similarly performing heavily optimized TTS models.
  • F5-TTS achieves state-of-the-art zero-shot ability compared with existing systems trained on industry-scale data.

Limitations

The authors identify two limitations: mel-spectrogram sequences remain much longer than text, and the system lacks fine-grained control over paralinguistic details such as emotion.

  • Mel-spectrogram sequences remain much longer than the text modality, limiting efficiency and motivating more efficient continuous representations.
  • F5-TTS lacks fine-grained control of paralinguistic details, including emotion.

Ethics Statements

The paper describes F5-TTS as a research project trained on large-scale public multilingual speech data, alongside comparisons with multiple TTS baselines and experimental configurations.

  • F5-TTS is trained on large-scale public multilingual speech data and can synthesize highly natural speech with speaker similarity.
  • The experiments also reference Seed-TTS and evaluate implementations or reported results from several large-scale TTS systems.
  • UTMOS scores are evaluated with an open-source MOS prediction model and measure naturalness objectively.
  • Small-model configuration tables specify Transformer and ConvNeXt dimensions, layers, heads, hidden-size multiples, and GFLOPs evaluation.

B.2 Ablation study on Input Condition

The input-condition ablation compares common text-and-audio prompting, ground-truth duration, and text-only input using small models, while Sway Sampling is evaluated across benchmarks and solvers.

  • Input conditions: The input-condition ablation uses text plus audio prompts, ground-truth duration, and text-only input with the audio prompt removed.
  • Input conditions: The ablation evaluates 155M small-model checkpoints trained on WenetSpeech4TTS Premium at 800K updates.
  • Sway Sampling: Sway Sampling consistently improves faithfulness to prompt text, speaker similarity, and naturalness in zero-shot generation.
  • Sway Sampling: Table 5 compares base-model results with and without Sway Sampling on LibriSpeech-PC and Seed-TTS English and Chinese test sets using the midpoint solver.
  • ODE solvers: Euler, midpoint, and Heun-3 ODE solvers are compared with different Sway Sampling coefficients; Euler is faster and typically slightly better at larger NFE with Sway Sampling.

B.5 ELLA-V Hard Sentences Evaluation

The ELLA-V evaluation tests TTS robustness on 100 difficult textual patterns, while additional comparisons use reported objective results and LibriSpeech-based benchmarks. The reported analysis highlights mispronunciation, word skipping, and the absence of endless repetition as distinct error patterns.

  • ELLA-V evaluates robustness using 100 difficult textual patterns, with generated samples included in the demo page.
  • The evaluation compares F5-TTS with reported objective results from E1 TTS and includes leading autoregressive and non-autoregressive TTS baselines.
  • Mispronunciations partially explain the higher WER on hard sentences, including substitutions such as “yogis” to “yojus” and “cavorts” to “caverts.”
  • A high deletion rate indicates word skipping on stacks of repeating words, whereas a low insertion rate indicates freedom from endless repetition.
  • WER varies across speakers because prompts can produce distinct utterances, causing the ASR model to transcribe one speaker correctly but fail on another.
  • With Vocos, reproduced multilingual E2 TTS obtains WERs of 2.92 on LibriSpeech-PC test-clean and 2.66 with Sway Sampling at 32 NFE.

B.7 Training and Inference Performance with Different Dataset Scales

The study examines whether F5-TTS remains stable when trained on substantially different English dataset scales. Results from small models indicate stable learning of speech-text alignment without grapheme-to-phoneme conversion, including on limited data.

  • F5-TTS 158M small models are trained on LibriTTS with 585 hours and LJSpeech with 24 hours to assess training stability across dataset scales.
  • Both small-model experiments use the base-model configuration, except for a batch size of 307,200 audio frames, equivalent to 0.91 hours.
  • The evaluation averages three random-seed generations using CFG strength 2, Sway Sampling coefficient −1, and 32 NFE steps.
  • F5-TTS small models enable stable training to learn speech-text alignment without grapheme-to-phoneme conversion, including on single-speaker LJSpeech in-set tests.

C Subjective Evaluation Details

The subjective evaluation measures generated-speech quality and speaker similarity through CMOS and SMOS ratings. It uses native English and Mandarin evaluators, standardized instructions, and comparisons with reference speech.

  • Twenty native evaluators judge English and Mandarin speech across 30 rounds with randomly selected utterances from all test sets and model variants.
  • The instructions ask judges to listen through all files, rate against the reference, and assess overall impression rather than isolated aspects.
  • CMOS rates overall quality relative to reference speech on an integer scale from −3 to +3, then reports average differentials from ground-truth speech.
  • SMOS uses a 1-to-5 scale with 0.5-point intervals, where higher scores indicate better speaker similarity to the prompt speech.
  • The authors encourage releasing evaluation samples and increasing evaluator counts to improve rigor, transparency, and fairness.
Loading 2410.06885v3…