F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

1 Introduction

• Parallel NAR synthesis enables fast inference, but alignment remains challenging, and rigid text–speech alignment can hinder naturalness (Eskimez et al., 2024; Anastassiou et al., 2024).Optimal-transport flow matching (Lipman et al., 2022) is widely used in recent TTS research.Source: §1 · pp. 1–2
• E2 TTS (Eskimez et al., 2024) replaces phoneme and duration prediction with padded characters, producing natural speech but exhibiting text–speech alignment robustness problems.Without explicit phoneme-level duration modeling, models learn word or phoneme lengths from total sequence length, improving prosody and rhythm. The authors attribute E2 TTS alignment failures to entangled semantic and acoustic features that re-ranking alone cannot resolve.Source: §1 · p. 2
• F5-TTS combines a Diffusion Transformer with ConvNeXt V2 (Woo et al., 2023), omitting phoneme alignment, duration prediction, a text encoder, and semantically infused codecs.Character sequences are padded to speech length and refined before concatenation with speech input. Training uses text-guided speech infilling and conditional flow matching.Source: §1 · p. 2, Figure 1 · p. 2
• F5-TTS demonstrates stronger robustness and greater faithfulness to text prompts while maintaining comparable speaker similarity.The authors support these improvements with ablation studies.Source: §1 · p. 3
• Sway Sampling improves naturalness, intelligibility, and speaker similarity through inference-time flow-step sampling and can extend to existing flow-matching models without retraining.During inference, the model and an ODE solver generate speech from sampled noise.Source: Figure 1 · p. 2, §1 · p. 3

2.1 Flow Matching

• Flow matching trains a neural vector field to transform a simple initial distribution into an approximation of the data distribution.Training covers the entire flow-step and data range. At inference, an ODE solver integrates from sampled noise; higher numbers of function evaluations (NFE) improve accuracy but increase computation.Source: §2.1 · p. 3
• Conditional flow matching uses Gaussian paths and has identical parameter gradients to the flow matching objective (Lipman et al., 2022).The affine flow maps initial noise to training data, with optimal transport choosing psi_t(x) = (1 - t)x + tx_1. A log-SNR reformulation with x_0 prediction (Kingma and Gao, 2024) makes CFM equivalent to cosine-schedule v-prediction (Salimans and Ho, 2022).Source: §2.1 · p. 3

2.2 Classifier-Free Guidance

• Doubled inference time accompanies classifier-free guidance (Ho and Salimans, 2022), which requires conditional and unconditional forward passes.Condition dropout during training enables linear extrapolation of these predictions at inference to balance fidelity and diversity. Unlike classifier guidance proposed by Dhariwal and Nichol (2021), this avoids an explicit classifier and its gradients.Source: §2.2 · p. 3

3.1 Pipeline

• Training reconstructs masked speech segments from surrounding audio and the full transcript, using noisy mel spectrograms and masked speech as model inputs.The noisy input is (1 − t)x0 + tx1, where x0 is Gaussian noise and x1 is the target mel spectrogram. Reconstruction targets m ⊙ x1, conditioned on the unmasked speech and extended text sequence.Source: §3.1 · p. 4
• English alphabets and symbols, or full pinyin for Chinese, form character sequences padded with filler tokens to match the number of mel frames.Full pinyin is chosen to facilitate Chinese zero-shot generation.Source: §3.1 · p. 4
• Inference uses reference audio for speaker characteristics and target text for content, estimating generation duration from the target-to-reference character-count ratio.The method assumes the combined character sequence is no longer than the mel sequence and pads it with filler tokens as during training.Source: §3.1 · p. 4
• An ODE solver integrates sampled noise into mel features conditioned on reference mel features and a concatenated, extended character sequence, discarding the reference portion before vocoder conversion to waveform.During inference, ordered flow steps can be uniformly sampled from 0 to 1, with their number determined by the NFE setting.Source: §3.1 · p. 4

3.2 F5-TTS

• F5-TTS combines DiT (Peebles and Xie, 2023) with ConvNeXt V2 (Woo et al., 2023) text refinement to improve convergence, alignment, and generation robustness.ConvNeXt processes characters before concatenation with noisy and masked speech, mitigating mismatched effective information lengths without phoneme-level duration prediction, explicit alignment, or an extra text encoder. DiT uses adaLN-zero rather than U-Net-style skip connections; flow time conditions adaLN-zero, and an additional mean-pooled text token is unnecessary.Source: §3.2 · pp. 4–5
• Sway Sampling applies nonuniform inference steps after uniform-step training, enabling fewer function evaluations while maintaining performance and transferring to other CFM models without retraining.The strategy improves faithfulness to reference text and speaker similarity. Negative s allocates more inference steps near t = 0, giving the ODE solver more precise initial integration where speech–text alignment is established.Source: §3.2 · pp. 4–5, §3 · p. 4

Datasets

• Approximately 95K hours of filtered English–Chinese Emilia (He et al., 2024) data train base models; ablations use 945 Mandarin hours from WenetSpeech4TTS (Ma et al., 2024).Evaluation uses LibriSpeech-PC test-clean (Meister et al., 2023) and Seed-TTS test-en/test-zh (Anastassiou et al., 2024), containing 1088 and 2020 samples respectively. To address unreleased prompts and inconsistent evaluation subsets, the authors release a 1127-sample LibriSpeech-PC subset with 4-to-10-second recordings.Source: §4 · pp. 5–6

Training

• 1.2M updates train the base models on eight NVIDIA A100 80G GPUs, using 307,200-audio-frame batches and AdamW (Loshchilov, 2017).Training uses a 7.5e-5 peak learning rate, 20K-update linear warmup, subsequent linear decay, gradient clipping at 1, and 0.1 attention/FFN dropout. F5-TTS has 335.8M parameters across a 22-layer DiT and four-layer ConvNeXt V2; reproduced E2 TTS has 333.2M parameters in a 24-layer flat U-Net Transformer.Source: §4 · p. 6
• 70% to 100% of mel frames are randomly masked for infilling training, with separate speech-only and joint speech–text conditioning drops.English uses alphabets and symbols; Chinese uses jieba and pypinyin to produce full pinyins, within a 2546-entry vocabulary including filler and multilingual characters; audio uses 100-dimensional log mel features at 24 kHz with hop length 256. CFG training first drops masked speech at rate 0.3, then drops masked speech together with text at rate 0.2 (Le et al., 2024), hypothesized to encourage text alignment.Source: §4 · p. 6

Inference

• Inference uses EMA weights, an Euler ODE solver for F5-TTS, and a midpoint solver for E2 TTS as described in Eskimez et al. (2024).The pretrained Vocos vocoder (Siuzdak, 2023) converts generated log mel spectrograms into audio.Source: §4 · p. 6

Metrics

• Cross-sentence evaluation reports Word Error Rate (WER), speaker similarity (SIM-o), Comparative Mean Opinion Scores (CMOS), and Similarity Mean Opinion Scores (SMOS).WER uses Whisperlarge-v3 (Radford et al., 2023) for English and Paraformer-zh (Gao et al., 2023b) for Chinese. SIM-o (Le et al., 2024) measures cosine similarity between synthesized and target speech embeddings extracted by a WavLM-large-based speaker verification model (Chen et al., 2022).Source: §4 · p. 6

Baselines

• Comparisons span leading autoregressive and non-autoregressive TTS systems, including a reproduced E2 TTS baseline (Eskimez et al., 2024).Mainly autoregressive comparators are VALL-E 2 (Chen et al., 2024), MELLE (Meng et al., 2024), FireRedTTS (Guo et al., 2024a), and CosyVoice (Du et al., 2024b). Other non-autoregressive comparators are Voicebox (Le et al., 2024), NaturalSpeech 3 (Ju et al., 2024), DiTTo-TTS (Lee et al., 2024), MaskGCT (Wang et al., 2024), and Seed-TTSDiT (Anastassiou et al., 2024).Source: §4 · p. 6

5 Experimental Results

• 2.42 WER at 32 NFE and 2.53 WER with 0.15 RTF at 16 NFE demonstrate F5-TTS’s accuracy–efficiency trade-off on LibriSpeech-PC test-clean.Results average three generation seeds, using CFG strength 2 and Sway Sampling coefficient −1. RTF measures generation of 10-second speech on an NVIDIA RTX 3090.Source: §5 · p. 6, Table 1 · p. 7
• 0.31 CMOS/3.89 SMOS on Seed-TTS test-en and 0.21 CMOS/3.83 SMOS on test-zh accompany near-ground-truth WER and comparable SIM.F5-TTS surpasses some larger-scale baselines while estimating duration from prompt transcript/text lengths without an extra predictor. The best-performing Seed-TTS uses orders-larger model and dataset scales, including several million training hours.Source: §5 · pp. 6–7, Table 2 · p. 7

5.1 Ablation of Model Architecture

• 4.17 WER and 0.54 SIM for F5-TTS versus 9.63 WER and 0.53 SIM for E2 TTS demonstrate stronger alignment at 800K updates.F5-TTS uses 32 NFE without Sway Sampling; the approximately 155M-parameter models train on 945 hours of Mandarin speech. Evaluation uses Seed-TTS test-zh.Source: Figure 2 · p. 7, §5.1 · p. 7
• 7% of E2 TTS test samples consistently fail (WER>50%) despite re-ranking, while removing the audio prompt eliminates these failures.The authors infer semantic–acoustic entanglement and argue that slow convergence leaves out-of-domain supervised fine-tuning or tremendous pretraining necessary.Source: §5.1 · pp. 7–8
• Text refinement enables DiT alignment: removing it prevents alignment, whereas MMDiT (Esser et al., 2024) learns quickly but collapses into repetitions and unstable timbre/prosody.Architectures with fewer skip connections permit faster training and inference, but successful alignment still requires the appropriate text representation and backbone.Source: §5.1 · p. 8
• +1.61 WER and +0.01 SIM from F5-TTS+Conv2Audio indicate substantially worse alignment robustness for slightly higher speaker similarity.F5-TTS+LongSkip and E2 TTS+Conv2Text also significantly degrade performance, showing that these architectural additions do not transfer directly between backbones.Source: §5.1 · p. 8

5.2 Ablation of Sway Sampling

• More negative Sway Sampling coefficients improve performance by emphasizing early flow steps, which establish prompt-faithful speech before later steps focus on intermediate output.This inference-time strategy applies to existing conditional flow-matching models without retraining.Source: §5.2 · p. 8, Figure 3 · p. 8
• Sway Sampling overrides leaked utterances to follow conflicting text prompts, whereas uniform sampling follows leaked content; it can also override leaked timbre.The experiment mixes Gaussian noise with duplicated audio-prompt mel features at t′ = 0.1, then resumes inference after skipping earlier flow steps.Source: §5.2 · p. 8

6 Conclusion

• F5-TTS achieves state-of-the-art zero-shot ability compared with existing works trained on industry-scale data.Its text-in, speech-out pipeline uses fully non-autoregressive flow matching with a diffusion transformer.Source: §6 · p. 8
• 0.15 test-time RTF makes F5-TTS competitive with heavily optimized TTS models of similar performance, while its design also enables faster training.The design enables faster inference as well.Source: §6 · p. 8
• ConvNeXt text modeling and test-time Sway Sampling improve speech-generation robustness and inference efficiency.ConvNeXt models the text, while Sway Sampling operates at inference time.Source: §6 · p. 8
• The authors plan to open-source code and models to enhance transparency and facilitate reproducible research.Source: §6 · p. 8

• Mel-spectrogram sequences remain much longer than text despite F5-TTS's faster training and inference and simple system design.Source: Limitations · p. 9
• F5-TTS lacks fine-grained control over paralinguistic details, such as emotion, despite strong zero-shot generation and reference-audio mimicry.Source: Limitations · p. 9
• More efficient, potentially universal continuous representations compatible with highly expressive speech synthesis remain a critical research direction for improving efficiency and performance.Source: Limitations · p. 9

Ethics Statements

• F5-TTS’s potential misuse for voice-identification spoofing motivates calls for watermarking and audio-output detection.The research model is trained on large-scale public multilingual speech data and can synthesize highly natural speech with high speaker similarity. The statement recommends safeguards rather than reporting their implementation.Source: Ethics Statements · p. 9

A Baseline Details

• VALL-E 2 (Chen et al., 2024) and MELLE (Meng et al., 2024) are compared using results reported in Meng et al. (2024).Both train on 50K hours of English Libriheavy (Kang et al., 2024) data; VALL-E 2 retains VALL-E’s architecture (Wang et al., 2023a) and adds repetition-aware sampling. MELLE autoregressively generates continuous-valued tokens using variational inference, with MELLE-Rx variants predicting multiple mel-spectrogram frames per step.Source: A Baseline Details · p. 13
• Voicebox (Le et al., 2024), NaturalSpeech 3 (Ju et al., 2024), and DiTTo-TTS (Lee et al., 2024) provide non-autoregressive baselines with published scores.Voicebox uses flow-matching infilling (330M parameters; 60K English hours), NaturalSpeech 3 uses factorized speech representations and diffusion (500M; Librilight (Kahn et al., 2020), 60K English hours), and DiTTo-en-XL uses cross-attention DiT with pretrained-language-model alignment (740M; 55K English hours). Voicebox scores come from Le et al. (2024) and Ju et al. (2024); the other comparisons use their respective model publications.Source: A Baseline Details · p. 13
• FireRedTTS (Guo et al., 2024a) and CosyVoice (Du et al., 2024b) are evaluated using official code and pretrained checkpoints.FireRedTTS combines a 400M-parameter autoregressive text-to-semantic model with a token-to-waveform model about half that size, trained with 248K hours of labeled speech. CosyVoice combines autoregressive text-to-token generation with flow-matching diffusion, has around 300M parameters, and trains on 170K hours of multilingual speech.Source: A Baseline Details · p. 13
• MaskGCT (Wang et al., 2024) and Seed-TTS (Anastassiou et al., 2024) use reported results; E2 TTS (Eskimez et al., 2024) is reproduced.MaskGCT uses mask-and-predict semantic and acoustic stages without precise text–speech alignment; the reproduced 333M multilingual E2 TTS avoids phoneme-level alignment, and both train on roughly 100K Chinese–English speech data from Emilia (He et al., 2024). Seed-TTSDiT is fully non-autoregressive, with an undisclosed training-data quantity described as orders of magnitude larger than previous TTS systems.Source: A Baseline Details · p. 13

B Experimental Result Supplements

• UTMOS (Saeki et al., 2022) measures naturalness objectively using an open-source MOS prediction model.The supplementary section’s reported UTMOS scores are evaluated with this predictor.Source: B Experimental Result Supplements · p. 13

B.1 Small Model Configuration

• Small-model configuration notation specifies architecture dimensions, depth, and hidden-size multiples; GFLOPs are evaluated using the thop Python package.Transformer entries list model dimension, layer count, head count, and hidden-size multiples. ConvNeXt entries list model dimension, layer count, and hidden-size multiples.Source: §B.1 · p. 14
• F5-TTS uses an adaLN DiT backbone with ConvNeXt V2 blocks, whereas E2 TTS uses a flat U-Net Transformer.F5-TTS+LongSkip connects the first and last Transformer layers (Lee et al., 2024). MMDiT (Esser et al., 2024) uses a double-stream Transformer, with the listed configuration describing one stream.Source: §B.1 · p. 14

B.2 Ablation study on Input Condition

• Input-condition ablations compare standard text-and-audio prompts, ground-truth rather than estimated duration, and text-only input with a blank audio prompt.The supplied passages describe these conditions but provide no numerical outcomes.Source: §B.2 · p. 14
• 155M small-model checkpoints trained on WenetSpeech4TTS Premium at 800K updates underpin all input-condition evaluations.Reported scores average results from three random seeds.Source: §B.2 · p. 14, Table 4 · p. 14

B.3 Sway Sampling Effectiveness on Base Models

• Gains on E2 TTS (Eskimez et al., 2024) demonstrate that Sway Sampling also benefits an existing flow-matching TTS model.The authors interpret this transfer as evidence of applicability beyond their own model.Source: §B.3 · p. 14
• Sway Sampling consistently improves zero-shot text faithfulness (WER), speaker similarity (SIM), and naturalness (UTMOS).Evaluations cover LibriSpeech-PC test-clean, Seed-TTS test-en, and Seed-TTS test-zh, using the midpoint solver and Sway Sampling coefficient s = −1.Source: Table 5 · p. 14, §B.3 · p. 14

B.4 Comparison of ODE Solvers

• Solver comparisons cover first-order Euler, second-order midpoint, and third-order improved Heun (Heun-3), evaluated across different Sway Sampling coefficients.Evaluation datasets are LibriSpeech-PC test-clean, Seed-TTS test-en, and Seed-TTS test-zh.Source: §B.4 · p. 14, Table 6 · p. 15
• Euler is inherently faster and typically performs slightly better at larger NFE with Sway Sampling; without Sway Sampling, Euler results degrade.This comparison concerns F5-TTS inference.Source: §B.4 · p. 14, Table 6 · p. 15

B.5 ELLA-V Hard Sentences Evaluation

• 100 difficult textual patterns from ELLA-V (Song et al., 2024) test robustness, with comparisons against objective results reported in E1 TTS (Liu et al., 2024b).Generated samples are also provided on the demo page.Source: §B.5 · p. 15
• F5-TTS and reproduced E2 TTS are evaluated with 32 NFE and Sway Sampling, averaging results across three random seeds.Because E1 TTSDMD’s prompts were unavailable, the authors randomly sampled three-second audio prompts from their LibriSpeech-PC test-clean set.Source: §B.5 · p. 15
• Hard-sentence errors include mispronunciations and skipped repeated words, while low insertion rates indicate no endless repetition.Mispronunciation partly explains the higher WER relative to commonly used test sets; high deletion rates indicate skipping within stacks of repeating words.Source: §B.5 · p. 15
• Prompts from different speakers produce distinct utterances that ASR may transcribe correctly for one speaker but incorrectly for another.The authors give “quokkas” being transcribed as “Cocos” as an example.Source: §B.5 · p. 15

B.6 Comparison of Vocoders and between PC and non-PC

• Vocoder comparisons use pretrained BigVGAN (Lee et al., 2022) and Vocos (Siuzdak, 2023), alongside evaluation on non-capitalized, punctuation-free LibriSpeech text.The non-PC version corresponds to an ordinary LibriSpeech test-clean subset. Table 8 covers F5-TTS Base on LibriSpeech-PC, LibriSpeech-non-PC, Seed-TTS test-en, and Seed-TTS test-zh.Source: §B.6 · p. 15, Table 8 · p. 16
• 2.66 WER with Sway Sampling versus 2.92 without is reported for reproduced multilingual E2 TTS on LibriSpeech-PC test-clean.Evaluation uses 32 NFE, Vocos, and Hubert-large-based (Hsu et al., 2021) ASR.Source: §B.6 · p. 15, Table 8 · p. 16

B.7 Training and Inference Performance with Different Dataset Scales

• 158M-parameter models were trained on 585-hour LibriTTS (Zen et al., 2019) and 24-hour LJSpeech.Training used the configuration from Sec.5.1 and Appendix B.1, with batches of 307,200 audio frames (0.91 hours). Every 100K updates took approximately 8 hours on 8 NVIDIA H100 SXM GPUs.Source: §B.7 · p. 16
• F5-TTS maintained stable training and learned speech–text alignment without grapheme-to-phoneme conversion across varying training-data amounts.The experiments tested whether training remained stable with limited data.Source: §B.7 · p. 16
• 32 NFE steps, CFG strength 2, and Sway Sampling coefficient −1 were used, with scores averaged across three random seeds.Because LJSpeech is single-speaker, evaluation used 1000 in-set samples, generating 4–7 seconds of speech with an approximately 3-second prompt.Source: §B.7 · p. 16

C Subjective Evaluation Details

• 20 native evaluators participated in English and Mandarin CMOS evaluation over 30 rounds using randomly selected utterances across all three test sets and model variants.Evaluators received detailed scoring guidelines and were instructed to hear all test files and sets before rating.Source: C Subjective Evaluation Details · p. 16
• −3 to +3 integer scores measure CMOS overall quality relative to a reference, with results reported as average differences from ground-truth speech scores.Evaluators were instructed to judge only against the reference and assess the overall impression rather than individual aspects.Source: C Subjective Evaluation Details · pp. 16–17
• SMOS measures similarity between synthesized speech and prompt speech through judges’ ratings from 1 to 5 in 0.5-point increments, with higher scores indicating greater similarity.Source: C Subjective Evaluation Details · p. 17
• The authors recommend releasing evaluation samples when model checkpoints remain closed and recruiting more evaluators to support transparent, comprehensive, and fair subjective evaluations.They contrast evaluator coverage with DiTTo-TTS (Lee et al., 2024), which reported 6 SMOS and 12 CMOS ratings, and NaturalSpeech 3 (Ju et al., 2024), which used 12 native evaluators judging 20 CMOS samples and 10 SMOS samples.Source: C Subjective Evaluation Details · p. 17