Source-linked AI summary
Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech
Yentl Collin, Evan Dufraisse, Amr Mohamed, Amine Khelif Khelif, Dani Bouch, Guokan Shang
TL;DR
Flow-matching TTS inference requires many neural function evaluations, while trajectory-integrated flow-map targets trade target accuracy against training cost. LFMD constructs conditional TTS flow-map targets from local teacher velocity, TD-DP derives reusable schedules from teacher dynamics and map consistency, and soft-DTW temporal self-distillation refines one-step synthesis. Temporal self-distillation from the four-step LFMD model reaches 1.80% WER at one NFE on Seed-TTS, close to the 32-NFE teacher’s 1.76%.
Problem
Flow-matching TTS inference requires many neural function evaluations, while trajectory-integrated flow-map targets trade target accuracy against training cost.
Source: §1 · p. 1Method
LFMD constructs conditional TTS flow-map targets from local teacher velocity, TD-DP derives reusable schedules from teacher dynamics and map consistency, and soft-DTW temporal self-distillation refines one-step synthesis.
Source: §1 · p. 1Results
Temporal self-distillation from the four-step LFMD model reaches 1.80% WER at one NFE on Seed-TTS, close to the 32-NFE teacher’s 1.76%.
Source: §1 · p. 1Takeaways & Limitations
The refined one-NFE student approaches the teacher in Seed-TTS WER, though its SIM-o and UTMOS remain lower.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Flow-matching text-to-speech (TTS) models achieve high synthesis quality but require many neural function evaluations (NFEs) to integrate their generative trajectories. Recent few-step flow-map distillation approaches for TTS construct targets from numerically integrated teacher trajectories, creating a trade-off between target accuracy and training cost. We propose Local Flow-Map Distillation (LFMD), which adapts Eulerian Map Distillation to conditional TTS and avoids teacher trajectory integration during target construction. For inference, we derive a sampling schedule (TD-DP) from teacher dynamics and consistency of the learned maps, with a single cost graph supporting multiple NFE budgets without external audio-metric evaluation. Because scheduling offers no flexibility at one NFE, we refine this regime with alignment-aware temporal self-distillation using soft-DTW. Across Seed-TTS and LibriSpeech-PC, LFMD improves low-NFE synthesis over a matched integral-distillation baseline. On Seed-TTS, the refined student reaches 1.80% WER with 1-NFE, compared with 1.76% for its 32-NFE teacher.
1. INTRODUCTION
Few-step TTS can reduce the integration cost of flow-matching models, but trajectory-integrated flow-map targets trade target accuracy against training cost. The paper introduces LFMD, TD-DP, and temporal self-distillation to address few-step synthesis and its scheduling challenges.
1. INTRODUCTION
- LFMD uses only the teacher velocity at the departure state to construct local flow-map targets, avoiding numerical integration over the target interval.Detached forward-mode target construction, a progressive interval curriculum, and a short-interval teacher-velocity anchor address memory overhead and stabilize long transports.Source: §1 · p. 1
- TD-DP builds one teacher-derived interval-cost graph reusable across NFE budgets, using local map consistency and composition without an external audio metric.This avoids the repeated synthesis and budget-specific searches required by metric-driven O3S.Source: §1 · p. 1
- 1.80% WER at 1 NFE approaches the 32-NFE teacher’s 1.76% on Seed-TTS, while SIM-o (.607) and UTMOS (3.41) remain below the teacher.The refined student uses alignment-aware temporal self-distillation.Source: §1 · p. 1
- Soft-DTW temporal self-distillation accommodates local timing differences between multistep targets and lower-step predictions, outperforming frame-wise self-distillation in the reported experiments.The one-NFE regime has no intermediate sampling step for a schedule to optimize, so the paper refines it through distillation from a stronger frozen four-step LFMD model.Source: §1 · p. 1
2. RELATED WORK
Prior work studies finite-time flow maps, inference-time step selection, and metric-based or adversarial refinement. The paper positions LFMD, TD-DP, and alignment-aware self-distillation against limitations in those approaches.
2.1. Flow Maps and Integral Velocity Distillation
- Integral-velocity TTS methods supervise average velocity from teacher displacements, but their targets depend on solver, sampling grid, and NFE budget; LFMD instead uses local teacher velocity.More teacher steps increase training cost, while coarse integration can reduce target accuracy over long intervals.Source: §2.1 · p. 2
2.2. Inference-Time Step Selection
- TD-DP uses teacher local dynamics and student map composition as interval costs, letting one graph support schedule search across NFE budgets without an external speech evaluator.O3S requires repeated synthesis and scoring and repeats its search for each budget.Source: §2.2 · p. 2
2.3. Metric Optimization and Adversarial Fine-Tuning
- The paper uses soft-DTW self-distillation to accommodate timing differences between multistep targets and lower-step predictions, without an external evaluator or discriminator.The authors report that this formulation substantially outperforms frame-wise self-distillation at one NFE.Source: §2.3 · p. 2
3. METHODOLOGY
The methodology defines conditional flow maps for speech, then trains them with local teacher dynamics rather than integrated teacher trajectories. It combines consistency-based schedule selection with temporal self-distillation for low-NFE synthesis.
3.1. Conditional Flow Maps
- LFMD models interval transport with an Euler-form map whose student predicts average velocity, matching identity transport at zero interval and teacher velocity as the interval shrinks.The teacher is conditioned on text and an audio prompt; the student absorbs classifier-free guidance and uses one conditional evaluation per interval.Source: §3.1 · p. 2
3.2. Local Flow-Map Distillation
- A departure-time flow-map identity lets LFMD learn long transports from local teacher velocity without rolling out the teacher over the target interval.Training progressively expands sampled intervals to the full horizon while ramping derivative correction; detached forward-mode targets avoid retaining a reverse-mode graph for the JVP branch.Source: §3.2 · p. 2
3.3. Teacher-Derived Step Scheduling
- TD-DP builds an interval-cost graph from local teacher consistency and map-composition consistency, then uses dynamic programming to select schedules for each NFE budget.The same graph supports separate budget paths without recomputing edge costs or using an external speech evaluator.Source: §3.3 · p. 3
3.4. Temporal Self-Distillation for Low-NFE Synthesis
- Temporal self-distillation trains lower-NFE predictions toward a stronger frozen four-step LFMD model, using four target evaluations instead of a 32-step teacher rollout.Band-constrained soft-DTW accommodates local timing differences; full-resolution and source regularizers preserve acoustic detail and limit drift, while inference retains the same architecture and NFE budget.Source: §3.4 · p. 3
4. EXPERIMENTS
Experiments compare few-step LFMD with matched integral-distillation students and teacher references, assess a reusable scheduler, and test self-distillation at low NFE.
4.1. Experimental Setup
- The setup trains F5-TTS teachers on Emilia or LibriTTS and compares matched LFMD/IMF students using three-seed WER, SIM-o, and UTMOS, with IMF targets from 16-step Sway integration.Emilia-trained models use Seed-TTS test-en; LibriTTS-trained models use LibriSpeech-PC test-clean.Source: §4.1 · p. 3
4.2. Few-Step Synthesis
- LFMD approaches the 32-NFE teacher at three and four NFEs, surpasses its WER while trailing SIM-o and UTMOS, and gains most over IMF at one and two NFEs.LFMD and IMF perform similarly at three and four NFEs; Table 1 reports teacher and few-step student results.Source: §4.2 · p. 3, Table 1 · p. 4
- LFMD updates are approximately 1.5, 2.5, and 4× faster than IMF with 8, 16, and 32 teacher steps; IMF's low-NFE results are sensitive to teacher discretization.With 16 teacher steps, uniformly sampled trajectories yield approximately 35% WER at one student NFE, versus approximately 8% with Sway-sampled trajectories.Source: §4.2 · p. 4
4.3. Scheduler Experiments
- TD-DP and O3S achieve broadly similar synthesis quality, while TD-DP reuses one interval-cost graph across NFE budgets without auxiliary scoring or repeated waveform generation.The graph is computed once rather than requiring a separate search for each budget.Source: §4.3 · p. 4
4.4. Fine-Tuning at Low NFE
- At one NFE, soft-DTW refinement reaches 1.80% WER, close to the 32-NFE teacher's 1.76%, and self-distillation also lowers WER at two and three NFEs.The frozen four-step LFMD model supplies the self-distillation target, requiring four evaluations rather than a 32-step teacher rollout.Source: §4.4 · p. 4
5. CONCLUSIONS
The paper concludes that LFMD enables strong low-NFE TTS without teacher trajectory integration for target construction, while TD-DP and self-distillation support schedule selection and lower-NFE generation.
5. CONCLUSIONS
- LFMD avoids numerical teacher trajectory integration, TD-DP reuses a teacher-derived cost graph across budgets, and self-distillation reaches 1.80% WER at one NFE on Seed-TTS versus 1.76% for the 32-NFE teacher.The cost graph requires neither external audio-metric evaluation nor budget-specific search.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study uses publicly available speech datasets and reports no new human-subject data collection.
6. COMPLIANCE WITH ETHICAL STANDARDS
- The study uses publicly available speech datasets and did not collect new human-subject data.The authors report no specific funding and no conflicts of interest.Source: §6 · p. 5