Source-linked AI summary
Harmonizing Spectral Evolution in Conditional Flow Matching for TTS
Isha Pandey, Varad Deshpande, Abhijat Bharadwaj, Ganesh Ramakrishnan
TL;DR
CFM-based TTS exhibits incoherent frequency evolution during ODE integration, a spectral-development issue that temporal discretization alone does not address. The paper introduces training-free DWT-based reweighting, reporting improved FAD and reduced NFE across evaluated architectures.
Problem
CFM-based TTS models show incoherent frequency evolution during ODE integration, with approximation components growing while detail components lag.
Source: §2.1 · p. 2Method
FSB uses DWT sub-band reweighting during ODE integration to suppress excessive approximation-band growth and boost lagging details.
Source: §1 · p. 1, §2.2 · p. 2Results
FSB reduces required NFE from 32 to 26 and improves FAD by up to 61% across F5-TTS, Matcha-TTS, and IndicF5, without degrading intelligibility and perceptual quality.
Source: §5 · p. 4Takeaways & Limitations
The low-dimensional schedules require one-time calibration and transfer across languages.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Conditional Flow Matching (CFM) models for text-to-speech (TTS) suffer from incoherent frequency evolution during inference. While similar spectral imbalances are addressed in diffusion models for other domains, those generic solutions fail to generalize to the inherently uncoordinated acoustic dynamics of CFM. We demonstrate that this issue can be effectively mitigated by introducing a novel training-free frequency-selective boosting strategy. Using the Discrete Wavelet Transform (DWT), our method dynamically modulates mel-spectrogram sub-bands during ODE integration, synchronizing spectral development by penalizing aggressive low-frequency growth and boosting lagging high-frequency details. Validated across diverse architectures (Matcha-TTS, F5-TTS, IndicF5), our approach reduces the required Number of Function Evaluations (NFE) from 32 to 26 and improves Frechet Audio Distance (FAD) by up to 61%, all without compromising mean opinion scores, speaker similarity, and speech intelligibility.
1. INTRODUCTION
The introduction identifies incoherent spectral development during CFM inference as a shared limitation and motivates frequency-aware correction. It presents FSB as a training-free, step-dependent intervention.
1. INTRODUCTION
- CFM inference has focused mainly on temporal discretization, while acoustic frequency development has received comparatively little attention.The paper identifies incoherent frequency evolution as a shared limitation across CFM-based acoustic models.Source: §1 · p. 1
- DWT analysis finds approximation-band energy growing while detail bands develop more slowly across fixed-step Matcha-TTS and Sway-sampled F5-TTS.The imbalance persists despite the different ODE time grids.Source: §1 · p. 1, Figure 2 · p. 2
- FSB uses step-dependent DWT sub-band reweighting to suppress excessive approximation-band growth and boost lagging detail components.The method is training-free and adapts its modulation strength to the integration stage.Source: §1 · pp. 1–2
- The paper reports that principled sub-band coordination yields gains across architectures and languages, whereas constant-strength and linear interventions fail to correct the imbalance.The schedules require one-time calibration and can transfer across related architectures and languages.Source: §1 · p. 2
2. METHODOLOGY
The methodology characterizes asynchronous DWT sub-band development in CFM-based TTS and introduces FSB to reweight bands during ODE integration. Its schedules are adapted to each model’s observed trajectory profile.
2.1. Sub-band Evolution in CFM-based TTS
- F5-TTS shows continued LL and HL growth, while Matcha-TTS has near-linear LL growth and delayed detail-band development.The analysis tracks the ℓ2-norms of DWT coefficients in the approximation and three detail sub-bands.Source: §2.1 · p. 2
2.2. Frequency-Selective Boosting
- FSB decomposes each intermediate mel-spectrogram, scales selected DWT sub-bands, and reconstructs the modified state at each ODE evaluation.Its clipped, scaled inverse-trajectory weighting suppresses aggressively growing components and increases the relative contribution of energy-deficient components.Source: §2.2 · p. 2
- Schedules are calibrated from aggregate sub-band growth curves and validation metrics, then fixed for inference without retraining.The calibration grid-searches 2–4 scalar parameters per modulated sub-band; IndicF5 reuses the F5-TTS schedule unchanged for Tamil and Bengali.Source: §2.2 · p. 3
- F5-TTS uses exponential attenuation for LL and HL, leaving LH and HH unchanged; Matcha-TTS applies linear corrections to LH and HL, leaving LL and HH unchanged.The schedules reflect the models’ different trajectory profiles.Source: §2.2 · p. 3
3. EXPERIMENTAL SETUP
The setup evaluates F5-TTS, IndicF5, and Matcha-TTS using model checkpoints and speech datasets spanning English and Indic languages.
Model Specifications
- F5-TTS and IndicF5 use approximately 350M-parameter checkpoints, while Matcha-TTS uses a single-speaker LJSpeech-trained checkpoint.The provided setup passage ends after identifying the F5-TTS checkpoint and does not specify its source.Source: §3 · p. 3
Evaluation Materials
- Table 1 is identified as covering frequency-domain interventions on F5-TTS.The supplied caption does not state the table's rows, columns, or results.Source: Table 1 · p. 3
Evaluation Datasets
- Evaluation covers LibriTTS-clean and -other for F5-TTS, IndicVoices-r for Tamil and Bengali, and LibriSpeech test text for Matcha-TTS.The listed sample sizes are approximately 5,000 per LibriTTS split, 300 IndicVoices-r samples, and 400 Matcha-TTS samples.Source: §3 · p. 3
4. RESULTS AND ANALYSIS
The results report that 26-NFE FSB improves FAD over matched 26-NFE baselines and can outperform 32-NFE inference, with gains across models and languages. Schedule and trajectory diagnostics further examine the method's configuration and stability.
4.1. Quality–Efficiency Trade-off
- FAD falls from 1.04 to 0.51 with 26-NFE FSB versus the 26-NFE baseline, and FSB outperforms the 32-NFE baseline on all metrics.Extending FSB to 32 NFE yields only marginal further gains; on an NVIDIA H200, 26-NFE FSB reduces average runtime by 16.4% (1.119 s vs. 1.339 s) and RTF by 15.9%.Source: §4.1 · p. 3
4.2. Cross-model and Perceptual Evaluation
- FAD reductions versus matched 26-NFE baselines are 48% on LibriTTS-clean, 52% on LibriTTS-other, and 61% on Matcha-TTS.WER/CER and speaker similarity remain close to 32-NFE references; predicted MOS exceeds the 32-NFE Matcha-TTS score (4.37 vs. 4.31). Human MOS pilot results favor 26-step FSB on intelligibility and perceptual quality and tie it with the 32-step baseline on naturalness (3.80 vs. 3.81).Source: §4.2 · pp. 3–4, Table 2 · p. 4, Table 3 · p. 4
4.3. Ablation: Reweighting Schedules
- The exponential LL schedule achieves the lowest FAD on both splits; the linear schedule is competitive, while the degree-3 polynomial schedule fails catastrophically.The comparison uses the cutoff-26/32 diagnostic, with other sub-band coefficients held fixed.Source: §4.3 · p. 4, Table 4 · p. 4
4.4. Ablation: Cross-lingual Transfer
- On IndicF5, FSB halves FAD for Tamil (7.58→4.37) and Bengali (9.30→4.32), while predicted MOS and speaker similarity also improve for both languages.IndicF5 reuses the F5-TTS schedule unchanged; ASR metrics are mixed, and ground-truth recordings have WER of 62–65%, making ASR unreliable in this setting.Source: §4.4 · p. 4
4.5. Trajectory Stability
- FSB (LL+HL) maintains baseline-level mean acceleration (1.01×) while reducing peak acceleration to 0.80×, without corrective spikes.LL-only modulation has higher divergence than LL+HL (r=0.089 vs. 0.049); Pre-Emphasis increases acceleration by more than 6× with strongly exponential divergence (R2=0.971).Source: §4.5 · p. 4, Table 5 · p. 4
5. CONCLUSION
The paper identifies incoherent DWT sub-band evolution in CFM-based TTS and proposes training-free Frequency-Selective Boosting (FSB) to coordinate spectral development during ODE integration. Across F5-TTS, Matcha-TTS, and IndicF5, it reduces the required NFE from 32 to 26 and improves FAD by up to 61% without degrading intelligibility or perceptual quality.
Conclusion
- FSB reduces the required NFE from 32 to 26 and improves FAD by up to 61% across F5-TTS, Matcha-TTS, and IndicF5, without degrading intelligibility or perceptual quality.The schedules require one-time calibration and transfer across languages; future work includes automatic schedule estimation and extension to waveform-domain flow models.Source: §5 · p. 4