Source-linked AI summary
DriftTTS: One-stage Single Step Text-to-Speech via Generative Drifting
Jaehyun Ahn, Dohyun Kim, Heeseung Kim
TL;DR
Score- and flow-matching TTS often requires iterative decoding, while existing low-NFE approaches depend on pretrained TTS models. DriftTTS uses generative drifting with segment-level training and achieves comparable quality to multi-step systems in a single NFE.
Problem
Score- and flow-based TTS typically requires repeated function evaluations, and existing low-NFE approaches rely on pretrained TTS models.
Source: §1 · p. 1Method
Results
About 4.2× and 1.3× faster than Matcha-TTS at ten steps and RapFlow-TTS at two steps, DriftTTS achieves comparable WER and MOS with one NFE.
Source: §4.3 · p. 4Takeaways & Limitations
DriftTTS reaches multi-step quality in a single-stage model without a pretrained TTS model, avoiding inference cost that grows with the number of steps.
Source: §5 · p. 4, §4.3 · p. 4Direct kernel computation in high-dimensional mel space can be unstable, and rigorous analysis of this behavior remains future work.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Text-to-speech models built on score matching and flow matching produce natural speech, but only by calling the decoder many times, which makes synthesis slow. Existing ways to cut the number of calls either distill from a multi-step teacher or add a stage on top of a pretrained model, leaving single-step TTS without a pretrained TTS model unexplored. We question whether pretraining a TTS model is necessary, and propose DriftTTS, which adopts generative drifting, an objective that moves iterative sampling from inference into training. Drifting assumes samples of fixed shape, whereas speech has variable length; we address this issue with segment drifting. On LJSpeech, DriftTTS matches the multi-step quality of Matcha-TTS and RapFlow-TTS in a single decoder call, while being faster. To our knowledge, this is the first application of generative drifting to text-to-speech.
1. INTRODUCTION
Existing TTS systems reduce inference cost imperfectly: many require repeated decoding, while low-NFE approaches rely on pretrained TTS models. DriftTTS applies generative drifting without a pretrained TTS model and performs on par with existing few-step systems in a single NFE.
1. INTRODUCTION
- IntMeanFlow and RapFlow-TTS achieve low-NFE synthesis, but both depend on a pretrained TTS model to provide a generation trajectory.IntMeanFlow uses one NFE; RapFlow-TTS uses two and matches or surpasses ten-step Matcha-TTS.Source: §1 · p. 1
- Score- and flow-based TTS can produce high-quality speech, but typically require ten or more function evaluations, while iterative sampling remains an inference bottleneck.Low-NFE image-generation methods include MeanFlow and Consistency Flow Matching.Source: §1 · p. 1
- DriftTTS applies generative drifting to TTS without relying on a pretrained TTS model, moving iterative sampling from inference into training.The paper identifies this as the first application of generative drifting to TTS.Source: §1 · p. 1
- DriftTTS performs on par with existing few-step TTS models using one NFE, and ablations show that the feature extractor and segment-level drifting are both necessary.Without either component, training leads to loudness divergence or false convergence.Source: §1 · p. 1
2. GENERATIVE DRIFTING
Generative drifting learns a pushforward map from prior samples to the data distribution by training generated samples toward their drifted counterparts. DriftTTS adapts this approach to speech using fixed-length segments and feature-space comparisons.
2. GENERATIVE DRIFTING
- Generative drifting treats a neural network’s outputs as a pushforward distribution and defines each sample’s drift as its change between training steps.A zero drift corresponds to a fixed point of the update.Source: §2 · p. 2
- The drifting field combines attraction toward data samples with interaction among generated samples, using a kernel to measure sample similarity.In practice, expectations are estimated over minibatches.Source: §2 · p. 2
- For high-dimensional data, generative drifting computes the field in an extractor’s feature space and trains samples toward fixed drifted targets.The objective equals the squared magnitude of the drifting field, with the drifted sample treated as fixed during backpropagation.Source: §2 · p. 2
- DriftTTS duration-repeats encoded text representations to form the prior, then compares a single-call generated segment with a target segment in MAE feature space.This segment-based pipeline adapts drifting to speech generation.Source: Figure 1 · p. 2
3. DRIFTTTS
DriftTTS adapts Matcha-TTS to predict target mel-spectrograms directly in one decoder step. Segment drifting handles variable-length speech by training on fixed-length segments conditioned on aligned text representations.
3.1. Architecture of DriftTTS
- The decoder directly predicts the target mel-spectrogram instead of iteratively predicting and integrating a velocity field, removing its timestep embedding.DriftTTS retains the text encoder and duration predictor of the Matcha-TTS architecture.Source: §3.1 · p. 2
3.2. Segment Drifting
- Segment drifting samples fixed-length prior and target speech segments during training to accommodate speech's variable length.The method addresses text conditioning by redefining the prior as the encoder output and providing decoder noise as a stochastic condition.Source: §3.2 · pp. 2–3
- Because each aligned prior corresponds to one target, target segments serve as positive samples; generated segments serve as negatives with self-interactions masked.Attraction to another speech segment could otherwise provide an incorrect training signal, even if the generated distribution matched the data distribution.Source: §3.2 · p. 3
- The feature extractor is a masked autoencoder trained on mel-spectrograms, and its corresponding encoder-layer representations are compared in pairs.Training applies 4×4 masks with 50% probability.Source: §3.2 · p. 3
- The drifting loss averages across temperature and MAE-layer combinations because summing them increases text-encoder gradients and reduces the prior loss's relative influence.The final objective trains the text encoder and duration predictor with prior and duration losses, and the decoder with the drifting loss.Source: §3.2 · p. 3
4. EXPERIMENTS
On LJSpeech, DriftTTS reaches quality comparable to multi-step Matcha-TTS and RapFlow-TTS in one step, without pretrained TTS, and runs faster than those multi-step settings. Ablations show that the feature extractor and segment-level drifting matter to training stability and speech quality.
4.1. System
- DriftTTS and its MAE were trained on LJSpeech, and HiFi-GAN converted predicted mel-spectrograms to speech.Training used 80-frame decoder segments and batches of 64 text-speech pairs per GPU.Source: §4.1 · p. 3
4.2. Evaluations
- The evaluation compares single-step DriftTTS with single-step baselines and with ten-step Matcha-TTS and two-step RapFlow-TTS.WER measures pronunciation accuracy, MOS perceptual naturalness, and RTF synthesis speed excluding the vocoder.Source: §4.2 · p. 3
4.3. Results
- DriftTTS runs about 4.2× faster than ten-step Matcha-TTS and 1.3× faster than two-step RapFlow-TTS while achieving comparable WER and MOS.It reaches this quality with a single NFE and single-stage training without a pretrained TTS model.Source: Table 1 · p. 4, §4.3 · p. 4
- Removing the MAE causes every metric to collapse as predicted mel-bin energy diverges; the authors leave rigorous analysis of this behavior to future work.Table 2 reports ablations of the feature extractor, drifting unit, and temperature.Source: Table 2 · p. 4, §4.3 · p. 4
- Frame drifting retains reasonable WER but sharply lowers UTMOSv2 and DNSMOS, as frames can match incorrect targets without preserving natural neighboring-frame connections.Segment drifting avoids this by tying neighboring frames together; Fig. 3 depicts the frame-wise prediction and target for a validation utterance.Source: §4.3 · p. 4, Figure 3 · p. 4
5. CONCLUSION
DriftTTS moves iterative sampling into training and predicts target mel-spectrograms directly in one step without a pretrained TTS model. Segment drifting adapts this approach to speech, while multispeaker and zero-shot synthesis remain future work.
5. CONCLUSION
- DriftTTS predicts the target mel-spectrogram directly in a single step without a pretrained TTS model.It is built on generative drifting, which moves iterative sampling from inference to training.Source: §5 · p. 4
- Segment drifting samples fixed-length training segments and uses the encoded text representation as the prior.This removes inference cost that grows with the number of steps while performing on par with existing models' multi-step results.Source: §5 · p. 4
- Multispeaker and zero-shot synthesis, and reducing dependence on the trained feature extractor, remain future work.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The listening test used Amazon Mechanical Turk, obtained participant consent, and collected only non-sensitive evaluation data.
6. COMPLIANCE WITH ETHICAL STANDARDS
- The online Amazon Mechanical Turk listening survey obtained participant consent and collected only non-sensitive evaluation data.Source: §6 · p. 5