Source-linked AI summary
DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift
Mohammad Nur Hossain Khan, Subrata Biswas, Bashima Islam
TL;DR
Whether competitive few-step TTS can avoid per-sample regression, distillation, adversarial training, and prescribed trajectories remained unexamined. DriftTTS applies distribution-matching drift with on-policy rollout, and its NFE=4 system scored 4.18 MOS in a fully paired blind test, comparable to Matcha-TTS and ground truth.
Problem
Whether a competitive few-step TTS model could avoid per-sample regression, adversarial discrimination, and consistency along a prescribed trajectory had not been examined, to the authors' knowledge.
Source: §1 · p. 1Method
DriftTTS trains a TTS decoder with distribution-matching drift in a mel-domain feature space and uses on-policy rollout for few-step generation without a generative teacher, distillation, or adversarial training.
Source: §1 · pp. 1–2Results
4.18 MOS for DriftTTS at NFE=4 was comparable to Matcha-TTS (3.96) and ground truth (4.22) in a fully paired blind listening test.
Source: §1 · p. 2Takeaways & Limitations
The results show that competitive few-step speech synthesis can be achieved without a pretrained generative teacher or distillation.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Few-step neural text-to-speech models often rely on short- ened diffusion or flow-matching schedules, or on distillation from pretrained multi-step teachers. To avoid these depen- dencies, we present DriftTTS, a few-step mel-spectrogram generator trained without a generative teacher, distillation, or adversarial discrimination. DriftTTS uses a distribution- matching drift objective in a mel-domain feature space defined by raw mels and a frozen masked-autoencoder encoder pretrained on the same LJSpeech training split. On-policy rollout trains the decoder on its own interme- diate states and supports inference up to the trained roll- out depth. On LJSpeech, DriftTTS at NFE=4 achieves 3.87 dB MCD and 3.7% WER, compared with 3.85 dB and 3.4% for Matcha-TTS. In a fully paired blind listen- ing test, DriftTTS obtains 4.18 MOS, compared with 3.96 for Matcha-TTS and 4.22 for ground truth. These results demonstrate competitive few-step synthesis without a pre- trained generative teacher. Code can be found at https: //github.com/BASHLab/driftTTS.git
1. INTRODUCTION
Prior few-step TTS approaches use per-sample regression, distillation, adversarial training, or prescribed trajectories; DriftTTS adapts distribution-matching drift to TTS with local positive pools and on-policy rollout. Its fully paired blind test reports NFE=4 naturalness comparable to Matcha-TTS and ground truth.
1. INTRODUCTION
- Diffusion and flow-matching TTS use per-sample targets, while one-step routes include distillation; whether competitive few-step TTS could avoid these dependencies remained unexamined.The cited approaches include Grad-TTS, Matcha-TTS, CoMoSpeech, and ProDiff.Source: §1 · p. 1
- DriftTTS replaces per-sample targets with a distribution-matching drift, using a condition-specific local positive pool and avoiding a teacher, discriminator, or prescribed trajectory.The local positive pool is built around each paired target segment.Source: §1 · pp. 1–2
- Random-depth on-policy rollout applies drift at the model's own visited states, allowing inference at any NFE up to the trained depth; step count was the largest quality lever in experiments.The feature space combines raw mels with a frozen masked-autoencoder encoder pretrained on the LJSpeech training split, which is discarded at inference.Source: §1 · pp. 1–2
- 4.18 MOS at NFE=4 was comparable to Matcha-TTS (3.96) and ground truth (4.22) in a fully paired blind listening test.The comparison is reported for DriftTTS at NFE=4.Source: §1 · p. 2
2. DRIFTTTS
DriftTTS replaces the prescribed noise-to-data field with a distribution-matching drift objective in feature space. On-policy rollout trains the decoder on its own visited states, and inference can run the same model up to its trained depth.
2. DRIFTTTS
- Generated-to-real attraction and generated-to-generated repulsion form a distribution-matching drift that replaces the prescribed noise-to-data field.The distance matrices are normalized by their batch means, and each scale is normalized by its batch RMS.Source: §2 · p. 2
- The drift objective regresses toward a displaced representation in a frozen mel-domain feature space, with the force pulled back to update the text encoder and decoder.The encoder remains fixed, and the normalized loss value is not used to monitor convergence.Source: §2 · p. 2
2.1. Training
- The text side follows the Grad-TTS/Matcha-TTS framework, using Monotonic Alignment Search and prior and duration-prediction losses.The prior loss compares the frame-level proxy with the target mel-spectrogram.Source: §2.1 · p. 2
- Training generates eight samples per text condition and matches them against the paired target plus seven Gaussian-perturbed target views.The segments are 192 frames, and the perturbations use σ = 0.05 in normalized mel space.Source: §2.1 · p. 2
- Drift features combine raw 80-bin mels with block-3 and block-6 activations from a frozen MelMAE pretrained on the LJSpeech training split.MelMAE is used only during training; its reconstruction decoder is discarded and it adds no inference cost.Source: §2.1 · p. 2
- On-policy training evaluates the drift at states reached by the current model after a randomly sampled number of rollout steps, up to depth K.The repulsive component is concentrated at early rollout states and acts primarily as an early-rollout shaping term.Source: §2.1 · p. 2
2.2. Inference and architecture
- At inference, generation starts from Gaussian noise and uses replacement updates; the model can run at any NFE up to trained depth K, with K = 4 in the primary configuration.A pretrained HiFi-GAN vocoder converts the generated mel-spectrogram to waveform.Source: §2.2 · p. 3, Figure 1 · p. 3
3. EXPERIMENTS
Experiments on LJSpeech compare DriftTTS with established TTS systems and examine objective, feature-space, and rollout-depth choices. DriftTTS is competitive with Matcha-TTS at four evaluations, while ablations show sensitivity to training depth and modeling choices.
3.1. Experimental Setup
- Evaluation uses the official LJSpeech partition, a shared HiFi-GAN vocoder, and objective measures plus a fully paired blind MOS test.The study compares DriftTTS and Matcha-TTS at NFE 1, 2, and 4, and Grad-TTS at NFE=10.Source: §3.1 · p. 3
3.2. Main Results
- At NFE=4, DriftTTS reaches 3.87 dB MCD and 3.7% WER versus Matcha-TTS’s 3.85 dB and 3.4%, while using fewer parameters.Table 1 reports the LJSpeech evaluation; all synthesized systems use the same HiFi-GAN vocoder.Source: §3.2 · p. 3, Table 1 · p. 4
- DriftTTS at NFE=4 scores 4.18 ± 0.16 MOS, versus 3.96 for Matcha-TTS and 4.22 for ground-truth recordings.The fully paired blind listening test had 15 raters evaluate the same 20 utterances across five systems.Source: §3.2 · p. 3, Table 2 · p. 4
- Both DriftTTS and Matcha-TTS exceed Grad-TTS-10 at 3.55 using fewer evaluations; at NFE=4, DriftTTS’s decoder takes 7.08 ms versus Matcha-TTS’s 11.22 ms.The decoder latency corresponds to a 1.58× speedup for DriftTTS.Source: §3.2 · p. 4
3.3. Ablations and Analysis
- Removing MelMAE features raises MCD from 3.873 to 4.065 dB; replacing drift with ℓ2 regression raises it to 4.309 dB.The raw-mel-only drift model also outperforms the MelMAE-based ℓ2 model, with MCD 4.065 versus 4.309 dB.Source: Table 3 · p. 4, §3.3 · p. 4
- Increasing training depth from K=2 to K=4 reduces MCD from 4.76 to 3.87 dB and WER from 7.05% to 3.7%, while UTMOS rises from 2.818 to 4.253.The results motivate K=4 as the primary configuration.Source: Table 4 · p. 4, §3.3 · p. 4
- Most test utterances exceed the 192-frame training window without degradation, but the authors restrict inference to NFE ≤ K.474 of 500 test utterances exceed the training window; a K=2 model’s MCD rises from 4.76 dB at NFE=2 to 5.81 dB at NFE=4.Source: §3.3 · p. 4
4. CONCLUSION
DriftTTS uses a distribution-matching drift objective without a pretrained generative teacher, distillation, or adversarial training. On LJSpeech, it achieves objective quality close to Matcha-TTS and a 4.18 MOS at NFE=4; ablations support the importance of the drift objective and combined mel-domain features.
4. CONCLUSION
- DriftTTS achieves competitive few-step speech synthesis without a pretrained generative teacher or distillation.The model is trained with a distribution-matching drift objective and without adversarial training.Source: §4 · p. 4