Source-linked AI summary
Fr\'echet Distance Loss on Speech Representations for Text-to-Speech Synthesis
Ho-Lam Chung, Kuan-Po Huang, Bo-Ru Lu, Hung-yi Lee
TL;DR
Few-step TTS training losses do not directly constrain the distribution of sampled speech, leaving the outputs of compressed samplers insufficiently matched to high-quality speech. SR-FD matches frozen Whisper and CTC feature moments from deployment-sampler speech to three offline content targets. On Seed-TTS English, four-step SR-FD lowers WER below both original four-step and ten-step baselines.
Problem
Local training objectives do not directly constrain whether complete utterances sampled by a compressed TTS model follow the distribution of high-quality speech.
Source: §I · p. 1Method
SR-FD fine-tunes a tokenizer-free flow-matching TTS model by matching generated-speech Whisper and CTC feature moments to three offline reference targets.
Source: §I · p. 1, §III · p. 2, C. Speech Representation Targets · p. 3Results
Four-step SR-FD reaches 1.41% WER, versus 2.23% for the original four-step baseline and 1.74% for the ten-step baseline, with both reductions statistically significant.
Source: A. Main Seed-TTS Result · p. 5Takeaways & Limitations
On Seed-TTS English, gains mainly reflect fewer content substitutions across prompt lengths while speaker similarity and quality proxies remain near the ten-step level.
Source: B. Speaker Similarity, Quality… · p. 5, §VI · p. 6
Abstract
from arXiv · showhide
Few-step diffusion and flow-matching text-to-speech (TTS) models are usually trained with local objectives, such as conditional flow matching, reconstruction, and stop prediction. These losses provide stable optimization, but they never ask whether sampled speech follows the distribution of high-quality speech. We propose Speech Representation Fr'echet Distance loss (SR-FD), a training-time distributional regularizer for tokenizer-free flow-matching autoregressive TTS. During fine-tuning, the model synthesizes speech with the same few-step sampler used at deployment, and SR-FD matches the mean and covariance of frozen Whisper and CTC features of this speech to reference statistics computed offline from three complementary content targets. The loss requires no discriminator and no inference-time computation. On Seed-TTS English, four-step SR-FD fine-tuning reduces WER from the original four-step VoxCPM2 baseline's 2.2279% to 1.4147%, a 36.5% relative reduction, and also surpasses the original ten-step baseline at 1.7366%; both gains are significant under an utterance-level paired bootstrap. Speaker similarity and objective quality proxies are preserved at the ten-step level, and an error analysis shows the gain comes from content substitutions across all prompt lengths. SR-FD is thus an intelligibility-improving distributional regularizer for few-step TTS.
I. INTRODUCTION
Few-step TTS objectives supervise local trajectories but do not directly constrain the distribution of complete sampled utterances. SR-FD matches generated-speech representation statistics to offline content targets and improves four-step intelligibility on Seed-TTS English.
I. INTRODUCTION
- Local flow-matching, reconstruction, and stop-prediction objectives do not directly constrain the population distribution of complete utterances produced by a compressed sampler.The resulting drift may affect content, duration, prosody, and acoustic quality.Source: §I · p. 1
- SR-FD matches generated-speech representation statistics to offline reference statistics using the same low-step sampler used at inference.It combines three complementary targets across Whisper semantic and CTC posterior spaces, without adding inference-time computation.Source: §I · pp. 1–2
- Four-step SR-FD reaches 1.41% WER, 36.5% below the four-step baseline and 18.5% below the ten-step baseline, with both reductions significant under paired bootstrap.Speaker similarity is preserved, and a blinded listening test finds no reliable listener preference against the ten-step baseline.Source: §I · p. 2
II. RELATED WORK
Prior work reduces generation steps or compares feature distributions, while SR-FD applies a lightweight distributional regularizer to the outputs of a few-step TTS sampler. The method builds on VoxCPM2’s continuous flow-matching acoustic decoder and existing fine-tuning losses.
II. RELATED WORK
- SR-FD turns Gaussian feature-statistic comparison into a differentiable training loss and complements step-reduction methods by targeting the distribution produced by the few-step sampler.It uses frozen encoders and precomputed moments, without adversarial or learned components.Source: §II · p. 2
III. METHOD
- SR-FD adds a training-time objective that matches generated speech feature means and covariances to offline reference moments, without a discriminator or inference-time computation.The pipeline uses the deployment-time four-step sampler and sends gradients only to LoRA weights.Source: §III · p. 2, Figure 1 · p. 2
A. Base Model and Training Objective
- VoxCPM2 decodes continuous AudioVAE latents with a diffusion transformer trained by conditional flow matching, and inference integrates its learned velocity field from noise using Euler steps.The fine-tuning objective also includes stop prediction and inherited auxiliary losses; SR-FD is added on top.Source: A. Base Model and Training Objective · pp. 2–3
B. Matching the Sampled-Speech Distribution
SR-FD addresses a mismatch in standard fine-tuning by evaluating generated speech with the deployment-time sampler rather than relying on teacher-forced frames.
B. Matching the Sampled-Speech Distribution
- Teacher-forced fine-tuning never includes the few-step sampler in its loss, so training can look healthy while the four-step sampler drifts.The model sees ground-truth history and predicts the next frame.Source: B. Matching the Sampled-Speech… · p. 3
- During each update, SR-FD differentiably generates a complete short utterance with the deployment-time four-step sampler.This makes the sampled speech itself part of the training objective.Source: B. Matching the Sampled-Speech… · p. 3
- Frozen extractors map generated audio to utterance-level feature vectors used only during training, constraining the sampler's deployment distribution rather than the supervised trajectory.Each vector's dimension is the corresponding extractor's feature dimension.Source: B. Matching the Sampled-Speech… · p. 3
C. Speech Representation Targets
SR-FD uses frozen Whisper and wav2vec 2.0 CTC extractors with three content targets. The targets anchor low-step semantic content, teacher phonetic content, and natural speech content.
C. Speech Representation Targets
- Whisper features represent semantic content, while wav2vec 2.0 CTC features represent phonetic content.Source: C. Speech Representation Targets · p. 3
- The three targets combine an ASR-verified four-step Whisper anchor with teacher-generation and real-speech targets in the CTC space.The low-step anchor uses generations whose ASR transcripts match the target text; the teacher target transfers content behavior from ten-step VoxCPM2, while real-speech statistics tie CTC features to natural speech.Source: C. Speech Representation Targets · p. 3, Table I · p. 3
- The real-speech CTC target is intended to keep the CTC branch from simply copying teacher-specific artifacts.Source: C. Speech Representation Targets · p. 3
D. Reference and Generated Moments
Reference moments are computed offline from target-audio features, and only the resulting statistics are retained. Generated moments use a recent-feature queue to estimate covariance without retaining gradients beyond the current batch.
D. Reference and Generated Moments
- Reference audio is encoded into features, from which the target-specific mean and covariance statistics are computed offline and stored.The reference audio is not used again after these moments are stored.Source: D. Reference and Generated Moments · p. 3
- Whisper covariance is rank-deficient because its feature dimension is high relative to the available sample count, so the method adds a small diagonal term and symmetrizes before the matrix square root.The absolute Whisper Fréchet value is biased by rank deficiency and is not used for model selection; target validation instead uses the WER ablation.Source: D. Reference and Generated Moments · p. 3
- A queue of recent generated features supplies population-scale covariance estimates while keeping memory cost at one batch.Earlier-step features are detached, so only the current mini-batch keeps gradients.Source: D. Reference and Generated Moments · pp. 3–4
E. The SR-FD Loss
SR-FD compares generated and reference Gaussian feature moments across extractors and targets. Detached normalization balances target contributions by gradient scale, while duration gating excludes strongly mismatched samples from the loss.
E. The SR-FD Loss
- SR-FD computes a Fréchet distance between generated and reference Gaussian moment estimates for each extractor and target.Source: E. The SR-FD Loss · p. 4
- Each distance term is divided by its own detached value so differently scaled feature spaces do not let the largest raw distance dominate.The normalized terms have magnitude near one while retaining a gradient direction that reduces Fréchet distance, balancing targets by gradient scale.Source: E. The SR-FD Loss · p. 4
- A duration gate admits samples only when duration is close to the target, excluding strongly mismatched speech that may inject noise into moment matching.At test time, the extractors, queues, reference moments, and Fréchet computation are absent.Source: E. The SR-FD Loss · p. 4
IV. EXPERIMENTAL SETUP
Fine-tuning uses a LibriTTS-derived manifest and offline reference statistics, while evaluation uses the Seed-TTS English test set. The full set supplies the primary WER metric; a smaller gate subset is reserved for screening and target-ablation checkpoint selection.
IV. EXPERIMENTAL SETUP
- Fine-tuning uses a 767-row LibriTTS-derived manifest with target text, continuation-prompt information, and per-utterance weights for hard lexical examples and teacher-consistent rows.Source: A. Data and Evaluation Protocol · p. 4
- Reference moments come from real LibriTTS speech, ten-step VoxCPM2 teacher generations, and ASR-verified four-step generations.The CTC targets each use 4,999 utterances at dimension 72; the Whisper target uses 1,000 utterances at dimension 960.Source: A. Data and Evaluation Protocol · p. 4
- The primary WER evaluation uses all 1,088 Seed-TTS English test prompts and 11,805 reference words; the 200-prompt, 2,070-word gate subset is only for screening and target-ablation checkpoint selection.Source: A. Data and Evaluation Protocol · p. 4
B. Systems Compared
The evaluation compares four- and ten-step VoxCPM2 with four-step SR-FD, alongside selected published flow-matching results. It uses a fixed decoding protocol and evaluates recognition, speaker identity, objective quality proxies, and perceptual preference.
B. Systems Compared
- The comparison includes pretrained VoxCPM2 at four and ten steps, four-step SR-FD, and best-published test-en results for F5-TTS and ARCHI-TTS.Audio for the published systems was unavailable for similarity and proxy scoring.Source: B. Systems Compared · p. 4
C. Optimization and Decoding
- Fine-tuning first adapts LoRA with Lbase, then runs 1,600 steps with SR-FD; evaluation uses shared decoding settings, differing by inference-step count.The reported SR-FD model is the step-1600 three-target checkpoint selected on the full-set WER frontier; ablation checkpoints use the 200-prompt gate subset.Source: C. Optimization and Decoding · pp. 4–5, D. Metrics and Selection · p. 5
D. Metrics and Selection
- Evaluation uses upstream Seed-TTS English WER and utterance-level paired bootstrap over 1,088 prompts, with SIM and objective proxies for speaker identity and quality.A blinded pairwise listening test compares four-step SR-FD with the ten-step baseline; the SR-FD checkpoint is selected on the full-set WER frontier.Source: D. Metrics and Selection · p. 5
A. Main Seed-TTS Result
Four-step SR-FD lowers Seed-TTS English WER below both original VoxCPM2 baselines. Speaker similarity and objective quality proxies are preserved, and the loss adds no inference-time computation.
B. Speaker Similarity, Quality Proxies, and Cost
- SIM rises from 0.74 to 0.76, matching the ten-step baseline; UTMOS and DNSMOS recover most of the four-to-ten-step gap.The proxies are not human ratings; the listening test reports no reliable preference and supports equivalence within the prespecified ±10-point margin.Source: B. Speaker Similarity, Quality… · p. 5, Table III · p. 5
A. Main Seed-TTS Result
- 167/11805 = 1.41% WER for four-step SR-FD, versus 263/11805 = 2.23% for four-step VoxCPM2 and 205/11805 = 1.74% for ten-step VoxCPM2.The reductions versus the four-step and ten-step baselines are statistically supported by paired bootstrap.Source: Table II · p. 5, A. Main Seed-TTS Result · p. 5
B. Speaker Similarity, Quality Proxies, and Cost
- Four-step SR-FD has an aggregate real-time factor of 0.2285, about 8.6% below the ten-step baseline at 0.2501.The authors claim zero added inference cost from the loss, not a large speedup from step reduction.Source: B. Speaker Similarity, Quality… · p. 5
C. Blinded Listening Test
The listening test supports practical perceptual equivalence between four-step SR-FD and the ten-step baseline within a ±10-point margin. Error analyses and target ablations examine where the WER gain comes from and which references contribute.
C. Blinded Listening Test
- Listeners show no reliable preference between four-step SR-FD and the ten-step baseline; TOST supports equivalence within the prespecified ±10-point margin.This supports practical equivalence under that margin, not a tighter claim.Source: C. Blinded Listening Test · p. 5
D. Where the Improvement Comes From
- SR-FD reduces every error type versus both baselines: substitutions fall to 142 from 203 for four-step and 168 for ten-step; deletions fall from 43 to 21.Insertions remain rare, and the analysis reports no trade of one error type for another.Source: D. Where the Improvement Comes From · p. 5, Table IV · p. 6
- SR-FD is below both baselines in every reference-length bucket, with its largest gain over four-step VoxCPM2 on short prompts.The model also makes occasional regressions on prompts both original baselines transcribe correctly, so text fidelity does not improve on every prompt.Source: D. Where the Improvement Comes From · pp. 5–6, Table V · p. 6
E. FD Target Ablation
- Removing any of the three reference targets worsens full-set WER; removing the low-step Whisper anchor causes the largest increase, from 167 to 182 errors.Removing the real-speech and teacher CTC targets raises errors to 176 and 175, respectively; the one-error difference is too small to rank them.Source: E. FD Target Ablation · p. 6
F. Is the Fréchet Distance Itself a Good Diagnostic?
Raw Fréchet distance is not a reliable checkpoint selector, even though SR-FD improves few-step intelligibility. The conclusion reports lower four-step WER than both the four-step and ten-step baselines, while recommending external metrics for selection.
F. Is the Fréchet Distance Itself a Good Diagnostic?
- Across 16 checkpoints, raw FD was weakly associated with gate WER: real-speech CTC FD was strongest (ρ = 0.383, p = 0.143), while anchor FD showed none (ρ = −0.021).The conclusion therefore says FD alone cannot select checkpoints or models; external WER remains necessary.Source: F. Is the Fréchet Distance Itself a… · p. 6, §VI · p. 6
- The leave-one-target-out ablation selects checkpoints on a 200-prompt gate subset, with UTMOS and DNSMOS staying within 0.005 of the full model.Source: Table VI · p. 6
- External CTC FD to real target speech fell from 0.00795 for the original four-step baseline to 0.00681 for SR-FD, while the ten-step baseline reached 0.00517 despite higher WER.The paper treats this separate-extractor, 200-prompt pass as an external sanity check.Source: F. Is the Fréchet Distance Itself a… · p. 6
VI. CONCLUSION
- 1.41% WER for four-step SR-FD is below the four-step baseline’s 2.23%, ten-step baseline’s 1.74%, and 32-step ARCHI-TTS reference’s 1.47%.Both gains over the VoxCPM2 baselines are bootstrap-significant.Source: §VI · p. 6
- The WER gain mainly reflects fewer content substitutions across prompt lengths, and weakens whenever any reference target is removed.Source: §VI · p. 6