Source-linked AI summary
OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching
Hieu-Nghia Huynh-Nguyen, Ngoc Son Nguyen, Huynh Nguyen Dang, Thieu Vo, Truong-Son Hy, Van Nguyen
TL;DR
Diffusion-based TTS can be computationally inefficient, while flow-matching approaches may add training constraints. OZSpeech uses a learned prior for one-step OT-CFM and reports improved WER and faster inference, though naturalness remains challenging.
Problem
Diffusion-based TTS uses computationally inefficient multi-step sampling, and training constraints can make approaches to reducing those steps resource-demanding.
Source: §1 · p. 2Method
OZSpeech reformulates OT-CFM to generate speech in one step from a learned prior, using acoustic-prompt prosody and detail codes while masking content codes.
Source: §1 · p. 2, §3.3 · p. 4Results
Takeaways & Limitations
The method offers lower-computation synthesis while balancing intelligibility and acoustical quality.
Source: §5 · p. 8Rounding real-valued ground-truth phoneme durations to integer values degrades synthesized speech quality in the temporal domain.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Text-to-speech (TTS) systems have seen significant advancements in recent years, driven by improvements in deep learning and neural network architectures. Viewing the output speech as a data distribution, previous approaches often employ traditional speech representations, such as waveforms or spectrograms, within the Flow Matching framework. However, these methods have limitations, including overlooking various speech attributes and incurring high computational costs due to additional constraints introduced during training. To address these challenges, we introduce OZSpeech, the first TTS method to explore optimal transport conditional flow matching with one-step sampling and a learned prior as the condition, effectively disregarding preceding states and reducing the number of sampling steps. Our approach operates on disentangled, factorized components of speech in token format, enabling accurate modeling of each speech attribute, which enhances the TTS system's ability to precisely clone the prompt speech. Experimental results show that our method achieves promising performance over existing methods in content accuracy, naturalness, prosody generation, and speaker style preservation. Audio samples are available at our demo page https://ozspeech.github.io/OZSpeech_Web/.
1 Introduction
Zero-shot TTS must reproduce an unseen speaker’s acoustic characteristics from a prompt, but existing approaches trade off sampling efficiency, training cost, and reliability. OZSpeech uses a learned prior with one-step OT-CFM and reports faster inference alongside improved WER.
1 Introduction
- Diffusion-based TTS can produce natural-sounding audio but its multi-step sampling is computationally inefficient for real-time use.Consistency Models reduce sampling to one step but require extensive training; Shortcut Models add training constraints.Source: §1 · p. 2
- OZSpeech reformulates OT-CFM to start from a learned prior rather than Gaussian noise, avoiding a full range of t values and extra training constraints.The vector field regresses trajectories from the learned prior toward target distributions.Source: §1 · p. 2
- OZSpeech is proposed as a zero-shot TTS system to synthesize speech matching the acoustic characteristics of a prompt from an unseen speaker.The paper identifies autoregressive sampling as a source of potentially infinite repetition in applications requiring precision and reliability.Source: §1 · pp. 1–2
- 2.7 −6.5 times faster inference than other methods accompanies multi-fold improvement in WER and a 29%-71% model size.OZSpeech’s WER also remains stable as audio-prompt noise increases.Source: §1 · p. 2
2 Related Work
Zero-shot TTS research includes autoregressive and diffusion-based systems, alongside approaches that represent speech with discrete codec tokens. Neural codecs provide compressed acoustic representations used in codec-based generation.
2 Related Work
- Zero-shot TTS generates speech in an unseen speaker’s voice from a few seconds of audio prompt; research includes autoregressive and diffusion-based approaches.The paper cites diffusion models (Ho et al., 2020; Song et al., 2021) and flow matching (Lipman et al., 2023; Liu et al., 2023) as related paradigms.Source: §2 · p. 2
- Neural codecs learn discrete speech tokens while preserving reconstruction quality and low bitrate; SoundStream uses residual vector quantizers to represent speech with multiple tokens.SoundStream uses a vector-quantized variational autoencoder, first introduced by (van den Oord et al., 2017).Source: §2 · p. 2
- VALL-E represents speech as discrete codec codes and frames TTS as conditional codec language modeling.The passage identifies VALL-E as a pioneering codec-based approach and lists follow-on work.Source: §2 · p. 3
3 Method
OZSpeech generates speech from text and an acoustic prompt by starting flow matching from learned prior codes rather than noise. It combines phoneme-conditioned code generation, prompt-based attribute transfer, and factorized codec representations.
3.1 Problem Statement
- FACodec decomposes speech into speaker identity and code sequences for prosody, content, and acoustic detail, while the system combines text and acoustic prompts to generate speech.The prior-code and vector-field stages produce prior codes and transition them toward target codes.Source: §3.1 · p. 3
3.2 Prior Codes Generation Modeling
- The Prior Codes Generator creates code sequences hierarchically, conditioning the first sequence on phoneme embeddings and later sequences on preceding codes.Its prior loss trains content codes conditioned on phonemes while other codes converge toward mean representations; a duration predictor estimates acoustic-token counts per phoneme.Source: §3.2 · p. 3
3.3 One-Step Optimal Transport Flow Matching for Zero-Shot TTS
- The one-step OT-CFM formulation starts from a learned prior state xpr rather than Gaussian noise x0, enabling sampling in as few as one step as the prior approaches the target.The reformulation treats the intermediate state as learnable and t as an unknown prior-dependent time variable.Source: §3.3 · pp. 3–4, §3.1 · p. 3
- The vector-field estimator uses prompt prosody and acoustic-detail codes while masking prompt content codes to prevent unwanted content transfer.The input also includes the prior representation and masked acoustic-prompt content; random Gaussian noise is added for robustness and diversity.Source: §3.3 · p. 4
- All six quantizer sequences are folded into the hidden dimension and modeled together, rather than processed independently.Quantizer embeddings identify each sequence in hidden space to help prevent confusion between quantizers.Source: §3.3 · p. 5
- Joint training combines prior and duration losses with flow-matching and anchor losses; the anchor loss regularizes embeddings and helps prevent collapse.The anchor loss also reduces distances between states and enables efficient sampling.Source: §3.3 · p. 5
4 Experiments
OZSpeech is evaluated on LibriSpeech test-clean across prompt lengths and metrics covering speech quality, speaker similarity, robustness, prosody, and latency. It shows strong WER and one-step efficiency results, while prompt strategy and noisy-prompt experiments probe robustness and generalization.
4.1 Experiment Setup
- Experiments use LibriTTS training data and LibriSpeech test-clean benchmarking, measuring quality, speaker similarity, robustness, prosody, and sampling latency.The metrics include UTMOS, SIM-O, SIM-R, WER, pitch and energy measures, NFE, and RTF.Source: §4.1 · p. 5
4.2 Main Results
- 58%, 44%, and 44%: OZSpeech reduces WER relative to the next-best method for 1s, 3s, and 5s prompts, respectively.It establishes a new WER state of the art across all three prompt lengths and maintains consistent WER as prompt length changes.Source: §4.2 · p. 6, Table 1 · p. 6
- Across prompt lengths, OZSpeech ranks best or second-best on pitch and energy accuracy and error, but lacks an obvious advantage on UTMOS, SIM-O, and SIM-R.The authors attribute small UTMOS degradation versus some baselines to differences in neural-codec trade-offs between acoustic and semantic representations.Source: §4.2 · p. 6
- 29%-71%: OZSpeech is this size of the other models; its trainable parameters are 17%-43% of theirs, and it is almost 3 times faster than F5-TTS.OZSpeech uses one sampling step, compared with 200 for NaturalSpeech 2 and 32 for F5-TTS.Source: §4.2 · p. 7, Table 2 · p. 7
4.3 Ablation Study
- Arbitrary Segment outperforms First Segment across all metrics, and the authors adopt it for training to improve robustness through more diverse speech contexts.The proposed explanation is that hiding prompt position avoids forcing transfer to the target beginning and supports smoother attribute transfer.Source: §4.3 · p. 7, Table 3 · p. 7
4.4 Noise Tolerance Analysis
- With clean-trained checkpoints on noisy 3-second prompts, OZSpeech's WER shows no or negligible degradation across tested SNRs, while baseline methods degrade across metrics.The noisy-prompt evaluation uses models trained on their original datasets without retraining on noisy samples.Source: §4.4 · p. 7, Table 4 · p. 8
- Noise-aware fine-tuning leaves clean-prompt performance largely unchanged and improves noisy-prompt non-WER metrics, with generally larger gains as SNR decreases; WER is unaffected.The authors report feasibility of synthesizing noise-free speech conditioned on noisy prompts while preserving key speech attributes.Source: §4.4 · p. 8
5 Conclusion
OZSpeech combines one-step flow matching from a learned prior with zero-shot TTS, balancing intelligibility and acoustic quality. The authors identify naturalness and broader language, modality, and noise-filtering support as future work.
5 Conclusion
- OZSpeech uses single-step flow matching from a learned prior and balances speech intelligibility with acoustic quality.It reports multi-fold WER improvement, 2.7–6.5× faster inference, and a model size 29%–71% that of other models.Source: §5 · p. 8
- OZSpeech maintains consistent WER across different audio-prompt lengths and noise levels, and achieves competitive results against models trained on larger datasets.Source: §5 · p. 8
- Future work will explore adaptive noise filtering and expand OZSpeech toward multilingual and multimodal zero-shot speech synthesis.Source: §5 · p. 8
Limitations
The authors report naturalness distortions that they associate with rounding ground-truth phoneme durations to align with codec codes. They describe a trade-off between synthesis quality and computational efficiency.
Limitations
- Synthesized speech can exhibit slight distortions that degrade overall quality and challenge naturalness.Source: Limitations · p. 9
- Rounding real-valued ground-truth phoneme durations to integers for codec alignment degrades synthesized speech quality in the temporal domain.Source: Limitations · p. 9
- The duration predictor requires ground-truth phoneme durations, and the authors propose exploring alternative alignment methods while noting a quality–efficiency trade-off.Source: Limitations · p. 9
Potential Risks
The supplied material describes zero-shot TTS risks alongside FACodec’s factorized speech representations and the basic Flow Matching and conditional optimal-transport path formulations.
Potential Risks
- Zero-shot TTS can generate speech in novel voices from little or no training data; the stated risks include fraud, fabricated media, privacy violations, and legal disputes.Source: Potential Risks · p. 9
A.1 FACodec
- FACodec uses factorized vector quantizers to represent content, prosody, and acoustic details as discrete sequences, while extracting timbre separately for waveform reconstruction.Source: §A.1 · p. 12
- The generated six-sequence representation is constrained to pre-trained FACodec subspaces and decoded with the prompt’s timbre representation.Source: §A.1 · p. 12
A.2 Flow Matching
- Flow Matching models a probability path from a source distribution to a target by regressing a neural velocity field, which transforms samples along that path.Source: §A.2 · pp. 12–13
- Conditional optimal-transport paths condition on real training data to make the flow-matching objective tractable, using a path between source and target samples.Source: §A.2 · p. 13
B Method Details
Training uses matched acoustic prompts from the target, then limits loss computation to target-speech velocity rather than prompt velocity.
B Method Details
- Training randomly clones a 1–3-second ground-truth segment to serve as both prompt and target, matching their attributes.The prompt transfers prosody and acoustic detail to output quantizers.Source: B Method Details · p. 13
- The velocity sequence is truncated to exclude acoustic-prompt components, focusing subsequent loss computation on the target velocity.The stated goal is to compute the drift needed to generate x1, not to backpropagate across the full sequence.Source: B Method Details · p. 13
- Anchor Loss Lanchor approximates target distribution x1 conditioned on the estimated target x̃1.The passage introduces this loss formulation and points to the computation of x̃1.Source: B Method Details · p. 13
C Training Details
The training setup specifies compact prior-code and vector-field networks, joint optimization, and LibriTTS-based training and evaluation data.
C Training Details
- The Prior Code Generator uses 256-dimensional, four-head encoder and shared-decoder layers, with two FFT blocks each and six quantizer-specific blocks.Its output dimensionality is 1024.Source: C Training Details · p. 14
- The Vector Field Estimator is a four-layer Transformer with 1024 hidden dimensions, 32 heads, and a 4096-dimensional feed-forward network.The two components are jointly trained on four 80GB A100 GPUs with batch size 16 and AdamW at learning rate 10^-4.Source: C Training Details · p. 14
D.1 Dataset Details
- Training uses 500 hours of LibriTTS, with MFA phoneme-duration alignment and FACodec codes derived from 16-kHz waveforms.FACodec decomposes each audio second into six quantizers, each containing 80 discrete speech units.Source: §D.1 · p. 14
- Evaluation uses 2.2 hours of LibriSpeech test-clean audio filtered to 4–10 seconds, with same-speaker prompts of 1, 3, or 5 seconds.Prompts are randomly selected from another sample by the same speaker.Source: §D.1 · p. 14
D.2 Metrics Details
The evaluation covers speech quality, speaker similarity, robustness, prosody, and sampling efficiency, using four zero-shot TTS baselines.
D.2 Metrics Details
- RTF measures the time to produce one second of speech, while NFE counts guiding-function evaluations during sampling.RTF is measured end-to-end on an NVIDIA 80GB A100 GPU.Source: §D.2 · p. 14
- UTMOS predicts mean opinion scores for speech quality, while SIM-O and SIM-R measure speaker similarity to the prompt and its FACodec reconstruction.SIM-O and SIM-R use cosine similarity of WavLM-TDCNN speaker embeddings and range from -1 to 1.Source: §D.2 · p. 14
- WER assesses word pronunciation against the text prompt, and prosody measures assess pitch and energy alignment with the acoustic prompt.Prosody evaluation uses categorical F0 and energy levels as well as RMSE.Source: §D.2 · p. 15
D.3 Baselines Details
- The baselines are VoiceCraft, NaturalSpeech 2, F5-TTS, and VALL-E, using pretrained checkpoints for the first three and a reproduced VALL-E model.The cited passages identify each baseline’s training data or implementation setup.Source: §D.3 · p. 15
E Extra Experiments
The 100M-parameter Small model is over 31% smaller than the 145M Base model and has comparable performance across metrics except pitch (F0).
E Extra Experiments
- Over 31% smaller, OZSpeech-Small (100M parameters) shows comparable performance to OZSpeech-Base (145M) across metrics except pitch (F0).For 1s prompts, F0 RMSE is 23.94 for Small versus 27.70 for Base; for 3s prompts, it is 13.54 versus 11.96.Source: E Extra Experiments · p. 15
- For 1s prompts, F0 RMSE is 23.94 for Small versus 27.70 for Base, a 13.6% improvement for Small.For 3s prompts, Small has a 13.2% relative decline in F0 RMSE: 13.54 versus 11.96 for Base.Source: E Extra Experiments · p. 15
- At 3s prompt length, Small’s F0 RMSE is 13.54, compared with 11.96 for Base.The passage reports this as a 13.2% relative decline for Small.Source: E Extra Experiments · p. 15
F Analysis
OZSpeech delivers competitive performance without a clear overall advantage over baselines, balancing sampling speed with speech synthesis capability. Its WER is strong, while UTMOS and speaker-similarity comparisons vary by metric and prompt length.
F Analysis
- OZSpeech’s WER distribution stays close to zero across 1s, 3s, and 5s prompts, outperforming the compared models.NaturalSpeech 2 is second-best; F5-TTS falls behind as prompt length increases, with its distribution approaching 0.5.Source: F Analysis · p. 16
- With one sampling step, OZSpeech achieves promising performance despite baselines having larger models and longer inference times.The paper identifies balancing sampling speed and synthesis capability as its primary objective.Source: F Analysis · p. 16
- VoiceCraft consistently outperforms OZSpeech on UTMOS, while VALL-E achieves the best result for the 1s prompt.VoiceCraft and OZSpeech maintain stable distributions across prompt lengths.Source: F Analysis · p. 16
- For SIM-O, VoiceCraft leads as prompt length increases, followed by VALL-E and then OZSpeech.At 1s, VoiceCraft and OZSpeech have almost equivalent distributions.Source: F Analysis · p. 16
- Across prompt lengths, OZSpeech leads on WER but not UTMOS or longer-prompt SIM-O, yielding competitive rather than clearly superior overall performance.The paper frames this as a trade-off between sampling speed and speech synthesis capability.Source: F Analysis · p. 16, Figure 2 · p. 17