Source-linked AI summary
VoXtream2: Full-stream TTS with dynamic speaking rate control
Nikita Torgashov, Gustav Eje Henter, Gabriel Skantze
TL;DR
Full-stream TTS must produce speech with minimal delay while adapting speaking rate as text arrives incrementally. VoXtream2 addresses this with zero-shot streaming synthesis, dynamic speaking-rate control, classifier-free guidance, and prompt-text masking, achieving competitive results with 4× real-time speed and 74 ms initial latency.
Problem
Existing TTS systems commonly assume static speaking rates and require full-utterance text, limiting incremental speech generation for interactive systems.
Method
VoXtream2 combines full-stream zero-shot TTS with frame-level dynamic speaking-rate control, distribution matching, classifier-free guidance across conditioning signals, and prompt-text masking for textless audio prompts.
Results
VoXtream2 achieves competitive zero-shot TTS performance despite using less training data and a smaller model, while running 4 times faster than real time with 74 milliseconds of initial latency.
Takeaways & Limitations
Dynamic speaking rate can be updated mid-utterance while maintaining stable intelligibility and voice cloning in the operating range, and textless prompting reduces reliance on prompt transcription and prompt rate.
Takeaways & Limitations
Generated speaking rate remains influenced by acoustic prompt rate, and some prompt-target rate combinations increase WER.
Abstract
from arXiv · showhide
Full-stream text-to-speech (TTS) for interactive systems must start speaking with minimal delay while remaining controllable as text arrives incrementally. We present VoXtream2, a zero-shot full-stream TTS model with dynamic speaking-rate control that can be updated mid-utterance on the fly. VoXtream2 combines a distribution matching mechanism over duration states with classifier-free guidance across conditioning signals to improve controllability and synthesis quality. Prompt-text masking enables textless audio prompting, removing the need for prompt transcription. Across standard zero-shot benchmarks and a dedicated speaking-rate test set, VoXtream2 achieves competitive objective and subjective results against public baselines despite a smaller model and less training data. In full-stream mode, it runs 4 times faster than real time with 74 ms first-packet latency on a consumer GPU.
1. Introduction
Interactive TTS requires incremental, low-latency speech generation while adapting speaking rate within an utterance. VoXtream2 addresses these needs with full-stream zero-shot synthesis and dynamic speaking-rate control.
- Motivation: Current systems commonly use static or utterance-level speaking-rate control, unlike human speech, whose rate varies within sentences.These variations reflect cognitive load, discourse structure, and communicative intent.
- Motivation: Most existing TTS systems require full-utterance text, limiting their use in real-time conversational agents and speech-to-speech translation.Streaming systems must produce short waveform chunks with minimal delay as text arrives incrementally.
- Approach: VoXtream2 introduces full-stream zero-shot TTS with dynamic speaking-rate control that can be modified while speech is generated.The model applies distribution matching and classifier-free guidance for fine-grained frame-level control.
- Reported capabilities: 4 times faster than real time with 74 milliseconds of initial latency, VoXtream2 supports low-delay full-stream interaction on a user-grade GPU.The system is designed for seamless interaction as text unfolds and speaking rate changes dynamically.
2. Related Work
Prior work addresses speaking-rate control through global or local duration manipulation, prompt-based conditioning, and text or acoustic inputs. Full-stream TTS reduces latency, but dynamic control and practical prompting remain limited or underexplored.
- Static control: Static speaking-rate control has been implemented through sentence-level descriptors, encoder manipulation, and duration-predictor conditioning or scaling.These approaches modify speech pace through explicit or implicit duration changes.
- Static control: Speaking-rate-control systems either use detailed text descriptions without voice cloning or combine acoustic prompts with text instructions for voice and style control.The related approaches trade off control descriptions and voice-cloning capabilities in different ways.
- Dynamic control: Dynamic speaking-rate control is underexplored; one prior method uses multi-round inference and becomes unintelligible when downsampling exceeds 40% of the original input length.The cited method operates at the word level by resampling input speech tokens.
- Full-stream TTS: Full-stream TTS accepts incrementally generated text and emits small waveform chunks before the full text is available, minimizing first-packet latency.Existing zero-shot full-stream systems can run faster than real time but require acoustic-prompt transcription, reducing practicality.
3. Method
VoXtream2 extends full-stream VoXtream with dynamic speaking-rate control, textless acoustic prompting, and conditioning guidance. Its rate controller matches duration distributions during generation while balancing rate adherence against intelligibility.
- Model architecture: VoXtream2 adds dynamic speaking-rate control to the full-stream VoXtream architecture, allowing rate changes during generation.The model uses a modified VoXtream architecture with autoregressive temporal and depth transformers.
- Model architecture: The temporal transformer jointly predicts semantic and six duration tokens, enabling finer-grained duration control.Duration tokens combine a phoneme-advance shift state in [0, 2] with one or two phonemes per frame.
- Prompt text masking: Prompt-text masking removes transcription requirements by replacing prompt text with special tokens while retaining classifier-free guidance.The model is trained to rely solely on the acoustic prompt, and prompt masking also enables any-language-to-English translingual capability.
- Acoustic prompt enhancement: Increasing speaker similarity through CFG degrades signal quality when acoustic prompts contain noise or recording artifacts.Prompt enhancement with Sidon is applied before generation and does not increase generation latency.
- Speaking rate control: Speaking-rate control matches six-bin duration-state distributions against a target distribution and a sliding-window accumulated distribution.The accumulated distribution uses the previous 3 seconds of generated speech, while the updated distribution is sampled with top-p = 0.9.
- Speaking rate control: β = 5 balances speaking-rate control and intelligibility: β = 10 follows the target more closely but increases WER through hallucinations.At β = 1, intelligibility impact is minimal but control is weak; the duration state is not guided with CFG to avoid rate increases observed in prior systems.
4. Experiment Setup
VoXtream2 is evaluated on standard zero-shot TTS benchmarks and a dedicated speaking-rate dataset using objective, subjective, and streaming-performance measures. The setup includes unseen speakers, controlled speaking-rate prompts, and comparisons with public streaming or duration-control baselines.
- Training and baselines: Training uses 47k hours of English Emilia speech and a filtered 15k-hour 22kHz HiFiTTS-2 subset.HiFiTTS-2 utterances shorter than 5 seconds or with WER above 10% were removed.
- Training and baselines: The model uses a Llama-3.2 transformer backbone with a 12-layer temporal transformer, 6-layer phoneme transformer, and 4-layer depth transformer.The depth-transformer weights remain frozen after pretraining on conversational speech.
- Training and baselines: Training lasted 28 hours on 2×NVIDIA H200 GPUs for 10 epochs, with 50-second random audio crops and CUDA-graph streaming optimizations.The Mimi codec streaming state is cached during generation.
- Training and baselines: Baselines include non-autoregressive, autoregressive duration-control, and autoregressive full-stream zero-shot TTS models, restricted to models with up to 1B parameters.CosyVoice2 additionally supports speaking-rate control through instructed generation.
- Datasets: Evaluation covers LibriSpeech-PC, SEED-TTS test-en, and the Emilia speaking-rate set, whose 62 unseen speakers provide slow, normal, and fast prompts.The prompt rates are 2.0 ± 0.5, 4.0 ± 0.6, and 5.6 ± 0.5 SPS, respectively.
- Evaluation metrics: Metrics include WER for intelligibility, cosine speaker similarity, UTMOS signal quality, MUSHRA-like naturalness ratings, speaking-rate correlation, RTF, and first-packet latency.Naturalness studies used 40 native listeners per study, while full-stream evaluation provides text word by word.
5. Results
VoXtream2 delivers competitive zero-shot TTS quality and stronger speaking-rate controllability across static and dynamic settings, while supporting efficient full-stream generation. Its control remains reliable across changing rates, though very slow speech increases errors through hallucinations, repetition, and filler insertion.
- Overall comparison: VoXtream2 achieves performance comparable to state-of-the-art systems despite its smaller model and substantially less training data.Subjectively, VoXtream2-B is on par with Kyutai-TTS and VoiceStar, and becomes most preferred with prompt enhancement.
- Full-stream performance: VoXtream2 achieves the lowest FPL and RTF among publicly available full-stream models.The comparison uses mixed precision on an RTX3090 GPU and evaluates Torch Compile, CUDA Graphs, and TensorRT.
- Full-stream: VoXtream2 attains the lowest WER for word-level streaming and shows consistent gains as text chunk size increases.At chunk size 4, it remains more accurate than CosyVoice2; KyutaiTTS benefits only marginally from additional context.
- Static speaking rate control: SRC substantially reduces dependence on prompt speaking rate, yielding a broader achievable range and smaller differences between prompt-speed conditions.Without SRC, output rate varies more strongly with slow, normal, and fast prompts.
- Static speaking rate control: Fillers occur more frequently at slow rates and nearly disappear at fast rates, with VoXtream2 uniquely inserting them automatically at slow rates.Filler-containing prompts further increase their occurrence, consistent with human speech.
- Dynamic speaking rate control: At slow speaking rates, WER increases substantially because underrepresented duration states cause hallucinations, repetition, and filler insertion.At fast rates, SPK-SIM decreases slightly while WER remains nearly unchanged.
- Dynamic speaking rate control: Dynamic SRC follows gradual changes and adapts rapidly to abrupt transitions, with reliable generation at 1 SPS and a maximum reliable rate around 5–6 SPS.Short-term fluctuations around the target rate resemble natural speaking-rate variation rather than control failure.
6. Ablations
Ablations show that dataset scaling, architectural changes, and conditioning guidance improve different quality dimensions, while inference settings affect full-stream latency and intelligibility. CFG’s effect under SRC appears mainly through intelligibility rather than speaking speed.
- Modular ablation: Dataset scaling and architectural changes improve all objective metrics over VoXtream.This comparison is reported on the SEED test-en dataset.
- Modular ablation: Text-conditioning CFG significantly reduces WER, while audio CFG improves SPK-SIM at the cost of higher WER and lower UTMOS.Speaker-embedding CFG further improves SPK-SIM and slightly reduces WER, again with a small UTMOS drop.
- Modular ablation: Speech enhancement for the acoustic prompt achieves the best UTMOS and WER, with a slight reduction in SPK-SIM.Increasing speaker-embedding conditioning instead yields the best SPK-SIM score.
- CFG under SRC: CFG does not significantly change speaking rate when it is excluded from duration-token sampling, but γtemp changes WER differently for slow and fast speech.Lower γtemp gives the best WER for slow speech, whereas larger values give the lowest WER for fast speech but reduce slow-speech intelligibility.
- Input streaming robustness: The streaming text rate and phoneme look-ahead determine full-stream latency and WER, with output streaming increasing both metrics relative to non-streaming evaluation.The TPS and look-ahead comparison is summarized in Table 7.
7. Limitations
VoXtream2 has two author-identified limitations: preprocessing is complex and the model remains partly coupled to the acoustic prompt’s speaking rate.
- The model’s generated speaking rate remains influenced by the acoustic prompt’s speaking rate.This residual coupling can increase WER for some prompt–target rate combinations, especially slow prompts paired with fast targets and fast prompts paired with slow targets.
- Approximately 35% of training data was discarded because of invalid phoneme alignments.The external phoneme aligner is unnecessary during generation but remains crucial for training-data preparation.
- Future work could use alternative alignment methods and dedicated prompt encoders or training objectives.These directions target the preprocessing dependency and incomplete disentanglement of prompt and target speaking rates.
8. Conclusion
VoXtream2 pairs full-stream zero-shot TTS with fine-grained speaking-rate control that can change during an utterance. The model also supports textless prompting, configurable quality–similarity–controllability trade-offs, and robust streaming under moderate token rates.
- Dynamic speaking-rate control can be updated mid-utterance while maintaining stable intelligibility and voice cloning in the operating range.The system supports smooth transitions between slow, normal, and fast speech.
- Prompt-text masking enables textless prompts and reduces sensitivity to prompt content and speaking rate.This removes the need for prompt transcription.
- Classifier-free guidance across conditioning signals improves intelligibility and speaker similarity while providing a lever to trade off similarity, signal quality, and controllability.The conclusion separately notes that changing γtemp affects speaking-rate-control outcomes by shifting intelligibility across target rates.
- At moderate token rates, effective look-ahead preserves quality in realistic LLM-driven text streams, while constrained look-ahead produces a latency–quality trade-off.
10. GenAI Usage Disclosure
The authors disclose using ChatGPT for selected writing, coding, and sentence-generation tasks, with all generated code and text verified by the authors.
- ChatGPT generated sentences for the Emilia speaking-rate test.
- ChatGPT was used to code the web interface for user studies.
- The authors used ChatGPT to polish the manuscript and thoroughly verified all AI-generated code and text.