Source-linked AI summary
X2Streaming-TTS: Causal Token-Level Text-to-Speech from Streaming Text with Speech-State Inheritance
Rime Wen, Zehan Liu, Shawn Qin, Lights Shi, Roy Gan, Hao Wang, Qian Wang
TL;DR
True token-level TTS must commit to uncertain text prefixes without revising emitted speech, while preserving continuity across segments. X2Streaming-TTS addresses this with causal commitment and speech-state inheritance, matching evaluated offline-baseline quality with median TTFTs of 15.8 ms for one request and 260.8 ms at 128 concurrent requests.
Problem
Token-level streaming TTS must irreversibly speak uncertain prefixes while deciding when to commit and how to continue acoustic state across segments.
Method
X2Streaming-TTS combines uncertainty-aware causal commitment with capacity-adaptive segmentation and causal speech-state inheritance, without accessing future text.
Results
15.8 ms single-request median TTFT and 260.8 ms median TTFT at 128 concurrent requests accompany synthesis quality comparable to evaluated offline baselines.
Takeaways & Limitations
X2Streaming-TTS demonstrates strict token-level synthesis with bounded acoustic context and quality comparable to the evaluated offline baselines.
Takeaways & Limitations
The segmentation theorem bounds segment count only and does not guarantee punctuation quality or physical KV-cache safety.
Abstract
from arXiv · showhide
Streaming text-to-speech is essential for low-latency spoken dialogue systems, yet many systems wait for sentence-level text and are therefore only pseudo-streaming. True token-level synthesis must generate speech from uncertain prefixes while maintaining perceptual continuity over an unbounded stream with bounded context. We present X2Streaming-TTS, a causal TTS framework that consumes asynchronously arriving text tokens and emits speech without accessing future input. To handle uncertain prefixes, we introduce causal commitment, which keeps ambiguous expressions provisional through uncertainty-aware buffering and performs capacity-adaptive, punctuation-aware segmentation. To preserve acoustic continuity, we further introduce causal speech-state inheritance, which carries the complete Code2Wav state and selected historical Talker states across segment boundaries. Together with an attention prior constraint, it blocks access to future positions while retaining bounded acoustic context. Experiments show that X2Streaming-TTS outperforms existing pseudo-streaming models on most subjective and objective metrics. Further analysis shows that causal commitment stabilizes online segmentation and reduces failures caused by insufficient context, while speech-state inheritance improves boundary continuity without degrading naturalness or speaker identity. X2Streaming-TTS thus achieves strict token-level synthesis with quality comparable to the evaluated offline baselines, a median time to first audio token (TTFT) of 15.8 ms for a single request, and a median TTFT of 260.8 ms at 128 concurrent requests. Our implementation is publicly available at https://github.com/X-Square-Robot/X2Streaming-TTS .
1 Introduction
X2Streaming-TTS addresses strict token-level speech synthesis as an irreversible decision problem under partial observability, where pronunciation, segmentation, and acoustic continuation must be handled without future text. It combines causal commitment with causal speech-state inheritance to achieve offline-comparable quality and low TTFT.
- Problem: Token-level TTS must commit speech from observed prefixes because delivered audio cannot be revised when future tokens change interpretation.Incomplete numbers, units, and symbols can be reinterpreted by later text.
- Problem: Streaming synthesis must jointly resolve pronunciation, efficient segmentation, and continuation, since punctuation-only or silence-reset boundaries can waste generation capacity or create perceptible seams.The challenges include ambiguous expressions, excessive short segments, and re-establishing pitch and timbre from scratch.
- Method: Causal commitment keeps ambiguous numbers, units, and symbols provisional while unifying capacity and boundary admission as constrained segmentation.It governs decisions through segment closure and prevents unresolved prefixes from being prematurely voiced.
- Method: Causal speech-state inheritance transfers complete waveform-decoder state and a fixed number of acoustic states across boundaries under a causal prior that excludes future positions.Removing it produces measurable pitch discontinuity at segment boundaries.
- Results: 15.8 ms median TTFT is achieved for a single request, while 260.8 ms is achieved at 128 concurrent requests, with quality comparable to evaluated offline baselines.The integrated system performs strict token-level synthesis without access to future text.
2 Related Work
Prior streaming TTS methods either synthesize after complete sentences or reduce context through limited lookahead and interleaving, while separate segmentation methods overlook acoustic cost and cross-segment speech state. X2Streaming-TTS addresses these gaps by coupling linguistic readiness with acoustic capacity under strict causality and transferring bounded speech state across boundaries.
- 2 Related Work: Complete-text streaming TTS begins after sentence availability and lowers latency through autoregressive acoustic generation, causal or chunk-aware decoding, and streaming vocoders.These methods build on discrete audio codecs and language-model-based zero-shot TTS architectures.
- 2 Related Work: Limited-lookahead systems reduce context using fixed lookahead, token-synchronous decoding, text–speech interleaving, and prosodic-boundary or pause prediction.Boundary-aware streaming generation additionally uses limited future words, boundary post-training, and bounded sliding prompts.
- 2 Related Work: Standalone text segmenters identify linguistically appropriate boundaries but ignore acoustic generation costs, so suitable boundaries may appear only after the acoustic budget is exhausted.SaT-3L accesses up to 48 future subwords and is used only as a boundary-quality reference.
- 2 Related Work: Existing methods wait for complete text, rely on future context, or ignore acoustic capacity and cross-segment state; X2Streaming-TTS couples readiness with acoustic cost under strict causality and transfers bounded speech state.Text segmentation alone does not define transferred speech state and cannot ensure cross-segment continuity.
3 X2Streaming-TTS
X2Streaming-TTS performs strict token-level speech synthesis through causal commitment and capacity-aware segmentation, while speech-state inheritance preserves bounded acoustic context across segment boundaries. Its causal attention prevents future access while retaining relevant historical states.
- Causal commitment: Causal commitment releases only pronunciation-stable text and closes segments before acoustic capacity is exhausted using uncertainty-aware buffering and punctuation-aware segmentation.Expressions involving numbers, units, symbols, abbreviations, and punctuation remain provisional until no permitted continuation can alter pronunciation; released text is never revised.
- Capacity-adaptive segmentation: CAPS selects punctuation-aware boundaries under cache constraints, adapts projected capacity with delayed feedback, and forces hard splits when eligible punctuation is unavailable.Capacity and thresholds are frozen when a segment opens, so feedback affects only subsequently opened segments.
- Capacity-adaptive segmentation: O(n^2) time is required for exact offline boundary optimization, or O(nC) when each segment contains at most C tokens.The optimization enforces the cache-footprint constraint G(Sk; ξ) ≤ B for every segment.
- Speech-state inheritance: Speech-state inheritance transfers the complete Code2Wav state and trailing H Talker states to warm-start waveform decoding and provide bounded historical memory.The transferred Code2Wav bundle includes KV, convolutional, transposed-convolutional, and frame-index states; H is fixed independently of stream length.
- Speech-state inheritance: Inheritance remains active only after a health check confirms complete input consumption, normal acoustic termination, ρk = Ak/nk ∈[1, 12], and an intact Code2Wav snapshot.Otherwise, both inherited state paths are cleared and generation restarts from the default state.
- Causal attention: Causal-prior attention assigns zero weight to future positions while preserving access to relevant historical states through content-based attention.For equal content similarity, the current position has prior odds of 10 : 1 over positions at distance at least five.
4 Experiments
Experiments evaluate recognition, speaker similarity, naturalness, segmentation, boundary continuity, symbol handling, long-text stability, and latency across held-out Mandarin passages and fixed-window protocols. X2Streaming-TTS shows strong streaming recognition, more effective causal commitment, improved boundary continuity, accurate resolution of ambiguous symbols, and low TTFT under concurrency.
- Evaluation Setup: Evaluations use 59 held-out Mandarin passages with 954 source-sentence boundaries, plus 60 fixed 10 s windows for stability experiments.Recognition uses Whisper and Paraformer; speaker similarity uses ECAPA-TDNN; naturalness uses UTMOS; boundary discontinuity uses ∆F0 and ∆E.
- Recognition Performance: Our method has lower recognition error than the offline reference in 3 of 8 conditions, with a maximum degradation of 0.62 percentage points in the others.Among streaming systems, it achieves the lowest error in 6 of 8 conditions; CosyVoice 3-S is better in SEED EN and long-text 2×.
- Causal Commitment: Causal commitment raises budget utilization from 11.77% to 76.93% versus punctuation-only segmentation and lowers hard-cap rate from 87.13% to 0.54% versus fixed windows.All evaluated policies produce zero estimated-budget and token-cap violations.
- Boundary Continuity and Stability: Our method achieves the best values across all five Table 3 columns, including PBD 0.1092, ∆F0 22.61 Hz, and ∆E 1.66 dB.FireRedTTS-2 has PBD 0.1915, while the proposed method’s bootstrap interval is disjoint from the nearest comparator’s interval.
- Symbol and Prefix-Ambiguity Evaluation: On symbol-intensive text, our method achieves CER 2.00% and fully correct reading 73.3%, while 93.33% of outputs preserve intended meaning.The strongest comparator reports CER 6.65% and fully correct reading 40.0%.
- Latency: Median TTFT rises from 15.8 ms for one request to 260.8 ms at 128 concurrent requests, remaining below 120 ms through 64 requests.The minimum-to-99th-percentile band widens at the highest tested loads, indicating greater request-level variation.
5 Conclusion
X2Streaming-TTS addresses irreversible token-level synthesis by deciding when prefixes are safe to speak and how to continue speech across segments. Its decomposition may extend to other irreversible online generation tasks.
- 5 Conclusion: Causal commitment delays unresolved expressions until their pronunciations are determined, then segments speech by balancing linguistic boundaries against acoustic capacity.The approach has a provable fragmentation bound under stated assumptions.
- 5 Conclusion: Streaming synthesis exploits denser text information than speech, allowing acoustic generation to process newly arriving text.
- 5 Conclusion: The same decomposition may extend to simultaneous translation, streaming recognition, and long-video generation.