Source-linked AI summary
X2-NativeCursor: Native-Token Text Progress Tracking for Incremental-Text Streaming Codec TTS
Zehan Liu, Carl Chen, Rime Wen, Kaiqi Fu, Altman Lin, Shawn Qin, Lights Shi, Roy Gan, Hao Wang, Qian Wang
TL;DR
Incremental-text streaming TTS needs online text progress tracking because arriving text does not indicate which words have been spoken, while waveform alignment adds streaming latency and acoustic computation. X2-NativeCursor observes native speech tokens before waveform decoding, maps stable spoken labels to original-text spans, and outputs a monotonic cursor. It achieves 0.151 Chinese-character MAE at 80-ms lookahead versus 1.253 for an online waveform baseline at 320 ms, with the improvement retained under a second reference and after adaptation to CosyVoice2.
Problem
Incremental-text streaming TTS requires online mapping from speech progress to original-text positions for highlighting, interruption handling, and dialogue-history updates.
Method
X2-NativeCursor uses an independently trained native-token observer with normalization-based span mapping, local matching, and monotonic cursor projection without changing the TTS generator.
Results
0.151 Chinese-character MAE at 80-ms lookahead versus 1.253 for an online waveform baseline at 320 ms, with lower tracking error under a second automatic reference and adaptation to CosyVoice2.
Takeaways & Limitations
Native speech tokens can support online cursor estimation before waveform decoding within the evaluated Qwen3-TTS and CosyVoice2 conditions.
Takeaways & Limitations
With three unseen voices, average MAE rises to 0.788 and varies from 0.320 to 1.213, indicating sensitivity to synthesis-voice changes.
Abstract
from arXiv · showhide
Incremental-text streaming text-to-speech (TTS) needs online text progress tracking for synchronized highlighting, interruption handling, and dialogue-history updates. Input text arrives before it is spoken, so text arrival alone cannot indicate speech progress. Existing waveform-based alignment requires complete audio or adds acoustic processing during streaming. We propose X2-NativeCursor, a lightweight observer that tracks progress from native speech tokens before waveform decoding without changing the TTS generator. Its normalization plan links spoken labels to their original-text spans. Text and native-token encoders feed a local matcher that estimates the current label position. A separate output rule converts revisable position estimates into a cursor that never moves backward. Mean absolute error against an automatic reference is 0.151 Chinese characters with 80-ms lookahead, versus 1.253 characters with 320-ms lookahead for an online waveform baseline. Alignment real-time factor also decreases from 0.3598 to 0.0180 relative to this baseline. Lower tracking error is retained under a second automatic alignment reference. We evaluate X2-NativeCursor on Qwen3-TTS and validate its adaptation to CosyVoice2 by training a separate observer for each backbone. Code is publicly available at https://github.com/X-Square-Robot/X2Streaming-TTS.
1 Introduction
X2-NativeCursor addresses the need for online text progress tracking in incremental-text streaming TTS, where text arrival does not reveal which words have been spoken. It tracks native speech tokens before waveform decoding and produces a monotonic original-text cursor without changing the TTS generator.
- Motivation: Incremental-text streaming TTS needs online mapping from speech frames to original-text positions for highlighting, interruption handling, and dialogue-history updates.Text arrives before speech, and codec-based words span varying numbers of speech-token frames.
- Motivation: Waveform-based alignment adds waiting and acoustic computation during streaming, while generator-integrated alignment can require architecture changes and retraining.Native speech tokens allow tracking before waveform decoding without re-encoding decoded audio.
- Approach: Its normalization plan preserves links from stable spoken labels to original-text spans, while a local matcher estimates the current label position.Labels are released only when their spoken reading is fixed, and the observer processes text and native-token features.
- Approach: X2-NativeCursor uses an independently trained observer that reads native speech-token frames without waveform input or generator changes.The design supports codec-based TTS by training a separate observer for each backbone.
- Results: 0.151 Chinese-character MAE at 80-ms lookahead compares with 1.253 for the online waveform baseline at 320 ms.The alignment real-time factor is 0.0180 versus 0.3598 for the baseline, and the gain holds under a second automatic reference.
2 Method
X2-NativeCursor combines stable text normalization, native-token encoding, local matching, and monotonic cursor projection to track original-text progress before waveform decoding. The observer uses bounded lookahead and is trained separately for each TTS backbone.
- Architecture: Figure 1 combines TNPlan, native-token encoding, local matching, and a backward-invariant cursor over original text.Dashed paths are used only during training.
- Spoken labels and text mapping: TNPlan converts incoming text into spoken labels, links each label to an original-text span, and delays release when later input could change the reading.This handles numbers, units, and symbols whose spoken forms may depend on later context.
- Native-token encoding: The text encoder represents spoken labels, while four dilated convolution blocks encode native speech-token frames with bounded lookahead.The main Qwen3-TTS setting uses one future frame, corresponding to 80 ms of lookahead.
- Local matching and cursor updates: For each frame, the local matcher scores nearby labels using native-token features, text features, and a location state.Seven candidate offsets are evaluated around the previous internal position, with learned weights and offset-specific parameters.
- Local matching and cursor updates: A masked softmax over candidate offsets produces the position update, while the output rule projects the furthest reached label position to the original-text cursor.The published cursor preserves the furthest position reached even when internal estimates move backward or skip labels.
- Training and inference: Training uses alignment, content, and rate losses, and inference updates the observer whenever a new lookahead-delayed native token arrives.Only the text encoder, native-token encoder, and local matcher are trained; each TTS backbone receives a separate observer.
3 Experimental Evaluation
The evaluation measures cursor accuracy, robustness, ablations, runtime, concurrency, and adaptation across codec-based TTS settings. X2-NativeCursor achieves substantially lower error and computation cost than online and native-token baselines while retaining performance under another reference and on CosyVoice2.
- Main results: 0.151 ± 0.005 Chinese-character MAE is achieved with Qwen3-ForcedAligner as the reference.The evaluation uses Qwen3-TTS speech and reports three-seed results as means and standard deviations.
- Main results: About 88% lower Chinese MAE and 44% lower English MAE are obtained than the online waveform baseline with 80-ms rather than 320-ms lookahead.Chinese MAE is also lower than every complete-audio baseline in the table.
- Main results: About 64% lower Chinese MAE is obtained than both native-token baselines while using less lookahead.The compared methods are cross-attention readout and CodecCTC+skip-DP, which use the same alignment supervision.
- Main results: Chinese-character MAE remains 0.206 under the independent MMS-FA reference after rescoring 604 aligned samples.X2-NativeCursor remains ahead of the online waveform and two learned native-token baselines under this reference.
- Ablation study: Removing the text encoder raises MAE from 0.343 to 1.465, while removing lookahead raises it to 0.471.Removing position and rate features has a smaller effect, raising MAE to 0.355; removing backward and skip moves mainly harms English.
- Runtime cost and concurrency: 0.0180 RTF versus 0.3598 for WindowMMS+PersistentCTC reduces alignment computation time by about 95%.The median observer cost is 1.33 ms per native frame, excluding waveform decoding.
- Runtime cost and concurrency: At 16 concurrent sessions, median cursor-update time is 2.45 ms and the 90th percentile is 4.92 ms, below the 80 ms represented by one native frame.These measurements were obtained on one NVIDIA A800-SXM4-80GB GPU.
- Adaptation to other codec-based TTS models: On CosyVoice2, the selected run achieves 0.284 Chinese-character MAE after retraining an observer while leaving the TTS generator unchanged.The other two training seeds achieve MAEs of 0.254 and 0.241.
4 Conclusion
X2-NativeCursor estimates raw-text progress from native speech tokens before waveform decoding while keeping the TTS generator frozen. On Qwen3-TTS, it substantially lowers teacher-relative Chinese-character error, and the approach also adapts to CosyVoice2.
- 0.151 Chinese-character MAE versus 1.253 for the online waveform baseline is achieved with one-quarter of the lookahead.The comparison uses 80-ms lookahead for X2-NativeCursor versus 320 ms for the baseline.
- X2-NativeCursor uses local native-token state updates and owner-span projection to track raw-text progress while keeping the TTS frozen.
- Lower tracking error holds against a second automatic reference, and retraining the observer adapts the method to CosyVoice2.
5 Compliance with Ethical Standards
The study uses synthetic speech produced by the systems under test and includes no human participants.
- The study uses synthetic speech produced by the systems under test and includes no human participants.