SyncSpeech: Efficient and Low-Latency Text-to-Speech based on Temporal Masked Transformer

Motivation

• The model targets practical deployment contexts including assistive communication, virtual assistants, and automatic video dubbing by improving TTS efficiency and latency.These application areas are identified as contexts where advances in TTS have increased practical impact.Source: §1 · p. 1
• AR TTS generates speech sequentially but is inefficient at high speech-token frame rates, whereas NAR TTS predicts in parallel but cannot generate speech incrementally.NAR models’ temporally unordered decoding leads to high first-packet latency, while streaming AR models still produce only one speech token per step.Source: §1 · p. 1

Approach

• SyncSpeech’s TMT paradigm combines AR temporal ordering with NAR parallel prediction for streaming text-to-speech generation.The approach predicts BPE-level durations and generates corresponding speech tokens within a unified decoding step, with complexity scaling linearly with the input text sequence.Source: §1 · p. 1

Results

• 6.4× and 8.8× RTF speedups for English and Mandarin, respectively, accompany comparable speech quality to existing mainstream TTS models.The reported speedups are measured against existing mainstream TTS models while maintaining comparable speech quality.Source: §1 · p. 1
• SyncSpeech reduces first-packet latency by 3.7× on English and 5.8× on Mandarin when integrated with upstream LLMs.The reported reductions are part of the model’s latency results for streaming scenarios.Source: §1 · p. 1

2.2. Other Modules

• The system combines a text-to-token model based on TMT with an off-the-shelf chunk-aware token-to-speech module built from CosyVoice2 components.The S3 tokenizer, conditional flow matching decoder, and HiFi-GAN vocoder produce waveforms from chunk-sized semantic tokens.Source: §2 · p. 2, §2.2 · p. 3

2.1. Temporal Masked Generative Transformer

• Random truncation and text look-ahead construct training sequences that simulate receiving streaming text while generating speech for the selected token.The truncated sequence contains text through n + q tokens and speech through the duration boundary a_n.Source: §2.1 · p. 2
• SyncSpeech’s TMT predicts the speech tokens for the current BPE token together with the next token’s duration in one decoding step.This unified prediction process enables synchronous generation from streaming text, except for duration prediction of the first text token.Source: §2.1 · p. 2
• The hybrid attention mask applies causal attention to text and special tokens, while speech tokens attend bidirectionally within their corresponding text-token span.This lets speech tokens access the total duration of their text token, improving speech robustness and naturalness.Source: §2.1 · p. 2
• During streaming inference, SyncSpeech repeatedly updates and pads the input sequence, predicts each new text token’s duration and speech tokens, and sends chunk-sized tokens to the decoder.Separate text and speech positional embeddings enable KV-cache acceleration even when streaming text tokens are inserted.Source: §2.1 · p. 3
• High-probability masked pre-training accelerates convergence and improves overall performance before fine-tuning with an inference-consistent prediction strategy.The masking rule is designed for high masking probability and consistency with inference, and the objectives jointly train masked speech and duration prediction.Source: §2.1 · p. 3

3.1. Experimental Settings

• FPL-L includes upstream text waiting and initial speech generation, with q controlling look-ahead and c counting iterations needed to exceed decoder chunk size.For SyncSpeech, FPL-L is defined as (q + 1) · dLLM + c · dTTS, while omitting the first term yields FPL-A.Source: §3.1 · p. 3

3.2. Main Results

• Comparable speech quality to CosyVoice models is observed across WER, speaker similarity, and MOS-N, with no significant MOS difference.The comparison used matched model sizes and data scale within each benchmark setting.Source: §3.2 · p. 3, Table 1 · p. 4
• 6.4× and 8.8× faster RTF than AR models on LibriSpeech and Seed, respectively, while complexity shifts from O(T) to O(L).The text-synchronous design depends on text length rather than speech sequence length, with L ≪ T in practical applications.Source: §3.2 · p. 4
• 3.7× and 5.8× faster FPL-A than AR models on LibriSpeech and Seed, respectively, with synthesis beginning after two text tokens.CosyVoice2 requires five text tokens and other baselines require the full sequence before starting synthesis.Source: §3.2 · p. 4

3.3. Analysis

• Removing high-probability masked pre-training increases WER by 1.17 and decreases UTMOSv2 by 0.15, while causal attention reduces robustness and naturalness.The hybrid mask supports bidirectional attention among speech segments corresponding to one text token and helps model token durations.Source: §3.3 · p. 4
• Top-k=3 sampling performs best for duration prediction, whereas greedy search is optimal for speech-token prediction.The authors hypothesize that strict text-synchronous alignment introduces temporal dependencies that make greedy search effective.Source: §3.3 · p. 4
• q = 1 gives the best WER, while q > 2 improves prosody but increases latency.The lower look-ahead setting reduces ambiguity and simplifies text-to-speech alignment; larger windows yield more natural pacing and pauses.Source: §3.3 · p. 4, Table 4 · p. 4

• SyncSpeech unifies temporally ordered generation and parallel inference through the TMT paradigm.Source: §4 · p. 4
• Evaluations show improved generation efficiency and reduced latency while maintaining comparable speech quality to current AR TTS systems.Source: §4 · p. 4