Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition

1 Introduction

• Prior unified speech-language models mostly operate offline, while streaming systems typically use separate task models; prior unified streaming evaluation was limited to single-speaker data.The authors state that, to their knowledge, no prior work demonstrated streaming recognition and synthesis using one shared decoder-only language-model backbone.Source: §1 · p. 1
• The model combines speech–text interleaving, a VibeVoice-family pretrained backbone, and two-stage training, with time-aligned interleaving yielding lower TTS word error rates than fixed-ratio interleaving in adaptation experiments.Separate streaming TTS and ASR models are adapted first; Uni-Stream is then initialized from adapted TTS and jointly trained, with task-specific distillation objectives.Source: §1 · p. 1
• Representation probing finds speech and text remain distinct across layers, while ASR and TTS representations are similar in early layers and diverge in later layers.The probing examines representations across tasks, modalities, and layers.Source: §1 · p. 2
• Uni-Stream uses shared weights for streaming ASR and TTS and achieves performance comparable to task-specific streaming models on both tasks.The model is initialized from an adapted TTS model and jointly trained on both tasks.Source: §1 · pp. 1–2
• The annotation pipeline transcribes and time-aligns 40,000 hours of English data, and the authors plan to release voted transcriptions and word-level timestamps for further research.The planned releases cover Emilia-EN transcriptions and timestamps for Emilia-EN, GigaSpeech, and LibriSpeech.Source: §1 · p. 2

2.1 Speech Language Models

• Unified speech language models perform recognition and synthesis with one language model, whereas streaming systems use incremental inputs but generally train separate recognition and synthesis models.Prior unified models' reported results, including progressive output, do not establish that both tasks run on streaming input; an earlier jointly trained streaming transducer was evaluated on a single-speaker corpus.Source: §2.1 · p. 2

2.2 Speech–Text Representations in Speech Language Models

• Speech carries prosody, speaker identity, and acoustic context beyond linguistic content shared with text, while cross-modal alignment need not imply a common representational structure.Prior studies describe language-aligned representations in multimodal models, but the relationship between alignment and effective information use remains open.Source: §2.2 · p. 2
• Speech-language-model studies find that codec representations encode phonetic, lexical-semantic, and accent information unevenly, and that retained acoustic information may not appear in task predictions.These findings motivate examining both what speech representations preserve and how the language model accesses that information.Source: §2.2 · pp. 2–3

3.1 Overview

• Uni-Stream uses shared weights for TTS and ASR, with a task token followed by interleaved text and speech chunks.The model is initialized from the adapted TTS model before joint training on both tasks.Source: §3.1 · p. 3

3.2 Streaming Speech-Text Interleaving

• For time-aligned TTS, each speech chunk corresponds to the first n words of its text chunk; lookahead words reappear in the next window.At inference, the model predicts speech-chunk boundaries that are determined by alignment during training.Source: §3.2 · p. 3

3.3 Unified Streaming Backbone

• The unified model uses the VibeVoice-TTS backbone with separate text and speech output heads for recognition and synthesis.The VibeVoice speech representations encode 24 kHz speech at 7.5 latent frames per second.Source: §3.3 · pp. 3–4, Figure 1 · p. 4

3.4 Two-Stage Training

• Training first adapts separate TTS and ASR specialists to streaming formats, then initializes the unified model from the TTS specialist and jointly trains both tasks.A synthesis anchor preserves inherited TTS capability, while recognition distillation supports ASR learning; the specialists are frozen during teacher-forcing training.Source: §3.4 · pp. 4–5

4.1 Experimental Setup

• Training uses Emilia-EN for TTS and ASR, plus GigaSpeech and LibriSpeech for ASR; annotation refinement and filtering retain approximately 40k hours of Emilia-EN for training.Word-end timestamps determine speech-segment boundaries for time-aligned TTS and text-emission chunks for streaming ASR.Source: §4.1 · p. 5

4.2 Stage I: Streaming Adaptation

• Time-aligned TTS has lower WER than fixed-ratio TTS (3.54% vs. 7.67%), while adapted ASR reduces word-emission delay to 0.66–0.67 s with test-clean WER of 3.56%.Many high-error fixed-ratio outputs completed the text but continued speaking, suggesting local stopping supervision contributes to time-aligned TTS’s lower WER.Source: §4.2 · p. 6, Table 2 · p. 6, Table 1 · p. 6

4.3 Stage II: Unified Streaming TTS and ASR

• At 300k steps, TTS WER is 3.20% with the TTS teacher, versus 4.35% for SFT only and 3.40% with both teachers; ASR WERs across configurations range from 3.63% to 3.68% on test-clean and 10.56% to 10.74% on test-other.Synthesis anchoring supports TTS quality, while additional ASR distillation benefits early recognition without a statistically significant final gain.Source: §4.3 · p. 7, Table 3 · p. 7
• Uni-Stream maintains 0.66 s mean ASR word-emission delay and 129 ms median TTS first-packet latency, with ASR and TTS real-time factors of 0.16 and 0.64.The latencies are comparable to Stage I specialists’ 0.66–0.67 s and 130 ms, respectively.Source: §4.3 · p. 7

4.4 Uni-Stream Representation Analysis

• Cross-task similarity falls from 0.78 to 0.19 for text and from 0.96 to 0.43 for speech between early and upper layers; speech–text similarity remains low and does not increase after joint training.Compared with initialization, upper-layer cross-task scores are lower by 0.42 for text and 0.24 for speech.Source: §4.4 · pp. 7–8, Figure 2 · p. 8

4.5 Comparison among SLM Baselines

• Uni-Stream uniquely combines streaming recognition and synthesis with shared weights; task-specific streaming systems still perform better on ASR, especially test-other, while TTS speaker similarity is comparable to Kyutai DSM-TTS but WER is higher.These comparisons are contextual, not controlled, because architectures and training data differ.Source: §4.5 · p. 8, Table 4 · p. 8

5 Conclusion

• Uni-Stream unifies streaming speech recognition and synthesis in a shared decoder-only language model using time-aligned speech–text interleaving and two-stage training.The accompanying transcriptions and word-level timestamps will be made public.Source: §5 · p. 9
• Performance is comparable to adapted task-specific models on both tasks; synthesis anchoring helps preserve TTS quality, while recognition distillation improves early ASR without a statistically significant final gain.Speech and text remain distinct under the similarity measure, and representations within each modality become more task-dependent in later layers.Source: §5 · p. 9

Recognition and synthesis quality

• The strongest specialized systems still outperform Uni-Stream on LibriSpeech test-other and synthesized-speech intelligibility.Future work will investigate stronger pretrained backbones and different joint-training strategies for these tasks while maintaining unified streaming capabilities.Source: Limitations · p. 9

Streaming configuration search

• Computational constraints limited configuration exploration, so chunk sizes, lookahead windows, and fixed-ratio text-to-speech ratios were selected empirically or from prior work.The authors say broader evaluation would characterize trade-offs among task performance, system latency, and training cost.Source: Limitations · p. 9

English-only evaluation

• Data preparation, training, and evaluation are limited to English, leaving effectiveness in other languages untested.The authors plan to extend the data pipeline and model training to multilingual settings.Source: Limitations · p. 9

Data and licensing

• Training uses Emilia-EN, GigaSpeech XL, and LibriSpeech train-960, while evaluation uses LibriSpeech and Seed-TTS-Eval under each corpus’s license and terms.Derived transcriptions and timestamps are automatically produced and will be released under the original licenses; the release excludes audio, and Emilia-EN is CC BY-NC 4.0.Source: Ethics Statement · p. 9

Potential misuse

• Zero-shot voice cloning and streaming ASR could facilitate impersonation, fraud, disinformation, or unauthorized transcription of private conversations.If model weights are released, the authors will restrict them to research use and require consent for voice prompts; they also encourage speech detection and watermarking.Source: Ethics Statement · p. 9

Evaluation

• All evaluations use automatic metrics, with no human subjects or crowd workers involved.Source: Ethics Statement · p. 9

Development

• AI coding agents assisted development, and the authors verified all resulting code.Source: Generative AI Disclaimer · p. 9

Manuscript preparation

• AI writing assistants proofread the manuscript, corrected grammar, and improved its language.Source: Generative AI Disclaimer · p. 9
• The authors reviewed and verified all AI-assisted text.Source: Generative AI Disclaimer · p. 10

A.1 Emilia-EN Annotation

• Emilia-EN transcripts are voted from the original transcript and recognizer hypotheses, with singleton insertions removed and utterances filtered by disagreement and alignment criteria.The pipeline merges word-boundary predictions with shared outlier rejection; it also requires monotone spans and applies MMS-FA score thresholds.Source: §A.1 · p. 12
• 24.4 ms is the three-way merge’s mean boundary error on TIMIT, below each individual aligner’s error.The evaluation uses 5,040 utterances with manual word boundaries; all aligners receive the reference word sequence.Source: §A.1 · p. 12, Table 5 · p. 12

A.2 ASR-only Corpora

• GigaSpeech XL and LibriSpeech train-960 retain their provided transcripts, which are normalized and aligned with Qwen3-ForcedAligner-0.6B.GigaSpeech segments with non-speech tags are discarded, and successful monotone alignments are retained subject to the ASR chunk-length limit.Source: §A.2 · p. 12

A.3 Word Boundaries and Sequence Construction

• Time-aligned TTS assigns each word the frames between consecutive cuts, while lookahead words provide text context only.Utterances are rejected if a segment exceeds 32 frames; ASR assigns words to chunks by their end-frame indices and limits each emission chunk to 32 text tokens.Source: §A.3 · pp. 12–13

• Training updates the backbone, modality connectors, and output heads while keeping both speech tokenizers frozen.The optimizer and schedule are reset at the start of Stage II.Source: §B.1 · p. 13
• Linear warm-up precedes cosine decay, and batches group similar-length examples under per-GPU token budgets, including padding.Source: §B.1 · p. 13
• TTS uses same-speaker voice prompts from another utterance, sampled durations of 3–10 s, and independent drops of text and voice conditioning at 0.2 and 0.1.The diffusion objective uses four noise samples per target frame.Source: §B.1 · p. 13

B.2 Evaluation Protocol

• TTS evaluation uses all 1,088 Seed-TTS-Eval English utterances; WER, SIM, and UTMOS use utterance averages, while ASR WER is pooled by LibriSpeech test set.TTS WER uses lowercased text with punctuation removed; ASR WER uses Whisper English normalization.Source: §B.2 · p. 13
• TTS first-packet latency includes prompt encoding and prefix processing; ASR word-emission delay excludes computation time, and RTF divides inference time by audio duration.Latency is measured on an H100 GPU; first-packet latency is the median over 100 batch-one utterances after warm-up.Source: §B.2 · p. 13

B.3 Stage II Comparisons

• No statistically significant differences were detected in the final paired Stage II comparisons, which used 1,000 bootstrap resamples and 95% confidence intervals.ASR WER pools test-clean and test-other; the specialist comparison uses θA for ASR and θT for TTS.Source: §B.3 · p. 13, Table 7 · p. 13

B.4 TTS Stopping Behavior

• 42 fixed-ratio versus 4 time-aligned TTS outputs exceeded the target text by at least three words, a 3.5 percentage-point difference (95% CI [2.4, 4.8]).The confidence interval uses paired bootstrap resampling by voice prompt.Source: §B.4 · p. 13