Source-linked AI summary
MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance
Xingjian Zhao, Zhe Xu, Qinyuan Cheng, Zhaoye Fei, Luozhijie Jin, Yang Wang, Hanfu Chen, Yaozhou Jiang, Qinghui Gao, Ke Chen, Ruixiao Li, Mingshu Chen, Ruiming Wang, Wenbo Zhang, Yiyang Zhang, Donghua Yu, Yang Gao, Xiaogui Yang, Yitian Gong, Yuanfan Xu, Yaqian Zhou, Xuanjing Huang, Xipeng Qiu
TL;DR
Spoken dialogue systems can lose speech information in cascaded pipelines, while text-guided generation still relies on an intermediate text representation. MOSS-Speech adds native speech understanding and generation to a pretrained text LLM through modality-based layer splitting and frozen pre-training. It reports state-of-the-art spoken question-answering results and near parity with text-guided methods, while supporting text and speech as native modalities.
Problem
Text-guided speech systems retain speech input cues but rely on intermediate text during generation, which adds latency, reduces efficiency, and restricts expressivity.
Source: §1 · p. 1Method
MOSS-Speech combines modality-based layer splitting and frozen pre-training to add native speech capabilities to a pretrained text LLM while largely retaining its text abilities.
Source: §1 · p. 2Results
The model reports state-of-the-art spoken question-answering results and near parity with text-guided methods, while supporting text and speech as input and output modalities.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
The authors conclude that end-to-end speech modeling can approach text-guided methods while addressing their latency and expressivity limitations.
Source: §7 · p. 10Chinese evaluations are not performed on models trained only in English.
Source: §4.1 · p. 8
Abstract
from arXiv · showhide
Spoken dialogue systems often rely on cascaded pipelines that transcribe, process, and resynthesize speech. While effective, this design discards paralinguistic cues and limits expressivity. Recent end-to-end methods reduce latency and better preserve these cues, yet still rely on text intermediates, creating a fundamental bottleneck. We present MOSS-Speech, a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance. Our approach combines a modality-based layer-splitting architecture with a frozen pre-training strategy, preserving the reasoning and knowledge of pretrained text LLMs while adding native speech capabilities. Experiments show that our model achieves state-of-the-art results in spoken question answering and delivers comparable speech-to-speech performance relative to existing text-guided systems, while still maintaining competitive text performance. By narrowing the gap between text-guided and direct speech generation, our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
1 Introduction
Cascaded and text-guided dialogue systems have limitations that motivate direct speech-to-speech modeling. MOSS-Speech combines modality-based layer splitting and frozen pre-training to add native speech abilities while retaining text capabilities.
1 Introduction
- Cascaded systems transcribe speech, generate a text response, and synthesize audio, discarding information in the original speech signal.They also constrain responses to content faithfully representable in text.Source: §1 · p. 1
- Text-guided systems preserve paralinguistic input cues, but intermediate text adds latency, reduces efficiency, and restricts expressivity.Non-verbal vocalizations such as laughter and hesitation lack natural text equivalents.Source: §1 · p. 1
- The approach combines modality-specific layer splitting with frozen pre-training to add native speech understanding and generation while largely retaining text capabilities.The paper says alignment transfers reasoning, world knowledge, and generalization from the text backbone to speech.Source: §1 · p. 2
- The paper presents a true speech-to-speech model that achieves state-of-the-art results on speech-to-speech benchmarks without intermediate text guidance.It also supports text and speech as both input and output modalities.Source: §1 · p. 2
2 Model Architecture
MOSS-Speech uses shared Transformer layers for multimodal fusion, then specializes later layers for text or speech generation. Its tokenizer is designed for efficient, streaming speech generation while retaining semantic and paralinguistic information.
2 Model Architecture
- The architecture adds modality-based layer splitting to an autoregressive Transformer for deep fusion of heterogeneous modalities and modality-specific generation.The split-then-specialize design uses early layers for joint fusion and final layers for modality-specific generation.Source: §2 · p. 2, §2.1 · p. 3
2.1 Modality-based Layer Split
- In a 28-layer model, speech–text similarity rises through the first 11 layers, stabilizes across the following 14, then decreases in the final 3.The model splits at block 32 of its 36-layer Transformer, routing the shared state to separate four-layer text and speech branches.Source: §2.1 · p. 3
2.2 Speech Tokenization
- The tokenizer targets a single-codebook, low-bitrate representation that supports efficient generation, semantic transfer, paralinguistic detail, and streaming.These objectives are intended to support low-latency processing and faithful speech reconstruction.Source: §2.2 · p. 3
- The tokenizer encoder uses ASR as its sole training objective and is made fully causal, rather than block-causal, for streaming support.The design addresses the reported mismatch between reconstruction-optimized tokens and LLM learning needs.Source: §2.2 · p. 4
- The decoder uses flow matching and compresses chunk size to reduce latency while maintaining reconstruction quality.The change addresses delays associated with CosyVoice 2's chunk-attention mechanism.Source: §2.2 · p. 4
3 Training Strategy
Training introduces speech into a pretrained text LLM through two-stage pre-training, followed by supervised fine-tuning on constructed multimodal data. The fine-tuning configurations cover speech and text as both inputs and outputs.
3.1 Pre-training
- Pre-training initializes from Qwen-3-8B and uses a two-stage pipeline to add speech while preserving the model's text capabilities.Stage 1 freezes the text backbone and trains newly introduced speech components; Stage 2 unfreezes a larger portion for cross-modal adaptation.Source: §3.1 · p. 4, §3.1.2 · p. 5
3.1.1 Data Collection and Processing
- The corpus starts with about 9 million hours of internet audio, filtered to roughly 4 million hours of speech, with podcasts used mainly for interleaved speech–text and video for unsupervised pre-training.The authors also synthesize speech–text pairs from English and Chinese text corpora to enrich the data.Source: §3.1.1 · pp. 4–5
3.1.2 Two-stage Pre-training
- Stage 1 freezes all Qwen-3-8B backbone parameters; Stage 2 tests three unfreezing configurations and adds text-only data to preserve linguistic competence.The three configurations achieve comparable results, and the authors use full-model unfreezing as the default for subsequent supervised fine-tuning.Source: §3.1.2 · p. 5
3.2.1 Data Adaptation and Construction
- The constructed supervised fine-tuning set contains over 1.5 million question–answer pairs, approximately 650,000 English and 860,000 Chinese.The process adapts text data for speech, synthesizes audio with multiple TTS systems, and filters entries whose ASR transcript WER is at least 0.2.Source: §3.2.1 · p. 6
3.2.2 Training Details
- Fine-tuning uses four input–output pairings: speech-to-speech, speech-to-text, text-to-speech, and text-to-text.The pairings share underlying content and are controlled by system prompts; training runs for two epochs.Source: §3.2.2 · p. 6
4 Evaluation
The evaluation covers the encoder and decoder, pre-trained speech and text capabilities, and supervised fine-tuning for question answering and speech quality. The reported codec results show competitive encoder WER and decoder gains in intelligibility and perceived quality against CosyVoice 2.
4.1 Tokenizer
- The encoder reaches 10.80% WER, versus GLM-4-Voice’s 9.17% with 2-second blocks, while operating in full streaming; it also beats non-streaming Mimi-8 (14.45%) and CosyVoice 2 (13.78%).Source: §4.1 · p. 7
4.2 Pre-training
- Pre-training is evaluated for coherent speech continuations and preservation of broad textual knowledge and reasoning.The speech benchmarks are StoryCloze and zh-StoryCloze; textual capability is evaluated on MMLU and CMMLU.Source: §4.2 · p. 8, Table 5 · p. 8
4.3 Supervised Fine-tuning
- The supervised fine-tuning evaluation measures question-answering ability and generated-speech quality.QA uses LLaMA-Question, Trivia QA, and Web Questions; speech quality is assessed with UTMOS.Source: §4.3 · p. 8, Table 6 · p. 8
5 Ablation Study
The ablation study finds that modality-based layer splitting and frozen pre-training improve speech modeling and preservation of text ability, while unfreezing strategies make relatively small differences.
5 Ablation Study
- The study isolates layer-splitting effects by comparing NF–NoSplit with NF, then frozen pre-training by comparing NF with FP–Full.It also tests full, shared-layer-only, and gradual reverse-order unfreezing after frozen pre-training.Source: §5 · p. 9
- Modality-based Layer Split improves speech modeling and textual ability preservation, Frozen Pre-training adds substantial gains, and unfreezing strategies differ relatively little.The table compares variants with and without modality separation and frozen pre-training, then compares full, shared-layer, and layerwise unfreezing.Source: §5 · p. 9
- The authors conclude that modality separation and freezing text parameters during pre-training are both critical to balancing speech learning with text-knowledge preservation.Different unfreezing schedules produce slight trade-offs, with smaller impact than layer splitting and frozen pre-training.Source: §5 · p. 9
6 Related Works
Related work spans codec design, speech-to-speech interaction models, and strategies for extending language models while retaining text capabilities.
6 Related Works
- Acoustic codecs preserve fine-grained detail but may lack semantic coherence, whereas semantic-oriented codecs favor linguistic content; this work follows the latter and adds streaming modules.The cited acoustic-codec examples include Zeghidour et al. (2021), Défossez et al. (2022), and Kumar et al. (2023); semantic-oriented examples include Du et al. (2024) and Zeng et al. (2024a).Source: §6 · p. 9
- Prior speech-interaction systems commonly retain text guidance, while Mini-Omni generates text and speech together but is described as limited without prior speech pre-training.The passage characterizes SpeechGPT, Moshi, Qwen-Audio, LLaMA-Omni, Freeze-Omni, and GLM-4-Voice as retaining forms of text prompts, instructions, or supervision.Source: §6 · pp. 9–10
- Prior approaches freeze LLM parameters or adapt progressively; the cited work associates freezing with preserved language modeling and progressive unfreezing with flexible multimodal integration.Examples include Freeze-Omni (Wang et al., 2025b), SpeechVerse, and Mini-Omni (Xie & Wu, 2024).Source: §6 · p. 10
7 Conclusion
The conclusion presents a speech-to-speech model that operates without intermediate text and reports state-of-the-art spoken question answering while retaining native text and speech modalities. The accompanying adaptation prompt specifies how Q&A content is adjusted for spoken delivery.
7 Conclusion
- The model supports speech-to-speech interaction without intermediate text and achieves state-of-the-art spoken question answering while supporting text and speech as native inputs and outputs.The authors describe its results as near parity with text-guided methods and its design as preserving text abilities while transferring linguistic and reasoning knowledge to speech.Source: §7 · p. 10
A Prompt for Supervised Fine-tuning Text Adaptation
- The adaptation prompt classifies each Q&A pair as PASSTHROUGH, ADAPT, or REJECT according to suitability for TTS.It directs the system to preserve meaning while rephrasing written forms for speech, adapting or rejecting content that cannot be rendered appropriately.Source: A Prompt for Supervised Fine-tuning… · p. 16
- The prompt converts or narrates non-vocal content when feasible and shortens material exceeding 200 words while preserving factual correctness and logical flow.Complex or block LaTeX may be read aloud if feasible or rejected; tables are to be summarized and URLs rephrased or removed.Source: A Prompt for Supervised Fine-tuning… · p. 16
B.1 HeatMaps
The layer-wise heatmaps show aligned-token similarity emerging in lower layers, then declining sharply at the final layer.
B.1 HeatMaps
- At layer 7 the similarity diagonal begins to appear, is clear by layer 11, and other-token similarity rises after layer 24 before all similarities drop at layer 27.This pattern is consistent with the Similarity Score figure.Source: §B.1 · p. 17
B.2 Samples
The samples are English text aligned with speech, and the section describes a layer-wise measure of text–speech similarity using aligned token pairs and a DTW path.
B.2 Samples
- The five random samples used to calculate the similarity score have English text content that is aligned.The section also includes sample text on language relationships and Pacific species, alongside material about speech–text alignment.Source: §B.2 · pp. 17, 19
- Speech–text alignment matches acoustic segments with text units over time, enabling the model to learn speech–language correspondences.The passage describes high-quality alignment as a prerequisite for better generalization and practical performance in large speech models.Source: §B.2 · p. 19
C Layer-wise Similarity Score.
- For each layer, the method forms cosine-similarity matrices from speech and text hidden states, then averages similarity along the optimal DTW path for each aligned pair.The path maximizes similarity under temporal constraints; the layer score also normalizes against non-aligned text-token similarities.Source: C Layer-wise Similarity Score · pp. 19–20
- A background normalization and a global coefficient are combined across pairs and layers to quantify relative text–speech alignment at each Transformer layer.The score is defined per layer from the DTW-based similarity and the background normalization.Source: C Layer-wise Similarity Score · p. 20
D Layer-wise Unfreeze Learning Rate Schedule
The schedule gives each of 32 transformer layers its own delayed warmup–cosine learning rate, staggering adaptation from later to earlier layers.
D Layer-wise Unfreeze Learning Rate Schedule
- Each transformer layer receives its own learning rate on a delayed warmup–cosine schedule, with u = s − d_i shifting the schedule by layer.Source: D Layer-wise Unfreeze Learning Rate… · p. 20
- 32 layers are indexed from 0 to 31, and adjacent layers are separated by k steps through d_i = (31 − i)k.Source: D Layer-wise Unfreeze Learning Rate… · p. 20
- The schedule uses a 2,000-step warmup, k = 5,000 steps, and η_min = 0.1η_max; T marks when all layers reach η_min.Source: D Layer-wise Unfreeze Learning Rate… · p. 20
- Higher layers unfreeze earlier while lower layers stay frozen longer, enabling controlled and stable adaptation of the pretrained text backbone.Source: D Layer-wise Unfreeze Learning Rate… · p. 21