Source-linked AI summary
MOSS-TTS Technical Report
Yitian Gong, Botian Jiang, Yiwei Zhao, Yucheng Yuan, Kuangwei Chen, Yaozhou Jiang, Cheng Chang, Dong Hong, Mingshu Chen, Ruixiao Li, Yiyang Zhang, Yang Gao, Hanfu Chen, Ke Chen, Songlin Wang, Xiaogui Yang, Yuqian Zhang, Kexin Huang, ZhengYuan Lin, Kang Yu, Ziqi Chen, Jin Wang, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu
TL;DR
Speech generation needs to generalize across speakers, languages, styles, and conditions while supporting control, low latency, and long-form stability. MOSS-TTS addresses this with discrete audio tokens, autoregressive modeling, and large-scale multilingual pretraining, yielding broad controllability and complementary trade-offs between its two architectures.
Problem
Speech generation is expected to generalize across speakers, languages, speaking styles, and acoustic conditions while supporting controllable, low-latency, and stable long-form synthesis.
Method
MOSS-TTS uses a high-quality audio tokenizer and purely autoregressive next-token modeling over aligned text and speech tokens, with two complementary generator architectures.
Results
The models support zero-shot voice cloning, duration and pronunciation control, multilingual code-switching, and stable long-form generation, with Local-Transformer generally stronger on speaker similarity and MOSS-TTS better suited to duration control and ultra-long generation.
Takeaways & Limitations
The recipe provides a clean, scalable path to speech generation with broad controllability, while exposing a practical choice between speaker preservation and long-context or control-oriented deployment.
Takeaways & Limitations
Speaker labels are meaningful only within individual recordings, and adjacent same-speaker segments are merged regardless of intervening silence duration.
Abstract
from arXiv · showhide
This technical report presents MOSS-TTS, a speech generation foundation model built on a scalable recipe: discrete audio tokens, autoregressive modeling, and large-scale pretraining. Built on MOSS-Audio-Tokenizer, a causal Transformer tokenizer that compresses 24 kHz audio to 12.5 fps with variable-bitrate RVQ and unified semantic-acoustic representations, we release two complementary generators: MOSS-TTS, which emphasizes structural simplicity, scalability, and long-context/control-oriented deployment, and MOSS-TTS-Local-Transformer, which introduces a frame-local autoregressive module for higher modeling efficiency, stronger speaker preservation, and a shorter time to first audio. Across multilingual and open-domain settings, MOSS-TTS supports zero-shot voice cloning, token-level duration control, phoneme-/pinyin-level pronunciation control, smooth code-switching, and stable long-form generation. This report summarizes the design, training recipe, and empirical characteristics of the released models.
1 Introduction
MOSS-TTS frames speech generation as scalable token prediction, addressing representation, long-context, streaming, and data-diversity challenges through a tokenizer–AR–pretraining recipe. The report introduces complementary architectures and demonstrates broad controllability and long-form capabilities.
- Speech foundation models must generalize across speakers, languages, styles, acoustic conditions, controllability settings, latency requirements, and long-form content.
- Scaling speech generation requires compact yet expressive tokens, stable long-sequence modeling compatible with streaming, and training signals that scale across noisy real-world data.
- The report advocates discrete tokens, autoregressive modeling, and large-scale pretraining as a clean, scalable path to speech quality and controllability.
- MOSS-TTS combines a causal discrete audio tokenizer, large-scale multilingual data, and efficient discrete autoregressive modeling for speech generation.
- MOSS-TTS supports zero-shot voice cloning, duration and pronunciation control, multilingual code-switching, and stable hour-scale long-form generation.
- The released architectures trade structural simplicity and scalability against modeling efficiency, speaker preservation, and time to first audio.
2 Related Work
Related work establishes discrete audio tokenization and autoregressive modeling as foundations for scalable speech generation, while surveying alternative TTS architectures and controllability mechanisms.
- Neural codecs learn encoder–quantizer–decoder stacks that compress audio while supporting downstream sequence modeling.
- Discrete audio tokens cast generation as token sequence modeling, enabling language-model-like scaling and supporting intelligible speech and zero-shot TTS.
- TTS research spans autoregressive, non-autoregressive, flow-based, diffusion-based, and end-to-end architectures with different speed, controllability, and quality tradeoffs.
- Modern TTS systems increasingly target voice cloning, speaking-rate or duration control, pronunciation control, and multilingual universal generation.
3 Audio Tokenizer
MOSS-Audio-Tokenizer is a causal Transformer audio codec designed to provide compact, variable-bitrate, semantically informative tokens for scalable autoregressive generation.
- Audio tokenizers bridge continuous audio and unified autoregressive generation by combining high-fidelity reconstruction with sequential modeling compatibility.
- MOSS-Audio-Tokenizer addresses limitations of external encoders, multistage pipelines, and architecture-specific biases through a CAT-based design.
- 24kHz audio is compressed to 12.5 fps using 32-layer RVQ, with variable bitrate from 0.125 to 4 kbps.
- The tokenizer uses a pure causal Transformer architecture intended to simplify implementation and improve scalability, with a 1.6-billion-parameter capacity.
- Pretraining on millions of hours of speech, music, and environmental audio supports generalization across audio domains.
- Its tokens preserve reconstruction quality while capturing semantic information suitable for autoregressive language modeling.
- Encoder, quantizer, decoder, semantic LLM, and discriminators are jointly optimized using reconstruction, adversarial, feature-matching, quantization, and audio-to-text objectives.
4 Architecture
MOSS-TTS uses discrete multi-stream audio tokens with autoregressive modeling, comparing a simple delay-pattern backbone with a more complex Local Transformer for different deployment tradeoffs.
- Overall architecture: MOSS-TTS models discrete audio tokens autoregressively, with the token modeling pattern determining complexity, scaling, latency, and synthesis quality.The system handles a 32-layer RVQ token block rather than expanding the sequence length to T × N_q.
- Architecture tradeoffs: The two architectures share the tokenizer and pretraining recipe but trade structural simplicity and scalability against modeling efficiency and stronger speaker similarity at smaller scale.The Local Transformer is used to emphasize voice-cloning quality, while MOSS-TTS is used for duration, pronunciation, and ultra-long generation.
- Delay Pattern: The Delay Pattern uses one Transformer backbone with multiple prediction heads and an RVQ-aware delay schedule, avoiding a T × N_q sequence-length increase.Each RVQ layer is shifted forward by j−1 frames, and channel predictions come from lightweight head projections.
- Delay Pattern: The delay architecture sums embeddings across RVQ layers at each delayed time step and feeds the resulting sequence, alongside text embeddings, to the backbone.The delayed audio representation has length T + N_q − 1, with separate embedding tables for the speech codebooks.
- Local Transformer: The Local Transformer sums RVQ-layer embeddings without temporal shifts, then autoregressively expands each backbone latent into the within-step token block.Its added frame-local loop increases complexity but provides a stronger inductive bias for frame-level token modeling.
5 Pretraining
MOSS-TTS pretraining uses a staged pipeline that standardizes raw recordings, creates speaker-consistent transcript pairs, and synthesizes targeted data to add voice-cloning, robustness, and pronunciation supervision.
- Data preprocessing: Raw open-domain recordings require acoustic standardization because heterogeneous formats and noise undermine diarization, ASR, and training consistency.Preprocessing includes denoising, format alignment, and volume normalization before later stages.
- Speaker diarization: Diarization produces speaker-labeled intervals, while speaker labels remain recording-local and are not linked across recordings.DiariZen is used for diarization after denoising.
- Segment consolidation: Consolidation removes segments shorter than 0.1 s, merges adjacent same-speaker segments without a gap threshold, and truncates output to at most one hour.The procedure preserves contiguous single-speaker coverage while preventing unbounded unit lengths.
- Transcript filtering: The filtered corpus is transcribed and quality-controlled before audio-transcript pairs are jointly filtered for consistency.This sequence forms the transcript-quality stage of the preprocessing pipeline.
- Data synthesis: Targeted synthesis addresses missing prompt-conditioned timbre transfer, robustness to noisy text, and phonetic-script supervision.Timbre-cloning pairs use prompt and target audio from the same speaker, while supplements add formatting-noise and phonetic-input examples.
- Pretraining curriculum: The curriculum introduces dense basic supervision first, control-oriented data during stable high learning rates, and long-context training only after short-context convergence.Phase 3 restores normal mixture proportions while decaying the learning rate from 2×10−4 to 2×10−6 for quality consolidation.
6 Evaluation
The evaluation covers tokenizer reconstruction and speech-generation capabilities, including voice cloning, multilingual transfer, duration and pronunciation control, and ultra-long synthesis. MOSS-Audio-Tokenizer outperforms open-source baselines on speech reconstruction, while MOSS-TTS demonstrates controllability and long-form generation with speaker drift as the main ultra-long bottleneck.
- 6.1 Audio Tokenizer: MOSS-Audio-Tokenizer consistently outperforms compared open-source baselines in speech reconstruction across evaluated bitrates.It remains competitive on general audio and music benchmarks.
- 6.1 Audio Tokenizer: The tokenizer’s reconstruction quality scales with bitrate, leveraging additional capacity through joint end-to-end optimization.Flexible RVQ-layer selection supports applications ranging from low-bitrate use to high-fidelity generation.
- 6.2 Voice Cloning: Continuation consistently improves speaker similarity over Clone, while MOSS-TTS-Local-Transformer preserves speakers more strongly despite using 1.7B parameters.In Continuation, it achieves the highest Chinese and English similarity scores among the open-source models in the table.
- 6.3 Multilingual Generation: MOSS-TTS remains competitive across several non-Chinese/English languages, with stable performance on German, Spanish, Italian, and Russian.The largest gaps occur in harder language pairs such as Japanese/Korean and some English continuation cases.
- 6.4 Duration Control: Overall relative duration error is around 0.7% from short to long utterances, achieved without dedicated duration-control fine-tuning.This demonstrates practically usable token-level duration control under a pretraining-only setup.
- 6.5 Ultra-Long Speech Generation: Ultra-long generation remains operational, but cumulative speaker drift over elapsed time is the dominant bottleneck rather than immediate lexical failure.Continuation substantially improves long-horizon speaker anchoring, especially in the longest Chinese and English buckets.
7 Conclusion
MOSS-TTS demonstrates a scalable open speech-generation foundation built from audio tokenization, autoregressive modeling, and large-scale multilingual pretraining. Its two architectures offer complementary tradeoffs, while evaluations identify speaker drift, multilingual coverage, and fine-grained controllability as remaining challenges.
- MOSS-TTS and MOSS-TTS-Local-Transformer implement a scalable speech-generation recipe combining a high-quality tokenizer, autoregressive modeling, and multilingual pretraining.
- The two models provide complementary operating points: MOSS-TTS favors simplicity, scalability, and long-context control, while Local-Transformer favors efficiency, speaker preservation, and faster first audio.
- Zero-shot cloning generally benefits from Local-Transformer speaker similarity, whereas MOSS-TTS is better suited to duration control and ultra-long generation.
- Long-horizon speaker drift, especially in English, remains the dominant ultra-long-generation failure mode rather than immediate lexical failure.
- The report identifies stronger long-context speaker anchoring, broader low-resource language coverage, and improved fine-grained controllability as key future directions.
- The released tokenizer, models, and unified autoregressive objective establish a practical foundation for open speech generation.