Source-linked AI summary
Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, Weizhen Bian, Zhen Ye, Sitong Cheng, Ruibin Yuan, Zhixian Zhao, Xinfa Zhu, Jiahao Pan, Liumeng Xue, Pengcheng Zhu, Yunlin Chen, Zhifei Li, Xie Chen, Lei Xie, Yike Guo, Wei Xue
TL;DR
Existing zero-shot TTS systems use complex multi-stage or multi-stream architectures and offer limited voice-attribute control and standardized evaluation. Spark-TTS addresses these gaps with BiCodec and a textual LLM, while VoxBox supports reproducible research and Spark-TTS demonstrates strong zero-shot intelligibility.
Problem
Existing codec-based TTS systems require complex multi-stage or multi-stream architectures, while voice creation, fine-grained attribute control, and standardized evaluation remain limited.
Method
Spark-TTS combines BiCodec’s semantic and fixed-length global tokens with a Qwen2.5-based textual language model for zero-shot synthesis and coarse- and fine-grained attribute control.
Results
Spark-TTS demonstrates significant superiority in zero-shot TTS intelligibility, ranking second only to Seed-TTS on Chinese CER and second only to F5-TTS on English WER.
Takeaways & Limitations
VoxBox provides a rigorously curated 100,000-hour annotated corpus for standardized TTS research and evaluation, while Spark-TTS supports specified-attribute voice generation and zero-shot synthesis.
Takeaways & Limitations
Spark-TTS has relatively lower speaker-similarity metrics in zero-shot TTS than multi-stage or NAR methods and lacks additional disentanglement constraints between global and semantic tokens.
Abstract
from arXiv · showhide
Recent advancements in large language models (LLMs) have driven significant progress in zero-shot text-to-speech (TTS) synthesis. However, existing foundation models rely on multi-stage processing or complex architectures for predicting multiple codebooks, limiting efficiency and integration flexibility. To overcome these challenges, we introduce Spark-TTS, a novel system powered by BiCodec, a single-stream speech codec that decomposes speech into two complementary token types: low-bitrate semantic tokens for linguistic content and fixed-length global tokens for speaker attributes. This disentangled representation, combined with the Qwen2.5 LLM and a chain-of-thought (CoT) generation approach, enables both coarse-grained control (e.g., gender, speaking style) and fine-grained adjustments (e.g., precise pitch values, speaking rate). To facilitate research in controllable TTS, we introduce VoxBox, a meticulously curated 100,000-hour dataset with comprehensive attribute annotations. Extensive experiments demonstrate that Spark-TTS not only achieves state-of-the-art zero-shot voice cloning but also generates highly customizable voices that surpass the limitations of reference-based synthesis. Source code, pre-trained models, and audio samples are available at https://github.com/SparkAudio/Spark-TTS.
1 Introduction
Spark-TTS addresses complexity, limited voice creation, coarse attribute control, and evaluation barriers in zero-shot TTS with a unified codec-LLM system, controllable voice generation, and VoxBox.
- Challenges: Existing codec-based TTS systems require dual models or complex multi-stream prediction, diverging from conventional text LLM frameworks.Semantic tokens also need acoustic prediction modules and lack integrated timbre control.
- Challenges: Reference-based systems lack precisely specified novel voices and fine-grained control over characteristics such as pitch modulation.The paper identifies insufficient attribute-control granularity as a remaining limitation.
- Approach: Spark-TTS combines BiCodec, Qwen2.5, and chain-of-thought prediction in a single codec LLM aligned with conventional text LLMs.BiCodec combines low-bitrate semantic tokens with fixed-length global tokens for linguistic content and time-invariant acoustic characteristics.
- Approach: Spark-TTS supports categorical and continuous voice attributes, including gender, pitch, and speaking speed, beyond traditional reference-based synthesis.The system integrates coarse-grained labels with fine-grained numerical values.
- Resources: VoxBox provides a rigorously curated and annotated 100,000-hour speech corpus for standardized TTS research and evaluation.The dataset was developed through systematic collection, cleaning, and attribute annotation.
2 Related Work
Related work spans semantic, acoustic, and multi-stream speech tokenizers and codec LLMs; Spark-TTS instead uses BiCodec single-stream tokens to unify TTS with text LLMs while adding controllable voice creation.
- Single-Stream Speech Tokenizers: Pure semantic tokens provide low-bitrate encoding but require an additional acoustic feature prediction module for speech synthesis.This reflects the trade-off between compactness and acoustic reconstruction.
- Single-Stream Speech Tokenizers: BiCodec differs from TiCodec by using semantic tokens as time-variant tokens and FSQ-based global embedding quantization with learnable queries and cross-attention.The design produces a relatively longer, more expressive, and flexible global representation.
- LLM-based Zero-Shot TTS: Codec-based zero-shot TTS commonly predicts single-stream codes before another model or diffusion process adds acoustic or continuous semantic features.This is one of two prevalent architecture categories described in the related work.
- LLM-based Zero-Shot TTS: A second category predicts multi-stream codes through parallel strategies or masked generative patterns.These approaches use specialized mechanisms to coordinate multiple code streams.
- LLM-based Zero-Shot TTS: Spark-TTS uses BiCodec single-stream tokens to simplify speech modeling within a text-LLM-compatible framework and add speaker-attribute-controlled voice creation.Compared with Llasa, it is reported to achieve higher zero-shot TTS performance with fewer parameters.
3 BiCodec
BiCodec decomposes speech into low-rate semantic tokens and fixed-length global tokens, then quantizes and decodes them into audio through separate tokenizer pathways and a shared reconstruction process.
- Representation: BiCodec represents speech with 50 TPS semantic tokens for linguistic content and fixed-length global tokens for speaker attributes and global characteristics.The two token types separate time-varying content from time-independent information.
- Tokenizer Structure: The Semantic Tokenizer uses wav2vec 2.0 features, while the Global Tokenizer extracts fixed-length representations from Mel spectrograms.Global representations are formed with cross-attention over learnable queries.
- Tokenizer Structure: BiCodec follows a VQ-VAE encoder-decoder framework in which discrete semantic and global tokens are reconstructed into a time-domain waveform.The decoder G produces the reconstructed signal from quantized representations.
- Quantization: The semantic tokenizer applies VQ, whereas the global tokenizer uses FSQ to represent time-independent global information and reduce VQ training-collapse risk.Factorized codes project semantic encoder outputs into a low-dimensional latent space before quantization.
- Training: BiCodec is trained end-to-end with GAN-based reconstruction, feature-matching, multi-scale mel-spectrogram, and codebook-related losses.The training objective also includes wav2vec 2.0 reconstruction loss to preserve semantic relevance.
4 Language Modeling of Spark-TTS
Spark-TTS uses a decoder-only language model with BiCodec tokens to support both zero-shot synthesis and attribute-driven voice creation. Its inference and training objectives organize semantic, global, fine-grained, and attribute information within a text-LLM-compatible framework.
- Model overview: Spark-TTS uses a decoder-only transformer unified with a typical textual language model, with Qwen2.5-0.5B2 as its backbone.BiCodec’s decoder directly converts language-model outputs into final audio, without flow matching for acoustic-feature generation.
- Inference: The system supports zero-shot TTS and voice creation from gender, pitch, and speed attributes.Without attribute labels, global tokens are extracted from reference audio; with labels, the model predicts attribute and speech tokens.
- Attribute control: Attribute tokenization combines coarse labels with fine-grained pitch and speed values, whose continuous values are rounded to the nearest integer.Gender is categorical, while pitch and speed each use five discrete levels alongside precise value tokens.
- Speech tokens: BiCodec’s speech tokenizer separates speech into global and semantic token streams, and its decoder reconstructs the waveform from both.Semantic tokens represent time-varying linguistic content, while global tokens encode speaker-related information.
- Training objective: Zero-shot training minimizes negative log-likelihood for semantic-token prediction conditioned on text and global speech-token prompts.The semantic output o is predicted using tokenized text T, global prompt G, and language-model parameters θ_LM.
- Training objective: Voice-creation training uses an attribute-label prompt and predicts fine-grained values, global tokens, and speech semantic tokens, with both objectives mixed during training.Each audio example is structured into separate zero-shot and control training samples.
5 VoxBox
VoxBox is an open-source, bilingual benchmark assembled from annotated and cleaned speech datasets. It provides gender, pitch, and speed labels for controllable TTS research and standardized comparison.
- Dataset overview: 102.5k hours of speech across 4.7 million audio files from 29 open datasets comprise VoxBox after data cleaning.The collection covers English and Chinese and includes both common TTS and speech-emotion-recognition datasets.
- Dataset overview: Each VoxBox audio file is annotated with gender, pitch, and speed to support voice creation and benchmark evaluation.The dataset is designed as an open, well-annotated resource for controllable TTS research.
- Gender annotation: 99.4% accuracy was achieved by the fine-tuned WavLM-large gender classifier on the AISHELL-3 test set.The classifier was then used to annotate datasets that previously lacked gender labels.
- Pitch annotation: Pitch values are extracted as average audio pitch, rounded to integers, and grouped into five levels using gender-specific Mel-scale percentile boundaries.The boundaries use the 5th, 20th, 70th, and 90th percentiles for male and female speech separately.
- Speed annotation: Speaking rate is measured in syllables per second after removing leading and trailing silence with voice activity detection.The rounded rate becomes a fine-grained speed token, while five speed levels are defined using percentile boundaries.
- Data cleaning: Lower-quality transcripts are cleaned by re-recognizing audio with FunASR and comparing the result against original scripts using word error rate.This process specifically addresses occasional inaccuracies in Whisper-based transcripts for Emilia.
6 Experiments
Experiments evaluate BiCodec reconstruction, Spark-TTS attribute control, and zero-shot TTS quality. Results show strong reconstruction and controllability, with high intelligibility and competitive quality across the reported evaluations.
- Reconstruction Performance: BiCodec achieves state-of-the-art reconstruction within the low-bitrate range below 1 kbps on most reported metrics.It ranks second on UTMOS behind StableCodec and on SIM behind X-Codec2.
- Reconstruction Performance: Global token sequences approaching length 32 improve reconstruction quality toward the unquantized benchmark.FSQ with learnable queries also substantially improves performance over the GVQ-based method.
- Control Capabilities: Spark-TTS significantly outperforms competing controllable TTS systems on gender control accuracy.The comparison includes VoxInstruct and Parler-TTS.
- Control Capabilities: Spark-TTS accurately aligns generated speech with specified coarse-grained and fine-grained pitch and speaking-rate attributes.Label-based control automatically generates attribute values, whereas manually specified values activate fine-grained control.
- Zero-shot TTS Performance: Spark-TTS demonstrates strong zero-shot intelligibility, ranking second to Seed-TTS for Chinese CER and second to F5-TTS for English WER.Its speaker similarity is weaker than multi-stage or NAR methods but significantly exceeds Llasa despite using 0.5B rather than 8B parameters.
- Zero-shot TTS Performance: Spark-TTS produces significantly higher-quality audio than the original recordings and outperforms CosyVoice2 on LibriSpeech test-clean.The comparison evaluates zero-shot TTS audio generation quality.
7 Conclusion
The paper presents BiCodec, Spark-TTS, and VoxBox as an integrated approach to efficient, controllable speech synthesis. The system combines compact semantic representation with global tokens for attribute control and supports both fine-grained control and zero-shot synthesis.
- 7 Conclusion: BiCodec combines 50-TPS semantic tokens with fixed-length global tokens to retain compression and intelligibility while encoding speaker attributes.It operates at 0.65 kbps and surpasses other codecs in the sub-1 kbps range.
- 7 Conclusion: Spark-TTS integrates Qwen2.5 to generate voices from specified attributes while supporting zero-shot synthesis.The paper identifies it as the first TTS model to combine fine-grained pitch and speaking-rate control with zero-shot TTS.
- 7 Conclusion: VoxBox is an open-source dataset with cleaned text and annotations for gender, pitch, and speaking rate.The dataset is intended to support comparative research in controllable speech synthesis.
Limitation
The main limitation is lower zero-shot speaker similarity than multi-stage or non-autoregressive methods. The authors attribute this partly to autoregressive inference variability and identify missing token disentanglement as a target for future work.
- Limitation: Spark-TTS has relatively lower zero-shot speaker similarity than multi-stage or NAR methods.The authors associate this limitation with greater speaker variability introduced by the autoregressive language model during inference.
- Limitation: BiCodec currently lacks additional disentanglement constraints between global tokens and semantic tokens.Future work proposes perturbing formants or pitch in semantic-token inputs to strengthen global-token control over timbre.
A.3 Additional Experiment
Additional experiments examine lower-bitrate reconstruction, global-token length, generalization, inference strategies, and controllable voice creation. The reported analyses compare tokenization choices and show that the second zero-shot inference strategy yields higher speaker similarity.
- Additional Experiment: BiCodec is evaluated at 25 TPS against Single-Codec on LibriSpeech and LibriTTS test-clean datasets.The downsampling operation reduces the semantic token rate for lower-bitrate comparison.
- Additional Experiment: Increasing global token length consistently improves reconstruction quality, approaching the benchmark at length 32.The comparison includes unquantized and group-VQ configurations on LibriTTS test-clean.
- Additional Experiment: BiCodec generalization is evaluated across a broader range of diverse datasets.The results are reported in Table 9.
- Additional Experiment: The second zero-shot inference strategy achieves higher speaker similarity than the alternative.It uses the synthesis text together with reference transcript, global tokens, and reference semantic tokens as a prompt.
- Additional Experiment: Spark-TTS supports coarse-grained control with attribute labels and fine-grained control with manually specified attribute values.The coarse-grained prompt predicts attribute values before global and semantic tokens.
C Compared Zero-shot Methods •
This section describes controllable speech-token generation and compares Spark-TTS with alternative TTS architectures and evaluation settings. It also outlines attribute annotation schemes and dataset distributions used for evaluation.
- Controllable generation: Spark-TTS generates global tokens before semantic tokens in a chain-of-thought sequence, with prompts containing text, attribute levels, and precise values.The fine-grained prompt format maps content and attribute controls to global and semantic token outputs.
- Compared zero-shot methods: Compared methods include flow-matching systems, two-stage semantic-plus-acoustic models, and Llasa’s single-stream codec-based autoregressive model.E2 TTS and F5-TTS predict Mel spectrograms, CosyVoice variants use flow matching after semantic-token prediction, and Llasa predicts single-stream codes directly.
- Reconstruction evaluation: BiCodec reconstruction is evaluated across bitrates, global-token lengths, and datasets including low-quality Chinese, expressive studio, and multilingual in-the-wild speech.The referenced tables cover bitrate and global-token-length variants and define Data-P, Data-S, and Data-M evaluation conditions.
- Metrics and attributes: Evaluation uses STOI, PESQ, UTMOS, and speaker similarity, while speech-rate and pitch categories are defined using percentile-based boundaries.PESQ is reported in wide-band and narrow-band settings, and SIM compares speaker embeddings from generated and prompt speech.
- Attribute annotation: VoxBox annotations include gender, pitch, speed, age, and emotion, with gender and age prediction models evaluated on AISHELL-3.The dataset also uses multiple model-derived emotion labels and confidence scores for samples lacking original emotion annotations.
E.5 Source Data
VoxBox combines numerous English, Mandarin, bilingual, multilingual, conversational, reading-style, and emotional speech sources. Its statistics and demographic distributions are summarized alongside an open-source speech-task training framework.
- Source diversity: The source collection spans Mandarin, English, bilingual, and multilingual corpora covering reading, conversational, and emotional speech.Examples include AISHELL-3, ESD, Expresso, Gigaspeech, HQ-Conversations, IEMOCAP, and Emilia.
- Source datasets: Additional sources include LibriSpeech, LibriTTS-R, M3ED, CASIA, MAGICDATA, MEAD, MELD, MER2023, MLS-English, and MSP-Podcast.These datasets provide reading-style, conversational, multilingual, and emotional speech across languages.
- Source datasets: The collection also includes NCSSD-CL, NCSSD-RL, RAVDESS, CREMA-D, SAVEE, TESS, VCTK, WenetSpeech4TTS, DailyTalk, EMNS, and EmoV-DB.The listed corpora add bilingual, multilingual, emotional, conversational, and multi-speaker speech resources.
- SparkVox toolkit: SparkVox is an open-source framework supporting vocoder, codec, TTS, and speech-understanding tasks, with file-processing tools for text and speech data.The Spark-TTS training code is planned for integration into SparkVox, whose simplified structure is illustrated in Figure 9.