Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis

Zhen Ye; Xinfa Zhu; Chi-Min Chan; Xinsheng Wang; Xu Tan; Jiahe Lei; Yi Peng; Haohe Liu; Yizhu Jin; Zheqi Dai; Hongzhan Lin; Jianyi Chen; Xingjian Du; Liumeng Xue; Yunlin Chen; Zhifei Li; Lei Xie; Qiuqiang Kong; Yike Guo; Wei Xue

Source-linked AI summary

Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis

Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi Dai, Hongzhan Lin, Jianyi Chen, Xingjian Du, Liumeng Xue, Yunlin Chen, Zhifei Li, Lei Xie, Qiuqiang Kong, Yike Guo, Wei Xue

arXiv:2502.04128v2Feb 6, 2025eess.AScs.AIcs.CLcs.MMcs.SD

TL;DR

Existing LLM-based TTS systems are often multi-stage, making compute-scaling choices across separate models difficult. Llasa uses a single Transformer with a speech tokenizer to study train-time and inference-time scaling, finding improvements in naturalness, prosody, and verifier-targeted speech qualities.

Problem
Many current LLM-based TTS systems use multiple stages and separate models, complicating analysis of where to scale training or inference compute.
Method
Llasa combines a single Llama-initialized Transformer with a speech tokenizer and studies training-time scaling plus verifier-guided inference-time search.
Results
Scaling training-time compute improves naturalness and prosody, while verifier-guided inference-time scaling improves emotional expressiveness, timbre consistency, and content accuracy.
Takeaways & Limitations
A unified LLM-aligned TTS system supports systematic study of both train-time and inference-time compute scaling and achieves strong zero-shot and state-of-the-art results.

Abstract

from arXiv · show

Recent advances in text-based large language models (LLMs), particularly in the GPT series and the o1 model, have demonstrated the effectiveness of scaling both training-time and inference-time compute. However, current state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring separate models (e.g., diffusion models after LLM), complicating the decision of whether to scale a particular model during training or testing. This work makes the following contributions: First, we explore the scaling of train-time and inference-time compute for speech synthesis. Second, we propose a simple framework Llasa for speech synthesis that employs a single-layer vector quantizer (VQ) codec and a single Transformer architecture to fully align with standard LLMs such as Llama. Our experiments reveal that scaling train-time compute for Llasa consistently improves the naturalness of synthesized speech and enables the generation of more complex and accurate prosody patterns. Furthermore, from the perspective of scaling inference-time compute, we employ speech understanding models as verifiers during the search, finding that scaling inference-time compute shifts the sampling modes toward the preferences of specific verifiers, thereby improving emotional expressiveness, timbre consistency, and content accuracy. In addition, we released the checkpoint and training code for our TTS model (1B, 3B, 8B) and codec model publicly available.

1. Introduction

Llasa aligns TTS with the minimalist Transformer-and-tokenizer paradigm of text LLMs to study training-time and inference-time scaling. The framework improves speech naturalness, prosody, and verifier-targeted qualities while providing a unified foundation for further research.

Motivation: TTS research often emphasizes customized architectures, whereas a standard Transformer-and-tokenizer framework can support broader investigations into scaling behavior.The text LLM community’s common design enables research on training-time scaling laws and inference-time scaling behaviors.
Llasa framework: Llasa uses a single Transformer initialized from Llama with an expanded vocabulary containing speech tokens and trains by next-token prediction.Its streamlined design may not always match highly customized TTS systems, but it creates a unified foundation beyond architecture exploration.
Scaling studies: Llasa systematically studies both training-time and inference-time compute scaling within one unified TTS framework.The framework examines scaling through model size or training data and through search with speech understanding verifiers.
Training-time scaling: Scaling training-time compute improves speech naturalness and expressive prosody, including patterns that capture meaning conveyed in text without explicit labels.The paper reports that larger models or more training data enhance prosody accuracy and complexity.
Inference-time scaling: Scaling inference-time compute with speech understanding verifiers improves emotional expressiveness, timbre consistency, and content accuracy by aligning outputs with verifier preferences.Evaluations on LibriSpeech, seed-tts-eval, and ESD demonstrate state-of-the-art results and search-based control of speaker identity or emotion.

2. Methods

Llasa aligns TTS with a standard LLM paradigm using a speech tokenizer and one Transformer, enabling unified investigation of train-time and inference-time scaling. Its experiments vary model size, training data, and inference search while evaluating text understanding and zero-shot speech generation.

Framework: Llasa keeps two main components—a tokenizer and a single Transformer-based LLM—to model discrete speech tokens autoregressively.The Transformer is initialized from an existing LLM such as Llama, while speech waveforms are converted into discrete tokens.
Training Objective: The model learns to predict each speech token conditioned on the text tokens and previously generated speech tokens using a negative log-likelihood objective.Training loss is calculated over the speech-token sequence.
Speech Tokenizer: X-codec2 fuses semantic and acoustic features into one codebook and uses a single vector quantizer for one-dimensional causal dependency.This design aligns speech-token generation with left-to-right autoregressive modeling and captures content, prosody, and timbre without extra decoding information.
Scaling Train-time Compute: Train-time scaling varies Transformer size at 250k hours or varies training data among 80k, 160k, and 250k hours with a 1B model.The model-size comparison uses Llasa-1B-250k, Llasa-3B-250k, and Llasa-8B-250k.
Evaluation: Evaluation measures text understanding across linguistic and expressive categories and zero-shot TTS generalization to unseen speakers, timbres, and emotional expressions.The text-understanding evaluation covers seven English categories and corresponding Chinese categories.
Scaling Inference-time Compute: Inference-time scaling generates multiple candidate speech sequences and uses reward models or verifiers to select higher-scoring outputs.Output reward models evaluate fully generated speech holistically, while beam search expands B beams by M tokens into N candidates and retains the top B.

3. Experiments

The experiments evaluate Llasa’s codec, TTS quality, and scaling behavior across training-time and inference-time compute. Scaling model size, data, or inference search generally improves naturalness, prosody, speaker similarity, emotion similarity, and transcription accuracy, while single-token reconstruction limits SIM-O.

Evaluation setup: Llasa’s experiments cover codec comparisons, TTS evaluation, train-time scaling, inference-time scaling, and baseline comparisons.The evaluation uses LibriSpeech, Seed-TTS-Eval4, and ESD, with WER, STOI, PESQ, speaker similarity, emotional similarity, and expert scores.
Codec evaluation: X-codec2 achieves the best performance at a token rate of 50 for most codec metrics.Low-token-rate codecs show improved performance over earlier DAC and EnCodec systems, but acoustic reconstruction remains limited.
Scaling train-time compute: Increasing model size from 1B to 8B and training data from 80k to 250k hours generally improves expert scores for Chinese and English TTS tasks.Larger models particularly improve emotion, poetry, and tongue twisters, while broader data coverage especially benefits rare characters and some English tasks.
Scaling train-time compute: Speaker similarity, word error rate, and emotional similarity consistently improve as model size and training data increase.These gains are reported as evidence of enhanced in-context learning ability.
Scaling inference-time compute: Best-of-N improves speaker similarity with more inference compute, while PRM beam search outperforms ORM under the same compute budget.A partial PRM strategy followed by ORM maintains WER near ground truth while achieving higher SIM than best-of-N.
Scaling inference-time compute: A WER-based verifier combined with partial PRM simultaneously improves speaker similarity and word error rate as inference-time compute increases.The paper attributes poor WER under larger PRM beams to insufficient candidate diversity and reports n = 2 seconds for partial PRM.
Scaling inference-time compute: Inference-time scaling generally benefits larger models more, but gaps remain minimal for some simple tasks and metrics.The authors suggest that smaller models may sometimes use inference-time compute more efficiently than relying only on larger training-time resources.
Comparison with baselines: Llasa achieves WER comparable to state-of-the-art TTS models under direct inference, but SIM-O remains limited by single-token codec reconstruction.The limitation concerns waveform reconstruction relative to mel-based vocoders or residual-vector-quantized codecs.

4. Extending to Speech Understanding Tasks

The paper extends its single-Transformer and discrete-token framework from TTS to automatic speech recognition. By swapping speech and text token order and training only on text-token prediction, the resulting ASR model is competitive on test-clean but weaker on test-other.

ASR framework: The ASR extension swaps speech and text token positions and applies cross-entropy loss only to text tokens.It retains the X-codec2 tokenizer, using speech tokens first and text tokens second.
ASR results: On LibriSpeech test-clean, the ASR model is competitive with Whisper Large v3.The paper presents this as evidence that an entirely discrete ASR paradigm based on quantized speech tokens can produce promising results.
ASR results: Performance on LibriSpeech test-other is weaker, likely because the training set is smaller and relatively clean and lacks data augmentation.The authors describe these factors as likely explanations rather than established causes.

5. Related work

Prior work established scaling in text LLMs and explored LLM-based TTS, but speech inference scaling remained underexamined. Llasa addresses this with a simpler single-Transformer design using discrete speech tokens.

Text LLMs motivate scaling training and inference compute, including repeat sampling, self-correction, and tree search for complex tasks.
Prior Base-TTS work did not separately analyze data scaling versus model scaling, evaluate multiple languages, or systematically investigate inference scaling in speech.
Earlier TTS systems commonly combine autoregressive language models with additional stages or diffusion components, complicating training and inference.
The related evaluations include continuation zero-shot synthesis, emotion similarity, and ASR performance on LibriSpeech test sets.
Llasa uses a single Transformer with discrete speech tokens, avoiding multi-stage complexity and the memory footprint of continuous representations.

6. Conclusion

Llasa aligns TTS with text LLM architectures through a single Transformer and tokenizer while examining both train-time and inference-time compute scaling. The paper reports improvements in naturalness, prosody, text comprehension, speaker similarity, emotional expressiveness, and content accuracy, alongside public model releases.

Llasa is a scalable TTS system using a single Transformer and tokenizer aligned with text LLM architectures.
Scaling model size and datasets improves speech naturalness, prosody, and text comprehension.
Inference-time scaling with speech understanding models as verifiers enhances speaker similarity, emotional expressiveness, and content accuracy.
The authors publicly release Llasa models to support further research.

A.1. Chinese Evaluation criteria

The Chinese evaluation criteria are presented as a dedicated table for assessing the Chinese test set.

Table 6 presents the evaluation criteria for the Chinese test set.
The criteria are organized as a table associated with Chinese-language evaluation.
This appendix section documents the criteria used for the Chinese evaluation.

A.2.3. CHINESE POETRY

This appendix section contains Chinese poetry and multilingual evaluation materials, alongside copied evaluation references and unrelated example prompts. The supplied passages mainly provide the textual content rather than experimental findings.

A.2.3. CHINESE POETRY: The Chinese poetry material includes tongue twisters and densely repeated Chinese sounds, words, and phrases.
A.2.3. CHINESE POETRY: Additional Chinese passages use literary allusions, named historical or cultural figures, and annotated pronunciations.
A.2.3. CHINESE POETRY: Several examples explicitly express emotions such as excitement, fear, anger, grief, joy, guilt, confusion, affection, and determination.