Source-linked AI summary
GLM-TTS Technical Report
Jiayan Cui, Zhihan Yang, Naihan Li, Jiankun Tian, Xingyu Ma, Yi Zhang, Guangyu Chen, Runxuan Yang, Yuqing Cheng, Yizhi Zhou, Guochen Yu, Xiaotao Gu, Jie Tang
TL;DR
Production TTS systems still face challenges in voice cloning, expressive prosody, pronunciation precision, and customization cost. GLM-TTS combines autoregressive text-to-token and diffusion token-to-waveform generation with reward alignment and pronunciation and voice controls; with 100k training hours, it reports state-of-the-art open-source benchmark performance.
Problem
Production TTS still faces limitations in low-resource voice cloning, nuanced emotion, pronunciation precision, reinforcement-learning stability, and scalable voice customization.
Source: §1 · p. 2Method
GLM-TTS uses a two-stage text-to-token autoregressive and token-to-waveform diffusion system, with GRPO rewards, LoRA voice customization, and hybrid phoneme-text input.
Source: §1 · pp. 2–3Results
With 100k training hours, GLM-TTS reports state-of-the-art open-source benchmark performance and improvements in pronunciation accuracy, emotional expressiveness, and speaker similarity.
Source: §1 · p. 2, §4 · p. 11Takeaways & Limitations
The system offers a practical framework for efficient, controllable speech synthesis, including scalable personalization and pronunciation control for polyphonic and rare words.
Source: §4 · p. 11English training data is less than half the Chinese training data; on test-en, GLM-TTS reports WER of 2.23% and SIM of 67.2 and notes potential for improvement with more English data.
Source: §3.2 · p. 8
Abstract
from arXiv · showhide
This work proposes GLM-TTS, a production-level TTS system designed for efficiency, controllability, and high-fidelity speech generation. GLM-TTS follows a two-stage architecture, consisting of a text-to-token autoregressive model and a token-to-waveform diffusion model. With only 100k hours of training data, GLM-TTS achieves state-of-the-art performance on multiple open-source benchmarks. To meet production requirements, GLM-TTS improves speech quality through an optimized speech tokenizer with fundamental frequency constraints and a GRPO-based multi-reward reinforcement learning framework that jointly optimizes pronunciation, speaker similarity, and expressive prosody. In parallel, the system enables efficient and controllable deployment via parameter-efficient LoRA-based voice customization and a hybrid phoneme-text input scheme that provides precise pronunciation control. Our code is available at https://github.com/zai-org/GLM-TTS. Real-time speech synthesis demos are provided via Z.ai (audio.z.ai), the Zhipu Qingyan app/web (chatglm.cn).
1 INTRODUCTION
GLM-TTS targets production TTS challenges with a two-stage system that reports state-of-the-art open-source benchmark performance using 100k training hours. Its contributions combine speech-tokenizer and reinforcement-learning improvements with lower-cost voice customization and explicit pronunciation control.
1 INTRODUCTION
- Production TTS systems still face constraints in low-resource voice cloning, nuanced emotion, pronunciation precision, reinforcement-learning stability, and affordable voice customization.The passage highlights polyphonic characters, rare words, and dialects as pronunciation challenges.Source: §1 · p. 2
- With 100k training hours, GLM-TTS reports state-of-the-art performance on open-source benchmarks, compared with systems trained on 1M and 1.1M hours.The comparison names CosyVoice 3 (1M hours) and FireRedTTS-2 (1.1M hours).Source: §1 · p. 2
- On Seed-TTS-eval zh, the optimized speech tokenizer achieves SIM = 76.1 and CER = 1.03%.The tokenizer uses fundamental-frequency constraints and an expanded 32k vocabulary.Source: §1 · p. 2
- The GRPO framework combines CER, SIM, Emotion, and Laughter rewards and reports gains in nuanced emotion expression, WER, and SIM across two benchmarks.The reported comparisons include commercial models on CV3-eval-emotion and Seed-TTS-eval.Source: §1 · p. 2
- LoRA voice customization adjusts 15% of parameters, uses 1 hour of single-speaker audio, and reduces training costs by 80% versus full fine-tuning.The report describes its performance as matching full-model-level performance.Source: §1 · p. 2
- Hybrid phoneme-text input targets polyphone and rare-word ambiguity, while Vocos2D uses 2D operations and DiT-style residual connections for waveform reconstruction.The hybrid scheme is reported to preserve prosodic naturalness; Vocos2D is described as improving frequency-subband modeling.Source: §1 · p. 3
2 METHODOLOGY
The methodology combines data preparation and tokenization with GRPO alignment, parameter-efficient voice customization, phoneme-level pronunciation control, and a redesigned vocoder.
2.1 DATA PROCESSING PIPELINE
- The data pipeline standardizes and segments audio, separates and denoises speech, diarizes speakers, filters transcripts by WER, optimizes punctuation, and extracts training features.For the WER filter, audio is retained only when WER is below 5%.Source: §2.1 · pp. 3–4
2.2 TEXT TOKENIZER
- Vocabulary pruning removes text tokens longer than two Chinese characters to reduce extreme text-to-acoustic alignments and support autoregressive generation at 25 Hz.The method describes hard speech-to-text length-ratio constraints alone as insufficient for optimal convergence.Source: §2.2 · p. 4
2.3 SPEECH TOKENIZER
- The Whisper-VQ tokenizer doubles its rate from 12.5 Hz to 25 Hz, expands its vocabulary from 16k to 32k, adds pitch estimation, and removes causal constraints.Training data also expands to include dialect and singing-voice datasets.Source: §2.3 · p. 4
2.4 SPEECHLM RL-ALIGNMENT
- GRPO combines CER, SIM, Emotion, and Laughter rewards with hierarchical fusion, dynamic resampling, and adaptive gradient clipping to address reward imbalance and training instability.Resampling can be triggered up to three times when batch rewards become homogeneous; clipping tightens early and relaxes later.Source: §2.4 · pp. 4–5
2.5 LORA FOR PREMIUM VOICE CUSTOMIZATION
- LoRA fine-tuning adjusts about 15% of backbone parameters for voice similarity and naturalness comparable to full fine-tuning, using about 1 hour of high-quality single-speaker audio.The report also says controlling the tuned-parameter ratio, for example above 15%, supports generalization and stability across scenarios.Source: §2.5 · pp. 5–6
2.6 PHONEME-IN
- Phoneme-in maintains vocabularies for polyphones and rare characters, trains mixed phoneme-text inputs, and replaces targeted characters with phonemes at inference while preserving prosody.Standard characters may be probabilistically replaced during training; vocabulary-targeted characters remain text to preserve semantic context.Source: §2.6 · p. 6
2.7 VOCOS2D
- Vocos2D replaces frame-wide 1D convolutions with 2D frequency-subband processing, using per-frequency embeddings and spectrogram shortcuts; training removes MPD and adds discriminator augmentation.The dataset includes high-quality singing voice data to expand pitch-range coverage.Source: §2.7 · pp. 6–7
3 EXPERIMENTS AND RESULTS
GLM-TTS improves on tokenizer and voice-cloning evaluations, while RL alignment experiments expose trade-offs among pronunciation, similarity, and emotional expressiveness. Phoneme input reduces hard-case pronunciation errors, and Vocos2D outperforms Vocos across the reported metrics.
3.1 SPEECH TOKENIZER EVALUATION
- The GLM-TTS tokenizer significantly outperforms GLM-4-Voice on Chinese recognition and shows minor improvements on English recognition.Table 1 compares the tokenizers on ASR tasks using WER/CER, where lower values are better.Source: §3.1 · pp. 7–8, Table 1 · p. 8
- GLM-TTS built on its tokenizer outperforms the GLM4-Voice-tokenizer variant on speaker similarity and character error rate.The comparison is on seed_test_zh.Source: §3.1 · p. 8, Table 2 · p. 8
3.2 VOICE CLONING RESULTS ON SEED-TTS-EVAL
- After multi-reward GRPO, GLM-TTS_RL reaches CER=0.89% and SIM=76.4 on test-zh, narrowing the gap with leading closed-source systems.GLM-TTS without RL reports CER of 1.03% and SIM of 76.1; MiniMax-Speech reaches CER of 0.83%, while Seed-TTS reaches SIM of 79.6.Source: §3.2 · p. 8, Table 3 · p. 9
- On test-en, GLM-TTS reports WER of 2.23% and SIM of 67.2 despite using less than half as much English as Chinese training data.The report suggests additional English data could further improve performance.Source: §3.2 · p. 8
3.3 SPEECH-LM RL-ALIGNMENT
- Clip-Higher yields positive gains, while Dynamic Sampling improves SIM and CER but lowers EMO on the internal emotion test set.The authors attribute the EMO decline to its more extreme, near-bimodal distribution and stop including SIM variance in resampling.Source: §3.3 · p. 8, Table 4 · p. 9
- More aggressive dynamic ϵh, ϵl, and T settings enhance emotional expressiveness but reduce pronunciation clarity and risk reward hacking.The experiments begin with values 0.3, 0.2, and 1, respectively, and increase them linearly during training.Source: §3.3 · p. 9, Table 5 · p. 10
- Increasing the laughter reward decreases CER and similarity scores, while enhancing laughter synthesis can improve emotional expressiveness.The laughter reward gives credit when laughter is detected and ASR transcribes the segment as a deletion; it gives none when ASR transcribes the corresponding text.Source: §3.3 · pp. 9–10, Table 6 · p. 10
3.4 EFFECTIVENESS OF PHONEME-IN
- On the internal hard-case dataset, phoneme input lowers PER from 13.23% with text-only input to 5.14%.The set emphasizes polyphonic characters, rare words, and ambiguous contexts.Source: §3.4 · p. 10, Table 7 · p. 10
4 CONCLUSION
GLM-TTS combines competitive speech quality with methods aimed at production deployability and controllable generation. Its conclusion highlights multi-reward GRPO, LoRA customization, and hybrid phoneme-text input as central components.
4 CONCLUSION
- With 100k training hours, GLM-TTS achieves competitive pronunciation accuracy and speaker similarity, while multi-reward GRPO improves pronunciation, emotional expressiveness, and speaker similarity without sacrificing training stability.The report presents GRPO as aligning synthesized speech with human perceptual preferences.Source: §4 · p. 11
- LoRA-based premium voice customization reduces data and computational costs to support scalable personalization in production environments.The report describes the strategy as optimized for practical deployability.Source: §4 · p. 11
- Hybrid phoneme-text input provides precise, controllable pronunciation for polyphonic and rare words, especially in Chinese professional and educational TTS applications.The report frames this as addressing a requirement in those applications.Source: §4 · p. 11