Source-linked AI summary
GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, Jie Tang
TL;DR
Spoken chatbots need both semantic competence and expressive, low-latency speech, but prior systems are constrained by speech-data scarcity and limited prosodic control. GLM-4-Voice combines a low-bitrate tokenizer, streaming decoder, and 1-trillion-token speech-text pre-training, then fine-tunes conversational speech generation. It achieves strong performance across speech modeling, spoken question answering, ASR, TTS, and conversational interaction.
Problem
Existing spoken chatbots face high-latency pipelines, scarce speech data, and limited ability to generate expressive speech styles and prosody.
Method
GLM-4-Voice uses a 12.5Hz supervised tokenizer, flow-matching decoder, speech-text pre-training on 1 trillion tokens, and streaming-thoughts conversational fine-tuning.
Results
GLM-4-Voice outperforms baselines on evaluated speech language modeling and spoken question-answering tasks except Topic-StoryCloze in S→S, while achieving similar ASR and TTS ability to Whisper-Large-V3 and CosyVoice.
Takeaways & Limitations
The model supports end-to-end spoken interaction with fluent, low-latency, nuanced responses and strong performance across speech language modeling, ASR, TTS, and spoken question answering.
Abstract
from arXiv · showhide
We introduce GLM-4-Voice, an intelligent and human-like end-to-end spoken chatbot. It supports both Chinese and English, engages in real-time voice conversations, and varies vocal nuances such as emotion, intonation, speech rate, and dialect according to user instructions. GLM-4-Voice uses an ultra-low bitrate (175bps), single-codebook speech tokenizer with 12.5Hz frame rate derived from an automatic speech recognition (ASR) model by incorporating a vector-quantized bottleneck into the encoder. To efficiently transfer knowledge from text to speech modalities, we synthesize speech-text interleaved data from existing text pre-training corpora using a text-to-token model. We continue pre-training from the pre-trained text language model GLM-4-9B with a combination of unsupervised speech data, interleaved speech-text data, and supervised speech-text data, scaling up to 1 trillion tokens, achieving state-of-the-art performance in both speech language modeling and spoken question answering. We then fine-tune the pre-trained model with high-quality conversational speech data, achieving superior performance compared to existing baselines in both conversational ability and speech quality. The open models can be accessed through https://github.com/THUDM/GLM-4-Voice and https://huggingface.co/THUDM/glm-4-voice-9b.
1 Introduction
Existing text LLMs miss emotional and expressive aspects of human communication, while conventional spoken chatbots incur high latency from ASR–LLM–TTS pipelines. GLM-4-Voice addresses these limitations with end-to-end speech modeling and large-scale speech-text pre-training.
- Text-based LLMs focus on semantic and logical communication but do not capture emotions and subtle vocal nuances.
- Voice interaction offers a more natural and engaging medium because human communication extends beyond semantics.
- Traditional spoken chatbots combine ASR, LLM processing, and TTS, a pipeline hindered by high latency.
- SpeechLMs process speech input and output end to end, but available speech data is much scarcer than online text corpora.
- GLM-4-Voice uses 1 trillion speech-text pre-training tokens, including synthetic interleaved data, unsupervised speech, and supervised speech-text datasets.It is further fine-tuned with a streaming-thoughts template that alternates text and speech tokens for seamless, low-latency responses.
2 Related Work
Related work develops speech language models and speech-based chatbots by scaling speech pre-training or adapting text LLMs. These approaches face limited speech data, incomplete speech generation, or insufficient control over expressive prosody.
- Speech language models are pretrained on unsupervised speech data, with prior work scaling speech training to 7 million hours.
- Speech data is relatively scarce compared with text corpora, limiting how fully speech models can leverage text-LLM capabilities.
- Speech-to-speech systems built by adding speech modules to LLMs often control content but not speaking styles and prosody.
- Instruction-only fine-tuning can produce simultaneous text and speech responses, but without speech pre-training the quality of both modalities is limited.
3 Architecture
GLM-4-Voice combines a low-bitrate supervised speech tokenizer, a streaming speech decoder, large-scale speech-text training, and decoupled inference. The architecture targets semantic accuracy, expressive speech, and low-latency interaction.
- 3 Architecture: GLM-4-Voice targets spoken chatbots that understand speech semantically and follow instructions for paralinguistic output.
- 3.1 Speech Tokenizaion: The single-codebook supervised tokenizer captures semantic information at 175bps while preserving high-quality reconstruction with minimal architectural changes.
- 3.1 Speech Tokenizaion: A vector-quantized bottleneck is inserted into a pretrained Whisper encoder, whose codebook vectors are trained with EMA and low-usage vectors are reset.
- 3.1 Speech Tokenizaion: The selected 12.5Hz tokenizer preserves enough semantic information for accurate ASR, motivating its use in GLM-4-Voice.
- 3.2 Speech Decoder: The decoder uses a speech-token encoder, conditional flow matching, and HiFi-GAN, trained in two stages from varied-quality speech and then high-quality single-speaker speech.
- 3.2 Speech Decoder: 175 bitrate achieves MOSNet 3.39 and WER 8.43 with the 12.5Hz decoder, providing an efficiency–quality balance.
- 3.3 Inference: Streaming inference alternates text and speech generation, using the text answer to guide spoken output while reducing the delay of waiting for complete text.
4 Training Procedure
GLM-4-Voice is trained in two stages: large-scale multimodal pre-training followed by conversational speech fine-tuning. The procedure combines synthetic and real speech data while separating text and speech-output learning to address their different learning speeds.
- Pre-training: Text pre-training data are mixed in to maintain text performance while extending the language model’s speech modeling ability.The model is initialized from GLM-4-9B-Base with an expanded vocabulary containing speech tokens.
- Pre-training: 1 trillion tokens combine 30% text data with unsupervised speech, supervised speech-text, and interleaved speech-text data for multimodal pre-training.The interleaved data are synthesized from text pre-training data, while unsupervised speech comprises 700k hours and supervised data includes ASR and TTS examples.
- Conversational fine-tuning: Conversational fine-tuning uses quality-filtered multi-turn spoken dialogues with shortened responses and synthesized speech outputs suited to verbal interaction.The data are primarily derived from text-based conversations, with code and mathematics removed and speech-input diversity enhanced for real-world voice chat.
- Conversational fine-tuning: Each conversational turn contains speech input, text input, text output, and speech output, enabling a decoupled speech-to-speech training procedure.The streaming thoughts template is used to reduce latency.
- Conversational fine-tuning: Training splits samples between speech-to-text and text-plus-speech-to-speech objectives because text outputs are learned faster than speech outputs.Loss masking alternately focuses on text output from speech input and speech output conditioned on both speech and text.
5 Evaluation
The evaluation tests GLM-4-Voice on speech language modeling, spoken question answering, ASR, TTS, and conversational quality. The base model generally outperforms baselines, while textual guidance remains advantageous for question answering.
- Base model evaluation: Speech language modeling evaluates continuation selection on spoken StoryCloze and spoken Topic-StoryCloze after transforming the textual benchmark into spoken form.Spoken Topic-StoryCloze is described as easier than spoken StoryCloze.
- Base model evaluation: Spoken question answering tests closed-book factual answering on Web Questions, Llama Questions, and TriviaQA using synthesized speech contexts and continuations.The evaluation covers both speech-to-speech and speech-to-text generation settings.
- Base model evaluation: GLM-4-Voice outperforms baselines on all evaluated speech language modeling and spoken question answering tasks in both S→S and S→T settings, except Topic-StoryCloze in S→S.Compared with Moshi, it performs better on spoken question answering with both textual and spoken answers.
- Base model evaluation: S→T accuracy is consistently higher than S→S accuracy, especially for spoken question answering, indicating that textual guidance remains necessary for intelligent speech chatbots.The method nevertheless reduces the gap between spoken and textual answers, particularly on Llama Questions.
- ASR / TTS: GLM-4-Voice achieves ASR and TTS ability similar to Whisper-Large-V3 and CosyVoice baselines.English and Chinese recognition use Whisper-Large-V3 and Paraformer-Large text predictions, respectively.
- Chat model evaluation: Chat quality is evaluated with GPT-4o using response-quality scores for General QA and normalized answer accuracy for a knowledge task.The judged texts are audio transcriptions produced by Whisper-Large-V3.
6 Conclusion
GLM-4-Voice is presented as an end-to-end spoken chatbot for natural, expressive voice interaction. Its architecture and large-scale speech-text pre-training bridge speech and text, while conversational fine-tuning improves fluent, low-latency, nuanced responses.
- Conclusion: GLM-4-Voice combines a 12.5Hz supervised speech tokenizer, a flow-matching speech decoder, and 1 trillion tokens of speech-text pre-training.The model is designed to bridge text and speech modalities for end-to-end spoken interaction.
- Conclusion: The model achieves strong performance across speech language modeling, ASR, TTS, and spoken question answering.The conclusion reports this as a cross-task outcome rather than a single benchmark result.
- Conclusion: High-quality conversational fine-tuning enhances fluent, low-latency, and nuanced responses.The paper presents open availability as supporting further exploration of practical spoken AI systems.
General QA
The passage instructs an impartial evaluation of an AI assistant’s response, emphasizing helpfulness, relevance, accuracy, depth, creativity, and detail. It also specifies a required rating format on a 1-to-10 scale.
- The overall task is framed as judging the quality of an AI assistant’s response to a displayed user question.
- The evaluator should assess the assistant’s response for helpfulness, relevance, accuracy, depth, creativity, and level of detail.
- The evaluation should begin with a short explanation and remain as objective as possible.
- The final rating must use the prescribed format, such as “Rating: [[5]]”, with a score from 1 to 10.
- The assistant’s answer is delimited by “[The Start of Assistant’s Answer]” and “[The End of Assistant’s Answer]”.
Knowledge
The task is to judge whether an answer expresses the same meaning as at least one reference answer, then return the judgment in a specified JSON format.
- An answer is correct when it expresses or contains the same meaning as at least one reference answer.
- The judgment should consider the question, reference answers, and answer being evaluated.
- The response must be JSON with a one-sentence analysis and a judgment field.
- The judgment field must contain either "correct" or "incorrect".
- The input identifies separate sections for reference answers and the answer to be judged.