GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot

• Text-based LLMs focus on semantic and logical communication but do not capture emotions and subtle vocal nuances.Source: §1 · p. 1
• Voice interaction offers a more natural and engaging medium because human communication extends beyond semantics.Source: §1 · p. 1
• Traditional spoken chatbots combine ASR, LLM processing, and TTS, a pipeline hindered by high latency.Source: §1 · p. 1
• SpeechLMs process speech input and output end to end, but available speech data is much scarcer than online text corpora.Source: §1 · pp. 1–2
• GLM-4-Voice uses 1 trillion speech-text pre-training tokens, including synthetic interleaved data, unsupervised speech, and supervised speech-text datasets.It is further fine-tuned with a streaming-thoughts template that alternates text and speech tokens for seamless, low-latency responses.Source: §1 · p. 2

• Speech language models are pretrained on unsupervised speech data, with prior work scaling speech training to 7 million hours.Source: §2.2 · p. 2
• Speech data is relatively scarce compared with text corpora, limiting how fully speech models can leverage text-LLM capabilities.Source: §2.2 · p. 2
• Speech-to-speech systems built by adding speech modules to LLMs often control content but not speaking styles and prosody.Source: §2.3 · pp. 2–3
• Instruction-only fine-tuning can produce simultaneous text and speech responses, but without speech pre-training the quality of both modalities is limited.Source: §2.3 · p. 3

3 Architecture

• GLM-4-Voice targets spoken chatbots that understand speech semantically and follow instructions for paralinguistic output.Source: §3 · p. 3

3.1 Speech Tokenizaion

• The single-codebook supervised tokenizer captures semantic information at 175bps while preserving high-quality reconstruction with minimal architectural changes.Source: §3 · p. 3
• A vector-quantized bottleneck is inserted into a pretrained Whisper encoder, whose codebook vectors are trained with EMA and low-usage vectors are reset.Source: §3.1 · pp. 3–4
• The selected 12.5Hz tokenizer preserves enough semantic information for accurate ASR, motivating its use in GLM-4-Voice.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Speech Decoder

• The decoder uses a speech-token encoder, conditional flow matching, and HiFi-GAN, trained in two stages from varied-quality speech and then high-quality single-speaker speech.Source: §3.2 · p. 4
• 175 bitrate achieves MOSNet 3.39 and WER 8.43 with the 12.5Hz decoder, providing an efficiency–quality balance.Source: §3.2 · p. 5

3.3 Inference

• Streaming inference alternates text and speech generation, using the text answer to guide spoken output while reducing the delay of waiting for complete text.Source: §3.3 · p. 5

Pre-training

• Text pre-training data are mixed in to maintain text performance while extending the language model’s speech modeling ability.The model is initialized from GLM-4-9B-Base with an expanded vocabulary containing speech tokens.Source: §4.1 · p. 6, §4.1.1 · p. 6
• 1 trillion tokens combine 30% text data with unsupervised speech, supervised speech-text, and interleaved speech-text data for multimodal pre-training.The interleaved data are synthesized from text pre-training data, while unsupervised speech comprises 700k hours and supervised data includes ASR and TTS examples.Source: §4.1.1 · p. 6, §4.1 · p. 6

Conversational fine-tuning

• Conversational fine-tuning uses quality-filtered multi-turn spoken dialogues with shortened responses and synthesized speech outputs suited to verbal interaction.The data are primarily derived from text-based conversations, with code and mathematics removed and speech-input diversity enhanced for real-world voice chat.Source: §4.2.1 · p. 7
• Each conversational turn contains speech input, text input, text output, and speech output, enabling a decoupled speech-to-speech training procedure.The streaming thoughts template is used to reduce latency.Source: §4.2.2 · p. 7
• Training splits samples between speech-to-text and text-plus-speech-to-speech objectives because text outputs are learned faster than speech outputs.Loss masking alternately focuses on text output from speech input and speech output conditioned on both speech and text.Source: §4.2.2 · p. 7

Base model evaluation

• Speech language modeling evaluates continuation selection on spoken StoryCloze and spoken Topic-StoryCloze after transforming the textual benchmark into spoken form.Spoken Topic-StoryCloze is described as easier than spoken StoryCloze.Source: §5.1 · pp. 7–8
• Spoken question answering tests closed-book factual answering on Web Questions, Llama Questions, and TriviaQA using synthesized speech contexts and continuations.The evaluation covers both speech-to-speech and speech-to-text generation settings.Source: §5.1 · pp. 7–8
• GLM-4-Voice outperforms baselines on all evaluated speech language modeling and spoken question answering tasks in both S→S and S→T settings, except Topic-StoryCloze in S→S.Compared with Moshi, it performs better on spoken question answering with both textual and spoken answers.Source: §5.1 · p. 8
• S→T accuracy is consistently higher than S→S accuracy, especially for spoken question answering, indicating that textual guidance remains necessary for intelligent speech chatbots.The method nevertheless reduces the gap between spoken and textual answers, particularly on Llama Questions.Source: §5.1 · p. 8

ASR / TTS

• GLM-4-Voice achieves ASR and TTS ability similar to Whisper-Large-V3 and CosyVoice baselines.English and Chinese recognition use Whisper-Large-V3 and Paraformer-Large text predictions, respectively.Source: §5.1 · p. 8

Chat model evaluation

• Chat quality is evaluated with GPT-4o using response-quality scores for General QA and normalized answer accuracy for a knowledge task.The judged texts are audio transcriptions produced by Whisper-Large-V3.Source: §5.2 · pp. 8–9

Conclusion

• GLM-4-Voice combines a 12.5Hz supervised speech tokenizer, a flow-matching speech decoder, and 1 trillion tokens of speech-text pre-training.The model is designed to bridge text and speech modalities for end-to-end spoken interaction.Source: §6 · p. 9
• The model achieves strong performance across speech language modeling, ASR, TTS, and spoken question answering.The conclusion reports this as a cross-task outcome rather than a single benchmark result.Source: §6 · p. 9
• High-quality conversational fine-tuning enhances fluent, low-latency, and nuanced responses.The paper presents open availability as supporting further exploration of practical spoken AI systems.Source: §6 · p. 9

• The overall task is framed as judging the quality of an AI assistant’s response to a displayed user question.Source: General QA · p. 14
• The evaluator should assess the assistant’s response for helpfulness, relevance, accuracy, depth, creativity, and level of detail.Source: General QA · p. 14
• The evaluation should begin with a short explanation and remain as objective as possible.Source: General QA · p. 14
• The final rating must use the prescribed format, such as “Rating: [[5]]”, with a score from 1 to 10.Source: General QA · p. 14
• The assistant’s answer is delimited by “[The Start of Assistant’s Answer]” and “[The End of Assistant’s Answer]”.Source: General QA · p. 14

• An answer is correct when it expresses or contains the same meaning as at least one reference answer.Source: Knowledge · p. 14
• The judgment should consider the question, reference answers, and answer being evaluated.Source: Knowledge · p. 14
• The response must be JSON with a one-sentence analysis and a judgment field.Source: Knowledge · p. 14
• The judgment field must contain either "correct" or "incorrect".Source: Knowledge · p. 14
• The input identifies separate sections for reference answers and the answer to be judged.Source: Knowledge · p. 14