Source-linked AI summary
Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction
Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, Jianhua Xu, Haoze Sun, Zenan Zhou, Weipeng Chen
TL;DR
Existing cascaded speech systems introduce delays, conversion errors, and loss of paralinguistic information, while audio integration can reduce reasoning capability. Baichuan-Audio addresses this with unified audio-text modeling, a multi-codebook tokenizer, an independent audio head, and two-stage pre-training, achieving strong real-time interaction, speech understanding, and question-answering performance.
Problem
Cascaded speech systems can introduce delays, accumulated conversion errors, and lost paralinguistic information, while integrating audio can reduce reasoning capabilities relative to text-only language models.
Method
Baichuan-Audio combines a 12.5 Hz multi-codebook speech tokenizer, independent audio head, alternating text-audio generation, audio decoding, and two-stage pre-training with interleaved data.
Results
Baichuan-Audio supports real-time speech interaction and outperforms previous or same-size open-source models across reported intelligence, ASR, and audio-understanding evaluations.
Takeaways & Limitations
The framework preserves semantic and acoustic speech information while retaining intelligent responses and enabling integrated speech comprehension and generation.
Abstract
from arXiv · showhide
We introduce Baichuan-Audio, an end-to-end audio large language model that seamlessly integrates audio understanding and generation. It features a text-guided aligned speech generation mechanism, enabling real-time speech interaction with both comprehension and generation capabilities. Baichuan-Audio leverages a pre-trained ASR model, followed by multi-codebook discretization of speech at a frame rate of 12.5 Hz. This multi-codebook setup ensures that speech tokens retain both semantic and acoustic information. To further enhance modeling, an independent audio head is employed to process audio tokens, effectively capturing their unique characteristics. To mitigate the loss of intelligence during pre-training and preserve the original capabilities of the LLM, we propose a two-stage pre-training strategy that maintains language understanding while enhancing audio modeling. Following alignment, the model excels in real-time speech-based conversation and exhibits outstanding question-answering capabilities, demonstrating its versatility and efficiency. The proposed model demonstrates superior performance in real-time spoken dialogue and exhibits strong question-answering abilities. Our code, model and training data are available at https://github.com/baichuan-inc/Baichuan-Audio
1 Introduction
End-to-end audio interaction models address delays, accumulated conversion errors, and lost paralinguistic information in cascaded systems, while Baichuan-Audio targets real-time speech interaction without sacrificing language capabilities.
- Cascaded ASR–LLM–TTS systems can lose paralinguistic information, introduce processing delays, and accumulate errors across conversions.
- Existing end-to-end systems differ in how they align modalities, decode speech, and preserve or model audio capabilities.
- Integrating audio modalities often reduces reasoning capabilities relative to textual large language models.
- Baichuan-Audio combines a tokenizer and stream-matching decoder to support end-to-end audio input and output for real-time speech interaction.
- Its 12.5 Hz multi-codebook tokenizer retains semantic and acoustic information, while an independent audio head captures unique audio features.
- The model processes text and audio inputs into high-quality text and speech outputs while maintaining intelligent responses.
2 Related works
Audio language-model research relies on speech tokenizers to connect continuous audio with discrete language models and increasingly focuses on end-to-end spoken interaction. Baichuan-Audio is presented within this shift toward unified audio-text generation.
- Speech tokenizers bridge audio segments and discrete language models by transforming continuous speech into discrete tokens.
- Self-supervised speech models capture semantic information, while neural acoustic codecs preserve broader audio information through discrete encoding.
- End-to-end speech interaction models have become a central focus for seamless and efficient multimodal communication.
- Moshi generates audio and text tokens simultaneously through a multi-stream output mechanism for full-duplex spoken dialogue.
- Baichuan-Audio’s overview depicts an audio LLM alternating text and audio token prediction before flow-matching decoding produces speech.
3 Baichuan-Audio
Baichuan-Audio combines an audio tokenizer, audio LLM, and decoder for end-to-end real-time speech interaction. Its tokenizer balances semantic and acoustic information, while reconstruction, alignment, and flow-matching components support speech quality and intelligibility.
- Architecture: The architecture comprises the Baichuan-Audio Tokenizer, an audio LLM, and an audio decoder that together process audio into tokens and reconstruct speech.The tokenizer captures semantic and acoustic information before the decoder converts audio representations into speech waveforms.
- Audio Tokenization: The 8-layer RVQ tokenizer operates at 12.5 tokens per second to balance semantic and acoustic information for controllable bilingual real-time conversations.It is designed to support both Chinese and English speech interaction.
- Audio Tokenization: The tokenizer extracts Whisper Encoder features, applies 4× downsampling, and uses a symmetric decoder with 4× and 2× upsampling to produce Mel-spectrogram representations.The reconstruction objective combines coarse and refined Mel targets with L1 and L2 losses.
- Training Objectives: Multi-scale Mel losses use different hop lengths and window sizes to preserve audio detail and improve reconstruction fidelity and training stability.The pretrained LLM is kept fixed for text-audio alignment, and a 1.5-billion-parameter LLM is selected for continued pretraining.
- Audio Decoding: The flow-matching decoder uses a Pre-Net and conditional U-Net decoder to generate 80-dimensional Mel spectrograms, which HiFi-GAN converts into waveforms.The decoder is trained while the AudioEncoder, VQ layers, and AudioDecoder remain fixed.
- Results: 4.05 UTMOS, up from 3.43 toward the 4.08 ground-truth score, accompanied reconstruction quality improvements while WER decreased from 2.84 to 2.78 on LibriSpeech-dev.The evaluation measures subjective audio perception with UTMOS and content quality with Whisper ASR WER.
4 Experiment
The experiments evaluate Baichuan-Audio’s speech intelligence, transcription, translation, speech synthesis, and audio understanding against text and multimodal baselines. Results show strong sentence continuation, ASR, and audio-understanding performance, while two-stage training mitigates intelligence degradation.
- General Intelligence Evaluation: Baichuan-Audio is evaluated on story continuation and commonsense reasoning to measure whether speech models preserve general intelligence.The evaluation uses sStoryCloze and a constructed sCMMLU benchmark, with text-to-text modeling as the baseline.
- General Intelligence Evaluation: Baichuan-Audio consistently outperforms previous models on sentence continuation evaluation.The results are reported in Table 5, alongside comparisons with the T→T and S→T settings.
- General Intelligence Evaluation: Two-stage training mitigates intelligence degradation relative to single-stage training, while S→T accuracy remains bounded by the T→T text-model accuracy.The model’s speech reasoning capability depends on the underlying pure-text LLM, making T→T an upper bound for the current architecture.
- ASR/TTS Evaluation: 3.2% WER on Fleurs test-zh is lower than Whisper-large-v3 at 12.4% and Qwen2-Audio-Base at 4.3%.Baichuan-Audio-Base also reports 7.2% WER on WenetSpeech test_net and 8.5% on test_meeting, with additional Chinese-dialect evaluation on KeSpeech.
- Audio Understanding Evaluation: Baichuan-Audio outperforms same-size models in AlpacaEval S→T, scoring 77.4, and surpasses GLM-4-Voice across S→S audio-understanding benchmarks.In S→S, its reported leads over GLM-4-Voice are 11.4 in Reasoning QA and 20.7 in AlpacaEval.
5 Conclusion
Baichuan-Audio integrates speech comprehension and generation in an end-to-end audio language model. Its design combines speech tokenization, an independent audio head, and two-stage pre-training to support real-time interaction while preserving language capabilities.
- Baichuan-Audio integrates speech comprehension and generation in an end-to-end audio large language model.
- Multi-codebook discretization at 12.5 Hz preserves both semantic and acoustic information in speech tokens.The speech signals are processed via a pre-trained ASR model.
- An independent audio head processes audio tokens, while interleaved two-stage pre-training balances audio modeling with preservation of language capabilities.
- Text-guided aligned speech generation supports speech interaction while retaining the model’s foundational cognitive abilities.
A.1 Open-Source Data for training
Table 9 provides an open-source dataset summary for the paper’s training data.
- Table 9 is an open-source dataset summary.
- The table concerns datasets used for training.
- The dataset information is presented in tabular form as Table 9.