Source-linked AI summary
Kimi-Audio Technical Report
KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang, Zaida Zhou
TL;DR
Kimi-Audio targets the absence of a universal, openly available model for diverse audio processing tasks. It combines hybrid audio representations, a multimodal LLM, and streaming detokenization, and reports state-of-the-art results across several audio benchmarks while releasing code, checkpoints, and evaluation tools.
Problem
Prior audio systems often focus on specific task types, rely mainly on downstream fine-tuning, or provide no source code and checkpoints, leaving universal open audio modeling limited.
Method
Kimi-Audio combines discrete semantic tokens and continuous acoustic features with a shared-and-specialized audio LLM and a chunk-wise flow-matching detokenizer.
Results
Kimi-Audio achieves state-of-the-art performance across speech recognition, audio understanding, audio-to-text chat, and speech conversation benchmarks.
Takeaways & Limitations
The released model code, checkpoints, and evaluation toolkit make the system and its cross-task evaluation resources available to the community.
Takeaways & Limitations
Evaluation still lacks benchmarks for the quality and coherence of generated audio responses.
Abstract
from arXiv · showhide
We present Kimi-Audio, an open-source audio foundation model that excels in audio understanding, generation, and conversation. We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation. Specifically, we leverage a 12.5Hz audio tokenizer, design a novel LLM-based architecture with continuous features as input and discrete tokens as output, and develop a chunk-wise streaming detokenizer based on flow matching. We curate a pre-training dataset that consists of more than 13 million hours of audio data covering a wide range of modalities including speech, sound, and music, and build a pipeline to construct high-quality and diverse post-training data. Initialized from a pre-trained LLM, Kimi-Audio is continual pre-trained on both audio and text data with several carefully designed tasks, and then fine-tuned to support a diverse of audio-related tasks. Extensive evaluation shows that Kimi-Audio achieves state-of-the-art performance on a range of audio benchmarks including speech recognition, audio understanding, audio question answering, and speech conversation. We release the codes, model checkpoints, as well as the evaluation toolkits in https://github.com/MoonshotAI/Kimi-Audio.
1 Introduction
Kimi-Audio addresses the lack of a universal, openly available audio foundation model by unifying diverse audio tasks through an LLM-based architecture, data, and training effort. Comprehensive evaluation reports state-of-the-art performance across multiple audio tasks, alongside released code, checkpoints, and evaluation tools.
- Motivation: Previous audio systems typically target separate task types, emphasize downstream fine-tuning over audio pre-training, or lack public code and checkpoints.These limitations span audio understanding, generation, and speech conversation systems.
- Contribution: Kimi-Audio is an open-source audio foundation model designed to handle a variety of audio processing tasks.The report organizes its construction around architecture, data, and training.
- Architecture: Its architecture uses discrete semantic audio tokens and continuous acoustic vectors as input, while combining audio and text tokens at output.The design aims to support both audio perception and generation capabilities.
- Training: Pre-training combines text-only and audio-only learning, audio-to-text mapping, and audio-text interleaving, followed by supervised fine-tuning for efficiency and task generalization.These tasks separately learn modality knowledge, connect audio with text, and further bridge the modalities.
- Evaluation: Kimi-Audio achieves state-of-the-art performance across speech recognition, audio understanding, audio-to-text chat, and speech conversation benchmarks.The evaluation toolkit was developed to support comprehensive and fair comparison across audio LLMs.
- Open Source: The authors release Kimi-Audio’s code, checkpoints, and evaluation toolkit to support community development and comparison.The toolkit addresses challenges involving metrics, evaluation protocols, and inference hyper-parameters.
2 Architecture
Kimi-Audio combines discrete semantic tokens, continuous acoustic features, a multimodal audio LLM, and a flow-matching detokenizer in one architecture. Its streaming design uses chunk-wise processing and look-ahead context to improve boundary quality while producing waveforms incrementally.
- Overview: Kimi-Audio uses an audio tokenizer, an audio LLM, and an audio detokenizer as its three primary architectural components.The tokenizer provides audio representations, the LLM produces text and/or audio tokens, and the detokenizer reconstructs waveforms.
- Audio Tokenizer: The hybrid tokenization strategy combines efficient discrete semantic tokens with continuous acoustic vectors to represent speech signals.Continuous Whisper features complement the semantic representation with detailed acoustic information.
- Audio Tokenizer: A vector-quantized Whisper encoder produces discrete speech tokens at 12.5Hz, while downsampled 50Hz Whisper features are added to their embeddings.The adaptor downsamples continuous features from 50Hz to 12.5Hz before they enter the audio LLM.
- Audio LLM: The audio LLM uses shared transformer layers for multimodal inputs, then parallel text and audio heads to predict corresponding outputs.Predicted audio semantic tokens are passed to the detokenizer for waveform synthesis.
- Audio LLM: The shared transformer layers and text head inherit pre-trained text-LLM weights, while the audio head is initialized randomly.This initialization retains text capabilities while the model learns audio processing and generation.
- Audio Detokenizer: The detokenizer converts 12.5Hz semantic tokens into 50Hz mel-spectrograms with flow matching, then uses a vocoder to generate waveforms.Its chunk-wise streaming framework processes audio chunks autoregressively and applies causal prompts from prior chunks.
- Audio Detokenizer: The training-free look-ahead mechanism appends future semantic tokens to each chunk during detokenization, retaining only the current chunk’s mel-spectrograms.It addresses intermittent boundary issues and delays generation of the first chunk by n tokens.
3 Data
Kimi-Audio’s data pipeline converts diverse, largely unannotated audio into multimodal training data with long-range context. Its post-training data spans audio understanding, speech conversation, and audio-to-text chat.
- Approximately 13 million hours of raw audio cover audiobooks, podcasts, interviews, acoustic events, music, environmental sounds, vocalizations, and multilingual content.
- Raw audio commonly lacks transcriptions, language labels, speaker annotations, and segmentation boundaries, while containing noise, reverberation, and speaker overlap.
- The processing pipeline generates high-quality multimodal audio-text annotations and preserves consistent long-range context in long-form audio.
- Audio annotation: Diarization post-processing merges same-speaker clusters, reassigns mixed-speaker chunks, and merges adjacent segments under duration and silence constraints.Clusters are merged above cosine similarity 0.6; adjacent chunks are reassigned below 0.5 similarity, and merging stops beyond 27 seconds or across silences over 2 seconds.
- Audio annotation: The refined diarization produces more accurate and consistently sized speaker turns than baseline diarization output.
- Post-training data: Supervised fine-tuning data covers audio understanding, speech conversation, and audio-to-text chat, including ASR, AQA, AAC, SER, SEC, and ASC tasks.The audio-understanding data includes 55,000 hours of in-house ASR data and 5,200 hours of in-house audio data for AAC/AQA.
4 Training
Kimi-Audio is continually trained to learn audio and text knowledge separately, map between modalities, and model interleaved audio-text sequences. The recipe initializes an audio LLM from Qwen2.5 7B and uses large-scale audio and text pre-training.
- Pre-training comprises unimodal audio/text learning, audio-text mapping through ASR and TTS, and audio-text interleaving tasks.The interleaving tasks predict semantic audio tokens, text tokens, or both from alternating audio-text sequences.
- The pre-training pipeline splits raw audio into segments containing audio and transcriptions, extracting continuous acoustic vectors and discrete semantic tokens for each segment.
- Unimodal pre-training: Text and audio unimodal pre-training use next-token prediction on text tokens and discrete audio semantic tokens, respectively.
- Audio-text mapping: ASR predicts text tokens from audio-text sequences, while TTS predicts audio semantic tokens from text-audio sequences.
- Audio-text interleaving: The model delays initial semantic-audio predictions with 6 blank tokens to trade off generation quality and latency.
- The audio LLM is initialized from Qwen2.5 7B and pre-trained for one epoch on 585B audio tokens and 585B text tokens.
4.2 Supervised Fine-tuning
Supervised fine-tuning equips Kimi-Audio for instruction following across diverse audio tasks using natural-language instructions and paired audio/text instruction forms. The detokenizer is trained with flow matching and chunk-wise fine-tuning.
- Natural-language instructions replace special task-switching operations, and each instruction is constructed in both audio and text versions.
- Each data source is fine-tuned for 2–4 epochs using ablation-informed training choices.
- Audio detokenizer: The audio detokenizer is pre-trained on about 1M hours of audio to learn diverse timbre, prosody, and quality.
- Audio detokenizer: Chunk-wise detokenizer fine-tuning uses dynamic chunks from 0.5 to 3 seconds, followed by fine-tuning on high-quality single-speaker recordings.
5 Inference and Deployment
Kimi-Audio’s real-time speech conversation deployment streams client audio to modular server services and returns generated audio chunks during inference. The architecture separates tokenization, LLM inference, and detokenization to support scalable, low-latency production operation.
- Real-time workflow: For each conversation round, the client streams user audio, server-side VAD detects completion, and inference begins after a commit signal.
- Real-time workflow: During inference, the client receives generated audio chunks in real time and plays them back to the user.
- Server inference: Server inference tokenizes input audio into semantic tokens and acoustic vectors, combines them with prompts and conversation history, and detokenizes generated tokens into waveform audio.
- Production deployment: The production architecture uses an RTC service, inference scheduler, tokenizer service, LLM service, and detokenizer service.
- Production deployment: Load balancers and multiple inference instances process tokenizer, detokenizer, and LLM requests in parallel for scalability.
- Production deployment: The modular deployment is intended to meet real-time speech-interaction demands while maintaining low latency and high availability.
6 Evaluation
The evaluation addresses reproducibility and metric challenges in audio foundation model benchmarking with an open-source toolkit, then compares Kimi-Audio across speech, understanding, chat, and conversation tasks. Results show strong performance across these evaluations, including leading scores on several benchmarks and expressive speech generation.
- Evaluation Toolkit: Audio foundation model evaluation is hindered by inconsistent metrics, inference configurations, and limited benchmarks for generated-audio quality.Examples include differing WER normalization, simplistic exact-match assessment for audio question answering, sensitivity to decoding and prompting parameters, and absent benchmarks for generation quality and coherence.
- Evaluation Toolkit: The open-source evaluation toolkit standardizes WER, adds GPT-4o-mini judging for audio question answering, and supports fairer comparisons across models.It also provides shared inference parameters and prompting recipes to improve reproducibility.
- Evaluation Toolkit: The toolkit includes a speech-conversation benchmark covering emotion, speed, accent, empathy, storytelling, and tongue twisters, and is released for community use.The authors position the toolkit as infrastructure for more reliable and comparable benchmarking.
- Evaluation Results: Kimi-Audio attains ASR error rates of 1.28 on LibriSpeech test-clean and 2.42 on test-other, while also reaching SOTA results on AISHELL-1 (0.60) and AISHELL-2 ios (2.56).The report also describes lowest error rates on both WenetSpeech test-meeting and test-net and robustness on an internal Kimi-ASR test set.
- Evaluation Results: Kimi-Audio leads several audio understanding and audio-to-text chat evaluations, and records a 3.90 overall speech-conversation average, above Step-Audio-chat (3.33), GPT-4o-mini (3.45), and GLM-4-Voice (3.65).It leads MMAU sound and speech categories, MELD emotion understanding, and several VoiceBench and OpenAudioBench tasks; GPT-4o remains higher at 4.06 overall.
7 Related Work
Existing audio-language systems often specialize in understanding, generation, or conversation, while Kimi-Audio unifies these capabilities in an open-source foundation model. Its hybrid representation, extensive multimodal pretraining, and instruction fine-tuning support broad audio tasks and state-of-the-art benchmark results.
- Prior systems commonly focus on understanding, generation, or conversation rather than supporting these capabilities universally.
- Kimi-Audio combines speech recognition, audio understanding, audio generation, and speech conversation within one open-source framework.
- Its hybrid input representation combines continuous acoustic features with 12.5Hz discrete semantic tokens for audio perception and efficient modeling.
- 13 million hours of curated speech, music, and environmental sound support multimodal pretraining across audio-only, text-only, audio-to-text, and interleaved tasks.
- Instruction-based fine-tuning produces state-of-the-art results on ASR, general audio understanding, audio-text chat, and speech conversation benchmarks.
- Kimi-Audio is described as fully open-source, pre-trained, instruction-followable, and real-time capable.
8 Challenges and Future Trends
The report identifies unresolved challenges for more capable audio foundation models. It emphasizes richer audio descriptions and better representations as directions beyond transcription-centered training and current token designs.
- From Audio Transcription to Audio Description: Current audio-text pretraining often derives text from speech transcription, capturing spoken content while omitting emotion, style, timbre, tone, scenes, and non-linguistic sounds.
- From Audio Transcription to Audio Description: Audio captions are proposed alongside transcriptions to describe richer context and support more nuanced multimodal audio processing.
- Better Audio Representations: Current representations use semantic or acoustic tokens, but semantic tokens obtained through ASR-oriented auxiliary losses may miss rich acoustic details.
Core Contributors
The report lists the contributors to Kimi-Audio and states that the contributor list is ordered alphabetically by last name. Project leads are marked separately.
- The contributor list includes Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, and Wei Song.
- The listed contributors also include Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, and Xinyu Zhou.
- Additional contributors are Y. Charles Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, and Yangyang Liu.
- The remaining listed contributors are Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, and Yutong Zhang.
- The contributor list is alphabetized by last name, and project leads are marked with a hash symbol.