PaperlayerView exact revision on arXiv

Source-linked AI summary

Kimi-Audio Technical Report

KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang, Zaida Zhou

arXiv:2504.18425v1eess.AScs.AIcs.CLcs.LGcs.MMcs.SD

TL;DR

Audio modeling has often treated tasks separately, and prior audio LLMs fell short of a universal model. Kimi-Audio addresses this gap with a broad audio foundation model that achieves state-of-the-art performance across several audio tasks.

  • Problem

    Prior audio LLMs often targeted individual task types, lacked audio pre-training, or withheld code and checkpoints, leaving a need for an open, broadly capable audio foundation model.

    Source: §1 · p. 2

  • Method

    Kimi-Audio combines architecture, data, and training work in an audio LLM that produces audio semantic tokens and corresponding text tokens.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    Kimi-Audio achieves state-of-the-art performance across speech recognition, audio understanding, audio-to-text chat, and speech conversation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors release Kimi-Audio's code, checkpoints, and evaluation toolkit to boost community development.

    Source: §1 · p. 3

    Evaluation of generated audio response quality and coherence remains limited by a lack of benchmarks.

    Source: §6.1 · p. 15

Abstract

from arXiv · show

We present Kimi-Audio, an open-source audio foundation model that excels in audio understanding, generation, and conversation. We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation. Specifically, we leverage a 12.5Hz audio tokenizer, design a novel LLM-based architecture with continuous features as input and discrete tokens as output, and develop a chunk-wise streaming detokenizer based on flow matching. We curate a pre-training dataset that consists of more than 13 million hours of audio data covering a wide range of modalities including speech, sound, and music, and build a pipeline to construct high-quality and diverse post-training data. Initialized from a pre-trained LLM, Kimi-Audio is continual pre-trained on both audio and text data with several carefully designed tasks, and then fine-tuned to support a diverse of audio-related tasks. Extensive evaluation shows that Kimi-Audio achieves state-of-the-art performance on a range of audio benchmarks including speech recognition, audio understanding, audio question answering, and speech conversation. We release the codes, model checkpoints, as well as the evaluation toolkits in https://github.com/MoonshotAI/Kimi-Audio.

1 Introduction

Kimi-Audio is presented as an open-source audio foundation model for a range of audio-processing tasks, addressing limitations of task-specific systems and building on the suitability of audio for LLM-based modeling.

Motivation and evaluation

  • The report identifies prior systems as limited by task-specific scope, little audio pre-training, or unavailable code and checkpoints, and introduces an evaluation toolkit for comprehensive, fair benchmarking.The report also describes deployment and serving in Kimi APP.Source: §1 · pp. 2–3

Architecture

  • The architecture combines an audio tokenizer and detokenizer with an audio LLM, using 12.5Hz audio tokens and continuous acoustic input features alongside discrete semantic tokens.Discrete text tokens are concatenated at output to enhance generation capability.Source: §1 · p. 2

Data

  • The data strategy pairs large-scale audio pre-training with quality-processing pipelines and task-specific SFT data constructed from open, accessible sources without data purchases.The pre-training pipeline includes speech enhancement, diarization, transcription, and filtering.Source: §1 · p. 2

Training

  • Training initializes the audio LLM from a pre-trained LLM, then uses text-only/audio-only, audio-to-text, and audio-text interleaving pre-training tasks before SFT designed for efficiency and generalization.The tasks aim to learn from text and audio domains and bridge the gap between them.Source: §1 · pp. 2–3

Evaluation and release

  • Kimi-Audio achieves SOTA performance in speech recognition, audio understanding, audio-to-text chat, and speech conversation, and its code, checkpoints, and evaluation toolkit are open-sourced.The toolkit is intended to support fair comparison across audio LLMs.Source: §1 · p. 3

2 Architecture

Kimi-Audio combines hybrid audio tokenization, a shared-and-specialized audio LLM, and flow-matching detokenization in one architecture for audio understanding, generation, and conversation.

2.1 Overview

  • The unified model links a 12.5Hz semantic-token and acoustic-feature tokenizer, an audio LLM that generates text and audio tokens, and a waveform detokenizer.Figure 2 depicts these three components as the system architecture.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 Audio Tokenizer

  • Discrete semantic tokens are combined with downsampled continuous Whisper features, joining semantically focused representation with acoustic detail.A vector-quantization layer produces tokens at 12.5Hz; Whisper features are downsampled from 50Hz to 12.5Hz and added to token embeddings. The semantic tokens use the supervised speech-tokenizer approach proposed by GLM-4-Voice [84], derived from an ASR model.Source: §2.2 · pp. 3–4

2.3 Audio LLM

  • The audio LLM processes multimodal inputs through shared transformer layers, then uses separate autoregressive text and audio heads to generate outputs.Shared transformer layers and the text head are initialized from a pretrained text LLM, while the audio head is initialized randomly.Source: §2.3 · pp. 4–5

2.4 Audio Detokenizer

  • The streaming detokenizer uses chunk-wise autoregressive flow matching and look-ahead tokens to address intermittent audio quality at chunk boundaries.It converts 12.5Hz semantic tokens to 50Hz mel-spectrograms before vocoding; for each chunk, it appends future tokens from the next chunk, generates audio, and retains only the current chunk’s mel-spectrograms. The training-free mechanism delays the first chunk by n tokens.Source: §2.4 · p. 5

3 Data

Section 3 describes a broad audio and text pre-training corpus, a pipeline for creating long-form contextual audio annotations, and diverse SFT data for audio understanding, speech conversation, and audio-to-text chat.

3.1 Pre-Training Data

  • The pre-training corpus combines text-only, audio-only, and text-audio data with approximately 13 million hours of raw audio spanning speech, music, environmental sounds, and multilingual information.Audio sources include audiobooks, podcasts, and interviews.Source: §3.1 · p. 6
  • The annotation pipeline generates multimodal audio-text data with long-form annotations and consistent long-range context, unlike pipelines focused on short segments.It addresses missing transcripts, language and speaker labels, segmentation boundaries, and audio artifacts.Source: §3.1 · p. 6
  • Refined diarization produces more accurate, consistently sized speaker turns than baseline diarization output.The process merges speaker clusters above 0.6 cosine similarity, reassigns 1.5-second chunks when adjacent-chunk similarity is below 0.5, and merges same-speaker segments until length exceeds 27 seconds or silence exceeds 2 seconds. Speech enhancement is applied to a random half of examples; transcription retains English and Mandarin, using Whisper-large-v3 and Paraformer-Zh respectively.Source: §3.1 · pp. 6–7

3.2 SFT Data

  • SFT data targets instruction following and audio processing through audio understanding, speech conversation, and audio-to-text chat.The audio understanding collection covers ASR, AQA, AAC, SER, SEC, and ASC, and includes 55,000 hours of in-house ASR data plus 5,200 hours for AAC/AQA.Source: §3.2 · p. 7, §3.2.1 · p. 8

3.2.2 Speech Conversation

  • Speech-conversation data comprises multi-turn queries synthesized from text using prompts randomly selected from more than 125K timbres, plus assistant responses synthesized in one Kimi-Audio speaker’s timbre with appropriate styles and emotions.The speaker recording defines over 20 styles and emotions, with each emotion divided into five intensity levels. Kimi-TTS generates speech with only a 3-second prompt, while Kimi-VC converts diverse speech into the speaker’s timbre while preserving styles, emotions, and accents.Source: §3.2.2 · p. 9

3.2.3 Audio-to-Text Chat

  • Audio-to-text chat data converts user queries from open-source text-domain SFT data into speech with varied timbres while retaining text assistant responses.Preprocessing removes difficult or lengthy text, rewrites it colloquially, and converts complex single-turn question-answer examples into simpler multi-turn instructions.Source: §3.2.3 · pp. 9–10

4 Training · 4.1 Pre-training

Kimi-Audio pre-training combines unimodal learning, audio-text mapping, and interleaving tasks to learn audio and text knowledge and align the modalities. Its training recipe initializes from a pre-trained LLM and trains on both audio and text tokens.

4.1 Pre-training

  • Audio is segmented and represented with continuous acoustic features and discrete semantic tokens alongside corresponding text, forming the basis for the pre-training tasks.The model pads the shorter audio or text sequence so both have the same length.Source: §4.1 · p. 10

4.1.1 Audio/Text Unimodal Pre-training

  • Text uses next-token prediction on MoonLight [44] data, while audio uses next-token prediction on discrete semantic audio tokens.Source: §4.1.1 · p. 11

4.1.2 Audio-Text Mapping Pre-training

  • ASR maps audio to text and trains loss on text tokens, while TTS maps text to semantic audio tokens and trains loss on those audio tokens.Source: §4.1.2 · p. 11

4.1.3 Audio-Text Interleaving Pre-training

  • Audio-to-semantic-token interleaving alternates audio segments with semantic audio tokens and calculates loss only on the semantic tokens.Source: §4.1.3 · p. 11
  • Audio-to-text interleaving alternates audio segments with text segments and calculates loss only on the text tokens.Source: §4.1.3 · p. 11
  • Combined audio-semantic-token-and-text interleaving delays the first several semantic-token predictions with six blank tokens to ease concurrent prediction and balance generation quality against latency.Source: §4.1.3 · p. 11

4.1.4 Pre-training Recipe

  • Pre-training starts from Qwen2.5 7B [76], uses 585B audio tokens and 585B text tokens for one epoch, and freezes the Whisper large-v3 [58] acoustic feature extractor for about 20% of tokens before joint fine-tuning.The task weights are 1 : 7 : 1 : 1 : 1 : 1 : 2; AdamW [48] uses cosine-decayed learning rates from 2e−5 to 2e−6 with 1% warmup.Source: §4.1.4 · p. 12

4.2 Supervised Fine-tuning · 4.3 Training of Audio Detokenizer

Supervised fine-tuning uses varied natural-language instructions and about 300K hours of data, while the audio detokenizer is trained in three stages, ending with high-quality single-speaker recordings.

4.2.1 Formulation

  • Supervised fine-tuning uses natural-language task instructions without special task-switching operations to support diverse downstream tasks.Source: §4.2.1 · p. 12
  • Each training sample randomly receives either a text instruction or an audio version generated zero-shot by Kimi-TTS from the text.Source: §4.2.1 · p. 12
  • For robust instruction-following, training randomly selects among 200 ASR instructions or 30 instructions for other tasks, using about 300K hours of supervised fine-tuning data.Source: §4.2.1 · p. 12

4.2.2 Fine-tuning Recipe

  • Each data source is fine-tuned for 2–4 epochs based on comprehensive ablation experiments.Source: §4.2.2 · p. 12
  • The fine-tuning recipe uses AdamW [48], cosine-decayed learning rates from 1e−5 to 1e−6, and 10% of tokens for warmup.Source: §4.2.2 · p. 12

4.3 Training of Audio Detokenizer

  • The detokenizer’s first stage pre-trains its flow-matching model and vocoder on about 1M hours of audio to learn diverse timbre, prosody, and quality.Source: §4.3 · p. 12
  • The second stage applies chunk-wise fine-tuning with dynamic chunk sizes from 0.5 to 3 seconds on the same pre-training data [32].Source: §4.3 · p. 12
  • The final stage fine-tunes the detokenizer on high-quality single-speaker recording data from the Kimi-Audio speaker.Source: §4.3 · p. 12

5 Inference and Deployment

Kimi-Audio illustrates deployment through real-time speech conversation, whose infrastructure and engineering demands exceed those of other audio tasks. Its workflow and modular production architecture connect client streaming to model inference and scalable audio generation.

5.1 Workflow of Real-Time Speech Conversation

  • The client streams user audio to the server, where VAD detects the end of speech and triggers inference; generated audio chunks stream back for playback.The workflow is illustrated in Figure 4.Source: Figure 4 · p. 13, §5.1 · pp. 13–14
  • Server inference tokenizes input into semantic tokens and acoustic vectors, combines them with prompts and conversation history, generates output tokens, then detokenizes them into audio.Source: §5.1 · p. 14

5.2 Production Deployment

  • The RTC service uses WebRTC for stable, low-latency client connections, forwarding incoming audio to the scheduler and returning generated audio chunks.Source: §5.2 · p. 14
  • The Inference Scheduler maintains conversation history and orchestrates tokenization, input construction, LLM generation, and detokenization for each interaction round.It stores output tokens in the ongoing history to maintain dialogue continuity.Source: §5.2 · p. 14
  • Load-balanced, parallel Tokenizer, Detokenizer, and LLM instances support a modular architecture designed to scale real-time speech interactions while maintaining low latency and high availability.The core components are computationally intensive and require scalable, efficient infrastructure.Source: §5.2 · pp. 14–15

6 Evaluation

The evaluation combines an open-source toolkit designed to improve the fairness and reproducibility of audio-model comparisons with assessments of Kimi-Audio across speech recognition, audio understanding, audio-to-text chat, and speech conversation.

6.1 Evaluation Toolkit

  • The toolkit addresses inconsistent metrics, sensitivity to inference settings, and missing generation benchmarks with standardized WER, GPT-4o-mini judging, and shared inference recipes.It provides a unified platform for comparing models and versions.Source: §6.1 · p. 15
  • The open-source toolkit supports evaluation of understanding, generation, and conversation, and includes a released speech-conversation benchmark covering control, empathy, and varied styles.The authors encourage community use and contributions to models, datasets, and standardized evaluation protocols.Source: §6.1 · pp. 15–16

6.2.1 Automatic Speech Recognition

  • Kimi-Audio reports LibriSpeech WERs of 1.28 on test-clean and 2.42 on test-other, and Mandarin ASR WERs of 0.60 on AISHELL-1 and 2.56 on AISHELL-2 ios.The report also says it achieves the lowest WERs on WenetSpeech test-meeting and test-net and confirms robustness on an internal Kimi-ASR test set.Source: Table 4 · p. 16, §6.2.1 · p. 18

6.2.2 Audio Understanding

  • Kimi-Audio scores 73.27 on MMAU sound category, 60.66 on MMAU speech category, and 59.13 on MELD speech emotion understanding.The report also says it leads on VocalSound and Nonspeech7k sound classification and TUT2017 and CochlScene acoustic scene classification.Source: Table 5 · p. 17, §6.2.2 · p. 18

6.2.3 Audio-to-Text Chat

  • On OpenAudioBench, Kimi-Audio achieves state-of-the-art results on AlpacaEval, Llama Questions, and TriviaQA, with competitive results on Reasoning QA and Web Questions.On VoiceBench, it outperforms compared models on AlpacaEval (4.46), CommonEval (3.97), SD-QA (63.12), MMSU (62.17), OpenBookQA (83.52), Advbench (100.00), and IFEval (61.10).Source: Table 6 · p. 18, §6.2.3 · p. 19

6.2.4 Speech Conversation

  • Kimi-Audio’s speech-conversation average is 3.90, exceeding Step-Audio-chat (3.33), GPT-4o-mini (3.45), and GLM-4-Voice (3.65), but below GPT-4o (4.06).Excluding GPT-4o, it scores highest on emotion, empathy, and speed control, while GLM-4-Voice is slightly better on accent control.Source: §6.2.4 · p. 19, Table 7 · p. 19

7 Related Work

Prior audio models often specialize in understanding, generation, or speech dialogue, while recent universal models still face limits in scope, pretraining, or accessibility. Kimi-Audio brings these capabilities together in an open-source model with broad audio pretraining and instruction-based fine-tuning.

ASR and Audio Understanding

  • Whisper-based systems including Qwen-Audio [10], Qwen2-Audio [11], SALMONN [63], and OSUM [21] strengthen speech understanding, but generally do not natively generate audio.These systems combine Whisper [58] as an audio encoder with large language models.Source: §7 · pp. 19–20

TTS and Audio Generation

  • AudioLM [3], VALL-E [70], LLASA [80], UniAudio [77], and VoiceBox [37] generate high-fidelity audio, but typically lack understanding, conversation, or instruction-following speech interaction.These approaches use neural codecs with autoregressive language models, with some extending them through hybrid tokenization or flow matching.Source: §7 · p. 20

Speech Conversation and Real-Time Dialogue

  • Moshi [14], GLM-4-Voice [84], Mini-Omni [72], OmniFlatten [86], LLaMA-Omni [18], and Freeze-Omni [71] support real-time or duplex dialogue, but may sacrifice generality or language modeling quality.Their approaches include interleaved or parallel text/audio decoding, progressive training, streaming decoders, and multi-task alignment; the passage attributes limitations to speech-only data and limited pretraining.Source: §7 · p. 20

Toward Universal Audio-Language Foundation Models

  • Baichuan-Audio [41], Step-Audio [28], and Qwen2.5-Omni [74] unify some audio-language capabilities, yet face limits in domain breadth, cost, accessibility, or raw-audio pretraining.Baichuan-Audio focuses on speech; Step-Audio uses 130B parameters and synthetic voice data; Qwen2.5-Omni emphasizes streaming inference.Source: §7 · p. 20

Kimi-Audio

  • Kimi-Audio pretrains on 13 million hours of speech, music, and environmental audio, then fine-tunes across tasks, achieving state-of-the-art results on several audio benchmark domains.Pretraining mixes audio-only, text-only, audio-to-text, and interleaved tasks to support generalizable audio reasoning and strong language abilities.Source: §7 · p. 20
  • Kimi-Audio is a fully open-source, pre-trained, instruction-followable, and real-time-capable model whose comprehensive coverage, scalable architecture, and broad task alignment make it a significant step toward general-purpose audio intelligence.Source: §7 · p. 21

8 Challenges and Future Trends

The report identifies remaining challenges and future directions for building more capable and intelligent audio processing systems.

From Audio Transcription to Audio Description

  • Pairing ASR-derived transcripts with audio captions would capture spoken content alongside paralanguage, acoustic scenes, and non-linguistic sounds for richer audio understanding and generation.The report argues that transcription alone focuses on what is said, while descriptive text adds context about the broader audio.Source: §8 · p. 21

Better Audio Representations

  • Integrated representations should combine transcription-oriented semantics with acoustic detail, since semantic tokens miss acoustic richness while reconstruction-trained acoustic tokens miss abstractive semantics.The proposed representations should capture speaker identity, emotion, and environmental sounds while retaining high-level abstractive information.Source: §8 · p. 21

Throw Away ASR and TTS in Audio Modeling

  • Training on native audio rather than ASR/TTS-based pseudo data could avoid their data-quality limits and enable models to surpass the ASR/TTS performance ceiling.The report attributes those limits to ASR text-recognition accuracy and TTS speech expressiveness, diversity, and quality.Source: §8 · p. 21

Appendix · A Contributions · Core Contributors

The Core Contributors section lists the Kimi-Audio contributors, notes that the roster is alphabetical by last name, and marks project leads with #.

Core Contributors

Related papers

Loading 2504.18425v1…