Source-linked AI summary
MiMo-Audio: Audio Language Models are Few-Shot Learners
Xiaomi LLM-Core Team, :, Dong Zhang, Gang Wang, Jinlong Xue, Kai Fang, Liang Zhao, Rui Ma, Shuhuai Ren, Shuo Liu, Tao Guo, Weiji Zhuang, Xin Zhang, Xingchen Song, Yihan Yan, Yongzhe He, Cici, Bowen Shen, Chengxuan Zhu, Chong Ma, Chun Chen, Heyu Chen, Jiawei Li, Lei Li, Menghang Zhu, Peidian Li, Qiying Wang, Sirui Deng, Weimin Xiong, Wenshan Huang, Wenyu Yang, Yilin Jiang, Yixin Yang, Yuanyuan Tian, Yue Ma, Yue Yu, Zihan Zhang, Zihao Yue, Bangjun Xiao, Bingquan Xia, Bofei Gao, Bowen Ye, Can Cai, Chang Liu, Chenhong He, Chunan Li, Dawei Zhu, Duo Zhang, Fengyuan Shi, Guoan Wang, Hailin Zhang, Hanglong Lv, Hanyu Li, Hao Tian, Heng Qu, Hongshen Xu, Houbin Zhang, Huaqiu Liu, Jiangshan Duo, Jianguang Zuo, Jianyu Wei, Jiebao Xiao, Jinhao Dong, Jun Shi, Junhao Hu, Kainan Bao, Kang Zhou, Linghao Zhang, Meng Chen, Nuo Chen, Peng Zhang, Qianli Chen, Qiantong Wang, Rang Li, Shaohui Liu, Shengfan Wang, Shicheng Li, Shihua Yu, Shijie Cao, Shimao Chen, Shuhao Gu, Weikun Wang, Wenhan Ma, Xiangwei Deng, Xing Yong, Xing Zhang, Xu Wang, Yifan Song, Yihao Zhao, Yingbo Zhao, Yizhao Gao, Yu Cheng, Yu Tu, Yudong Wang, Zhaojun Huang, Zhengju Tang, Zhenru Lin, Zhichao Song, Zhipeng Xu, Zhixian Zheng, Zihan Jiang
TL;DR
Existing audio language models generally require task-specific fine-tuning, unlike humans’ few-shot adaptation to new audio tasks. MiMo-Audio scales lossless next-token speech pretraining beyond 100 million hours and adds instruction tuning with thinking data, yielding broad few-shot generalization and strong benchmark performance. The approach also has documented limitations in complex audio generation and spoken-dialogue stability.
Problem
Existing audio language models typically rely on task-specific fine-tuning, whereas humans can generalize to new audio tasks from few examples or simple instructions.
Method
MiMo-Audio scales lossless next-token speech pretraining to over one hundred million hours, then applies diverse instruction tuning and thinking data for audio understanding and generation.
Results
MiMo-Audio-7B models show few-shot generalization across diverse speech tasks, while MiMo-Audio-7B-Instruct achieves open-source SOTA across audio understanding and spoken-dialogue benchmarks.
Takeaways & Limitations
Scaling lossless speech pretraining provides a supported path toward general-purpose speech intelligence beyond task-specific training.
Takeaways & Limitations
MiMo-Audio-Base remains suboptimal for speech generation with background music and complex sound events, while MiMo-Audio-Instruct has unstable spoken-dialogue quality, pronunciation, style control, and prompt compliance.
Abstract
from arXiv · showhide
Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with only a few examples or simple instructions. GPT-3 has shown that scaling next-token prediction pretraining enables strong generalization capabilities in text, and we believe this paradigm is equally applicable to the audio domain. By scaling MiMo-Audio's pretraining data to over one hundred million of hours, we observe the emergence of few-shot learning capabilities across a diverse set of audio tasks. We develop a systematic evaluation of these capabilities and find that MiMo-Audio-7B-Base achieves SOTA performance on both speech intelligence and audio understanding benchmarks among open-source models. Beyond standard metrics, MiMo-Audio-7B-Base generalizes to tasks absent from its training data, such as voice conversion, style transfer, and speech editing. MiMo-Audio-7B-Base also demonstrates powerful speech continuation capabilities, capable of generating highly realistic talk shows, recitations, livestreaming and debates. At the post-training stage, we curate a diverse instruction-tuning corpus and introduce thinking mechanisms into both audio understanding and generation. MiMo-Audio-7B-Instruct achieves open-source SOTA on audio understanding benchmarks (MMSU, MMAU, MMAR, MMAU-Pro), spoken dialogue benchmarks (Big Bench Audio, MultiChallenge Audio) and instruct-TTS evaluations, approaching or surpassing closed-source models. Model checkpoints and full evaluation suite are available at https://github.com/XiaomiMiMo/MiMo-Audio.
1 Introduction
MiMo-Audio extends scaled next-token prediction to speech by combining lossless information flow, massive pretraining data, and a unified understanding-generation framework. The resulting models show few-shot generalization across speech tasks, while post-training adds instruction following and thinking capabilities.
- Existing audio language models lack the flexible, generalizable vocal intelligence that humans use across speakers, accents, environments, and social settings.
- Scaling next-token prediction to over one hundred million hours of speech is proposed as a route to broad task generalization and emergent few-shot abilities.The approach aims to let models acquire atomic skills during training and rapidly adapt to new speech tasks at inference.
- MiMo-Audio uses a lossless speech-information architecture, a patch encoder and decoder for 200-token-per-second audio, and a two-stage strategy unifying understanding and generation.The patch encoder downsamples four consecutive RVQ timesteps to a 6.25Hz representation before the LLM, while the decoder reconstructs the full 25Hz sequence.
- MiMo-Audio-7B-Base exhibits few-shot speech learning, strong SpeechMMLU performance, and generalization to unseen tasks including voice conversion, style transfer, denoising, and speech translation.It performs under speech input and output while closely approaching text-based MMLU and causing only minor degradation in text performance.
- The work contributes a 100-million-hour scaling result, a generative speech-pretraining blueprint, and thinking mechanisms for speech understanding and generation.Post-training uses a diverse instruction-tuning corpus and thinking data across audio understanding and generation.
2.1 MiMo-Audio-Tokenizer
MiMo-Audio-Tokenizer is designed to preserve both semantic and acoustic information while remaining suitable for language modeling. Its staged objectives combine audio-to-text alignment, multiscale reconstruction, commitment, and adversarial waveform refinement.
- Semantic tokens align well with language but lose fine acoustic detail, whereas acoustic codec tokens reconstruct audio faithfully but align poorly with text semantics.
- MiMo-Audio-Tokenizer addresses this trade-off with a unified model trained from scratch to improve cross-modal alignment and speech reconstruction quality.The stated strategy is to scale both model parameters and training data to alleviate semantic-acoustic representation conflict.
- Architecture: The tokenizer combines an audio encoder, discretization module, audio decoder, and vocoder, with a Transformer encoder, RVQ quantization, and causal decoding.The encoder uses bidirectional attention, while the decoder uses causal self-attention for speech synthesis.
- Training: Stage 1 jointly trains audio-to-text alignment, multiscale mel-spectrogram reconstruction, and RVQ commitment losses.The reconstruction objective uses mel-spectrogram scales e = {5, 6, 7}, with weights λ_A2T=10.0, λ_recon=1.0, and λ_commit=1.0.
- Training: Stage 2 freezes the tokenization representation and uses adversarial and feature-matching objectives to improve fine-grained waveform reconstruction.The multitask GAN recipe combines reconstruction, adversarial, and discriminator feature-matching losses.
- Evaluation: MiMo-Audio-Tokenizer achieves the highest PESQ-NB/WB, SIM, and STOI scores across both ZH and EN Seed-TTS-Eval splits at comparable bitrate.These gains are measured on the downstream modeling codebooks and are reported as preserving full speech-information fidelity.
2.2 MiMo-Audio
MiMo-Audio jointly models text tokens and audio patches in a unified autoregressive sequence, supporting arbitrary combinations of text and audio modalities. Its architecture uses a patch encoder, an LLM backbone, and a patch decoder, with delayed audio-token generation to address cross-codebook dependencies.
- MiMo-Audio: MiMo-Audio jointly models text and audio tokens, autoregressively predicting either modality to support arbitrary text-audio combinations.The input is an interleaved sequence of text tokens and audio patches.
- MiMo-Audio: Audio frames are grouped into contiguous patches to reduce the granularity mismatch between audio and text modalities.Each audio patch contains G consecutive frames.
- MiMo-Audio: The model comprises a patch encoder, an LLM backbone, and a patch decoder that respectively represent audio patches, process multimodal inputs, and generate audio patches.The patch encoder aggregates audio-token representations into a hidden representation for the LLM.
- Patch Decoder: The patch decoder generates audio tokens autoregressively within each patch using separate output heads for the RVQ codebooks.The decoder uses the same embedding tables as the patch encoder and one output head per RVQ codebook.
- Patch Decoder: Layer-specific delays shift RVQ codebook tokens across time, mitigating poor generation quality caused by cross-layer token dependencies.The delayed patch preserves its delay pattern during decoding, while empty tokens are disregarded.
3 Pre-Training
MiMo-Audio uses large-scale, diverse audio pre-training and evaluates whether this scaling yields few-shot generalization across speech intelligence, audio understanding, and speech-to-speech tasks.
- Data: The pre-training corpus combines unimodal text and speech with multimodal speech–text data, using a pipeline for collection, processing, annotation, and quality control.The audio data spans hundreds of millions of hours and diverse sources and content domains.
- Data: Dual semantic and non-semantic annotation supplies fine-grained attributes for targeted filtering and training.Semantic labels assess conversational quality, knowledge density, and reasoning, while audio captions describe timbre, emotion, and environment.
- Evaluation: The evaluation measures modality-invariant knowledge, auditory comprehension and reasoning, and speech-to-speech generation with few-shot in-context protocols.SpeechMMLU uses parallel text and speech input/output splits, while speech-to-speech evaluation conditions on paired speech exemplars.
- Results: Few-shot abilities emerge sharply after approximately 0.7 trillion training tokens across SpeechMMLU, voice conversion, and speech-to-speech translation.Performance rises non-linearly from a negligible baseline, then improves steadily before stabilizing.
- Results: MiMo-Audio achieves the highest SpeechMMLU scores in S2S (69.1), S2T (69.5), and T2S (71.5) among the compared models.It is reported as the only evaluated model sustaining high-level performance across all speech reasoning tasks.
- Results: MiMo-Audio achieves an MMAU overall score of 66.0, 5.7 points above Step-Audio2 mini-base’s 60.3.Its subdomain scores are balanced across speech (67.6), sound effects (65.2), and music (65.3).
- Results: Few-shot prompting reveals speech-to-speech generation and modality-invariant knowledge emerging at similar training scales.The reported alignment supports generalization to transformations of speaker identity, emotion, and speaking rate, alongside speech continuation across varied scenarios.
4 Post-Training
MiMo-Audio’s post-training activates audio understanding, generation, and dialogue capabilities through diverse instruction data and systematic evaluation across these task families.
- Training: Post-training integrates diverse open-source and constructed data for audio understanding, speech generation, spoken dialogue, and instruction-following TTS.The corpus includes cleaned and augmented understanding data, caption-conditioned generation data, and stylistically varied spoken conversations.
- Training: All model parameters are fine-tuned on 100 billion tokens spanning ASR, TTS, audio understanding, spoken dialogue, instruction-following TTS, and text dialogue.Speech generation and dialogue continue using text-guided interleaving of text tokens and speech patches.
- Evaluation: MiMo-Audio-7B-Instruct is evaluated on audio understanding, spoken dialogue, speech recognition, and speech generation.The evaluation setup is summarized in Table 7 and includes benchmarks such as MMSU, MMAU, MMAR, and MMAU-Pro.
MMAU-Pro
MiMo-Audio is evaluated across audio understanding, spoken dialogue, speech recognition, and speech generation benchmarks. Results show leading open-source performance across these task families, with competitiveness against closed-source systems.
- Audio Understanding: MiMo-Audio-7B-Instruct leads open-source models on MMSU and MMAU audio understanding benchmarks.Its overall scores also outperform Gemini 2.5 Flash and Gemini 1.5 Pro on these benchmarks.
- MMAU-Pro: MiMo-Audio-7B-Instruct leads on MMAU-Pro and MMAR, with results close to Gemini 2.5 Flash.These benchmarks target more challenging audio reasoning tasks.
- Spoken Dialogue: MiMo-Audio-7B-Instruct achieves the best open-source performance on Big-Bench-audio and Multi-Challenge-Audio, approaching gpt-4o.Scores are 72.90 (S2T) and 60.20 (S2S) on Big-Bench-audio, and 15.15 (S2T) and 10.10 (S2S) on Multi-Challenge-Audio.
- Speech Recognition and Generation: MiMo-Audio-7B-Instruct performs strongly on both ASR and TTS among open-source large speech models.It outperforms gpt-4o-mini-tts on InstructTTS English and Chinese subsets and is positioned as a leading open-source controllable TTS solution.
5 Conclusion
The conclusion presents massive-scale, lossless audio pretraining with next-token prediction as a viable route to general-purpose speech intelligence. MiMo-Audio validates this approach through few-shot generalization, a unified model blueprint, and strong benchmark performance.
- 5 Conclusion: Over 100 million hours of lossless audio pretraining enables MiMo-Audio to move beyond task-specific fine-tuning.The authors frame this as a viable path toward general-purpose speech intelligence.
- 5 Conclusion: MiMo-Audio exhibits few-shot generalization to voice conversion, style transfer, and speech editing without task-specific training.The authors describe this emergence as a speech-domain “GPT-3 moment.”
- 5 Conclusion: The proposed blueprint combines a unified high-fidelity audio tokenizer, scalable architecture, and phased training strategy.The blueprint is presented alongside the empirical validation of speech-domain generalization.
- 5 Conclusion: MiMo-Audio-7B-Instruct achieves state-of-the-art performance on multiple benchmarks and rivals closed-source systems.This supports the paper’s conclusion that the methodology can build versatile audio language models.
6 Limitations and Future Work
The paper identifies limitations in in-context learning, spoken-dialogue stability, and the thinking mechanism’s effects across audio domains. Future work focuses on broader audio generation and reinforcement-learning-based improvements.
- Limited In-Context-Learning Performance: MiMo-Audio-Base remains constrained on speech generation with background music and complex sound events.The authors aim to improve its general audio-generation capability.
- Unstable Spoken Dialogue Performance: MiMo-Audio-Instruct exhibits timbre discontinuities, unstable audio quality, mispronunciations, and inconsistent system-prompt compliance in speech dialogue.Complex symbols and formulas are especially prone to mispronunciation, while style control is unstable.
- Limited Thinking Performance: The thinking mechanism improves speech-related understanding but degrades sound and music understanding.The authors attribute this phenomenon to hallucinations introduced during the thinking process and propose reinforcement learning.
A Contributions and Acknowledgments
The acknowledgments thank contributors from multiple Xiaomi teams and note that authors within each role are listed alphabetically by first name.
- A Contributions and Acknowledgments: The paper acknowledges contributions from the Xiaomi LLM-Plus, NGK, MiChat, Mify, Data Platform, and CloudML teams.It also thanks contributors not explicitly listed in the paper.
- A Contributions and Acknowledgments: Authors within each role are listed alphabetically by their first name.
Core Contributors
The passage lists core contributors to the paper, including Dong Zhang, Gang Wang, and Jinlong Xue.
- Core contributors listed include Dong Zhang, Gang Wang, and Jinlong Xue.
- The contributor list also names Kai Fang, Liang Zhao, and Rui Ma.
- Additional named contributors include Shuhuai Ren, Shuo Liu, Tao Guo, and Weiji Zhuang.
Deployment & Evaluation
The supplied passages list additional contributors associated with the paper.
- The additional contributor list includes Bowen Shen, Chengxuan Zhu, and Chong Ma.
- Further listed contributors include Chun Chen, Heyu Chen, Jiawei Li, Lei Li, and Menghang Zhu.
- The contributor list continues with Peidian Li, Qiying Wang, Sirui Deng, Weimin Xiong, and Wenshan Huang.
- Other named contributors include Wenyu Yang, Yilin Jiang, Yixin Yang, Yuanyuan Tian, and Yue Ma.
- Additional contributors are listed across the passages, including Yue Yu, Zihan Zhang, Zihao Yue, and many others.