MiMo-Audio: Audio Language Models are Few-Shot Learners

• The paper proposes scaling next-token speech pretraining for task generalization, addressing prior models’ limited breadth with lossless speech-information flow and over 100 million hours of data.The authors describe this as the first empirical evidence that scaling lossless, compression-based speech pretraining to this scale unlocks emergent task generalization.Source: §1 · pp. 4, 6
• MiMo-Audio combines a 1.2B-parameter, 25Hz tokenizer producing 200 tokens per second with a patch encoder and decoder that present 6.25Hz sequences to the LLM.The tokenizer uses eight RVQ layers and semantic and reconstruction objectives; the patch encoder groups four consecutive timesteps before the decoder generates the full-rate sequence.Source: §1 · pp. 4–5
• A two-stage training strategy moves from speech understanding to unified understanding and generation, using a curated corpus exceeding 100 million hours and a benchmark spanning reasoning and speech-to-speech tasks.The benchmark assesses modality-invariant general knowledge, auditory comprehension and reasoning, and diverse speech-to-speech generation tasks.Source: §1 · p. 5
• MiMo-Audio-7B-Base shows few-shot speech learning, approaches text-based MMLU on SpeechMMLU, and generalizes to unseen tasks including voice conversion, style transfer, and speech translation.It also generates coherent monologues and multi-speaker dialogues in formats such as talk shows, debates, podcasts, and game commentaries.Source: §1 · p. 5
• MiMo-Audio-7B-Instruct achieves open-source SOTA on MMSU, MMAU, MMAR, and MMAU-Pro, outperforming open-source models on spoken dialogue benchmarks and matching GPT-4o-mini-tts on instruction-following TTS.Post-training uses a diverse instruction-tuning corpus and chain-of-thought data for audio understanding and generation; MiMo-TTS-7B was trained on over 7 million hours to produce speech dialogue data.Source: §1 · pp. 5–6

2.1 MiMo-Audio-Tokenizer

• MiMo-Audio-Tokenizer is trained from scratch to capture semantic and acoustic information, addressing the trade-off between text alignment and fine-grained waveform reconstruction.Semantic tokens align well with linguistic content but lose acoustic detail, while codec tokens reconstruct audio well but align less effectively with text.Source: §2.1 · p. 6

2.1.1 Architecture

• Its encoder uses a 32-layer, 20-head bidirectional Transformer with 2× downsampling at both ends and adds layer-3 hidden states to the final output.The model dimension is 1280, and the FFN inner dimension is 5120.Source: §2.1.1 · pp. 6–7, Figure 2 · p. 7
• A 20-layer residual vector quantizer uses codebooks of size 1024 in its first two layers and 128 thereafter; a causal audio decoder supports streaming generation, while the Vocos-design vocoder replaces ConvNeXt with a Transformer.The Transformer has 16 layers and uses sliding-window attention with window sizes [40, 10], providing receptive fields of [6.4s, 1.6s].Source: §2.1.1 · p. 7

2.1.2 Training

• Stage 1 scales training to over 11 million hours and jointly learns audio reconstruction and audio-to-text alignment, with the tokenizer and LLM trained from scratch.The stage combines a multi-scale mel-spectrogram reconstruction objective with next-token prediction for the LLM text output.Source: §2.1.2 · pp. 7–8
• In stage 2, the tokenizer is frozen while discriminators train the decoder and vocoder with reconstruction, adversarial, and feature-matching losses.Training uses MPD and MS-STFT discriminators under a Hinge-GAN framework.Source: §2.1.2 · pp. 7–9

2.1.3 Evaluation

• On Seed-TTS-Eval, MiMo-Audio-Tokenizer achieves the highest PESQ-NB/WB, SIM, and STOI scores on both ZH and EN splits, substantially outperforming baselines at comparable bitrate.Evaluation uses the first eight codebooks, matching the audio tokens used by downstream MiMo-Audio.Source: Table 1 · p. 9, §2.1.3 · p. 9
• Reconstruction is evaluated on Seed-TTS-Eval ground-truth recordings using speaker similarity, STOI, and PESQ, with quality decoded from the first eight codebooks.Mimi is evaluated under the same codebook protocol for consistency.Source: §2.1.3 · p. 9

2.2 MiMo-Audio

• MiMo-Audio interleaves text tokens with patches of G audio frames and autoregressively predicts the next text token or audio patch.Audio frames use eight RVQ codebooks; patching addresses their lower information density relative to text tokens.Source: §2.2 · p. 10

2.2.1 Patch Encoder

• The patch encoder aggregates embeddings from each frame’s RVQ codebooks, then uses a six-layer bidirectional Transformer to encode local context within each patch.It concatenates frame outputs and projects them to the LLM’s input dimensionality.Source: §2.2.1 · pp. 10–11

2.2.2 Large Language Model

• MiMo-7B-Base (Xiaomi, 2025) serves as the LLM backbone, processing either text embeddings or encoded audio patches and routing hidden states to text prediction or audio generation.Source: §2.2.2 · p. 11

2.2.3 Patch Decoder

• To address poor audio quality from simultaneous RVQ-layer prediction, the decoder applies layer-specific delays and autoregressively generates delayed patches, inspired by Copet et al. (2023).The 16-layer causal Transformer uses independent output heads for the RVQ codebooks and preserves the delay pattern during decoding.Source: §2.2.3 · pp. 11–12

3.1 Data

• The speech-data pipeline combines collection, automated processing, multidimensional annotation, and quality control to build large-scale, high-quality, diverse audio data.Source: §3.1 · p. 12

3.1.1 Data Preprocessing

• The corpus spans hundreds of millions of in-the-wild audio hours across varied sources and topics, processed with normalization, diarization, VAD, ASR, and audio-quality assessment.Sources include podcasts, audiobooks, news, interviews, and conferences; topics range from daily communication and entertainment to business, arts, and scientific research.Source: §3.1.1 · pp. 12–13

3.1.2 Data Labeling

• A dual-dimension annotation system labels semantic quality and non-semantic audio attributes, enabling fine-grained data filtering and more targeted training.Semantic scores assess conversational quality, knowledge density, and reasoning; audio captions describe timbre, emotional style, and background environment.Source: §3.1.2 · p. 13

3.1.3 Data Curation

• Curation removes noisy, low-quality, or unsafe segments and samples higher-quality data using combined semantic and non-semantic scores.Source: §3.1.3 · p. 13

3.2 Training

• Starting from MiMo-7B-Base, progressive two-stage pre-training aims to preserve text capabilities while adding speech understanding and generation.Source: §3.2 · p. 13

3.2.1 Understanding Training

• Stage 1 trains the patch encoder and LLM for speech understanding on 2.6T tokens, with loss computed only on text tokens.The dataset combines 1.2T text tokens and 1.4T speech-related tokens across four task formats; the patch encoder learning rate is 2e-4 and the LLM learning rate is 3e-5.Source: §3.2.1 · pp. 13–14

3.2.2 Understanding-Generation Joint Training

• The second stage trains all model parameters on 5T tokens for integrated speech understanding and generation, using text-guided interleaving to improve speech-generation quality.The data includes seven task formats; generation alternates text and speech patches at a fixed 5:5 ratio, then generates remaining speech tokens after text is complete.Source: §3.2.2 · p. 14

3.3.1 Few-shot In-context Learning

• The few-shot protocol assesses speech–text competence across modality-invariant general knowledge, auditory comprehension and reasoning, and speech-to-speech generation.The evaluation follows a GPT-3–style paradigm and uses in-context examples rather than task-specific fine-tuning.Source: §3.3.1 · p. 14
• SpeechMMLU2 provides 8,549 questions across 34 subjects in four parallel text- and speech-input/output splits for cross-modal evaluation.Questions and options are synthesized from MMLU while preserving semantic content, enabling same-question comparisons across text-to-text, speech-to-text, text-to-speech, and speech-to-speech.Source: §3.3.1 · p. 15
• Few-shot MMAU evaluation extends auditory assessment beyond speech semantics to information extraction and reasoning across speech, environmental sounds, and music.MMAU is used to probe additional dimensions of the acoustic world after large-scale speech–text pretraining.Source: §3.3.1 · p. 15
• The speech-to-speech protocol tests whether paired speech exemplars alone can support diverse downstream tasks without parameter updates.The paper motivates this protocol with the hypothesis that high-fidelity audio-token compression during pretraining induces in-context learning; example tasks are listed in Table 5.Source: §3.3.1 · pp. 15–16, Table 5 · p. 16

3.3.2 Speech Continuation

• MiMo-Audio-7B-Base generates semantically coherent speech continuations while preserving speaker identity, timbre, prosody, environmental acoustics, and non-speech audio.Prompts span single-speaker domains such as comedy and lectures, as well as multi-speaker debates, interviews, and theatrical performances.Source: §3.3.2 · p. 16

Emergent Ability

• After performance remained negligible below approximately 0.7 trillion training tokens, few-shot benchmark abilities surged nonlinearly and then improved before stabilizing.The reported tasks included 5-shot SpeechMMLU (T2S and S2S), 16-shot voice conversion, and 16-shot speech-to-speech translation.Source: §3.4 · p. 17

Speech Intelligence

• 69.1 on SpeechMMLU-S2S, 69.5 on SpeechMMLU-S2T, and 71.5 on SpeechMMLU-T2S were the highest scores reported for each task.MiMo-Audio was the only evaluated model described as sustaining high-level performance across all speech reasoning tasks.Source: §3.4 · p. 17
• 3.4 points was MiMo-Audio’s modality gap, versus 22.3 for Step-Audio2 minibase, 58.9 for Kimi-Audio-base, and 39.2 for Baichuan-Audio-base.The gap is defined as the difference between a model’s text2text and speech2speech scores.Source: §3.4 · pp. 17–18

General Audio Understanding

• 66.0 on MMAU overall, 5.7 points above Step-Audio2 mini-base (60.3), with consistently high scores across speech, sound effects, and music.The subdomain scores were 67.6 for speech, 65.2 for sound effects, and 65.3 for music.Source: §3.4 · p. 18

Speech Task Generalization

• At 16-shot prompting, speech-generation abilities and SpeechMMLU T2S/S2S emerged at similar training scales, with generalization to speaker identity, emotion, and speaking-rate transformations.Voice conversion and speech-to-speech translation are reported under this setting; other less automatically evaluable generation tasks are shown qualitatively.Source: §3.4 · p. 18

Speech Continuation

• MiMo-Audio-Base continued speech across scenarios including streaming, teaching, recitation, singing, talk shows, and debates, producing coherent, prosodically connected audio without parameter adaptation.Reported examples include appropriate audience cheers in talk shows, consistent accents in dialect speech, and smooth prosody in two-person debates.Source: §3.4 · p. 18

4.1 Data

• The data strategy uses supervised instruction fine-tuning datasets to activate the pretrained model’s understanding and generation capabilities across tasks.Source: §4.1 · p. 19

4.1.1 Audio Understanding

• Audio-understanding data combines open-source speech, sound, and music datasets with an LLM-based cleaning and augmentation pipeline to address label noise and narrow task paradigms.The resulting diverse data includes audio captioning and question answering.Source: §4.1.1 · p. 19

4.1.2 Speech Generation

• Speech-generation instruction data pairs audio captions with a high-quality speech subset, training the model to generate matching audio for controllable, instruction-following speech.Source: §4.1.2 · p. 19

4.1.3 Spoken Dialogue

• Spoken-dialogue data covers single- and multi-turn user queries and assistant replies, with screened text used to ensure reliable content quality.Colloquial question-answer pairs are stylistically rewritten, then synthesized with appropriate style and emotion using randomly selected voice prompts spanning many timbres.Source: §4.1.3 · p. 19

4.2 Training

• Post-training fine-tunes all model parameters on 100 billion tokens across six task formats, including ASR, TTS, understanding, spoken dialogue, instruction-following TTS, and text dialogue.ASR, TTS, and text-dialogue data come from open-source collections; the other tasks use high-quality datasets described in Section 4.1.Source: §4.2 · p. 19
• Generation and spoken-dialogue training interleaves text and speech patches at a fixed 5:5 ratio, with distinct loss weights for text and audio tokens.Training uses a context length of 8192 and a batch size of 2.1M tokens.Source: §4.2 · pp. 19–20

4.3 Evaluation

• The systematic evaluation covers audio understanding, spoken dialogue, speech recognition, and generation, with task-specific configurations listed in Table 7.Source: §4.3 · p. 20, Table 7 · p. 20

4.3.1 Audio Understanding

• Audio understanding is evaluated with MMSU for multitask spoken understanding, MMAU for speech, sound, and music, and MMAR and MMAU-Pro for mixed-audio reasoning and audio knowledge.Source: §4.3.1 · p. 20

4.4 Results

• MiMo-Audio-7B-Instruct outperforms all open-source models and Gemini 2.5 Flash and Gemini 1.5 Pro overall on MMSU and MMAU, and leads MMAU-Pro and MMAR close to Gemini 2.5 Flash.These results establish strong performance on both standard audio question answering and more challenging audio reasoning tasks.Source: Table 8 · p. 21, §4.4 · p. 23
• 72.90 (S2T), 60.20 (S2S), 15.15 (S2T), and 10.10 (S2S) are MiMo-Audio-7B-Instruct's Big-Bench-audio and Multi-Challenge-Audio scores, best among open-source models and close to gpt-4o.It ranks second only to gpt-4o on Big-Bench-audio and leads the open-source group on both Multi-Challenge-Audio versions.Source: Table 8 · p. 21, §4.4 · p. 23
• MiMo-Audio-7B-Instruct performs similarly to Step-Audio2-mini and Kimi-Audio-Instruct on ASR and TTS benchmarks, and beats gpt-4o-mini-tts on both InstructTTS English and Chinese subsets.The authors describe its overall InstructTTS results as especially competitive and position it as a leading open-source controllable TTS solution.Source: Table 9 · p. 22, §4.4 · pp. 23–24

MMAU-Pro

• Audio5 responses are GPT-scored, while MultiChallenge (Sirdeshmukh et al., 2025) tests final-turn answers from dialogue histories in S2T and S2S forms.Spoken Audio5 responses are transcribed with Whisper-Large-V3 (Radford et al., 2023) before GPT-4o-mini evaluation. MultiChallenge filters nonspoken formats, then synthesizes remaining dialogue with consistent voices selected from a pool of 250; S2T provides history as text and S2S entirely as speech.Source: MMAU-Pro · p. 22

4.3.3 Speech Recognition and Generation

• MiMo-Audio-7B-Instruct is evaluated on LibriSpeech test-clean and AISHELL-1 ASR (WER), plus SeedTTS and InstructTTSEval for conventional and style-controlled TTS.SeedTTS includes English, Chinese, and a Chinese hardcase subset; TTS WER compares ASR transcripts with reference text, while InstructTTSEval uses Gemini-based scoring for instruction alignment.Source: §4.3.3 · pp. 22–23

5 Conclusion

• Pretraining on over 100 million hours of lossless audio shows scaling next-token prediction can support general-purpose speech intelligence beyond task-specific fine-tuning.The conclusion presents this as a viable path for audio language models.Source: §5 · p. 24
• Few-shot capabilities emerged after a critical data threshold, enabling voice conversion, style transfer, and speech editing without task-specific training.The paper frames this as an achievable “GPT-3 moment” in the speech domain.Source: §5 · p. 24
• The work presents a blueprint combining a unified high-fidelity audio tokenizer, scalable architecture, and phased training strategy; MiMo-Audio-7B-Instruct achieves state-of-the-art results on multiple benchmarks and rivals closed-source systems.Source: §5 · p. 24
• The research offers a foundational methodology for versatile audio models that understand and generate speech with human-like adaptability.The conclusion describes this as a step toward more natural, flexible, and intelligent systems.Source: §5 · p. 24

Limited In-Context-Learning Performance

• In-context learning remains suboptimal for speech generation with background music and complex sound events, motivating work to improve general audio generation.The pretrained model can perform novel tasks beyond its training scope, but these scenarios remain challenging.Source: §6 · p. 24

Unstable Spoken Dialogue Performance

• Spoken dialogue can show timbre discontinuities, unstable audio quality, mispronunciations, and inconsistent system-prompt compliance; future work will use reinforcement learning to improve stability.Mispronunciations are especially common for complex symbols and formulas, and dialogue style control is unstable.Source: §6 · p. 24

Limited Thinking Performance

• Thinking improves speech-related understanding but degrades sound and music understanding, with hallucinations during thinking identified as the cause; reinforcement learning is planned to improve audio understanding.Source: §6 · p. 24

A Contributions and Acknowledgments

• The acknowledged contributors include Xiaomi LLM-Plus, NGK, MiChat, Mify, Data Platform, CloudML, and people not explicitly listed.Authors within each role are listed alphabetically by first name.Source: A Contributions and Acknowledgments · p. 30