MinMo: A Multimodal Large Language Model for Seamless Voice Interaction

1 Introduction

• Seamless voice interaction calls for comprehensive audio understanding, including paralinguistic cues and audio events, natural expressive speech, relevant responses, and interruption-capable full-duplex conversation.In full-duplex interaction, the system listens while speaking and can continue its response or concede the turn to address a new user query.Source: §1 · pp. 1–2
• MinMo surpasses prior SOTA models across ASR, S2TT, spoken question answering, vocal sound classification, emotion recognition, language identification, age recognition, and gender detection.The comparison covers MinMo (approximately 8B parameters) and speech-text multimodal models across the listed tasks.Source: Figure 1 · p. 2, §1 · p. 3
• Native models such as Moshi (Défossez et al., 2024) face long speech-token sequences and speech-text data imbalance, while aligned models such as LLaMA-Omni (Fang et al., 2024) and Freeze-Omni (Wang et al., 2024b) have limited speech data and task evaluations.The paper notes that aligned systems retain text LLM capabilities but lack systematic evaluation of nuanced instruction following and full-duplex conversation.Source: §1 · pp. 2–3
• MinMo’s autoregressive streaming voice decoder mixes LLM hidden states with speech tokens at a fixed ratio, balancing structural simplicity and competitive generation performance.The paper reports that the decoder outperforms previous models and enables speech responses matching user-specified emotions, dialects, and speaking rates, as well as specific voices, with 98.4% instruction-following accuracy.Source: §1 · p. 3
• MinMo adapts a pretrained text LLM through multi-stage alignment on 1.4 million hours of diverse speech data, with minimal loss of original text-LLM capabilities.Its training covers Speech-to-Text, Text-to-Speech, and Speech-to-Speech tasks; the paper also reports methods to mitigate catastrophic forgetting.Source: §1 · p. 3
• A full-duplex prediction module uses the text LLM’s semantic understanding to decide whether to continue speaking or yield, listen, and respond to a new query.Speech-to-text latency is approximately 100ms; full-duplex latency is approximately 600ms in theory and 800ms in practice.Source: §1 · pp. 2–3

Multimodal Spoken Dialogue Models

• Generic audio-understanding models such as Qwen2-Audio (Chu et al., 2024) and SALMONN (Tang et al., 2024) use speech or audio encoders with pre-trained text LLMs, but voice interaction has not been systematically explored.Qwen2-Audio uses multi-task pre-training and instruction-based fine-tuning; SALMONN uses Q-Former and LoRA for modality alignment.Source: §2 · pp. 3–4
• Turn-based systems include PSLM (Mitsui et al., 2024), which relies on ASR and can lose paralinguistic information, and end-to-end Llama-Omni and Mini-Omni, which were not trained for full-duplex communication.PSLM generates speech and text tokens in parallel to reduce speech-generation latency but suffers reduced response quality; Mini-Omni generates text and audio tokens simultaneously while padding tokens so text guides audio generation.Source: §2 · pp. 4–5
• Full-duplex prior work includes collaborative systems such as VITA (Fu et al., 2024) and end-to-end models such as dGSLM (Nguyen et al., 2022) and LSLM (Ma et al., 2024b), each with stated limitations.Native models including Moshi (D´efossez et al., 2024) and GLM-4-Voice (Zeng et al., 2024) face speech-text sequence and training-balance challenges; aligned models aim to retain pre-trained text-LLM capabilities.Source: §2 · p. 5

Text Style-Controllable Speech Synthesis

• Prior systems address acoustic style through paralinguistic features or style-controllable TTS, but no aligned multimodal model had demonstrated style-controllable voice generation before MinMo.MinMo’s streaming voice decoder enables user-specified emotions, dialects, speaking rates, and voice mimicry; prior approaches include Moshi (D´efossez et al., 2024) and GLM-4-Voice (Zeng et al., 2024).Source: §2 · pp. 5–6

3.1 Model Architecture

• MinMo combines a pretrained text LLM with speech encoders, streaming speech-generation modules, and a predictor for real-time turn-taking in an approximately 8B-parameter model.The predictor decides whether to respond or pause ongoing output to process user audio; end-to-end audio latency is approximately 600 ms on an L20 GPU.Source: §3.1 · pp. 6–7

3.2 Streaming Voice Decoder

• The streaming voice decoder projects LLM context and user instructions into semantic vectors, then generates speech autoregressively in batches of 15 tokens for waveform synthesis.It interleaves five semantic vectors with fifteen speech tokens and uses a chunk-aware flow-matching model and mel-to-wave vocoder for streaming synthesis (Du et al., 2024b).Source: §3.2 · p. 7

3.3 Tasks and Training Data

• Speech-to-text training uses approximately 1.2 million hours of paired data spanning recognition, translation, language identification, emotion recognition, event detection, speaker analysis, and speech smoothing.The data is organized in ChatML format with natural-language task instructions.Source: §3.3 · pp. 8–9
• Text-to-speech data includes 170,000 hours of paired speech in four languages and approximately 1,000 hours of instruction-controlled audio covering attributes such as emotion, rate, dialect, and role-playing.The paired data matches that used to train CosyVoice 2.Source: §3.3 · p. 9
• Speech-to-speech training data includes approximately 10,000 hours of simulated multi-turn conversation and 100 hours of style-controllable conversation.Simulation combines synthesized user and assistant speech from text dialogues with real speech queries and generated responses, and also synthesizes style-varied dialogues.Source: §3.3 · pp. 9–10
• Speech-to-ControlToken data combines approximately 3,000 hours of real voice interactions with about 1,000 hours of simulated voice chat, annotated with heuristic duplex labels.The labels cover assistant turn-taking, user turn-taking after a sampled time gap, and user back-channels.Source: §3.3 · p. 10

3.4 Model Training

• Four progressive alignment stages train MinMo for speech understanding, speech generation, speech-to-speech conversation, and duplex interaction while retaining the text LLM’s capabilities.Speech-to-text alignment uses Pre-align, Full-Align, and instruction fine-tuning; later stages update the output projector and voice-token LM, then train the duplex predictor on 4,000 hours of conversation.Source: §3.4 · pp. 10–11

4.1 Speech Recognition and Translation

• MinMo outperforms Whisper Large v3 and Qwen2-Audio on most multilingual speech-recognition test sets and maintains consistent performance with or without language-identification prompts.The Common Voice results show that Whisper Large v3 and Qwen2-Audio depend more strongly on language-identification prompts than MinMo does.Source: §4.1 · pp. 12–13, Table 5 · p. 13
• MinMo reaches 85.3% language-identification accuracy across 102 Fleurs languages and outperforms SeACo-Paraformer on contextual-biasing ASR accuracy and hard-case hotword recall.Its language-identification training covers all 102 Fleurs languages; contextual biasing is trained through prompts containing hotwords.Source: §4.1 · p. 15, Table 7 · p. 15, Table 9 · p. 15

4.2 Speech Analysis and Understanding

• On Air-Bench, MinMo leads all baselines on language, gender, age, emotion, and vocal-sound classification, but Qwen-Audio outperforms it on sound-question classification.MinMo also outperforms all baselines on gender detection and age estimation. The authors note that sound-question evaluation can penalize answers that do not match the benchmark’s expected options.Source: §4.2 · pp. 16–17, Table 12 · p. 17
• In zero-shot evaluation across low-resource languages, MinMo achieves the best F1 score on most languages, including some absent from its original training data.The evaluation uses EmoBox utterances and reference labels across 32 datasets in 14 languages.Source: §4.2 · p. 16, Table 13 · p. 17

4.3 Speech-to-Text Enhancement

• MinMo performs better in subjective punctuation-insertion and inverse-text-normalization evaluations, while its spoken-language smoothing performance is comparable to Qwen2.5-7B.For punctuation and inverse text normalization, GPT-4 Turbo ranks outputs across randomized options and multiple scoring rounds; smoothing is assessed with objective metrics and human and LLM rankings.Source: §4.3 · pp. 18–19, Table 15 · p. 18

4.4 Voice Generation

• MinMo exceeds GLM-4-Voice in instruction-following voice-generation accuracy, particularly for dialects and role-playing, but its voice decoder has slightly lower Chinese content consistency and speech quality than CosyVoice 2.0-SFT.English content consistency is similar, with a slightly lower NMOS score; the authors report that this reduction does not significantly hinder human understanding.Source: §4.4 · p. 19, Table 18 · p. 19, Table 17 · p. 19

4.5 Voice Chat

• MinMo achieves state-of-the-art results in speech-to-speech question answering and on speech-to-text Llama Questions and Web Questions, though its speech-to-speech scores are lower than speech-to-text.The authors attribute the speech-to-speech gap to text-rich answers, demands on text-to-speech, and the ASR model used in evaluation. On Alpaca and ChitChat, MinMo largely maintains the base model’s conversational ability, though its response scores remain below ground truth.Source: §4.5 · p. 20, Table 19 · p. 21, Table 20 · p. 21
• In duplex evaluation, MinMo approaches 99% turn-taking prediction at K=10 on human-machine conversations, while actual human-human assistant turn-taking declines and user back-channel accuracy is 70%–80%.Average response delay is 250 ms for user turn-taking and around 660 ms for assistant turn-taking; the estimated end-to-end assistant-turn delay is approximately 600 ms.Source: §4.5 · pp. 21–22, Table 21 · p. 22, Table 22 · p. 22
• The voice-chat latency table covers average latency for assistant and user turn-taking across multiple full-duplex voice-chat datasets.The supplied passage does not include the table’s latency values.Source: Table 23 · p. 22

• Trained on over 1.4 million hours of speech, MinMo achieves state-of-the-art performance across spoken dialogue, multilingual speech recognition, and emotion recognition benchmarks.Source: §5 · p. 22
• MinMo’s multi-stage alignment balances audio understanding and generation while minimizing catastrophic forgetting in text-based LLMs.Source: §5 · p. 22
• Using hidden-layer representations of the text model, MinMo’s novel streaming end-to-end audio-generation alignment enables a structurally simple, low-latency voice decoder with competitive performance.Source: §5 · p. 22
• MinMo follows user instructions to generate nuanced speech reflecting specified emotions, dialects, and speaking styles.Source: §5 · pp. 22–23
• MinMo supports full-duplex interactions with approximately 600ms latency.Source: §5 · p. 23

6 Limitations

• The text model receives only LoRA updates, and its language- and task-following ability needs improvement; broader updates with more high-quality text data warrant exploration.The paper proposes investigating whether such data and updates could enhance instruction-following.Source: §6 · p. 23
• Long-tail pronunciation errors in end-to-end audio generation partly stem from retained one-to-many LLM tokens and output-text symbols that cannot be converted effectively into speech.The paper identifies data scaling as a possible way to address these errors.Source: §6 · p. 23
• Instruction-controlled audio-generation efficiency needs improvement, partly because instruction data are limited and hidden-embedding-only end-to-end alignment restricts historical-information transmission.The paper attributes the efficiency limitation in part to the small size of current instruction data.Source: §6 · p. 23
• MinMo’s semantics-based duplex module still requires separate AEC and VAD modules; a fully end-to-end duplex model remains future work.Source: §6 · p. 23

A.1 Spoken Language Smoothing

• The faithfulness prompt scores ASR-error correction and preservation of the original meaning, using human-refined text as a reference for evaluating the model’s output.It uses 1–10 scores and asks for an evaluation summary with reasons and suggested modifications in JSON format.Source: §A.1 · pp. 28–29
• The faithfulness evaluation considers ASR and grammatical errors alongside whether the refined text remains faithful and readable.The prompt describes the source as potentially containing recognition errors, grammatical errors, and spoken or informal expressions.Source: §A.1 · p. 29
• The formality prompt evaluates grammar, written formality, and readability of the model’s refined output on a 1–10 scale.It requests an evaluation summary and suggested modifications in JSON format.Source: §A.1 · p. 29

A.2 Punctuation and Inverse Text Normalization

• For punctuation, the prompt asks models to compare options on punctuation alone, ignoring other differences and full-width versus half-width characters.It requires ranking-only output, with no explanation.Source: §A.2 · pp. 29–30
• The ranking prompts display results as option labels paired with ranks, such as a:1; b:3; c:2, and allow tied ranks.The punctuation and inverse-text-normalization prompts both specify this output convention.Source: §A.2 · p. 30
• The inverse-text-normalization prompt compares options only on inverse text normalization, excluding differences in wording and punctuation.It requests ranking results without explanation.Source: §A.2 · p. 30