Source-linked AI summary
MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijie Yan, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Yang, Tianyu Zhao, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Pei Zhang, Chong Zhang, Jinren Zhou
TL;DR
Seamless voice interaction requires accurate audio understanding, expressive and relevant responses, and full-duplex communication. MinMo adapts a pre-trained text LLM through multi-stage alignment on over 1.4 million hours of speech data, achieving state-of-the-art performance across diverse benchmarks while supporting duplex interaction and nuanced speech generation.
Problem
Seamless voice interaction requires systems to understand speech content, paralinguistic cues, and audio events while producing natural, relevant responses and supporting simultaneous two-way communication.
Method
MinMo adapts a pre-trained text LLM through multi-stage alignment across speech tasks and uses an autoregressive streaming voice decoder for speech generation.
Results
MinMo achieves state-of-the-art performance across multiple benchmarks, including spoken dialogue, multilingual speech recognition, speech translation, emotion recognition, and speaker analysis, while retaining most text-LLM capabilities.
Takeaways & Limitations
MinMo supports full-duplex interaction and instruction-controlled speech generation with specified emotions, dialects, speaking styles, and speaking rates.
Takeaways & Limitations
MinMo's text LLM receives only LoRA updates, leaving diverse language and task instruction-following in need of improvement, and its end-to-end audio generation has long-tail pronunciation errors.
Abstract
from arXiv · showhide
Recent advancements in large language models (LLMs) and multimodal speech-text models have laid the groundwork for seamless voice interactions, enabling real-time, natural, and human-like conversations. Previous models for voice interactions are categorized as native and aligned. Native models integrate speech and text processing in one framework but struggle with issues like differing sequence lengths and insufficient pre-training. Aligned models maintain text LLM capabilities but are often limited by small datasets and a narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal Large Language Model with approximately 8B parameters for seamless voice interaction. We address the main limitations of prior aligned multimodal models. We train MinMo through multiple stages of speech-to-text alignment, text-to-speech alignment, speech-to-speech alignment, and duplex interaction alignment, on 1.4 million hours of diverse speech data and a broad range of speech tasks. After the multi-stage training, MinMo achieves state-of-the-art performance across various benchmarks for voice comprehension and generation while maintaining the capabilities of text LLMs, and also facilitates full-duplex conversation, that is, simultaneous two-way communication between the user and the system. Moreover, we propose a novel and simple voice decoder that outperforms prior models in voice generation. The enhanced instruction-following capabilities of MinMo supports controlling speech generation based on user instructions, with various nuances including emotions, dialects, and speaking rates, and mimicking specific voices. For MinMo, the speech-to-text latency is approximately 100ms, full-duplex latency is approximately 600ms in theory and 800ms in practice. The MinMo project web page is https://funaudiollm.github.io/minmo, and the code and models will be released soon.
1 Introduction
MinMo targets seamless voice interaction by addressing broad audio understanding, expressive generation, intelligent response, and full-duplex communication challenges. It combines large-scale multi-task training with a simple streaming voice decoder while preserving text-LLM capabilities.
- 1 Introduction: Seamless voice interaction requires accurate content and paralinguistic understanding, expressive responses, relevant chatbot behavior, and full-duplex interruption handling.Full-duplex communication lets the system listen while speaking and respond when the user interrupts.
- 1 Introduction: Existing systems include native multimodal models and aligned models that retain a pretrained text LLM through intermediate text outputs.Aligned models face concerns about limited speech data, possible text-chat degradation, and insufficiently broad speech-task evaluation.
- 1 Introduction: 1.4 million hours of speech data support MinMo’s multi-stage alignment across speech-to-text, text-to-speech, and speech-to-speech tasks.The training covers diverse speech tasks and is designed to mitigate catastrophic forgetting while improving voice comprehension and generation.
- 1 Introduction: An autoregressive streaming Transformer voice decoder mixes LLM hidden states with speech tokens at a fixed ratio for audio generation.The design aims to balance structural simplicity, low latency, and voice-generation performance.
- 1 Introduction: MinMo achieves state-of-the-art performance across multiple open-source speech benchmarks while retaining most original text-LLM capabilities.Reported tasks include spoken dialogue, multilingual speech recognition, speech translation, emotion recognition, and speaker analysis.
2 Related Work
Related work spans generic audio-understanding models, spoken-dialogue systems, full-duplex architectures, and controllable speech synthesis. These approaches differ in modality alignment, duplex support, latency, response quality, and acoustic control.
- Multimodal Spoken Dialogue Models: Generic speech foundation models adapt pretrained text LLMs for audio understanding but have not been systematically explored for voice interaction.Examples include Qwen2-Audio and SALMONN, which use speech or audio encoders with modality-alignment mechanisms.
- Multimodal Spoken Dialogue Models: Voice-interaction systems include collaborative pipelines and end-to-end frameworks, with some supporting only traditional half-duplex voice chat.Collaborative systems may rely on ASR, monitoring models, or external TTS modules, introducing modality or response-quality trade-offs.
- Multimodal Spoken Dialogue Models: Mini-Omni generates text and audio tokens together while padding N tokens so text tokens precede and guide audio-token generation.
- Multimodal Spoken Dialogue Models: Native multimodal full-duplex models jointly model speech and text, whereas earlier end-to-end systems faced limitations in pretrained-LLM use, acoustic modeling, online operation, or response quality.The cited examples include dGSLM, LSLM, Moshi, GLM-4-Voice, SyncLM, IntrinsicVoice, and Omni-Flatten.
- Text Style-Controllable Speech Synthesis: Speech interaction requires understanding and generating acoustic attributes such as emotion, dialect, speaking rate, and speaker identity.Controllable synthesis systems investigate these attributes to support richer spoken communication.
3 MinMo
MinMo combines a pretrained text LLM with lightweight audio-alignment modules and a streaming voice decoder for multimodal speech interaction. Its multitask training spans speech recognition, synthesis, speech-to-speech dialogue, and interaction control.
- Model Architecture: MinMo uses a pretrained SenseVoice-large Voice Encoder, Qwen2.5-7B-instruct text LLM, and CosyVoice 2 components for audio understanding and generation.The Input Projector aligns and downsamples encoder outputs, while the Output Projector aligns LLM representations with the voice decoder.
- Streaming Voice Decoder: MinMo has approximately 8 billion parameters and an end-to-end audio-to-response latency of approximately 600 ms on an L20 GPU.The latency calculation accounts for text-token generation, speech-token generation, and waveform synthesis.
- Streaming Voice Decoder: The voice decoder interleaves five semantic vectors with fifteen speech tokens, then synthesizes waveform chunks from the generated tokens.A Voice Token LM autoregressively generates speech tokens, and a streaming token2wav synthesizer converts them into audio.
- Tasks and Training Data: Its multitask training covers Speech-to-Text, Text-to-Speech, Speech-to-Speech, and Speech-to-ControlToken categories.Speech-to-Text includes ASR, translation, language identification, emotion recognition, audio event detection, speaker analysis, and spoken language smoothing.
- Tasks and Training Data: Text-to-Speech training includes 170,000 hours of paired data across Chinese, English, Korean, and Japanese, plus approximately 1,000 hours of instruction-controlled audio generation.Instructions encode attributes such as emotion, speaking rate, dialect, and role-playing.
- Tasks and Training Data: Speech-to-Speech data includes approximately 10,000 hours of simulated multi-turn conversational speech and 100 hours of style-controllable multi-turn speech.The simulation uses text dialogue, synthesized user and assistant speech, and selected real speech queries to increase diversity and reduce the gap between synthesized and real audio.
- Tasks and Training Data: Speech-to-ControlToken data combines approximately 3,000 hours of real voice-interaction data with simulated text-dialogue data.The data models assistant turn-taking, user turn-taking, and user back-channel behavior.
4 Experiments
MinMo is evaluated across multilingual speech recognition and translation, speech analysis, voice generation, and duplex voice-chat tasks. Results show broad strengths, alongside task-specific limitations in complex audio QA, speech-to-speech quality, and some conversational evaluations.
- Speech Recognition and Translation: MinMo achieves superior ASR performance on most multilingual test sets compared with Whisper Large v3 and Qwen2-Audio.Its performance remains robust without language-identification prompts, unlike the baselines’ stronger dependence on LID information.
- Speech Recognition and Translation: MinMo consistently outperforms a cascaded Whisper-Qwen model on speech translation and reaches SOTA Chinese↔English and Japanese↔English performance.It maintains consistent performance across Fleurs and CoVoST2 despite excluding Fleurs from training augmentation.
- Speech Recognition and Translation: 85.3% language-identification accuracy on Fleurs’ 102 languages surpasses all previous models and includes accurate Cantonese identification.MinMo’s LID training data covers all 102 Fleurs languages.
- Speech Analysis and Understanding: MinMo outperforms all baseline models on gender detection and age estimation, while surpassing all baselines on Vocal Sound classification but trailing Qwen-Audio on complex sound question answering.The authors attribute the sound-QA gap partly to MinMo’s voice-interaction focus and evaluation post-processing that may misalign valid answers with fixed choices.
- Text Enhancement and Voice Generation: MinMo performs better in subjective punctuation-insertion and inverse-text-normalization evaluations, while its voice decoder shows slightly lower Chinese content consistency and speech quality than CosyVoice 2.0-SFT.On English, content consistency is similar but NMOS is slightly lower; the reported reduction does not significantly hinder human understanding.
5 Conclusion
MinMo combines multimodal understanding, speech generation, and full-duplex interaction through multi-stage alignment and extensive speech training. Its decoder supports low-latency, instruction-controlled speech generation with specified speaking styles.
- MinMo combines audio understanding, audio generation, and end-to-end duplex speech interaction through multi-stage alignment over 1.4 million hours of audio data.The training covers a wide range of speech tasks and adapts a pre-trained text LLM.
- MinMo achieves state-of-the-art performance across diverse benchmarks while balancing audio understanding and generation and minimizing catastrophic forgetting in the text LLM.
- MinMo’s voice decoder uses text-model hidden-layer representations to provide structurally simple, competitive, low-latency audio generation.
- Instruction-following controls MinMo’s speech generation for user-specified emotions, dialects, and speaking styles.
- MinMo supports full-duplex interactions with approximately 600ms latency, enabling simultaneous conversational exchange.
6 Limitations
MinMo’s limitations include incomplete instruction following, long-tail pronunciation errors, and inefficient instruction-controlled audio generation. The authors identify text-model updating, data scaling, and improved alignment as areas for further work.
- MinMo’s pre-trained text model receives only LoRA updates, and its diverse language- and task-following abilities need improvement.The authors suggest exploring more high-quality text data and more comprehensive text-model updates.
- Long-tail pronunciation errors remain in end-to-end audio generation.The authors attribute these partly to one-to-many LLM tokens and special symbols that cannot be effectively converted into speech.
- Instruction-controlled audio generation is not yet efficient because instruction data are small and alignment uses only hidden embeddings.
7 Authors (alphabetical order of family name)
This section lists the paper’s authors in alphabetical order of family name.
- The author list includes Zhijie Yan, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Yang, Tianyu Zhao, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Pei Zhang, Chong Zhang, and Jinren Zhou.
- The author list includes Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, and Fan Yu.
- The author list includes Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Zhifu Gao, Yabin Li, and Xiang Lv.
A Prompts for Voice Understanding Tasks
The appendix provides prompts for evaluating spoken language smoothing, punctuation insertion, and inverse text normalization. It specifies criteria, scoring procedures, and ranking formats for these voice-understanding tasks.
- A.1 Spoken Language Smoothing: The appendix introduces faithfulness and formality evaluation prompts for spoken language smoothing.
- A.1 Spoken Language Smoothing: Faithfulness evaluation asks whether refined ASR output removes recognition and grammatical errors while preserving the original content and improving readability.
- A.1 Spoken Language Smoothing: Formality evaluation scores grammatical accuracy and clear written style on a 1-to-10 scale.
- A.2 Punctuation and Inverse Text Normalization: The appendix introduces a prompt for evaluating punctuation and inverse text normalization.
- A.2 Punctuation and Inverse Text Normalization: Punctuation evaluation ranks three sentence options while focusing only on punctuation and ignoring other differences, including character width.
- A.2 Punctuation and Inverse Text Normalization: A separate formality prompt is included under the punctuation and inverse text normalization appendix subsection.
- A.2 Punctuation and Inverse Text Normalization: Inverse text normalization evaluation ranks three options while focusing only on inverse text normalization, excluding textual and punctuation differences.