Source-linked AI summary
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
Zhifei Xie, Changqiao Wu
TL;DR
Real-time speech interaction requires models that reason directly over audio and stream speech without latency from external TTS systems. Mini-Omni addresses this with end-to-end audio processing, parallel text-audio generation, minimal-data adaptation, and a speech-focused dataset, while reporting preserved language capabilities and strong fundamental-task performance.
Problem
Real-time voice interaction remains difficult because direct audio reasoning is challenging and common cascaded systems depend on external TTS, increasing latency.
Method
Mini-Omni combines end-to-end audio input and streaming output with parallel text-audio generation, batch-parallel inference, adapter-based training, and VoiceAssistant-400K.
Results
Mini-Omni is reported as an open-source real-time speech-interaction model that preserves original language capabilities during batch-parallel inference and performs strongly on fundamental multimodal tasks.
Takeaways & Limitations
The approach provides a reference for adding streaming speech capabilities to other language models with minimal training and modification.
Abstract
from arXiv · showhide
Recent advances in language models have achieved significant progress. GPT-4o, as a new milestone, has enabled real-time conversations with humans, demonstrating near-human natural fluency. Such human-computer interaction necessitates models with the capability to perform reasoning directly with the audio modality and generate output in streaming. However, this remains beyond the reach of current academic models, as they typically depend on extra TTS systems for speech synthesis, resulting in undesirable latency. This paper introduces the Mini-Omni, an audio-based end-to-end conversational model, capable of real-time speech interaction. To achieve this capability, we propose a text-instructed speech generation method, along with batch-parallel strategies during inference to further boost the performance. Our method also helps to retain the original model's language capabilities with minimal degradation, enabling other works to establish real-time interaction capabilities. We call this training method "Any Model Can Talk". We also introduce the VoiceAssistant-400K dataset to fine-tune models optimized for speech output. To our best knowledge, Mini-Omni is the first fully end-to-end, open-source model for real-time speech interaction, offering valuable potential for future research.
1 Introduction
Mini-Omni addresses the gap in real-time voice interaction by combining end-to-end audio input and streaming output with methods designed to preserve language-model capabilities. The paper also introduces a speech-focused dataset and evaluates core multimodal and inference behavior.
- Motivation: Real-time voice interaction remains a major gap because existing approaches often rely on cascaded text generation and external TTS, adding latency.The paper identifies direct audio reasoning, model complexity, modality alignment, and resource demands as additional challenges.
- Mini-Omni: Mini-Omni is presented as an open-source end-to-end model with audio input, audio streaming output, and real-time conversational capabilities.The model uses a simple architecture and addresses direct audio reasoning with a 0.5B model and limited synthesized audio data.
- Generation method: The proposed parallel generation paradigm simultaneously produces audio and text tokens while aiming to preserve the language model’s original capabilities.Batch-based parallel generation further enhances reasoning during streaming audio output, while SNAC is selected to retain audio quality.
- Training framework: “Any Model Can Talk” adds speech capabilities through adapters, pretrained models, and limited synthesized-data fine-tuning without changing the original model architecture.The approach uses a three-phase training process including annealing and supervised fine-tuning.
- Evaluation: Preliminary experiments found strong performance on textQA, ASR, text-to-speech response, and speechQA, while batch-parallel inference preserved the model’s original capabilities.The introduction states that further experiments were planned to provide additional details.
- Data: VoiceAssistant-400K provides over 400,000 GPT-4o-generated entries for speech-assistant supervised fine-tuning because many open-source QA datasets contain code or overly lengthy text.The dataset is designed for speech model outputs and voice-assistant tone development.
2 Related Work
Related work develops multimodal understanding and audio language modeling through encoders, adapters, and discrete audio tokens. Real-time speech interaction remains comparatively scarce, with many systems relying on external TTS or latency-inducing cascades.
- Multimodal Understanding: Multimodal understanding methods commonly use pretrained modality encoders and lightweight adapters to align non-text inputs with language models.Examples include LLaVA, Flamingo, and BLIP for visual understanding.
- Audio Language Modeling: Audio tokenization converts continuous signals into discrete tokens, enabling language models to perform speech-text tasks including ASR, TTS, music understanding, and sound editing.Related systems explore audio codecs, decoding methods, and conditioning techniques, including diffusion-based conversion from tokens to audio.
- Real-Time Human-Machine Interaction Models: Real-time interaction models remain scarce: some use Audio-Text-Text-Audio cascades that introduce latency, while others output text and depend on external TTS systems.The paper positions Mini-Omni as a fully end-to-end speech-to-speech conversational model.
3 Mini-Omni
Mini-Omni generates text and audio in parallel to transfer text reasoning into streaming speech while preserving the original model’s capabilities. Its training combines modality alignment, adaptation, and multimodal fine-tuning with batch-parallel decoding.
- Decoding Strategies: Mini-Omni simultaneously generates text and audio tokens so streaming speech can leverage text-based reasoning.Audio is generated through text-to-speech synthesis while text and audio are decoded in parallel.
- Decoding Strategies: Batch parallel decoding is introduced to improve reasoning performance during streaming audio output.The method addresses audio responses that tend to be simpler than text responses.
- Any Model Can Talk: The approach aims to preserve the original language model’s capabilities while adding real-time speech interaction.The authors report minimal impact of the audio modality on text capabilities and describe the method as adaptable to other text-capable models.
- Audio Encoding: Audio encoding uses speech features for input and multi-codebook audio tokens for output to retain audio detail and support streaming.Flattening audio tokens produced excessively long sequences that harmed streaming and destabilized learning.
- Any Model Can Talk: Mini-Omni uses three training stages: modality alignment, adaptation training, and multimodal fine-tuning.The first stage trains adapters with speech recognition and synthesis data; later stages train text capabilities from audio inputs and then fine-tune the entire model.
- Model Input Ids: Model inputs place text or audio in modality-specific sequences and use a task-dependent <answer> token to guide multimodal output.Audio input tokens and Whisper features are transformed through adapters before concatenation.
4 Experiments
Mini-Omni is evaluated on foundational speech capabilities using multimodal training data and case studies. The reported results show excellent audio comprehension, while speech-based reasoning remains weaker than text-based reasoning.
- Evaluation Scope: Mini-Omni’s foundational evaluation covers speech recognition and use-case examples for speech understanding and reasoning.The section describes datasets, processing methods, hyperparameters, core speech recognition results, and streaming examples.
- Datasets: Approximately 8,000 hours of speech recognition data, 2 million Open-Orca examples, and 1.5 million synthesized speech QA pairs support training.The VoiceAssistant-400K dataset was created with GPT-4o to avoid unsuitable code and symbolic outputs.
- Training Setup: Mini-Omni uses the Qwen2-0.5B base language model and trains on 8 A100 GPUs with batch size 192 per step.The training setup also uses a Whisper-small speech encoder and a cosine annealing learning-rate scheduler.
- Experimental Results: Mini-Omni’s speech recognition slightly lags Whisper-small’s decoder but still achieves an excellent level of audio comprehension.The comparison uses four LibriSpeech test sets and includes wav2vec2, Whisper-small, and VITA.
- Case Study: Speech-based reasoning is somewhat weaker than text-based reasoning, motivating the need for batch generation.The case studies demonstrate capabilities in speech understanding and reasoning.
5 Conclusion
Mini-Omni combines parallel text-and-audio generation with minimal additional data and modules to support efficient real-time speech interaction. The work also introduces Any Model Can Talk and VoiceAssistant-400K to facilitate speech adaptation and fine-tuning.
- Mini-Omni uses parallel text and audio generation to transfer text capabilities to streaming speech interaction.The approach supports direct speech-to-speech capabilities with high model and data efficiency.
- Batch-parallel generation further improves reasoning ability and efficiency during streaming audio output.
- Any Model Can Talk enables rapid speech adaptation of other models with minimal additional training.It uses pre- and post-adapters and limited additional data.
- The authors release VoiceAssistant-400K for fine-tuning speech output and progressively open-source the data, inference, and training code.