Source-linked AI summary
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, Jiancheng Gui, Luoyuan Zhang, Xian Sun, Fuwei Huang, Moye Chen, Zhuo Lin, Hanyu Liu, Qingxin Gui, Qingzhe Han, Yuyang Wen, Huiping Liu, Rongkang Wang, Yaqi Zhang, Hongliang Wei, Chi Chen, You Li, Kechen Fang, Jie Zhou, Yuxuan Li, Guoyang Zeng, Chaojun Xiao, Yankai Lin, Xu Han, Maosong Sun, Zhiyuan Liu, Yuan Yao
TL;DR
Current multimodal models remain turn-based and reactive, limiting timely adjustment and proactive assistance in evolving environments. MiniCPM-o 4.5 introduces Omni-Flow for real-time full-duplex interaction, approaching Gemini 2.5 Flash in vision-language capabilities while outperforming Qwen3-Omni-30B-A3B in omni-modal understanding and speech generation quality.
Problem
Current multimodal models separate perception from response and remain request-driven, limiting continuous adjustment and proactive interaction in evolving multimodal environments.
Method
MiniCPM-o 4.5 uses Omni-Flow, a unified streaming framework aligning multimodal inputs and outputs on a shared temporal axis for full-duplex interaction.
Results
MiniCPM-o 4.5 approaches Gemini 2.5 Flash in vision-language capabilities and surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and speech generation quality.
Takeaways & Limitations
The 9B model supports real-time full-duplex omni-modal interaction on edge devices with less than 12GB RAM during deployment.
Takeaways & Limitations
The model’s robustness in long, dynamic real-world streaming interactions requires further improvement and validation, and its proactive behavior remains relatively simple.
Abstract
from arXiv · showhide
Recent progress in multimodal large language models (MLLMs) has brought AI capabilities from static offline data processing to real-time streaming interaction, yet they still remain far from human-level multimodal interaction. The key bottlenecks are no longer modality coverage or latency alone, but the interaction paradigm itself. First, perception and response are still separated into alternating phases, preventing models from incorporating new inputs for timely adjustment during generation. Second, most current models remain reactive, responding only to explicit user requests instead of acting proactively in the evolving multimodal environment. We present MiniCPM-o 4.5, our latest effort towards human-like multimodal interaction, which mitigates these gaps by real-time full-duplex omni-modal interaction. It can see, listen, and speak simultaneously in real-time, while also exhibiting proactive behaviors such as issuing reminders or comments based on its continuous understanding of the live scene. The key technique behind MiniCPM-o 4.5 is Omni-Flow, a unified streaming framework that aligns omni-modal inputs and outputs along a shared temporal axis. This formulation converts conventional turn-based interaction into a full-duplex, time-aligned process, enabling simultaneous perception and response and allowing proactive behavior to arise within the same framework. With a total of 9B parameters, MiniCPM-o 4.5 approaches Gemini 2.5 Flash in vision-language capabilities, delivering state-of-the-art open-source performance at its scale. It also surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and delivers better speech generation, with significantly higher computation efficiency. Driven by its efficient architecture design and inference optimization, the model can perform real-time full-duplex omni-modal interaction on edge devices with less than 12GB RAM cost.
1 Introduction
MiniCPM-o 4.5 addresses the limits of turn-based, reactive multimodal interaction with real-time full-duplex streaming and proactive behavior. Its Omni-Flow framework aligns multimodal inputs and outputs temporally, while the 9B model delivers strong vision-language and omni-modal performance efficiently on edge devices.
- Motivation: Current MLLMs still separate perception and response into alternating phases, limiting continuous adjustment during interaction.The introduction identifies the interaction paradigm, rather than modality coverage or response latency alone, as the main challenge toward human-like interaction.
- Motivation: Human-like interaction requires token-level coupling of perception and response, enabling listening, watching, speaking, and writing in parallel.The proposed direction also replaces passive, turn-based response generation with continuous and proactive interaction driven by context.
- MiniCPM-o 4.5: MiniCPM-o 4.5 enables simultaneous real-time seeing, listening, and speaking, while issuing reminders or comments from continuous live-scene understanding.Figure 3 contrasts this full-duplex behavior with blocked information flow and passive responses in turn-based paradigms.
- Omni-Flow: Omni-Flow aligns multimodal inputs and outputs along a shared temporal axis, converting distinct interaction turns into unified streaming.This shared temporal formulation supports continuous perception during speech, real-time response updates, and proactive action.
- Results and contributions: 9B parameters enable MiniCPM-o 4.5 to approach Gemini 2.5 Flash in vision-language capabilities while running on edge devices with less than 12GB RAM.The model also supports switching between full-duplex omni-modal streaming and traditional turn-based interaction.
2 End-to-End Omni-Modal Architecture
MiniCPM-o 4.5 uses an end-to-end omni-modal architecture for both full-duplex Omni-Flow interaction and conventional turn-based inference. Its encoders, LLM backbone, and speech decoders connect through token-level hidden states and align input-output streams on a shared millisecond-level timeline.
- Architecture: The architecture has streaming multimodal encoders, a Qwen3-8B LLM backbone for omni-modal understanding and text generation, and speech decoders.These components are connected through token-level hidden states in an end-to-end trainable design.
- Visual Encoding: Visual inputs use LLaVA-UHD image partitioning, with 448×448 maximum resolution in full-duplex streaming mode and 2240×2240 otherwise.Each slice is encoded into 1024 tokens by a 0.4B SigLIP ViT and compressed into 64 tokens by a resampler.
- Audio Encoding: Audio is encoded chunk-by-chunk by a 0.3B Whisper Medium encoder, producing 50 feature tokens per second and 10 audio tokens per second after 5× temporal compression.A two-layer MLP projector performs the temporal compression to reduce the LLM token budget.
- Text Decoding: The LLM generates text and speech-generation hidden states at 3-4 decoding steps per second, matching human speech speed during real-time full-duplex interaction.This avoids directly generating speech tokens at typically about 25 tokens per second.
- Speech Generation: A lightweight ∼0.3B Llama speech token decoder combines reshaped LLM hidden states with decoder states for S3 token generation, followed by streaming flow-matching waveform synthesis.The waveform decoder synthesizes audio based on reference audio in the multimodal system prompt.
3 Omni-Flow
Omni-Flow converts turn-based interaction into a time-aligned full-duplex process by coordinating continuous visual, audio, and output streams. Its design choices and TAIL speech generation strategy improve responsiveness, stability, and synchronization with the evolving environment.
- Omni-Flow framework: Omni-Flow coordinates omni-modal input and output streams along a shared temporal axis, enabling simultaneous perception and speaking instead of alternating phases.It partitions continuous interaction into fine-grained time windows inspired by time-division multiplexing.
- Omni-Flow framework: The framework represents live visual observations, acoustic context, and assistant outputs as three continuously evolving, time-aligned streams.User requests enter primarily through env-audio as part of the continuously observed world state.
- Omni-Flow framework: Time-aligned visual, audio, and output tokens are grouped per chunk and serialized into a unified sequence for a standard causal language model.When no output is produced, the output sequence contains only a special [listen] token.
- Design ablations: A 1.0 s chunk provides the best balance because shorter chunks improve responsiveness but leave insufficient modeling capacity for stable decisions and coherent generation.The ablations also show that explicit group boundaries help, while separating speech control from content generation improves stability.
- Time-Aligned Interleaving (TAIL): TAIL adaptively interleaves text and speech so playback approaches the current interaction-time boundary without allowing text to run substantially ahead.A bounded look-ahead defers the last few text tokens in each chunk, supplying pronunciation and prosody context while preserving timely speech generation.
4 Data
MiniCPM-o 4.5 is trained on diverse speech, vision-language, text, and full-duplex multimodal data designed for broad capability coverage, high-quality generation, and real-world interaction. Its full-duplex samples time-align visual and audio inputs with text and speech outputs, while targeted task data supports continuous scene description and proactive reminding.
- Speech Data: Millions of hours of diverse unlabeled speech provide training data for zero-shot TTS, ASR, and multi-turn, multi-speaker dialogue.The corpus spans varied speakers, accents, and conversational patterns.
- Speech Data: Professional voice-actor recordings create colloquial spoken dialogues with varied emotion, speaking rate, and emphasis under consistent vocal identities.Dialogues originate from diverse seed queries and balance structured content with improvised expression.
- Vision-Language Data: Vision-language data expands MiniCPM-V 4.5’s data system across broader task types and real-world scenarios through improved captions, relevance filtering, document masking, and response curation.Additional data includes dense video captions for temporal events, human actions, and scene transitions, plus text-only instruction data for linguistic robustness.
- Full-Duplex Data: Full-duplex training samples contain visual input, audio input, output text, and output speech, with every component tagged by a time index.The dataset combines large-scale web data with a smaller set of high-quality instruction samples.
- Full-Duplex Data: Manually constructed full-duplex task scenarios support precise interaction capabilities including continuous scene description and proactive reminding.Large-scale web audio-video data is filtered for audio-visual relevance and quality by removing misleading shortcuts and noisy segments.
5 Training
MiniCPM-o 4.5 uses a staged training pipeline that progressively integrates speech while preserving pretrained visual and linguistic capabilities. Joint multimodal training, instruction tuning, and reinforcement learning then strengthen full-duplex interaction, instruction following, efficiency, and hallucination mitigation.
- Staged training: The pipeline progressively integrates speech into MiniCPM-V 4.5 while retaining individual-modality advantages and enabling efficient cross-modal generalization.The design emphasizes smooth and stable integration across modalities.
- Staged training: MiniCPM-o 4.5 combines pretrained Whisper and MiniCPM-V 4.5 components with randomly initialized speech modules, freezing pretrained components while training the additions.The added modules include an audio projector, an LLM-to-speech projector, and a speech decoder.
- Joint pretraining: The second stage unfreezes all parameters for balanced joint pretraining on vision-language, speech, and omni-modal data, including timeline-aligned proactive and full-duplex interactions.Different modality combinations are assigned to data-parallel ranks to maintain a fixed data ratio at every training step.
- Instruction tuning: Supervised fine-tuning uses large-scale instruction tuning followed by human-annotated refinement, with variable resolutions and frame rates supporting inference quality-efficiency trade-offs.The maximum frame resolution is randomly set to 0.2–0.4 megapixels.
- Reinforcement learning: Reinforcement learning combines GRPO for reasoning and instruction following with length, general-quality, and RLAIF-V rewards to improve efficiency, answer quality, code-mixing, and visual hallucination mitigation.The length reward is omitted for the first 480 training steps, while image-text hallucination mitigation transfers to omni-modal full-duplex interaction.
6 Evaluation
MiniCPM-o 4.5 is evaluated across vision-language, speech, text, and omni-modal streaming capabilities, achieving strong results across these areas. Its 9B-parameter model performs competitively with larger systems while supporting full-duplex interaction.
- Evaluation Setup: The evaluation spans four groups: vision-language understanding, speech understanding and generation, text capability, and omni-modal streaming interaction.Vision-language tests cover STEM and general reasoning, documents and OCR, multi-image reasoning, hallucination, and video understanding.
- Text Capability: MiniCPM-o 4.5 outperforms its Qwen3-Instruct-8B backbone in most text-only tasks, including reasoning, mathematics, coding, and instruction following.The results indicate that balanced textual and multimodal data preserves core text capabilities while adding multimodal capabilities.
- Vision-Language Understanding: 77.6 and 78.2 are MiniCPM-o 4.5’s average OpenCompass scores in instruct and thinking modes, respectively.With 9B parameters, it outperforms similarly sized InternVL3.5-8B and Qwen3-VL-8B, larger Qwen3-Omni-30B, and approaches Gemini 2.5 Flash and GPT-5.
- Vision-Language Understanding: MiniCPM-o 4.5 achieves the best document-parsing performance and outperforms all baselines on Mantis-Eval and MMSI-Bench.It is strong on English and Chinese OmniDocBench, on par with top-tier models on OCRBench, TextVQA, and DocVQA, and competitive on MUIRBench.
- Speech Understanding and Generation: MiniCPM-o 4.5 leads on CoVoST 2 en→zh, MELD, VoiceBench AlpacaEval, and Speech TriviaQA while remaining close to leading ASR systems.It achieves the lowest CER/WER on SeedTTS Test-ZH and SeedTTS Test-EN, performs best on Expresso and ESD, and improves long-form English generation on LongTTS.
- Omni-modal and Streaming Interaction: 54.4 is MiniCPM-o 4.5’s LiveSports-3K-CC win rate, exceeding LiveCC and StreamingVLM by 12.9 and 8.8 points, respectively.It also achieves the best results on five of seven omni-modal benchmarks: Daily-Omni, WorldSense, Video-Holmes, JointAVBench, and AVUT-Human.
7 Efficient Real-Time Inference
MiniCPM-o 4.5 delivers higher throughput and lower memory usage than Qwen3-Omni-30B-A3B under vLLM on a single NVIDIA RTX 4090. The llama.cpp-omni framework further supports efficient full-duplex streaming across multiple hardware platforms and operating systems.
- vLLM inference efficiency: 154.3 tokens/s with 19 GB memory usage: MiniCPM-o 4.5 runs in BF16 on a single NVIDIA RTX 4090, while Qwen3-Omni-30B-A3B runs out of memory.These results are reported under the standard vLLM setting.
- vLLM inference efficiency: 212.3 tokens/s: MiniCPM-o 4.5 achieves higher INT4 throughput, lower first-token latency, and nearly half the memory usage compared with Qwen3-Omni-30B-A3B.The comparison is conducted on a single NVIDIA RTX 4090 under vLLM.
- Streaming deployment: llama.cpp-omni enables smooth full-duplex streaming execution across multiple hardware platforms.The framework is tailored to MiniCPM-o 4.5’s streaming interaction paradigm.
- Streaming deployment: Compatibility is validated across macOS, Windows, and Linux, alongside a lightweight demo system.These deployment provisions extend beyond runtime efficiency to operating-system support and demonstration.
8 Conclusion
MiniCPM-o 4.5 is a 9B open-source MLLM for real-time full-duplex omni-modal interaction, enabled by Omni-Flow’s shared temporal alignment of multimodal inputs and outputs. The conclusion also notes limitations in robustness, speech-generation stability, and real-world streaming validation.
- Contributions: MiniCPM-o 4.5 is a 9B open-source MLLM for real-time full-duplex omni-modal interaction, continuously perceiving visual and auditory streams while generating speech responses.It requires less than 12GB RAM during deployment.
- Contributions: The model approaches Gemini 2.5 Flash in vision-language capabilities and delivers frontier image and video understanding performance among open-source MLLMs at this scale.These capabilities are reported alongside practical edge efficiency.
- Contributions: Omni-Flow aligns multimodal inputs and outputs along a shared temporal axis, providing a general formulation for full-duplex and proactive multimodal interaction.It is presented as the key technique behind MiniCPM-o 4.5.
- Limitations: Foundation capability and robustness in long, dynamic real-world streaming interactions still require further improvement and validation.The paper characterizes MiniCPM-o 4.5 as an early exploration of real-time full-duplex omni-modal interaction.
- Limitations: Speech generation in omni-modal streaming mode can occasionally be unstable, including mispronunciation or unintended mixing between English and Chinese.This instability is identified as a limitation of the current system.
9 Appendix · A Model Configuration
The appendix specifies MiniCPM-o 4.5’s architectural configuration, including component-level hyperparameters, a total of 9.34B learnable parameters, and bfloat16 precision.
- A Model Configuration: 9.34B learnable parameters comprise the full model.This count refers to the full model’s learnable parameters.
- A Model Configuration: bfloat16 precision is used by the full model.The stated precision applies to the full model configuration.
- A Model Configuration: Table 13 lists the architectural hyperparameters for each model component.The table is titled “Architectural hyperparameters of MiniCPM-o 4.5.”