Source-linked AI summary
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, Qipeng Guo, Haodong Duan, Xin Chen, Han Lv, Zheng Nie, Min Zhang, Bin Wang, Wenwei Zhang, Xinyue Zhang, Jiaye Ge, Wei Li, Jingwen Li, Zhongying Tu, Conghui He, Xingcheng Zhang, Kai Chen, Yu Qiao, Dahua Lin, Jiaqi Wang
TL;DR
Current MLLMs face difficulty sustaining simultaneous streaming perception, reasoning, and memory, while long contexts are impractical for retaining multimodal history. IXC2.5-OL separates those functions into coordinated modules, and reports competitive audio recognition and strong video benchmark results, including open-source StreamingBench SOTA.
Problem
Current MLLMs switch between perception and reasoning, and retaining multimodal history in long context windows is impractical for continuous, long-term use.
Source: §1 · p. 2Method
IXC2.5-OL combines specialized streaming perception, reasoning, and memory models, using compressed long-term video memory to retrieve relevant clips and short-term details for reasoning.
Source: §1 · p. 2, §3.2 · pp. 4–5Results
IXC2.5-OL reports competitive audio recognition and video SOTA among open-source models, including 73.79% on StreamingBench.
Source: §1 · p. 2Takeaways & Limitations
The system is presented as supporting real-time, responsive interaction with changing multimodal environments.
Source: §5 · p. 9The authors identify reducing system latency as future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Creating AI systems that can interact with environments over long periods, similar to human cognition, has been a longstanding research goal. Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding. However, the challenge of continuous and simultaneous streaming perception, memory, and reasoning remains largely unexplored. Current MLLMs are constrained by their sequence-to-sequence architecture, which limits their ability to process inputs and generate responses simultaneously, akin to being unable to think while perceiving. Furthermore, relying on long contexts to store historical data is impractical for long-term interactions, as retaining all information becomes costly and inefficient. Therefore, rather than relying on a single foundation model to perform all functions, this project draws inspiration from the concept of the Specialized Generalist AI and introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module: Processes multimodal information in real-time, storing key details in memory and triggering reasoning in response to user queries. (2) Multi-modal Long Memory Module: Integrates short-term and long-term memory, compressing short-term memories into long-term ones for efficient retrieval and improved accuracy. (3) Reasoning Module: Responds to queries and executes reasoning tasks, coordinating with the perception and memory modules. This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time.
1. Introduction
IXC2.5-OL addresses the challenge of continuous multimodal interaction by separating streaming perception, reasoning, and memory. It reports competitive audio recognition and strong video benchmark results, including state-of-the-art performance among open-source models on StreamingBench.
1. Introduction
- Current decoder-only MLLMs alternate perception and reasoning, while storing multimodal history in long contexts becomes impractical as video streams accumulate millions of tokens within hours.The paper identifies the cost and inefficiency of retaining historical clues in context as a barrier to continuous, long-term service.Source: §1 · p. 2
- IXC2.5-OL combines specialized streaming perception, reasoning, and memory models, processing video and audio separately while storing key details and triggering reasoning for human queries.The reasoning module handles queries and reasoning tasks in coordination with perception and memory.Source: §1 · p. 2
- The memory module integrates short- and long-term multimodal memory, continuously compressing short-term memories into information-rich long-term memories for retrieval.It supports retrieval of detailed short-term information as well as long-term historical cues.Source: §1 · p. 2
- IXC2.5-OL is competitive on audio recognition and achieves video SOTA among models under 10B parameters, with 66.2% M-Avg on MLVU and 73.79% on open-source StreamingBench.It also reports 68.7% overall accuracy on MVBench, 60.6% on Video-MME, and 1.42 on MMBench-Video.Source: §1 · p. 2
- The authors released the model parameters and inference and deployment source code, including the web frontend and backend.The stated aim is to foster development of the multimodal streaming interaction community.Source: §1 · p. 2
2. Related Works
Related work spans multimodal models for image, video, audio, and omni-modal understanding. The paper situates IXC2.5-OL among online video systems while noting that its current design handles audio and video separately.
2. Related Works
- Earlier LVLMs were primarily designed for single-image, multi-round conversations, whereas recent advancements [1] [30] [148] expanded their ability to process and understand multi-image inputs.Source: §2 · p. 3
- Video-understanding methods use sparse sampling or temporal pooling [44] [66] [77] [79] [133], compressed video tokens [49] [60] [63] [119] [144], memory banks [43] [98] [100], and language bridges [45] [50] [142].Online video work targets real-time stream understanding, but the cited approaches still lack simultaneous perception, memory, and reasoning.Source: §2 · p. 3
- Audio-understanding work models audio as Seq2Seq [93], explores streaming duplex speech [78] [114] [116] [134], and extends to audio-visual [59] [96] and unified audio-visual-text models.Source: §2 · p. 3
- Omni-modal models combine inputs such as video and audio to produce text, audio [13], or omni-modal outputs; IXC2.5-OL handles audio and video separately for now.The paper says future versions will incorporate joint training across modalities.Source: §2 · p. 3
3. Method
The method separates streaming perception, memory, and reasoning into asynchronous modules. It combines real-time audio/video processing with compressed-memory retrieval to support query answering.
3.1. Streaming Perception Module
- Audio translation maps audio into sound classes and speech text, while CLIP-L/14 encodes each real-time video frame into semantic features.The audio encoder is Whisper and the SLM is Qwen2-1.8B.Source: §3.1 · pp. 3–4
3.2. Multi-modal Long Memory Module
- Short-term clip memories and global memories are compressed into high-compressed long-term video memory, reducing the reasoning module’s context burden.The memory module is designed to handle extremely long video input without placing millions of tokens in the reasoning context.Source: §3.2 · pp. 4–5
- For each query, the module aligns question features with long-term memory, selects related clips, and trains retrieval on semantics- and reference-implicit questions.Reference-implicit questions use pronouns rather than naming objects directly.Source: §3.2 · p. 5
3.3. Reasoning Module
- Reasoning combines the question with retrieved video clips and memory, while instruction prediction lets the system wait on noisy non-questions.The module decides whether a question should receive a response.Source: §3.3 · p. 5
3.4. System Pipeline
- The frontend streams audio and video through SRS to backend threads; VAD interrupts playback and snapshots memory, while retrieved memories ground LLM responses sent through TTS.The video process reads frames at one frame per second and stores extracted memory.Source: §3.4 · pp. 5–6
4. Experiments
IXC2.5-OL is evaluated on Chinese and English speech recognition and on video benchmarks, including long-video understanding and real-time visual understanding. Results include leading performance on MLVU and StreamingBench, alongside competitive or comparable results on other video evaluations.
4.1. Audio Benchmarks
- IXC2.5-OL achieves lower WER than recent streaming audio LLMs VITA and Mini-Omni across Chinese and English benchmarks with a 1.5B LLM.The evaluation uses Wenetspeech for Chinese and LibriSpeech for English; Table 3 reports WER.Source: §4.1 · pp. 6–7, Table 3 · p. 7
4.2. Video Benchmarks
- IXC2.5-OL surpasses the previous MLVU SOTA by 1.3% for Video-XL and 1.6% for GPT-4o.The MLVU evaluation covers seven multiple-choice tasks on videos ranging from 3 minutes to 2 hours; Table 4 describes SOTA performance at the 7B scale.Source: §4.2 · pp. 7, 9, Table 4 · p. 7
- On Video-MME, IXC2.5-OL performs competitively, comparable to previous SOTA MiniCPM-V 2.6.The benchmark spans six visual domains and videos from 11 seconds to 1 hour.Source: Table 5 · p. 7, §4.2 · p. 9
- IXC2.5-OL improves 2.67% over LLaVA-OneVision on StreamingBench, leading open-source models but falling just short of Gemini 1.5 Pro.StreamingBench assesses real-time visual understanding across 18 tasks, 900 videos, and 4,500 human-curated QA pairs.Source: Table 6 · p. 8, §4.2 · p. 9
- On MMBench-Video, IXC2.5-OL achieves SOTA performance on perception tasks and comparable performance on overall evaluations.The benchmark uses GPT-4-based evaluation for open-ended answers.Source: Table 7 · p. 8, §4.2 · p. 9
- Despite its 7B parameter size, IXC2.5-OL outperforms the GPT-4 series and 72B LLaVA-OneVision on MVBench temporal understanding.MVBench comprises 20 challenging video tasks that cannot be effectively addressed using a single frame.Source: §4.2 · p. 9, Table 8 · p. 9
5. Conclusion
IXC2.5-OL is a real-time streaming model with long-term memory that advances multimodal text, audio, and visual capabilities. The authors identify reducing system latency as future work.
5. Conclusion
- IXC2.5-OL advances multimodal text, audio, and visual capabilities with long-term memory.Source: §5 · p. 9
- Real-time processing supports fluid, responsive interactions with changing multimodal environments.Source: §5 · p. 9
- Future work will focus on reducing system latency for a seamless user experience.Source: §5 · p. 9