Source-linked AI summary
AURA: Always-On Understanding and Real-Time Assistance via Video Streams
Xudong Lu, Yang Bo, Jinpeng Chen, Shuhan Li, Xintong Guo, Huankang Guan, Fang Liu, Dunyuan Xu, Peiwen Sun, Heyang Sun, Rui Liu, Hongsheng Li
TL;DR
Existing VideoLLMs are mainly offline, while streaming systems face challenges in unified response control and long-horizon context. AURA co-designs context management, streaming data, training, and deployment around a unified VideoLLM. It achieves state-of-the-art streaming results and runs a real-time ASR/TTS demo at 2 FPS on two 80G accelerators.
Problem
Most VideoLLMs are offline, and existing streaming approaches face limitations in unified response generation and long-horizon video-text context management.
Method
AURA is an end-to-end streaming visual interaction framework using context management, structured streaming supervision, training objectives, and deployment optimization.
Results
AURA achieves 73.1% overall accuracy on StreamingBench, surpasses MiniCPM-o-4.5 by 10.4%, and achieves 25.4% on OmniMMI.
Takeaways & Limitations
AURA supports continuous live-video observation with real-time and proactive question answering, while maintaining competitive performance on offline video understanding tasks.
Takeaways & Limitations
The evaluation excludes MiniCPM-o-4.5’s full-duplex mode because it often becomes silent and may produce irrelevant responses on video streams longer than two minutes.
Abstract
from arXiv · showhide
Video Large Language Models (VideoLLMs) have achieved strong performance on many video understanding tasks, but most existing systems remain offline and are not well-suited for live video streams that require continuous observation and timely response. Recent streaming VideoLLMs have made progress, yet current approaches often rely on decoupled trigger-response pipelines or are limited to captioning-style narration, reducing their effectiveness for open-ended question answering and long-horizon interaction. We propose AURA (Always-On Understanding and Real-Time Assistance), an end-to-end streaming visual interaction framework that enables a unified VideoLLM to continuously process video streams and support both real-time question answering and proactive responses. AURA integrates context management, data construction, training objectives, and deployment optimization for stable long-horizon streaming interaction. It achieves state-of-the-art performance on streaming benchmarks and supports a real-time demo system with ASR and TTS running at 2 FPS on two 80G accelerators. We release the AURA model together with a real-time inference framework to facilitate future research.
1. Introduction
AURA addresses the mismatch between offline VideoLLMs and live interaction by unifying continuous observation, response timing, and long-horizon context management in one streaming framework.
- Most existing VideoLLMs analyze buffered complete videos, limiting prompt responses to ongoing events in live assistants and interactive systems.
- Existing streaming systems use decoupled or unified architectures, but decoupled triggers can be inconsistent with the primary model’s contextual state and response generation.
- AURA enables one streaming VideoLLM to process frames continuously, remain silent when appropriate, and support real-time and proactive question answering.
- AURA co-designs context management, data construction, training, and deployment for stable long-horizon streaming interaction.
2. Related Work
Related work has progressed from offline VideoLLMs toward online interactive processing, but streaming systems must coordinate perception, decisions, responses, and limited context over time.
- Recent VideoLLMs improve token management, spatial-temporal modeling, and visual-language alignment for efficient video understanding.
- Streaming video understanding shifts from whole-video processing to online interaction, requiring synchronization of perception, decision-making, and response.
- AURA’s context-management mechanism addresses the streaming setting’s continuously growing video and interaction history within a limited context window.
3. Interactive Video Stream Context Management and QA Types
AURA organizes unbounded video and interaction history into a bounded streaming context, enabling asynchronous responses with distinct timing and multiplicity requirements.
- Context organization: AURA packages each video chunk with any contemporaneous question and marks response-free steps with the special <|silent|> token.
- Streaming QA types: Real-Time QA produces one immediate response, Proactive QA waits for sufficient evidence before one response, and Multi-Response QA produces multiple responses as events evolve.
- Context organization: A dual sliding-window strategy retains recent video and recent QA groups, preserving textual interaction history outside the video window.
- Context organization: The video window keeps recent N seconds, while the interaction window retains recent M QA groups, such as N=30 and M=10.
- Streaming QA types: Multi-Response QA continuously monitors ongoing events and responds without requiring repeated user queries.
- Streaming QA types: These three response categories form the foundation for data construction that teaches diverse streaming response behaviors.
4. Coarse-to-Fine Streaming Data Engine
AURA’s coarse-to-fine data engine converts videos and timestamped interactions into verified streaming supervision covering response timing, phrasing, difficulty, and contextual history.
- Pipeline overview: The data engine has five stages: Video Preparation, QA Synthesis, QA Refinement, Streaming Structuring, and Quality Verification.
- Video preparation: Videos span diverse public-source categories and are resampled to 2 FPS and re-encoded in H.264 for temporal coverage and decoding stability.
- QA synthesis: Separate synthesis pipelines handle single-response Real-Time and Proactive QA versus multiple-valid-answer Multi-Response QA.
- QA refinement: QA refinement increases Real-Time difficulty diversity and rewrites Proactive and Multi-Response questions into semantically equivalent phrasings.
- Streaming structuring: Streaming Structuring unrolls continuous interactions into timestamp-anchored samples, supervising one target answer while retaining earlier QA history as context.
- Quality verification: Quality Verification filters truncated-context samples whose answers lack visual grounding, factual correctness, temporal consistency, or hallucination-free support.
- Practical designs: The pipeline also inserts acknowledgments for delayed responses and combines multiple interaction patterns within individual videos.
5. Training and Inference Design
AURA’s training and inference design combines selective supervision, context management, multimodal streaming, and asynchronous deployment for low-latency, long-horizon assistance.
- Training objective: AURA introduces a loss that selects supervision and mitigates silent-turn bias in streaming interactions.The objective combines supervision selection with class reweighting.
- Training objective: Sliding-window samples supervise all silent assistant messages and only the last non-silent message, excluding earlier non-silent messages.This selection reflects which responses remain supported by retained video and history evidence.
- Training objective: Class balancing down-weights silent-message tokens by the inverse silence count while preserving supervision for learning when not to speak.This prevents abundant silent supervision from dominating informative response optimization.
- Inference system: The inference system combines continuous video and speech input with multimodal AURA inference, ASR transcription, and TTS speech output.Video chunks enter context independently or are paired with speech transcriptions as user messages before response generation.
- Inference system: Context truncation uses a predefined video window, while FIFO removal of old chunks can prevent reuse of previously computed KV caches.AURA, ASR, and TTS operate asynchronously alongside streaming output and resource-isolation optimizations.
6. Experiment
AURA is evaluated on streaming and offline video benchmarks, with comparisons covering accuracy, latency, and training-objective effects. It achieves the strongest reported streaming performance while preserving substantial offline capability and supporting real-time interaction metrics.
- Streaming benchmark results: AURA achieves the best overall performance on StreamingBench, OVO-Bench, and OmniMMI against open-source and proprietary baselines.The comparisons cover three streaming video benchmarks and report AURA as the strongest overall model on each.
- Streaming benchmark results: 73.1% overall accuracy on StreamingBench exceeds MiniCPM-o-4.5 by 10.4% and Gemini-1.5-Pro by 6.0%.AURA ranks first among open-source models across RTVU, OSU, and CU, and on 14 of 18 fine-grained subtasks.
- Streaming benchmark results: 65.3% overall accuracy on OVO-Bench improves over ViSpeak by 4.2% and exceeds Gemini-1.5-Pro by 2.3%.AURA is best among open-source models on RTVP and BT, while trailing the best model by 1.2% on FAR.
- Streaming benchmark results: 25.4% overall accuracy on OmniMMI surpasses all open-source and proprietary models, with first place on 5 of 9 fine-grained metrics.Non-streaming models lack Proactive Alerting capability, so PA is left blank for those models.
- Inference performance: AURA achieves the lowest average TTFT, while sliding-window pruning and prefix caching prevent latency and computed-token growth during sustained streaming.Without prefix caching, long prompt prefixes are recomputed; without the sliding window, multimodal context accumulates over time.
- Inference performance: 75.0 ms average server-side TTFT and approximately 312.2 ms end-to-end latency support real-time conversational interaction with ASR and TTS.The system also reports 7.3 ms per token, 12.6 tokens per response, and 93.0 ms average TTS first-chunk latency.
- Offline benchmark results: 58.8% on LongVideoBench, 68.1% on MVBench, and 65.1% on Video-MME indicate that streaming training largely preserves offline video understanding.AURA remains particularly close to its Qwen3-VL-8B-Instruct base model on MVBench, with modest drops on the other two benchmarks.
- Training-objective analysis: Reweighting silent messages and supervising only the final non-silent assistant message addresses excessive silence and insufficiently grounded targets.The design responds to default-loss over-generation of <|silent|> and context loss caused by sliding-window truncation.
7. Conclusion
AURA is an end-to-end streaming visual interaction framework for continuous live-video observation, real-time question answering, and proactive responses. It co-designs context management, data construction, training, and deployment to address selective silence, timely response, and long-horizon interaction.
- AURA continuously observes live video streams and supports both real-time question answering and proactive responses with a unified VideoLLM.
- Its framework introduces context management, streaming data construction, a silent-speech balanced loss, and real-time streaming inference.These components target unbounded video-text history, supervision for multiple QA settings, silent prediction behavior, and stable deployment.
- Experiments report state-of-the-art streaming-benchmark performance while maintaining competitive results on conventional offline video-understanding tasks.