PaperlayerView exact revision on arXiv

Source-linked AI summary

Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

Chang Nie, Jiaju Wei, Junlan Feng, Chaoyou Fu, Caifeng Shan

arXiv:2607.05511v1cs.CV

TL;DR

Long-horizon video agents incur costly latency from iterative reasoning and weak global context; Light-Omni uses dual contextual states for reflexive retrieval and action, outperforming M3-Agent by 2.4% accuracy with 12.1× speedup.

  • Problem

    Long-horizon video understanding lacks global context, causing retrieval to prioritize local similarity while ignoring global narrative structure.

  • Method

    Light-Omni maintains a consolidated multimodal global state and a conditioned latent state that drives actions and retrieval in a fast reflexive loop.

  • Results

    Light-Omni outperforms M3-Agent by 2.4% accuracy, with a 12.1× speedup and 2.6× lower GPU memory use.

  • Takeaways & Limitations

    Light-Omni provides a lightweight memory framework that improves the performance and efficiency of existing multimodal language models.

  • Takeaways & Limitations

    Offline memory construction remains computationally expensive for massive video streams despite being half the overhead of M3-Agent.

Abstract

from arXiv · show

Agentic video understanding equips models with long-term memory to autonomously process and respond to continuous, long-horizon multimodal streams. However, advanced video agents often rely on ``detective-style'' iterative reasoning for action control (e.g., $\mathtt{search}$) and evidence aggregation, incurring prohibitive costs and latency. We argue that such heavy reasoning primarily compensates for the lack of global context and semantic misalignment in retrieval. This paper introduces Light-Omni, a multimodal agent framework for reflexive and lightweight video understanding. It achieves this through dual contextual states that instantly build the required context in a single forward pass. First, we maintain a global state, a finite-sized multimodal script continuously consolidated from episodic memory, serving as the global context for Light-Omni. Through hierarchical merging, it preserves recent details while summarizing past events. Second, conditioned on this global context, we generate a parametric latent state that directly drives autonomous actions and produces retrieval embeddings, with minimal latency. Benefiting from this coupled design, Light-Omni achieves semantically aligned retrieval and reflexive responses while avoiding iterative reasoning. Extensive experiments validate the effectiveness of Light-Omni across multiple video benchmarks. Notably, it outperforms M3-Agent with an average 2.4% accuracy gain, a 12.1$\times$ speedup, and a 2.6$\times$ improvement in GPU memory efficiency. Furthermore, it serves as a memory system to enhance both the performance and efficiency of existing MLLMs. Project page: https://clare-nie.github.io/Light-Omni.

1. Introduction

Long-horizon video understanding is hindered by finite context, fragmented memory, and retrieval misalignment, prompting heavy iterative reasoning. Light-Omni instead uses global context and dual contextual states to enable reflexive responses and semantically aligned retrieval.

  • Motivation: Finite context windows make MLLMs effectively amnesiac on continuous, long-horizon video streams, motivating memory-augmented agentic video understanding.External databases or evolving memory banks compensate for missing context through retrieval.
  • Problem: Similarity-based retrieval over segmented clips with coarse metadata can lose temporal coherence and overlook global narrative structure.Short-term memory and sparse sampling break temporal coherence, while retrieval becomes a myopic hunt without global context.
  • Problem: Heavy reasoning mainly compensates for missing explicit global context and misalignment between query and memory distributions.Existing memory systems use fragmented substitutes rather than holistic, persistent representations of past observations.
  • Contribution: Light-Omni transitions video understanding from multi-step reasoning to reflexive response generation by maintaining global context and directly learning retrieval embeddings.The framework is motivated by context-driven memory evocation and avoids explicit retrieval intermediaries such as rewrites, conditions, and keywords.
  • Contribution: Dual contextual states bridge query–memory semantic gaps, enabling accurate and robust retrieval under noisy inputs without iterative reasoning.Light-Omni is also designed as a general memory system that integrates with existing MLLMs to improve efficiency and performance.

2. Related Work

Related work progresses from unified omni-modal models to retrieval- and memory-augmented video understanding, then to agentic systems using active reasoning and tool execution. However, agentic workflows increase latency and compensate for imperfect retrieval rather than resolving semantic misalignment between queries and memory representations.

  • Omni-Modal Models: Unified omni-modal models process interleaved video, audio, and text streams end-to-end, spanning open-source Qwen-Omni and proprietary Gemini series.These models represent a shift from vision-centric designs toward unified omni-modal architectures.
  • Retrieval- and Memory-Augmented Video Understanding: Retrieval-augmented methods address finite context windows in long-horizon video understanding using compressed frame features, sparse textual metadata, clip summaries, and ASR transcripts.Early approaches maintain static memory banks following retrieval-augmented generation paradigms, while later work aims to preserve temporal structure and event continuity.
  • Agentic Video Understanding: Agentic video methods replace passive retrieval with goal-oriented reasoning that decomposes queries, refines search intents, and iteratively invokes tools for evidence aggregation.Typical tools include ASR, search, and grounding.
  • Agentic Video Understanding: Agentic multi-step interaction increases latency and treats reasoning as compensation for imperfect retrieval instead of directly resolving query–memory semantic misalignment.This latency makes highly responsive agentic interactions difficult in practice.

3. Light-Omni Framework

Light-Omni uses dual contextual states and structured multimodal long-term memory to separate global-context consolidation from fast reflexive action and retrieval. Its hierarchical memory and latent-state design supports coherent, semantically aligned video understanding with low-latency inference.

  • Overview: Light-Omni decouples incremental global-context consolidation from real-time reflexive responses using dual contextual states and multimodal long-term memory.The system produces actions from the current context while updating memory after responding.
  • Multimodal Memory: Its memory stores omni-modal streams and interactions while supporting structured CRUD operations for dynamic maintenance and efficient retrieval.The memory system includes user profile, semantic, and episodic memory components.
  • Latent State: The latent state uses soft prompts and a single forward pass to generate semantically aligned retrieval embeddings and independently evaluate whether to act.A projection head rectifies the retrieval query in continuous latent space, enabling action control and retrieval together.
  • Global State: Light-Omni’s global state hierarchically merges episodic scripts so recent fine-grained details coexist with distant high-level summaries within a bounded context window.The merging strategy progressively summarizes historical context while preserving temporal continuity and coherence.
  • Efficiency Optimization: 42% cumulative latency reduction results from feature caching and redundancy pruning without compromising accuracy.The reported optimization instead yields a slight accuracy improvement.

4. Experiments

Across long-video benchmarks, Light-Omni delivers strong accuracy, robustness, and efficiency while also improving existing MLLMs as a multimodal memory system. Its hierarchical dual-state design supports scalable, low-latency interaction and effective long-term memory.

  • Long-video performance: 58.0% average accuracy on VideoMME-long and LVBench, with Light-Omni outperforming major commercial and open-source MLLMs.It exceeds GPT-4o at 48.1%, Gemini-2.0-Flash at 55.8%, and Qwen2.5-VL-72B at 56.1%.
  • Memory-system transfer: 4.9%, 2.5%, and 3.8% average accuracy gains for Qwen2.5-VL-7B, Qwen3-VL-8B, and Gemini-2.0-Flash, respectively.On LVBench, integrating Light-Omni with Qwen2.5-VL-7B accelerates inference by up to 7.2× and reduces memory footprint by 2.5×.
  • Retrieval robustness: 1.3% performance degradation under audio noise, compared with 5.1% for RAG and 3.7% for RAG-Rewrite.Light-Omni also produces sharper query-to-memory similarity matrices, indicating stronger semantic alignment under noisy inputs.
  • Latency and scalability: 11.36× lower overall system latency through single-call response generation, retrieval prefill, and decoupled memory consolidation.This design reduces online interaction overhead relative to M3-Agent while keeping offline memory construction substantially lighter.
  • Memory design and scalability: 66.10% accuracy for hierarchical Sg, outperforming Vanilla and Uniform by 2.99% and 1.99%, respectively.The default memory configuration uses Ns = 12 and Ne = 4, while six months of simulated interaction constrain global-state topics to three times a one-hour session and latency averages 2.37 seconds.

5. Conclusion

Light-Omni shifts agentic video understanding from slow, deliberate reasoning to fast, reflexive action through dual contextual states. Its versatile memory framework also enhances existing MLLMs for more responsive, practical multimodal agents.

  • Conclusion: Light-Omni shifts agentic video understanding from slow, deliberate reasoning to fast, reflexive action.The framework is presented as a new paradigm for agentic video understanding.
  • Conclusion: Dual contextual states bridge the semantic gap between queries and memory representations while enabling global coherence and semantically aligned evidence retrieval.The states provide non-parametric global coherence and semantically aligned retrieval.
  • Conclusion: Light-Omni’s versatile memory framework enhances existing MLLMs for more responsive and practical multimodal agents.The memory framework extends beyond video assistance to improve multimodal-agent responsiveness and practicality.

Supplementary Material for … A.3. Memory Capabilities Supported by Light-Omni

The supplementary material expands Light-Omni’s multimodal memory system, covering its components, storage and retrieval mechanisms, and supported memory-driven capabilities. It describes profile, semantic, and episodic memory alongside context-dependent retrieval and low-latency processing for continuous streams.

  • Supplementary Material for: The supplementary material complements the main paper with details on the multimodal memory system, training data, training procedures, online inference, and additional experiments.It is organized across Appendices A–D.
  • A. Multimodal Memory System: Light-Omni’s multimodal memory system consists of three distinct components with dedicated storage formats and update mechanisms.The components are elaborated in the supplementary memory-system sections.
  • A.1. Memory Storage and Management: User Profile memory stores identity information, preferences, and behavioral traits extracted from multimodal streams for personalized experiences.For visual input sampled at 1 fps, Buffalo_s7 performs face detection and recognition, with feature-level tracking supporting identity consistency.
  • A.1. Memory Storage and Management: Semantic Memory appends timestamped factual knowledge, relationships, and abstract concepts derived from generated topics while filtering redundant content.Each entry uses the format “[start_timestamp]-[end_timestamp]: content”.
  • A.1. Memory Storage and Management: Episodic Memory chronologically archives past events and raw multimodal inputs through narrative-style incremental additions that preserve temporal continuity without repetitive frames.Entries derived from generated topics include comprehensive contextual logs, though the supplied passage truncates the full list.
  • A.1. Memory Storage and Management: Unified identifiers such as <face_idx> establish cross-references and semantic connections across memory modules while maintaining global identity consistency.Table 5 provides concrete examples of the three memory types.
  • A.2. Memory Retrieval and Utilization: Runtime retrieval selects profiles for faces detected in the current visual input, whereas Semantic and Episodic Memory retrieval activates conditionally when the speech action is triggered.Memory entries are encoded into dense embeddings using Qwen3-Embedding-0.6B.
  • A.3. Memory Capabilities Supported by Light-Omni: Light-Omni supports five primary memory-driven capabilities and uses silero-vad to cap segments at 30 seconds for low-latency, proactive interaction over long horizons.The supplied passage begins describing historical visual and auditory retrieval but truncates the remaining capability list.

B. Training Dataset Construction · C. Optimization Details and Runtime Pipeline

Light-Omni addresses the lack of continuous, long-horizon interactive video data with a four-step synthesis pipeline that produces 46,166 samples from 2,118 videos. The dataset trains separate Memory, Generation, and Reaction adapters, with most samples allocated to agentic memory and reaction capabilities.

  • B. Training Dataset Construction: The dataset synthesis pipeline targets the absence of continuous, long-horizon interactive contexts in existing open-source video datasets.It begins by collecting public videos and randomly grouping them to simulate prolonged, multi-session user interactions.
  • B. Training Dataset Construction: 46,166 training samples are derived from 2,118 unique videos.The resulting dataset is partitioned into three subsets for training distinct capabilities.
  • B. Training Dataset Construction: The training dataset is heavily weighted toward the Memory and Reaction adapters to cultivate agentic behaviors.Figure 10 depicts the distribution across the three adapters.
  • B. Training Dataset Construction: The Memory Adapter uses 10,493 samples for topic generation, context merging, and profile updating.Its breakdown is 9,046 topic-generation, 648 context-merging, and 799 profile-updating samples.
  • B. Training Dataset Construction: The Generation Adapter uses 4,785 samples for fundamental question answering.These comprise 4,380 open-ended QA samples and 405 multiple-choice samples.
  • B. Training Dataset Construction: The Reaction Adapter uses 30,888 samples for latent state generation, speech actions, search actions, and retrieval optimization.Its breakdown is 10,798 speech-action, 4,637 search-action, and 15,453 retrieval-optimization samples.

C.1. Training Details … D.1. Online Video Understanding

Light-Omni uses independently optimized LoRA adapters and an asynchronous runtime pipeline that separates reflexive response generation from memory consolidation. On OVO-Bench, it achieves 54.51% average accuracy, outperforming Dispider and Flash-VStream.

  • C.1. Training Details: C.1. Training Details: Light-Omni adapts the Qwen2.5-Omni-7B backbone by independently optimizing different adapters with LoRA.The optimization pipeline uses the open-source ms-swift framework.
  • C.1. Training Details: C.1. Training Details: The LoRA adapters use rank r = 64, α = 128, dropout 0.05, and target all linear projection modules.These settings are specified for the adapter optimization procedure.
  • C.2. Runtime Pipeline: C.2. Runtime Pipeline: The pipeline asynchronously consolidates queued interactions by selectively updating the user profile and global state.Algorithm 1 specifies queue-based asynchronous memory consolidation after online processing.
  • D. Additional Experimental Results and Analysis: D. Additional Experimental Results and Analysis: Table 7 reports detailed OVO-Bench accuracy results for each task, with the best open-source results highlighted.The table is identified as the detailed evaluation on OVO-Bench.
  • C.2. Runtime Pipeline: C.2. Runtime Pipeline: An asynchronous architecture separates online preprocessing, reflexive response generation, and background memory updates.The online service performs speech activity detection and face recognition; a background process updates memory from queued interactions.
  • C.2. Runtime Pipeline: C.2. Runtime Pipeline: Light-Omni triggers reflexive actions and responses at speech end or when the silence interval exceeds 30 seconds.The resulting interaction data are pushed into a queue for lazy memory updating.
  • D.1. Online Video Understanding: D.1. Online Video Understanding: 54.51% average accuracy is achieved on OVO-Bench’s Real-Time Visual Perception and Backward Tracing tasks.The evaluation concerns Light-Omni’s online performance.
  • D.1. Online Video Understanding: D.1. Online Video Understanding: +3.05% and +25.41% are the gains over Dispider and Flash-VStream, respectively.Light-Omni is reported to significantly outperform these open-source online MLLMs.

D.2. Comparison with advanced Video Agents

Light-Omni outperforms leading proprietary, retrieval-augmented, and reasoning-based video agents across three long-video benchmarks. It delivers especially strong gains on HippoVlog while balancing high accuracy with lower computational overhead.

  • Overall comparison: Light-Omni achieves a +10.9% average accuracy gain over Qwen2.5-Omni-7B across comparisons with leading agentic baselines.The comparison includes proprietary MLLMs, RAG-based agents, and reasoning-based agentic frameworks.
  • Benchmark gains: On HippoVlog, Light-Omni improves over M3-Agent by 13.0% and WorldMM-8B by 8.8%.These gains demonstrate consistent advantages over sophisticated reasoning-based agents.
  • Efficiency: Light-Omni offers a superior balance of high accuracy and low computational overhead compared with retrieval-augmented and iterative-reasoning paradigms.The comparison highlights efficiency alongside accuracy rather than relying solely on iterative reasoning.

D.3. Ablation Analysis

The ablation study identifies distinct roles for Light-Omni’s core components: global state mainly reduces inference latency, episodic memory drives accuracy, and latent state provides a modest average gain while supporting retrieval robustness.

  • Component contributions: Removing episodic memory causes the largest accuracy drop, decreasing performance from 64.84% to 60.50%.The passage identifies episodic memory as the key driver of overall performance.
  • Component contributions: The latent state yields a modest average gain of 0.68% and plays a critical role in retrieval robustness.The supplied passage states the retrieval-robustness role but ends mid-word, so no further detail is available.

D.4. Redundancy Pruning Analysis

Light-Omni prunes redundant visual tokens at both temporal and spatial granularities. Across three benchmarks, lower retention generally preserves accuracy while reducing tokens, though higher retention increases latency linearly.

  • Pruning strategy: Redundancy pruning addresses temporal and spatial redundancy through inter-frame cosine-distance filtering and intra-frame saliency-based spatial-token selection.Inter-frame pruning compares mean feature embeddings of consecutive frame tokens, while intra-frame pruning retains the most informative tokens within each frame.
  • Performance–latency trade-off: Across three benchmarks, increasing token retention yields modest accuracy gains but causes linearly increasing response latency.The retention-ratio trade-off is illustrated in Figure 11.
  • Compression effectiveness: Over 75% of redundant visual tokens are eliminated without compromising core understanding capabilities.VideoMME-long accuracy remains highly resilient even at lower compression ratios.

E. Further Discussion · F. System Prompts for Light-Omni Framework

The discussion emphasizes realistic synthesized data, context maintenance and retrieval as the basis of Light-Omni’s impact, while noting backbone and offline-construction limitations. The appendix documents prompts for memory construction, response generation, profile updating, and global-state consolidation.

  • E. Further Discussion: Synthesized data prioritizes diverse source videos and carefully crafted in-situ dialogues with realistic audio acoustics and environmental noise.These principles aim to reflect real-world interactive scenarios and maximize realism.
  • E. Further Discussion: Light-Omni frames effective context maintenance and retrieval, rather than heavy reasoning, as the cornerstone of video understanding.Its dual contextual states support global-context construction and semantically aligned evidence aggregation.
  • E. Further Discussion: 2.5% average performance boost is achieved by using Light-Omni as a plug-and-play memory framework for Qwen3-VL-8B across VideoMME-long and LVBench.The discussion also notes that Qwen2.5-Omni-7B trails newer Qwen3 models in certain general capabilities.
  • E. Further Discussion: Light-Omni’s performance is constrained by its backbone model, and its offline memory construction remains a limitation.The passage identifies backbone capability and offline memory construction as limitations of the framework.
  • F. System Prompts for Light-Omni Framework: The appendix presents system prompts used at four framework stages: memory construction, response generation, user-profile updating, and global-state consolidation.Figures 12–15 document these prompts for reproducibility.
  • F. System Prompts for Light-Omni Framework: The memory-construction prompt assigns a Multimodal Memory Agent to synthesize inputs into a high-density log for the current window.The prompt explicitly defines the agent’s role and synthesis objective.
  • F. System Prompts for Light-Omni Framework: The memory-construction prompt supplies global memory and face profiles mapping face indices to identities as contextual inputs.These are represented by the placeholders {GLOBAL_MEMORY} and {INPUT_FACES}.
Loading 2607.05511v1…