PaperlayerView exact revision on arXiv

Source-linked AI summary

Native Active Perception as Reasoning for Omni-Modal Understanding

Zhenghao Xing, Ruiyang Xu, Yuxuan Wang, Jinzheng He, Ziyang Ma, Qize Yang, Yunfei Chu, Jin Xu, Junyang Lin, Chi-Wing Fu, Pheng-Ann Heng

arXiv:2606.19341v2cs.CVcs.CLcs.SD

TL;DR

Long-video understanding remains costly because passive multimodal systems process static or fixed-window inputs exhaustively as sequences grow. OmniAgent instead uses query-driven OTA interaction to distill selected audio-visual evidence into persistent memory, achieving state-of-the-art open-source performance across ten benchmarks, including 50.5% on LVBench versus 47.3% for the 10× larger Qwen2.5-VL-72B.

  • Problem

    Passive multimodal video systems process fixed-window or exhaustive inputs, while the computational burden of high-dimensional long sequences remains a central challenge.

  • Method

    OmniAgent formulates video understanding as POMDP-based iterative OTA interaction, using persistent textual memory and Agentic SFT plus TAURA-based reinforcement learning.

  • Results

    50.5% on LVBench versus 47.3% for the 10× larger Qwen2.5-VL-72B, with state-of-the-art open-source performance across ten benchmarks.

  • Takeaways & Limitations

    OmniAgent supports active perception as an intrinsic reasoning process, with performance improving as additional reasoning turns uncover evidence for complex queries.

Abstract

from arXiv · show

Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5% vs. 47.3%).

1. Introduction

OmniAgent reframes omni-modal video understanding as native active perception: an agent iteratively gathers query-relevant audio-visual evidence and distills it into persistent textual memory. It combines Agentic SFT with TAURA-based reinforcement learning and reports state-of-the-art open-source performance across ten benchmarks.

  • Framework: OmniAgent formulates audio-visual exploration as a POMDP-driven Observation-Thought-Action cycle that selectively gathers frames, audio, and audio-visual clips.The agent distills transient percepts into persistent textual memory until sufficient evidence supports an answer.
  • Framework: Persistent textual memory decouples reasoning complexity from video duration, enabling scalable reasoning over hour-long videos.This design contrasts with passive methods that process video frames uniformly.
  • Optimization: The two-stage optimization uses Agentic SFT for best-of-N trajectory bootstrapping with dual-stage quality control, followed by TAURA-based agentic reinforcement learning.TAURA uses turn-level entropy to steer credit assignment toward pivotal discovery turns.
  • Results: 50.5% on LVBench and 71.1% on MLVU demonstrate OmniAgent’s long-video comprehension performance among open-source models.The paper reports these results as part of state-of-the-art performance across ten benchmarks.
  • Results: 50.5% vs. 47.3% on LVBench shows that the 7B OmniAgent outperforms the 10× larger Qwen2.5-VL-72B.The comparison is reported directly in the introduction’s benchmark summary.

2. Related Work

Related work spans proprietary and open-source OmniLLMs and two main trajectories for agentic video understanding: orchestrating expert modules or applying image-based transformations. These approaches overlook video’s sequential nature, motivating OmniAgent’s POMDP formulation for native agentic capabilities and test-time scaling.

  • Omni Large Language Models: OmniLLM research combines proprietary systems such as GPT-4o and Project Astra with a growing range of open-source models for audio-visual understanding.The passage identifies both proprietary and open-source contributions but notes that existing passive methods struggle with limitations described subsequently.
  • Agentic Video Understanding: Agentic video understanding follows two trajectories: LLM orchestration of expert modules using pre-extracted contexts, and video analysis through temporal clipping, spatial zooming, or combinatorial cropping.The first trajectory relies on captions or summaries, while the second adapts “Think with Image” paradigms to video.
  • Agentic Video Understanding: Both trajectories treat video as a static information container or large image, neglecting sequential structure and hampering scaling.This limitation motivates a formulation that explicitly models video understanding as sequential decision-making.
  • OmniAgent: OmniAgent formulates video understanding as a Partially Observable Markov Decision Process, injecting native agentic capabilities into MLLMs for robust test-time scaling without external modules.Its procedure alternates query-conditional active perception with memory consolidation, while maintaining constant-order media overhead.

3. OmniAgent

OmniAgent frames omni-modal video understanding as native, query-driven active perception in a POMDP, iteratively converting selected audio-visual evidence into persistent textual memory. It bootstraps and optimizes this capability with agentic supervised fine-tuning and TAURA-based reinforcement learning that assigns credit at the turn level.

  • Native Active Perception: OmniAgent separates transient multimodal percepts from persistent textual memory, decoupling reasoning complexity from raw video duration.The agent selectively distills critical audio-visual cues into text rather than processing all video inputs uniformly.
  • Native Active Perception: The agent operates in a POMDP through iterative Observation-Thought-Action cycles, with each turn selecting information needed to answer the query.Observations summarize percepts, thoughts identify information gaps, and actions retrieve frames, audio, clips, or emit the final answer.
  • Agentic Supervised Fine-Tuning: 58K trajectories across three task categories bootstrap OmniAgent through best-of-N exploration and dual-stage quality control.The corpus covers MCQ, numerical reasoning, and temporal grounding; filtering verifies task-specific correctness, including IoU ≥0.5 for temporal grounding.
  • Agentic Reinforcement Learning: TAURA addresses advantage homogenization by converting trajectory-level rewards into turn-level attributions using continuous mean-token-entropy weighting.This prioritizes information-dense turns while preserving gradient flow for all tokens, with token credit assigned through each token’s corresponding turn index.

4. Experimental Results

OmniAgent achieves strong performance across long-video, audio-visual, and temporal-grounding benchmarks through active, query-conditional perception. Its gains persist with increased reasoning turns while maintaining sampling and duration efficiency.

  • Video Understanding and Reasoning: 50.5% on LVBench lets OmniAgent-7B outperform Qwen2.5-VL-72B (47.3%) and Qwen2.5-Omni-7B (43.0%).It achieves state-of-the-art performance among open-source models on long-video benchmarks.
  • Video Understanding and Reasoning: +7.2% on VideoMME and +5.7% on MLVU over dense-sampling LongVU demonstrate gains from retrieving missing evidence through active perception.OmniAgent reaches 71.1% on MLVU, compared with Video-R1’s 60.9%.
  • Audio-Visual Understanding and Reasoning: +4.7% on DailyOmni and +7.8% on OmniVideoBench over Qwen2.5-Omni show benefits from joint audio-visual reasoning.Audio events serve as temporal anchors for targeted visual sampling.
  • Temporal Grounding: +33.4% on LongVALE and +33.0% on VUE-TR over Qwen2.5-Omni demonstrate substantial temporal-grounding improvements.On-demand sampling progressively narrows localization from coarse to fine, and OmniAgent-7B surpasses GPT-4o and Gemini-2.5-Pro on VUE-TR.
  • Test-time Scaling Analysis: 59.6% versus 53.4% on VideoMME-Long as K increases from 6 to 52 confirms monotonic positive test-time scaling.Average turns saturate at approximately 11.7 despite the nearly 9× expansion of the upper bound.
  • Visual Sampling Efficiency: 50.5% accuracy with 203 sampled frames lets OmniAgent-7B outperform Qwen2.5-VL-72B at 47.3% using 768 frames.Compared with Zoom-Zero-7B’s fixed 256-frame entry cost, OmniAgent samples strictly on demand.

5. Conclusion · A. Detailed Mathematical Notation · B. Agentic Audio-Visual Interaction Environment

OmniAgent frames omni-modal video understanding as iterative, query-driven active perception whose reasoning complexity depends on task difficulty rather than video duration. The paper also formalizes its trajectory-, turn-, and token-level notation and documents the interaction environment’s implementation protocols for reproducibility.

  • 5. Conclusion: OmniAgent treats perception as an intrinsic, iterative reasoning process for omni-modal video understanding.Its central approach is query-driven information distillation rather than exhaustive preprocessing.
  • 5. Conclusion: Reasoning complexity is driven by task difficulty rather than video duration through active perception.The framework replaces exhaustive preprocessing with iterative, query-driven information distillation.
  • 5. Conclusion: Agentic SFT bootstraps active perception, while Agentic RL with TAURA refines it.These are the two training mechanisms identified for developing OmniAgent’s native agentic capabilities.
  • A. Detailed Mathematical Notation: The mathematical notation is organized across trajectory level (i), interaction turn level (k), and autoregressive token level (t).This hierarchy structures the notation used throughout the OmniAgent framework.
  • A. Detailed Mathematical Notation: At turn k, policy πθ conditions generation on distilled memory M_k−1 and previous transient percept E_k−1.After raw media percept E_k−1 is distilled into textual observation O_k, it is purged from the active context while the causal chain is preserved via M_k.
  • A. Detailed Mathematical Notation: The active context purges raw media percept E_k−1 after distillation into textual observation O_k, maintaining context efficiency via M_k.The framework preserves the causal chain through the distilled memory rather than retaining the raw percept.
  • A. Detailed Mathematical Notation: TAURA steers the global success signal A_i toward specific interaction turns during optimization.The passage describes this as part of the framework’s index mapping and dependency structure.
  • B. Agentic Audio-Visual Interaction Environment: The interaction environment Ω is documented through distributed infrastructure and robust media processing protocols for reproducibility.These implementation details characterize the Agentic Audio-Visual Interaction Environment.

B.1. Distributed Architecture via Ray and Verl … B.5. Agent Instruction Template

The appendix details OmniAgent’s distributed Ray/Verl implementation, robust FFmpeg perception operators, randomized exploration, memory consolidation, and instruction template. Together, these components support efficient, fault-tolerant, iterative multimodal interaction while limiting persistent memory growth.

  • B.1. Distributed Architecture via Ray and Verl: Ray and Verl provide a distributed actor-based environment that improves computational efficiency and prevents out-of-memory bottlenecks during large-scale reinforcement-learning training.A detached GlobalProcessor loads tokenizer and multimodal processor state once per physical node.
  • B.1. Distributed Architecture via Ray and Verl: A remote worker pool runs independent SingleVideoQAEnv instances, enabling asynchronous perception and trajectory generation across CPU cores.
  • B.2. Robust Perception Operators (FFmpeg Implementation): Two-stage FFmpeg seeking combines keyframe-based coarse seeking with accurate decoding to resolve sensing actions into percepts with sub-second precision.
  • B.2. Robust Perception Operators (FFmpeg Implementation): Visual, auditory, and clip operators enforce fidelity and continuity through boundary fallbacks, validated 16kHz PCM S16LE audio, and superfast-to-ultrafast encoding fallback.Visual extraction uses -q:v 2; clips use libx264 with CRF 20, while ffprobe validates requested audio intervals.
  • B.3. Exploration Incentives via Randomization: Agentic SFT randomizes physical constraints over discrete grids, sampling frame counts n ∈[30, 60], clip durations dclip ∈[30, 60] seconds, and audio durations daudio ∈[150, 300] seconds.Step sizes are 2 for frame and clip parameters and 10 for audio duration.
  • B.3. Exploration Incentives via Randomization: Structured error signals, including Err.TS OOB and Err.INVALID JSON, let the agent record mistakes in persistent memory and self-correct in later turns.
  • B.4. Memory Consolidation and History Purging: Memory consolidation removes prior media after textual observations are recorded, replacing multimodal entries with metadata-preserving summaries that decouple memory costs from video duration.Visual timestamps and audio or clip temporal headers remain available through the rewritten text trace.
  • B.5. Agent Instruction Template: The instruction template governs trajectory synthesis, reinforcement learning, and inference through a step-by-step Observe-Think-Action loop with runtime limits configured by Sec. 4.

C. Empirical Analysis: Entropy as a Proxy for Reasoning Criticality

The analysis validates TAURA’s motivation by examining agent reasoning traces. It quantifies the relationship between model uncertainty, measured by entropy, and reasoning criticality.

  • Motivation for TAURA: The study analyzes the agent’s reasoning traces to validate TAURA’s underlying motivation.This analysis is designed to test the rationale for using TAURA.
  • Uncertainty Measurement: The analysis uses entropy to quantify model uncertainty during reasoning.Entropy serves as the study’s measure of model uncertainty.
  • Reasoning Criticality: The experiment examines how model uncertainty relates to reasoning criticality.The relationship between entropy and reasoning criticality is the central quantity being investigated.

C.1. Methodology: Identifying Decision Forks

The methodology identifies pivotal Decision Forks in multi-turn agentic trajectories, distinguishing steps that alter search strategy or narrow hypotheses from routine execution. It uses model-based evaluation of successful VideoMME traces to select a single Top-1 Fork Step without human bias.

  • Decision Forks are pivotal trajectory junctures that dictate subsequent search strategy or significantly narrow the hypothesis space, unlike routine execution steps.
  • Gemini-2.5-Pro evaluated successful VideoMME trajectories using the query, full interaction trace, and final outcome to identify the single Top-1 Fork Step.
  • The Top-1 Fork Step is defined as a pivotal reasoning juncture where the process diverges into multiple potential paths.

C.2. Quantitative Analysis: The Entropy Gap

The entropy gap measures how the identified Fork Step’s mean token entropy differs from the trajectory average. Most fork steps show higher uncertainty, while lower or neutral gaps occur mainly in short-horizon or easy, linear queries.

  • C.2. Quantitative Analysis: The Entropy Gap: The analysis defines the entropy gap as the Fork Step’s mean token entropy minus the trajectory’s average entropy.For trajectory i and turn k, the quantities are denoted Hi,kfork and Hi, with ∆H = Hi,kfork − Hi.
  • C.2. Quantitative Analysis: The Entropy Gap: 79.2% of identified fork steps exhibit a positive entropy difference, indicating higher policy uncertainty during critical decisions.Examples include pivoting from scanning to verification, when the agent weighs potential reasoning paths.
  • C.2. Quantitative Analysis: The Entropy Gap: 20.8% of cases have negative or neutral entropy differences, typically involving short-horizon trajectories or easy queries.These reasoning paths are generally linear and deterministic.

C.3. Case Study: Entropy at a Fork Step

The case study shows that entropy spikes identify pivotal fork steps in which the agent filters candidates and switches modalities for verification. In the analyzed trajectory, entropy rose sharply during the audio-to-visual pivot and decreased after ambiguity was resolved.

  • Trajectory Analysis: H ≈0.397 characterizes routine scanning, when the agent remains confident during standard information gathering.This low-entropy state precedes the critical fork step.
  • Trajectory Analysis: H ≈0.927 marks the fork step where audio evidence rules out options A, B, and C and prompts visual verification of American Express.The agent chooses “get frames” rather than concluding immediately, making the spike a pivotal modality-switch decision.
  • Trajectory Analysis: H ≈0.790 follows subsequent verification, indicating that resolving the ambiguity returns the agent to a lower entropy state.The trajectory concerns identifying the company shown in the video but absent from the audio.
  • Implication: Entropy spikes serve as a reliable signal for identifying high-value reasoning steps that warrant amplified reinforcement.This interpretation is supported by the entropy spike’s alignment with the candidate-filtering and modality-switching decision.
  • Quantitative Analysis: Entropy Shift Distribution: 79.2% of critical fork steps exhibit higher uncertainty than the trajectory mean, linking entropy increases to critical reasoning steps.The distribution uses entropy difference ΔH = H_i,k^fork − H_i.

C.4. Inference Runtime and Action Analysis

On a 100-sample LVBench subset, OmniAgent offers a favorable accuracy–latency tradeoff, outperforming Qwen2.5-VL-72B in both wall-clock latency and accuracy. Qwen2.5-Omni-7B is fastest in single-shot inference but substantially less accurate.

  • Inference Runtime Analysis: 66.8 s vs. 75.1 s: OmniAgent has lower wall-clock latency than Qwen2.5-VL-72B on 100 LVBench samples.The comparison uses measured inference latency in seconds.
  • Inference Runtime Analysis: 51.0% vs. 47.0%: OmniAgent achieves higher accuracy than Qwen2.5-VL-72B on the same LVBench subset.Together with its lower latency, this establishes the reported accuracy–latency tradeoff.
  • Inference Runtime Analysis: 34.8 s: Qwen2.5-Omni-7B remains fastest in single-shot inference, but its accuracy is substantially lower at 41.0%.The passage contrasts single-shot speed with accuracy.

D. Qualitative Analysis

Qualitative examples show OmniAgent solving long-video questions through iterative, targeted observation: it revisits likely intro windows, cross-checks audio and visuals, and falsifies false temporal leads before grounding answers in decisive evidence. This process identifies Nish Parkar, resolves the Kyo-yuzen ambiguity using narration, and recovers the exact gavel-strike interval [1603, 1607].

  • Active Reasoning: OmniAgent locates the green-skirt host by revisiting intro-adjacent windows, checking repeated appearances and lower-thirds, and identifies the decisive text as “Nish Parkar”.It first visually confirms the host at multiple timestamps, then switches from teaser audio to targeted intro scanning when the initial audio check is inconclusive.
  • Audio-visual Collaborative Reasoning: OmniAgent answers the Kyo-yuzen multiple-choice question by combining targeted temporal queries with on-screen text and narration, determining that “more than 20 stages” makes option A false.The other statements are confirmed through narration: fine lines, steaming, and painting on cloth.
  • Temporal Grounding: [1603, 1607] OmniAgent exactly matches the ground-truth interval for John Brassard’s adjournment and gavel-strike event.The final interval covers “I am NOT happy” through “I am adjourning the meeting” and the gavel strike.
  • Temporal Grounding: OmniAgent falsifies a suspected gavel event near ∼700s and pivots to a later visually agitated segment before confirming the precise adjournment moment.Localized probing shows no on-camera gavel strike in the early candidate windows, motivating the later search.
Loading 2606.19341v2…