PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Perception Agent for Omnimodal Audio-Video Understanding

Keda Tao, Wenjie Du, Bohan Yu, Weiqiang Wang, Jian Liu, Huan Wang

arXiv:2512.23646v2cs.CV

TL;DR

Fine-grained cross-modal understanding and alignment remain challenging for audio-video models and agents. OmniAgent actively orchestrates modality-specific tools, using audio-guided event localization to guide its reasoning. Across audio-video benchmarks, it reports 10–20% higher accuracy than leading open- and closed-source models without training.

  • Problem

    Fine-grained cross-modal understanding and temporal or feature alignment remain challenging, while existing agents often rely on static workflows or costly frame-captioning.

    Source: §1 · p. 2

  • Method

    OmniAgent uses an active, self-reflective agent to orchestrate callable unimodal tools and select query-relevant audio or video information, including audio-guided event localization.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    10–20% accuracy without training over leading open- and closed-source models, including Qwen3-Omni-30B (Xu et al., 2025b) and Gemini2.5-Flash (Comanici et al., 2025), across several audio-video benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    OmniAgent's active, audio-driven selection of query-relevant information addresses cross-modal alignment and fine-grained understanding in audio-video reasoning.

    Source: §5 · p. 8

    Reliance on external models and extended contexts improves performance but constrains reasoning efficiency.

    Source: §5 · p. 8

Abstract

from arXiv · show

Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often face challenges in fine-grained cross-modal understanding and have difficulty with multimodal alignment. To address these limitations, we introduce OmniAgent, to our best knowledge, the first fully active perception agent that dynamically orchestrates specialized unimodal tools to achieve more fine-grained omnimodal reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, we demonstrate a paradigm shift from passive response generation to active multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent reasoning. Extensive empirical evaluations on three audio-video understanding benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and closed-source models by substantial margins of 10% - 20% accuracy without training.

1. Introduction

OmniAgent addresses fine-grained audio-video understanding and cross-modal alignment with an active agent that dynamically orchestrates modality-specific tools and audio-guided event localization. It reports state-of-the-art accuracy on audio-video benchmarks, exceeding leading open- and closed-source models by 10–20% without training.

1. Introduction

  • Cross-modal understanding remains difficult because of temporal and feature alignment challenges, while static agent workflows limit dynamic planning for fine-grained analysis.End-to-end multimodal models also face difficult joint audio-video alignment training and limitations in fine-grained understanding.Source: §1 · p. 2
  • Frame-captioning approaches incur computational overhead and may produce query-irrelevant captions, whereas audio can provide concise temporal grounding for salient events.The paper presents audio cues as useful for event localization and more comprehensive cross-modal understanding.Source: §1 · p. 2
  • OmniAgent uses active perception and self-reflection to orchestrate tools, select audio or video, and focus on relevant spatial and temporal ranges.The approach treats strong single-modal models as callable tools and shifts from passive response generation to active information inquiry.Source: §1 · pp. 2–3
  • 10–20% accuracy without training over leading open- and closed-source models, including Qwen3-Omni-30B (Xu et al., 2025b) and Gemini2.5-Flash (Comanici et al., 2025).The paper reports best accuracy on several audio-video understanding benchmarks.Source: §1 · p. 3
  • Audio-guided event localization and a modality-specific toolkit support fine-grained cross-modal reasoning.The contributions identify both the comprehensive toolkit and the audio-guided event localization algorithm.Source: §1 · p. 3

2. Related Work

Related work covers end-to-end multimodal models and video-understanding agents. The paper identifies fine-grained audio-video understanding and cross-modal alignment as gaps in these lines of work.

2.1. Omnimodal Large Language Models

  • End-to-end models such as Qwen3-Omni (Xu et al., 2025a) provide unified multimodal perception, but face alignment-training demands and fine-grained cross-modal challenges.The related-work discussion also identifies Gemini as a powerful closed-source multimodal baseline (Comanici et al., 2025; Team et al., 2024; 2023).Source: §2.1 · p. 3

2.2. Video Understanding Agent

3. OmniAgent

OmniAgent reframes audio-video understanding as active, sequential inquiry, dynamically orchestrating modality-specific tools to gather evidence and reason across modalities. Its design combines coarse-to-fine perception with iterative planning, observation, and reflection.

3. OmniAgent

  • OmniAgent turns audio-video understanding into active, sequential decision-making by dynamically orchestrating modality-specific tools rather than using passive retrieval or rigid workflows.The paper presents it as the first active-perception agent framework for omnimodal understanding.Source: §3 · p. 3

3.1. Overview and Problem Formulation

  • The agent selects actions from the query and accumulated memory, choosing a modality and level of detail to maximize relevant information.The formulation contrasts low-cost auditory cues with high-cost visual inspection.Source: §3.1 · p. 4

3.2. Modality-Aware Expert Toolset

  • The modality-aware toolset pairs sparse global and high-resolution clip video analysis with speech transcription, audio captioning, targeted audio QA, and audio-guided event localization.The tools vary in information density and computational overhead; the event tools use audio to identify sound events for temporal localization.Source: §3.2 · p. 4

3.3. Agentic Design

  • In the Think-Act-Observe-Reflect loop, the agent plans, executes a selected tool, updates memory with the observation, and reflects before continuing or answering.Reflection checks evidence sufficiency and cross-modal discrepancies; the agent can use audio for event localization and precise visual localization when needed.Source: §3.3 · p. 5
  • Planning dynamically selects a listen-or-watch strategy and tool parameters, including sub-questions or a target video time window.The agent uses accumulated contextual memory to support multi-step reasoning.Source: §3.3 · p. 5

4. Experimental Results

OmniAgent leads the reported audio-video benchmarks, while the analyses describe how tool choices and reasoning behavior support fine-grained understanding. The experiments evaluate benchmark performance, reasoning patterns, and component choices.

4.1. Experimental Settings

  • Evaluation covers Daily-Omni, OmniVideoBench, and WorldSense, with comparisons against open-source and closed-source models and other agents.The benchmarks cover short-form, long-form, and medium-length video settings.Source: §4.1 · p. 6

4.2. Comparison with SoTA Models

  • 82.71% overall accuracy on Daily-Omni exceeds Gemini-2.5-Flash-Thinking (72.7%) and Qwen3-Omni (72.08%).The paper also reports 10%-20% gains over competing agent-based architectures.Source: §4.2 · pp. 6–7
  • On OmniVideoBench, OmniAgent reports 59.1% overall accuracy and substantially surpasses end-to-end baselines; it also leads on WorldSense.The reported 59.1% is for OmniVideoBench; the passage gives no WorldSense accuracy value.Source: Table 2 · p. 6, §4.2 · p. 7, Table 3 · p. 7

4.3. Analyses on Reasoning Behaviors

  • Across backbones, agents first use TAGC (AGC) for global audio context and culminate with TVCA (VCA) for fine-grained visual evidence.With o3, TELO localizes events during intermediate reasoning, while TVCA and TAQ are favored for final resolution; GPT-4o and o4-mini often converge rapidly or favor one modality.Source: §4.3 · pp. 7–8

4.4. Ablation Study

  • Gemini-2.5 is reported to have stronger event temporal grounding than earlier Gemini iterations and open-source alternatives; using Gemini 2.5-Flash across tools yields better reasoning accuracy.The discussion identifies audio-visual temporal grounding as a future research direction.Source: §4.4 · p. 8
  • Removing Video Clip QA prompts repetitive reliance on Global QA and failure on fine-grained details; removing Audio QA and event localization causes a substantial performance decline.The paper describes Video Clip QA as indispensable for maintaining accuracy.Source: §4.4 · p. 8, Table 5 · p. 8

5. Conclusion

OmniAgent uses active tool orchestration and audio-driven event localization for audio-visual reasoning. The paper reports better performance than existing open- and closed-source OmniLLMs, while noting efficiency constraints from external models and extended contexts.

5. Conclusion

  • OmniAgent uses a recursive Think-Act-Observe-Reflect loop to accumulate multimodal evidence progressively.It actively orchestrates tools and departs from conventional static workflows.Source: §5 · p. 8
  • Audio-driven event localization helps the agent select query-relevant information across modalities to address alignment and fine-grained understanding.Source: §5 · p. 8
  • Reliance on external models and extended contexts improves performance but constrains reasoning efficiency.The authors envision training an omnimodal agentic model as future work.Source: §5 · p. 8

Impact Statement

The paper acknowledges privacy and data-sensitivity implications for in-depth analysis of real-world audio and video. Its evaluation is confined to open-source benchmark datasets, and the system prioritizes transparency and interpretability.

Impact Statement

  • In-depth analysis of real-world video and audio streams raises privacy and data-sensitivity considerations.Source: Impact Statement · p. 9
  • Evaluation is strictly confined to open-source benchmark datasets.The passage names Daily-Omni, WorldSense, and OmniVideoBench.Source: Impact Statement · p. 9
  • The system design prioritizes transparency and interpretability in tool invocation and decision-making.Source: Impact Statement · p. 9

Release Notes

The release notes distinguish an initial technical report from a full paper. The appendix contents list implementation details, behavior analyses, efficiency analyses, case studies, future work, and prompts.

Release Notes

  • Version 1 is a technical report presenting the idea, method, major experimental results, and discussions.Source: Release Notes · p. 9
  • Version 2 is a full paper presenting all experimental results, case studies, ideas, and details.Source: Release Notes · p. 9

Appendix Content

  • The appendix covers implementation, behavior and efficiency analyses, case studies, future work, and prompts for OmniAgent and its toolset.Source: Appendix Content · p. 13

A. More Implementation Details.

The appendix reports implementation settings and costs, then examines how OmniAgent selects tools across benchmarks. Its correct answers can draw on a single modality or both, depending on the query.

A. More Implementation Details.

  • OpenAI o3 synthesizes answers from accumulated evidence when OmniAgent exceeds its reasoning-step limit; interrupted queries are re-evaluated in a final phase.The setup uses official APIs for tool models and OpenAI models, with TVGA and TVCA video sampling at 2 and 5 FPS, respectively.Source: A. More Implementation Details · p. 13
  • DVD [Zhang et al., 2025b] was tested only on Daily-Omni because longer-video API costs were unaffordable; Daily-Omni [Zhou et al., 2025] could not be cross-evaluated.The Daily-Omni benchmark uses segmented video input; a metadata tool retrieves video duration and frame rate for subsequent tool calls.Source: A. More Implementation Details · p. 13
  • $0.05-$0.11 per question is the reported OmniAgent API cost across benchmarks using OpenAI, Qwen, and Gemini APIs.Costs vary with video length and query complexity.Source: A. More Implementation Details · p. 13

B.1. Explainability of Modal Tendency Reasoning

  • Correctly resolved queries are categorized as visual-only, audio-only, or mixed according to their tool requirements across Daily-Omni, WorldSense, and OmniVideoBench.Some WorldSense questions are correctly answered using unimodal tools.Source: §B.1 · pp. 13–14
  • OmniAgent infers the context required by a query and retrieves information from the corresponding modality, while Daily-Omni and OmniVideoBench typically require analysis of both modalities.The authors describe this as query-driven modality selection and broad adaptability for omnimodal understanding.Source: §B.1 · p. 14

B.2. Reasoning Behavior Analyses in Different Benchmarks

Tool-use patterns vary by benchmark, while the overall distribution remains stable as video duration increases. WorldSense and OmniVideoBench exhibit distinct modality-use patterns.

B.2. Reasoning Behavior Analyses in Different Benchmarks

  • Longer videos correlate with more iterative inference steps, but the overall distribution of tool use remains stable across benchmarks.Figure 7 presents inference-behavior statistics across three benchmarks.Source: §B.2 · p. 14, Figure 7 · p. 14
  • For vision-centric WorldSense queries, OmniAgent typically uses a global-to-local strategy, increasing Video Global QA invocations.The global-to-local behavior is illustrated in Figure 10.Source: §B.2 · p. 14
  • OmniVideoBench shows a balanced distribution between audio-based localization and coarse-grained video localization.Figure 7 summarizes inference behavior across the benchmarks.Source: §B.2 · p. 14, Figure 7 · p. 14

C. Efficiency Analyses

The efficiency analysis reports reduced visual-token redundancy and inference latency relative to DVD, alongside lower aggregate API costs for extended videos. Case studies show audio-guided localization in short clips and iterative evidence gathering in complex ones.

C. Efficiency Analyses

  • Approximately 10% of DVD’s aggregate API cost is incurred by OmniAgent in scenarios involving extended video durations.The authors note that their design prioritizes cross-modal alignment and fine-grained understanding over inference speed.Source: C. Efficiency Analyses · p. 14
  • OmniAgent reduces visual-token redundancy and inference latency relative to DVD in the Daily-Omni comparison.Table 6 compares token consumption and latency; evaluation on other benchmarks used a randomly sampled query subset because DVD was computationally prohibitive.Source: C. Efficiency Analyses · p. 14, Table 6 · p. 14

D.1. Audio-Video Reasoning Cases

  • Complex problems can require repeated iterations to find key evidence, with audio playing a crucial role in the reasoning process.Figure 8 illustrates a long-video case.Source: Figure 8 · p. 15
  • For fundamental questions on short video clips, OmniAgent uses audio cues to temporally localize events before subsequently analyzing corresponding visual information, illustrating audio-guided event localization.Source: §D.1 · p. 15
  • In a complex case, OmniAgent establishes global audio-visual context, jointly localizes the cue “no signal,” and uses QA tools to obtain decisive evidence.The case illustrates adaptive tool invocation and audio-guided event localization.Source: §D.1 · p. 15

D.2. Unimodal-based Reasoning Cases

OmniAgent can choose a modality suited to the query rather than forcing audio into visual-only tasks. The case studies also describe a failure mode in which poor initial evidence leads to redundant tool calls.

D.2. Unimodal-based Reasoning Cases

  • For visual queries, OmniAgent uses a coarse-to-fine global-to-local strategy, beginning with Global QA and then using Clip QA to verify fine-grained details.Audio is not forcibly integrated when extraneous, and visual cues alone can support event localization.Source: §D.2 · p. 15, Figure 10 · p. 16
  • For basic audio-visual event understanding, audio temporal cues can guide visual alignment, with accurate answers reached in 2–3 reasoning steps.Figure 9 illustrates the short-video reasoning case.Source: Figure 9 · p. 16

D.3. Analyses of Failure Cases

  • In highly complex cases, insufficient initial evidence or erroneous intermediate guidance can trigger 5–7 redundant Video Clip QA steps, even when audio eventually supplies the needed information.The authors associate this trajectory with computational overhead and uncertainty.Source: §D.3 · p. 16

E. Future Work

The future-work discussion identifies computational overhead, reasoning errors, and evaluation gaps, while proposing efficiency improvements, broader tool support, and more rigorous audio-visual benchmarks.

E. Future Work

  • Iterative reasoning increases computational overhead, while stochastic decisions and upstream tool errors can propagate; future work targets tool-calling agentic models and backbone acceleration and compression.The authors also identify tool-output accuracy, efficient memory, and tool-output polishing as future research directions.Source: E. Future Work · pp. 16–17, F. ⋆Insights and More Discussion · p. 18
  • More representative, rigorous benchmarks are needed to evaluate holistic audio-visual comprehension; OmniAgent is also reported to handle audio-video asynchrony robustly through adaptive, hypothesis-driven evidence gathering.The agent adapts whether it acquires audio or visual information to the query and evolving reasoning process.Source: F. ⋆Insights and More Discussion · p. 18
  • Context-driven invocation of single-modal tools is presented as scalable to additional modality-specific tools, while the authors report greater reasoning transparency and experimental insights.The authors describe broader tool integration as a way to address a wider range of multimodal understanding tasks.Source: F. ⋆Insights and More Discussion · p. 18

G.1. Agent System Prompt

  • The system prompt requires a THINK–ACT–OBSERVE–REFLECT loop, one tool call per action, selective cross-checking, and stopping once evidence is sufficient.It also asks the agent to prefer listening before looking and to provide a concise, user-facing answer without raw tool traces.Source: §G.1 · p. 19

G.2. Agent User Prompt

  • The user prompt maps audio and video questions to specialized tools, favors audio-first temporal localization, and allows plans to change based on observations.It recommends checking video duration before selecting valid time ranges.Source: §G.2 · p. 19

H.1. Video Global QA; H.2. Video Clip QA; H.3. Audio Global Caption; H.4. Audio ASR; H.5. Audio QA

  • Tool prompts support video-wide QA and clip-level temporal reasoning, audio summaries, timestamped ASR, and audio QA for questions grounded in sound.Clip QA is restricted to what is visible within the specified time range; ASR is instructed to timestamp speech segments.Source: §H.1 · p. 20, §H.2 · p. 20, §H.3 · p. 20

H.6. Event List

  • The Event List tool is prompted to produce a contiguous, medium-granularity semantic timeline, merging sustained scenes and separating definite context shifts.An example output divides audio into timed soundscape and speech segments.Source: §H.6 · p. 21

H.7. Event Location

  • Event Location is prompted to return precise timestamps, distinguish instantaneous events from sustained ones, obey requested counts, and report when an event is absent.Its output format asks for a timestamp, context or detail, and why it matches the query.Source: §H.7 · p. 22

Related papers

Loading 2512.23646v2…