PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

Yi Wang, Tianzi Wang, Peter Bell

arXiv:2610.38960v1cs.AIcs.CLeess.AS

TL;DR

Fine-grained AVQA requires grounding answers in sparse, cross-modal video evidence, and it remains unclear whether agents need broad specialized toolsets. The paper progressively prunes and consolidates tools into a two-tool workflow; across three benchmarks, it reports comparable-or-better accuracy than OmniAgent with substantially lower tool use and estimated cost.

  • Problem

    It remains unclear whether strong agentic AVQA requires broad specialized toolsets or can retain accuracy with fewer, consolidated tools.

    Source: §1 · p. 1

  • Method

    The authors use staged controlled experiments to prune specialized-tool OmniAgent and consolidate its capabilities into omni-modal captioning and question-answering tools.

    Source: §1 · p. 2

  • Results

    The two-tool agent achieves comparable-or-better accuracy than OmniAgent across Daily-Omni, AVUT, and WorldSense, with 72–79% fewer tool calls and 72–75% lower estimated inference cost.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The ablations indicate that broad audio-visual context and question-directed inspection capture most required capabilities, rather than an extensive tool count.

    Source: §7 · p. 8

    Controlled ablations use only the 125-example Daily-Omni validation subset, while complete benchmark evaluations use a single run, limiting confidence in small configuration differences.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Fine-grained audio-visual question answering (AVQA) requires models to resolve complex cross-modal correspondences and temporally grounding from sparsely distributed evidence. Omni-modal agents, which autonomously orchestrate perception tools to actively collect evidence, achieve strong performance, but it remains unclear whether active perception from both audio and visual modalities depends on broad specialized toolsets, especially audio or visual modality-specific tools. Such systems may also introduce long execution trajectories and substantial inference cost. We investigate whether comparable performance can be retained with a compact workflow exposing only essential perceptual capabilities. Starting from the specialized-tool OmniAgent, we conduct staged controlled experiments that prune the toolset and progressively consolidate modality-specific tools into omni-modal tools, motivating a two-tool agent comprising only Omni-modal captioning and Omni-modal QA. Across three widely used AVQA benchmarks, our compact agent achieves comparable or better accuracy than the specialized-tool agent while reducing inference cost by around 70%.

1 Introduction

Fine-grained AVQA must locate sparse evidence across audio and video, while broad specialized toolsets can increase execution length and cost. The paper tests whether a compact, consolidated agent can preserve performance with fewer tools.

1 Introduction

  • Fine-grained AVQA requires locating temporally sparse evidence and resolving cross-modal correspondences, while single-pass OmniLLMs may miss question-relevant evidence.These gaps motivate active perception through iterative planning, temporal localization, and external tools.Source: §1 · p. 1
  • Broad function-specific toolsets may improve evidence coverage but can also produce long trajectories, redundant reasoning and observations, and substantial inference cost.OmniAgent combines global audio captioning with speech recognition, event localization, and global and clip-level video understanding.Source: §1 · p. 1
  • The study asks whether pruning specialized tools preserves accuracy and whether modality-specific audio and visual tools can be consolidated into omni-modal tools.The authors examine these questions as OmniLLM capabilities continue to evolve.Source: §1 · p. 1
  • 78.86%, 80.57%, and 58.39% accuracy on Daily-Omni, AVUT, and WorldSense; comparable-or-better accuracy than OmniAgent with 72–79% fewer calls and 72–75% lower estimated cost.The compact agent also consistently outperforms its corresponding direct OmniLLM baseline, Gemini-2.5-Flash.Source: §1 · p. 2
  • Controlled ablations indicate that a broad modality-specific toolkit is unnecessary when global audio-visual captioning and question-directed perception retain the required capabilities.The staged study starts from specialized-tool OmniAgent and leads to complementary Omni Captioning and Omni QA tools for broad context and targeted evidence acquisition.Source: §1 · p. 2

2 Related Work

AVQA evaluates joint audio-visual perception, temporal evidence localization, and reasoning. Prior work ranges from modular captioning and reasoning to agentic evidence acquisition, but leaves open the role of specialized versus omni-modal tools.

2.1 Audio-Visual Question Answering

  • AVQA questions depend jointly on a video's auditory and visual streams, testing modality-specific perception, sparse evidence localization, temporal alignment, and cross-modal reasoning.Source: §2.1 · p. 2
  • Prior AVQA approaches include modular captioning and reasoning, question-conditioned descriptions, end-to-end OmniLLMs, and agents that iteratively acquire question-relevant evidence.Source: §2.1 · p. 2

2.2 Agentic AVQA

  • Agentic AVQA systems use modality-separated tools or omni-modal perception, but differences in backbones, training, and workflows leave the role of toolset breadth unresolved.ReAct-style systems repeatedly identify missing evidence, call a perception or retrieval tool, and update reasoning from observations.Source: §2.2 · pp. 2–3

3 Method

The method pairs a text-only planner with two omni-modal tools for broad context and question-directed evidence. Controlled, staged studies examine tool removal and modality consolidation while holding key evaluation procedures consistent.

3.1 Compact Agentic Audio-Visual Reasoning

  • A text-only planner uses Omni Captioning for timestamped broad audio-visual context and Omni QA for targeted evidence, then selects another tool or returns an answer based on interaction history.Caption-first is a soft instruction; subsequent actions remain dynamic, and the planner must answer from collected evidence when its valid-call budget is exhausted.Source: §3.1 · p. 3

3.2 Capability-Removal Study

  • The capability-removal study removes functional groups from the full C+Q+E+A+M toolkit on the same 125-question validation subset, including a minimal C+Q configuration.The groups are global audio captioning (C), question-directed audio/video answering (Q), event listing and localization (E), speech recognition (A), and video metadata (M).Source: §3.2 · pp. 3–4

3.3 Workflow and Modality-Consolidation Study

  • The modality-consolidation study stages workflow changes, replaces audio captioning and modality-specific QA with omni-modal tools, then tests whether metadata and clip-level QA are needed.Matched comparisons retain the planner, history handling, and final-answer procedure, distinguishing workflow, caption modality, QA specialization, and tool availability.Source: §3.3 · p. 4

4 Experiments

The experiments evaluate the compact agent on three AVQA benchmarks, using Daily-Omni for controlled workflow ablations and AVUT and WorldSense for cross-benchmark evaluation. They compare direct inference with a reproduced specialized-tool agent under matched protocols.

4.1 Evaluation Datasets

  • The final compact agent is evaluated on Daily-Omni, AVUT, and WorldSense, while controlled tool and workflow ablations use Daily-Omni.The full Daily-Omni benchmark has 1,197 questions; AVUT and WorldSense evaluations use 1,734 and 3,172 questions, respectively.Source: §4 · p. 4, §4.1 · p. 4

4.2 Planner and Shared Perception Backbone

  • The compact agent uses o3 as planner and Gemini-2.5-Flash as a shared perception backbone for timestamped omni-modal captioning and question answering.The planner first requests an omni-modal caption, then either seeks further evidence or returns an answer; the final system uses medium reasoning and allows at most six valid tool calls.Source: §4.2 · p. 4

4.3 OmniAgent Replication and Ablation Protocol

  • The reproduced OmniAgent retains its ReAct-style workflow and eight specialized tools, with an o3 planner and Gemini-2.5-Flash-based perception components.The replication uses high reasoning; the study isolates the reasoning-level change in an ablation.Source: §4.3 · p. 5
  • Controlled ablations use the same stratified 125-example Daily-Omni subset and matched settings, with tool-pruning configurations run three times.Full benchmark evaluations are single runs with a fixed seed; ablation results report mean and sample standard deviation.Source: §4.1 · p. 4, §4.3 · p. 5

4.4 Evaluation and Efficiency Accounting

  • The study reports exact multiple-choice accuracy and averages tool calls, tokens, and estimated API cost per question.Valid calls count executor-accepted invocations that return observations, including repeated tool calls; the final-answer round is excluded from call counts but its tokens are counted.Source: §4.4 · pp. 5–6

5 Results

The compact agent matches or exceeds the reproduced OmniAgent on overall accuracy while using substantially fewer resources. Results show especially large agentic gains on long videos and identify how tool pruning, modality consolidation, and workflow changes shape the final design.

5.1 Performance and Efficiency of the Compact Agent

  • The two-tool agent improves accuracy over reproduced OmniAgent by 1.33 points on Daily-Omni and 3.41 points on AVUT, while staying within 0.91 points on WorldSense.It reduces average calls by 72.2–78.7%, input tokens by at least 44%, output tokens by at least 63%, and estimated cost by at least 71%.Source: §5.1 · p. 6

5.2 Agent Performance by Video Durations

  • The compact agent’s advantage over direct inference reaches 9.68 points on AVUT and 16.88 points on WorldSense for videos longer than five minutes.The AVUT bucket has 31 questions, compared with 379 in WorldSense; in this duration range, the compact workflow beats OmniAgent on AVUT and remains within 0.79 points on WorldSense.Source: §5.2 · p. 6

5.3 Essential Tool-Group Selection

  • Removing audio-event and ASR tools yields the highest mean ablation accuracy, 78.93%, with C+Q+M, while removing QA drops accuracy to 62.13%.Removing global audio captioning yields 75.47%; the minimal C+Q configuration reaches 74.13%, supporting the retained metadata tool as useful for temporal grounding.Source: §5.3 · p. 6, Table 3 · p. 7

5.4 Modality Consolidation

  • Replacing audio-only captioning with omni-modal captioning and a caption-first instruction recovers accuracy to 77.87% while reducing calls to 2.397.System 6 reaches 78.40% with 1.941 calls after timestamps are included in captions; from Systems 0 to 6, accuracy rises 2.93 points while calls, output tokens, cost, and latency fall by 55.0%, 58.2%, 63.3%, and 66.9%.Source: §5.4 · p. 7

5.5 Compacting the Omni-Tool Workflow

  • System 9 retains 77.07% accuracy with 1.547 calls and 13.809k total tokens; relative to System 6, it uses 20.3% fewer calls and 39.3% less estimated cost.Removing clip-level QA in System 8 retains 77.60% mean accuracy, while adding the clip tool back to System 9 yields no accuracy gain; System 9 lies on the empirical accuracy-efficiency Pareto frontier.Source: §5.5 · pp. 7–8, Figure 2 · p. 8

6 Discussion

The discussion attributes strong performance to complementary passive and active evidence acquisition rather than a large specialized toolset. It also argues that modality consolidation must be co-designed with workflow and evidence placement.

6.1 Essential Perceptual Capabilities

  • Captioning supplies broad context while QA retrieves and verifies question-specific evidence; timestamps in omni-modal captions make metadata less useful.The discussion also describes ASR and audio-event tools as largely redundant given native audio understanding and caption prompts that request speech and salient sounds.Source: §6.1 · p. 8

6.2 Modality Consolidation and Workflow Co-design

  • Modality consolidation and workflow changes must be co-designed because compacting the pipeline can create an information bottleneck.Omni-modal captioning and a caption-first policy recover performance by placing richer evidence in the initial context; on longer WorldSense videos, adding clip QA reduces overall accuracy and underperforms in three of four duration bins.Source: §6.2 · p. 8

7 Conclusions

The study finds that a compact agent built around broad audio-visual context and question-directed inspection can retain strong AVQA performance. It matches or exceeds specialized-tool OmniAgent on two benchmarks, stays within 0.91 percentage points on WorldSense, and shows a prominent benefit on videos longer than five minutes.

7 Conclusions

  • The compact agent combines broad audio-visual context from captioning with question-directed inspection through QA, using a caption-first policy and six-call budget.The authors report that its accuracy improvement over a strong OmniLLM is prominent on videos longer than five minutes.Source: §7 · p. 8
  • 78.86%, 80.57%, and 58.39% accuracy on Daily-Omni, AVUT, and WorldSense, respectively; the agent matches or exceeds specialized-tool OmniAgent on the first two and remains within 0.91 percentage points on WorldSense.It reduces tool calls and estimated inference cost substantially relative to OmniAgent.Source: §7 · p. 8

Limitations

The study’s findings are bounded by limited repeated evaluation, reliance on one closed-source planner and perception backbone, and a tool-selection process that is not exhaustive and remains partly heuristic.

Limitations

  • Controlled ablations were repeated only on a 125-example Daily-Omni validation subset, while complete benchmark evaluations used a single run.Run-to-run variance makes small configuration differences statistically inconclusive; repeated full-benchmark evaluation would provide more reliable estimates.Source: Limitations · p. 9
  • The main experiments use only the closed-source, API-accessible planner o3 and shared perception backbone Gemini-2.5-Flash.Their changing behavior, availability, and pricing limit reproducibility and complicate separating workflow properties from model-specific behavior.Source: Limitations · p. 9
  • The joint design space of clip-level QA availability, tool-call budget, and planner reasoning effort was not searched formally or exhaustively.The authors do not claim the final configuration is globally optimal; a larger search could better characterize the accuracy-efficiency Pareto frontier.Source: Limitations · p. 9
  • Tool selection remains partly heuristic: functional groups are manually defined and configurations are chosen through staged empirical observations.This may overlook alternative tool combinations or interfaces and may depend on the initial taxonomy.Source: Limitations · p. 9

Ethical Considerations

The work evaluates existing AVQA benchmarks and does not collect human-subject data or release new video data. It identifies perception and reasoning errors as the main risks and limits its experiments to benchmark evaluation, not high-stakes deployment.

Ethical Considerations

  • The study uses existing AVQA benchmarks and neither collects new human-subject data nor releases new video data.Benchmark datasets are publicly accessible; the study relies on their creators’ curation rather than conducting an independent content audit.Source: Ethical Considerations · p. 9
  • Audio-visual perception and reasoning errors could omit evidence, hallucinate events, misinterpret speech, or misassociate audio and visual content.The authors identify possible harm in high-stakes decision-making, monitoring, or surveillance-like settings.Source: Ethical Considerations · p. 9
  • The system is evaluated only on benchmarks and is not intended for high-stakes deployment without additional validation and appropriate human oversight.The work also notes computational, financial, and environmental costs from large API-based models, while reporting reduced tool interactions, token use, and estimated inference cost relative to OmniAgent.Source: Ethical Considerations · p. 9

A WorldSense Clip-Level QA Ablation by Video Duration

The supplied passages describe the WorldSense clip-level QA analysis and implementation details for the final system, latency reporting, prompts, benchmark split, and artifact documentation.

A WorldSense Clip-Level QA Ablation by Video Duration

B.1 Model and Inference Configuration

  • The final system’s efficiency accounting includes one complete caption-generation call per question, even when the caption is cached or shared across questions.Ablation latency was measured during live rollout and includes captioning; reported settings were consistent across the three benchmarks.Source: §B.1 · p. 11
  • The planner may make at most six valid tool calls; the required first caption counts toward that budget, but the final-answer round does not.Gemini requests use a 180-second timeout, up to four attempts, and 30-second delays between attempts.Source: §B.1 · p. 11
  • Across experiments and development runs, API expenditure was approximately $500 for o3 and $100 for Gemini-2.5-Flash.Both models are closed-source, with undisclosed parameter counts and provider-side inference hardware.Source: §B.1 · p. 11

C Ablation Latency

  • Table 8 reports per-question retry-adjusted latency for Systems 0–11.The passage identifies the table’s scope but gives no latency values.Source: C Ablation Latency · p. 11

C.1 Prompts for the Proposed System

  • The final system uses ask_perception for whole-video omni-modal QA and ask_caption for whole-video omni-modal captioning.Tables 9 and 10 present the key instructions used by the system.Source: §C.1 · p. 11

D Daily-Omni Split Statistics

  • The 125-example Daily-Omni validation subset used for controlled ablations was stratified by six task types, with a distribution reported to closely match the full benchmark.The comparison is reported in Table 11.Source: D Daily-Omni Split Statistics · p. 11

E Benchmark and Artifact Documentation

  • Table 12 documents the composition, provenance, and documentation of the three evaluation benchmarks; the study uses released evaluation data and redistributes no new videos.No independent audit for personally identifying or offensive content was conducted; the study relies on benchmark creators’ curation and release procedures.Source: E Benchmark and Artifact Documentation · p. 11

F Video-Disjoint Daily-Omni Held-Out Evaluation

The held-out evaluation excludes videos represented in the Val125 development subset to assess the compact system on a video-disjoint partition. The compact system retains the highest accuracy there, with no decrease relative to its whole-benchmark result.

F Video-Disjoint Daily-Omni Held-Out Evaluation

Related papers

Loading 2610.38960v1…