PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning

Haibo Wang, Jiteng Mu, Jialu Li, Jingru Yi, Yuanjun Xiong, Jianming Zhang, Lifu Huang, Mingze Xu

arXiv:2610.02181v1cs.CV

TL;DR

Long audio-visual sequences can scatter brief sounds and visual details across modalities and time, while existing reasoning approaches may not revisit raw signals. OmniSeek makes evidence gathering an active, multi-turn process, trained with interleaved evidence trajectories and reinforcement learning that rewards reliance on both modalities. Experiments report improved performance across a wide range of audio-visual benchmarks, though extremely long videos can exceed the model’s audio-driven context capacity.

  • Problem

    Omni-LLMs passively encode entire audio-visual streams, where brief acoustic events and fine visual details can be diluted, and existing reasoning methods do not revisit raw signals.

    Source: §1 · p. 1

  • Method

    OmniSeek iteratively routes tool calls to audio or video segments and appends the retrieved raw clips, with training on OmniTraj-170K and reinforcement learning that rewards trajectories depending on both modalities.

    Source: §1 · pp. 1–2

  • Results

    OmniSeek improves performance across a wide range of audio-visual benchmarks requiring long-form and multi-hop reasoning.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results support active cross-modal evidence seeking as a way to address audio-visual reasoning over long-form and multi-hop inputs.

    Source: §6 · pp. 8–9

    For the base Qwen3-Omni-30B-A3B-Instruct, two hours of audio yields roughly 90K tokens, exceeding its 32K-token context limit.

    Source: §A.7 · p. 17

Abstract

from arXiv · show

We present OmniSeek, an agentic framework that transforms an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use. Rather than passively processing an entire audio-visual sequence in a single forward pass, OmniSeek makes evidence acquisition part of the reasoning process: it dynamically decides whether to look or listen, and over which temporal window, to retrieve sparse but critical evidence across different modalities within long contexts. Through an iterative multi-turn protocol, the retrieved raw audio or visual segments are appended back into the context to support subsequent reasoning. To cold-start this capability, we build a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with interleaved audio and visual evidence. We first supervise the model on these trajectories to instill multi-turn tool-use behavior, and then further optimize the policy via a two-stage reinforcement learning with verifiable rewards. Moreover, we introduce an Audio-Visual Necessity objective that explicitly rewards successful trajectories whose reasoning depends on both modalities, discouraging single-modality shortcuts. Extensive experiments across a wide range of benchmarks demonstrate that OmniSeek learns adaptive cross-modal evidence seeking and consistently improves audio-visual reasoning performance.

1. Introduction

OmniSeek addresses the limits of passive, single-pass audio-visual reasoning by making evidence acquisition an active, multi-turn process. It combines interleaved audio-visual reasoning trajectories with reinforcement learning designed to discourage single-modality shortcuts.

1. Introduction

  • Omni-LLMs can dilute brief sounds and fine visual details in long, passively encoded streams, while existing multimodal reasoning approaches remain text-bound and do not revisit raw signals.These limitations can leave models unable to isolate and compose available evidence, encouraging reliance on language priors or unimodal shortcuts.Source: §1 · p. 1
  • OmniSeek lets an Omni-LLM choose which modality and temporal region to inspect, append retrieved raw clips to context, and iteratively refine evidence before answering.Its loop interleaves reasoning, tool calls, and observations to gather complementary evidence across modalities and time.Source: §1 · p. 1, Figure 1 · p. 2
  • OmniTraj-170K provides multi-turn Chain-of-Thought trajectories with timestamped audio-visual evidence to teach models to interleave and reason over modalities.The corpus spans question types requiring explicit audio-visual integration.Source: §1 · p. 1
  • The Audio-Visual Necessity objective uses modality-specific attention masking to reward trajectories dependent on both modalities and discourage single-modality shortcuts without extra rollouts.Source: §1 · p. 2
  • OmniSeek learns adaptive reasoning behavior and achieves leading performance across a broad suite of audio-visual benchmarks.Source: §1 · p. 2

2. Related Work

Prior work has explored active visual and video inspection and unified omni-modal models, but existing omni-modal reasoning approaches still rely on text-based traces, coupled modalities, or single-turn retrieval. OmniSeek instead uses iterative tool interaction with independently routed audio and video evidence.

Thinking with Images/Videos

  • Image and video reasoning work adds proactive visual inspection, temporal zooming, or external tools, while Openo3-Video embeds spatio-temporal coordinates in its reasoning trace.Source: §2 · p. 2

Omni Large Language Models

  • Omni-LLMs unify multimodal perception and generation, while token-compression methods target the computational overhead of long audio-visual sequences.Source: §2 · p. 2
  • OmniTraj-170K’s data engine aligns audio-visual contexts, creates evidence-chain questions, and assembles multi-turn trajectories.Source: Figure 2 · p. 3
  • LatentOmni, OmniVideo-R1, and OmniReasoner primarily use text-based traces, coupled modalities, or single-turn retrieval; OmniSeek instead routes modalities independently through iterative tool use.Source: §2 · p. 3

3. OmniTraj-170K

OmniTraj-170K is built by aligning timestamped audio and visual annotations, generating questions from cross-modal evidence chains, and assembling multi-turn reasoning trajectories. Its trajectories span varied question types and video timescales, with localized evidence and multiple tool calls.

OmniTraj-170K construction

  • The data engine creates timestamp-aligned visual and audio contexts, then generates questions requiring both modalities with ordered evidence chains of 2 to 7 spans.Visual captions are produced over cohesive scenes of approximately 15 seconds; evidence spans specify modality, timestamp window, and text, and are arranged in logical retrieval order.Source: §3 · p. 3
  • Trajectory assembly interleaves model-generated reasoning with deterministically constructed tool calls and observations, then verifies valid calls and cross-modal evidence structure.The model generates reasoning steps that plan retrieval, respond to clips, and synthesize evidence; tool calls and observations follow the predefined evidence chain.Source: §3 · p. 4

Dataset Statistics

  • OmniTraj-170K contains 169,725 trajectories over 39,797 videos and covers 19 cross-modal question types.Videos span 122 categories and range from under a minute to tens of minutes.Source: §3 · p. 4, Figure 3 · p. 4
  • Most evidence spans last 3 to 10 seconds, and retrieved spans occur across the videos’ temporal positions.Source: §3 · p. 4
  • 76.1% of trajectories require two tool calls, while 23.9% require three or more to synthesize an answer.Source: §3 · p. 4

4. OmniSeek

OmniSeek iteratively retrieves audio and video evidence into its context, then trains the agent to use tools and depend on both modalities for successful answers.

4.1. Multi-turn Tool Protocol

  • Retrieved audio or video clips are appended to the context across turns, while separate tools enable targeted audio inspection and higher-resolution visual retrieval.The agent decides whether to retrieve more evidence or answer based on whether its accumulated evidence is sufficient.Source: §4.1 · pp. 4–5

4.2. Three-Phase Training Strategy

  • Training progresses from supervised learning on OmniTraj-170K to verifiable-reward exploration, then hard-example refinement with an Audio-Visual Necessity reward.The final phase uses 8K mined failures; successful tool use and correct answers earn a tool-use reward in Phase 2.Source: §4.2 · p. 5

4.3. Audio-Visual Necessity

  • Modality-specific attention masking estimates response-token dependence on audio and video, and the minimum modality necessity rewards joint dependence only for correct answers.Positive likelihood drops are aggregated per modality; computing the reward requires two lightweight no_grad passes and no additional rollouts.Source: §4.3 · pp. 5–6

5. Experiments

OmniSeek performs strongly across audio-visual and general video benchmarks, with especially large gains on long-form and complex reasoning tasks. Ablations and analyses examine training phases, retrieval depth, reasoning paradigms, and the OmniTraj-170K corpus.

5.1. Main results

  • 70.4% on MMOU and 44.2% on LVOmni exceed the base Qwen3-Omni-Instruct by +16.3% and +8.4%, respectively, with strong results also reported on complex audio-visual benchmarks.OmniSeek is reported as state-of-the-art or highly competitive among open-source models across a broad suite of omnimodal benchmarks.Source: §5.1 · pp. 6–7
  • 78.5% on Video-MME and 77.1% on MLVU outperform the base Qwen3-Omni-Instruct, indicating preserved and improved general video understanding.OmniSeek also exceeds the text-only OmniVideo-R1 on Video-MME, at 78.5% versus 73.6%.Source: §5.1 · p. 7, Table 3 · p. 7
  • Accuracy peaks at 3 tool calls on JointAVBench and 4 on Daily-Omni, while VideoHolmes and MMOU peak at ≥5 calls, reflecting task-dependent retrieval depth.The caption identifies Figure 5 as accuracy versus number of tool calls.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2. In-depth Analysis

  • Phase 1 SFT lowers Daily-Omni from 71.9% to 69.1% and WorldSense from 55.1% to 50.3%; Phase 2 RL recovers performance, and Phase 3 with G = 8 further improves complex-task results.The analysis reports VideoHolmes rising from 55.9% to 69.6% during Phase 2, then to 72.6% with Phase 3 at G = 8.Source: Table 2 · p. 7, §5.2 · p. 8
  • Increasing the Phase 3 rollout size from G = 8 to G = 16 raises Daily-Omni from 75.9% to 78.2% and FutureOmni from 56.4% to 58.1%.The table caption identifies P1, P2, and P3 as the SFT, Phase-2 RL, and Phase-3 RL stages, respectively.Source: Table 2 · p. 7, §5.2 · p. 8
  • The Audio-Visual Necessity reward adds +3.2% on WorldSense and +2.2% on OmniVideoTest versus standard RL, while the complete pipeline gains +8.1% on Daily-Omni and +7.3% on WorldSense over the base model.The complete-pipeline margins also include +4.1% on OmniVideoBench and +8.4% on LVOmni.Source: Table 2 · p. 7, §5.2 · p. 8
  • OmniSeek exceeds the same-two-stage-RL text-only CoT baseline by +13.5% on OmniVideoTest, +8.1% on WorldSense, and +7.0% on Daily-Omni.The comparison restricts the baseline to text generation, while OmniSeek retrieves targeted visual and audio clips.Source: §5.2 · p. 8, Table 4 · p. 8
  • A 100K OmniTraj-170K subset improves over zero-shot on LVOmni by +5.8% and Daily-Omni by +3.9%, and exceeds OmniVideo-100K on Daily-Omni by +2.1% and OmniVideoBench by +1.7%.OmniVideo-100K scores higher on OmniVideoTest (61.7% versus 59.2%), while the OmniTraj subset remains +4.7% above the base model there.Source: §5.2 · p. 8, Table 5 · p. 8

6. Conclusion

OmniSeek turns audio-visual reasoning into active, multi-turn evidence seeking rather than passive holistic encoding. The paper pairs this framework with interleaved-modality reasoning trajectories, three-phase training, and an Audio-Visual Necessity objective, and reports improvements on long-form and multi-hop benchmarks.

6. Conclusion

  • OmniSeek enables Omni-LLMs to adaptively retrieve cross-modal evidence across multiple turns instead of relying on a single holistic encoding.The framework is described as turning audio-visual reasoning from passive perception into active evidence seeking.Source: §6 · pp. 8–9
  • OmniTraj-170K supplies reasoning trajectories with interleaved modalities, alongside a three-phase training strategy and an Audio-Visual Necessity objective that discourages unimodal shortcuts.Source: §6 · p. 9
  • Experiments report improvements across a wide range of audio-visual benchmarks requiring long-form and multi-hop reasoning.Source: §6 · p. 9

A.1. More Implementation details

The appendix points to a table of implementation and hyperparameter details for the three-phase training strategy.

A.1. More Implementation details

  • Table 6 contains implementation and hyperparameter details for the three-phase training strategy.Source: §A.1 · p. 12

A.2. Benchmarks

The evaluation spans audio-visual reasoning, audio-centric understanding, long-form video, and general video benchmarks. Their descriptions specify distinct temporal, cross-modal, and task-coverage emphases.

A.2. Benchmarks

  • Daily-Omni, WorldSense, FutureOmni, and OmniVideo-Test assess temporal alignment, tightly coupled audio-video understanding, future forecasting, and long-term cross-modal dependencies.Source: §A.2 · p. 12
  • AVUT focuses on auditory information; VideoHolmes, OmniVideoBench, and JointAVBench target complex clue-based, synergistic, or strictly correlated audio-visual reasoning.Source: §A.2 · p. 12
  • LVOmniBench and MMOU cover long-form audio-video understanding, while Video-MME spans video domains and lengths and uses single-turn direct-answer evaluation without tools.Source: §A.2 · p. 12
  • LongVideoBench targets referring reasoning in long videos; MLVU spans multiple video genres and tasks, and LVBench tests long-term temporal understanding in videos of at least 30 minutes.Source: §A.2 · pp. 12–13

A.3. Visual Annotation Prompt

The visual-annotation prompt requires concrete, visual-only descriptions grounded in each segment. It also prescribes timestamps and rejects segments with overlaid speech subtitles.

A.3. Visual Annotation Prompt

  • Annotations describe concrete visible details and events, such as people, actions, objects, locations, camera focus, transitions, and readable on-screen text.The prompt excludes audio content and inferred details.Source: §A.3 · pp. 13–14
  • Each visual event receives an approximate timestamp range relative to the segment start, and events are sorted by timestamp.Use seconds; cover the event’s visible duration and use the full segment range if it persists throughout.Source: §A.3 · p. 14
  • Segments containing overlaid dialogue, lyric, or speech-transcription subtitles near the frame bottom are rejected rather than annotated.If no distinctive visual evidence exists, the prompt specifies an empty visual_events list.Source: §A.3 · p. 14

A.4. Evidence-Grounded QA Generation Prompt

The prompt generates multi-hop audio-visual QA from structured context, requiring explicit evidence chains grounded in timestamped audio and visual segments. It imposes constraints on modality use, evidence selection, and question format.

A.4. Evidence-Grounded QA Generation Prompt

  • QA generation uses structured context to create questions from 19 types, with an evidence chain whose timestamps and content come directly from Stage 1 annotations.The question, options, and answer must require both audio and visual information, not outside knowledge or language priors.Source: §A.4 · pp. 14–15
  • Segment captions are reconciled across the whole video using overall_description, since independent segment annotation may describe the same entity differently.Evidence spans must still be copied from visual_events or ASR, not from overall_description.Source: §A.4 · pp. 14–15
  • Each question has exactly four plain-text options, one correct, with plausible distractors and no answer leak.The prompt also requires answer, index, and label fields to agree.Source: §A.4 · p. 15
  • Each QA lists 2–7 evidence spans in logical retrieval order, including audio and video and at least one span that directly determines the answer.Evidence text and timestamps are taken from the corresponding ASR or visual-event annotation.Source: §A.4 · p. 15
  • Every evidence span must isolate one unambiguous event; skip a QA when a span combines distinct events relevant to the question.Spans should generally last 3–20 seconds, a guideline subordinate to the single-event requirement.Source: §A.4 · p. 15

A.5. Interleaved Trajectory Assembly Prompt

The trajectory-assembly prompt separates model-generated reasoning from tool execution, prescribing a logical retrieval order and evidence-grounded think steps. The appendix also reports benchmark breakdowns and discusses context-length limits and possible remedies.

A.5. Interleaved Trajectory Assembly Prompt

  • The assembly pipeline has the model generate reasoning while deterministically interleaving tool calls and observations from the chosen evidence spans.This restriction is intended to prevent tool-use hallucinations.Source: §A.5 · p. 16
  • The model outputs a logical permutation of all evidence spans and N+1 think strings: an opening plan, steps after retrieved clips, and a final connection from evidence to the question.The order reflects reasoning needs, not necessarily timestamp order.Source: §A.5 · p. 16
  • Think steps must refer to clips naturally, avoid indices and timestamp-based justifications, and form a grounded, coherent chain of reasoning.The prompt directs each non-first step to reflect the content just retrieved.Source: §A.5 · p. 17

A.6. Per-Benchmark Fine-Grained Results

  • The appendix reports fine-grained OmniSeek results across benchmarks using each benchmark’s official dimensions.The reported breakdowns are in Tables 7–16.Source: §A.6 · p. 17

A.7. Limitations, Discussions and Future Work

  • Extremely long videos face an audio-context bottleneck: two hours produce roughly 90K audio tokens, exceeding the base model’s 32K-token context.The stated audio tokenization rate is approximately 12.5 tokens per second at 16 kHz.Source: §A.7 · p. 17
  • When context overflows, instruction-following and formatting degrade; an 88-minute LVOmniBench failure example describes a repetitive think loop without tool calls or a valid answer.The model may hallucinate sensory observations instead of executing tools.Source: §A.7 · p. 17
  • Proposed directions include training on longer trajectories and asymmetric token compression; speeding up playback may fit the context window but distort fine-grained audio cues.Compression examples include discarding silent tokens or merging redundant acoustic features.Source: §A.7 · p. 17

Related papers

Loading 2610.02181v1…