PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniGAIA: Towards Native Omni-Modal AI Agents

Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Haoxuan Li, Hao Wang, Shijian Wang, Guanting Dong, Jiajie Jin, Yinuo Wang, Yuan Lu, Ji-Rong Wen, Zhicheng Dou, Zhouchen Lin

arXiv:2602.22897v3cs.AIcs.CLcs.CVcs.LGcs.MM

TL;DR

Current multimodal systems and benchmarks insufficiently support or measure unified perception, long-horizon reasoning, and tool use across video, audio, and images. The paper introduces the OmniGAIA benchmark and OmniAtlas agent, whose tool-integrated training improves open-model performance while substantial perception and hard-reasoning gaps remain.

  • Problem

    Existing multimodal LLMs and benchmarks are largely bi-modal or perception-centric, providing limited evaluation of unified omni-modal reasoning and multi-turn tool use.

  • Method

    OmniGAIA uses an event-graph pipeline for human-verified multi-hop tasks, while OmniAtlas combines active perception, tool-integrated reasoning, trajectory synthesis, masked supervised learning, and OmniDPO.

  • Results

    62.5 Pass@1 for Gemini-3-Pro versus 13.3 for Qwen3-Omni shows the benchmark’s difficulty, while OmniAtlas improves Qwen3-Omni from 13.3 to 20.8.

  • Takeaways & Limitations

    OmniGAIA remains challenging for current models, and effective tool use and long-horizon reasoning are identified as important capabilities beyond parameter scaling alone.

  • Takeaways & Limitations

    Visual and audio perception errors remain approximately 30%–50%, and the Hard subset requiring deep multi-hop reasoning remains a formidable challenge.

Abstract

from arXiv · show

Human intelligence naturally intertwines omni-modal perception -- spanning vision, audio, and language -- with complex reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g., vision-language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive benchmark designed to evaluate omni-modal agents on tasks necessitating deep reasoning and multi-turn tool execution across video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation agent under tool-integrated reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.

1 Introduction

The paper addresses the gap between bi-modal multimodal models and assistants that jointly perceive, reason over, and act across vision, audio, and language. It introduces OmniGAIA for evaluation and OmniAtlas for tool-integrated omni-modal reasoning.

  • Current multimodal LLM research remains dominated by bi-modal settings, limiting integrated handling of interwoven real-world modalities.
  • Existing benchmarks largely emphasize bi-modal perception and do not adequately measure multi-hop omni-modal reasoning or multi-turn external tool use.
  • OmniGAIA contains 360 tasks across 9 real-world domains using video-with-audio and image+audio inputs, requiring multi-turn tools and verifiable open-form answers.
  • Its event-graph pipeline mines multimodal signals, links cross-modal entities and events, expands evidence with retrieval and tools, and generates human-verified multi-hop questions.
  • OmniAtlas combines tool-integrated reasoning and active omni-modal perception with trajectory synthesis, supervised learning, and OmniDPO error correction.
  • 62.5 Pass@1 for Gemini-3-Pro versus 13.3 for Qwen3-Omni demonstrates OmniGAIA’s difficulty, while the OmniAtlas recipe raises Qwen3-Omni from 13.3 to 20.8.

2 Related Work

Recent foundation models increasingly unify text, vision, and audio through shared representations, while evaluation and agent research remain less developed for long-horizon omni-modal tool use.

  • Omni-modal models commonly map heterogeneous visual and acoustic inputs into a shared token space through unified tokenization-and-projection interfaces.
  • Existing omni-modal benchmarks largely emphasize short audio or video perception, leaving long-horizon reasoning and tool-integrated agency underexplored.
  • The paper positions OmniGAIA against this evaluation gap by targeting challenging omni-modal agentic tasks.

1. Data Collection

The benchmark’s data collection is motivated by the limited coverage of existing multimodal evaluation and draws on established agentic tool-use paradigms. Its construction targets richer omni-modal evidence and interaction.

  • Existing benchmarks largely focus on short audios or videos and perception-centric tasks, leaving long-horizon reasoning and tool-integrated agency underexplored.
  • OmniGAIA extends this landscape toward evaluation settings that combine richer modalities with long-horizon, tool-integrated behavior.
  • Prior autonomous-agent research studies reasoning and acting through external tools, including workflow-based and native agentic paradigms.

3 OmniGAIA: Benchmarking Omni-Modal General AI Assistants

OmniGAIA stress-tests unified vision, audio, and language understanding together with long-horizon reasoning and multi-turn tool use. Its event-graph pipeline builds structured evidence and expands it through cross-modal retrieval and external tools.

  • OmniGAIA evaluates unified perception across vision, audio, and language alongside long-horizon reasoning and multi-turn tool use in realistic scenarios.
  • The benchmark uses complementary video-with-audio and image+audio settings to reflect real-world omni-modal interactions.
  • Its video sources include broad-domain videos averaging 4 minutes and additional approximately 10-minute datasets for long-context reasoning.
  • The construction pipeline extracts time-aware signals from video, audio, and images, including clip descriptions, ASR, speaker diarization, audio events, and acoustic environments.
  • An explicit omni-modal event graph organizes entities, events, and cross-modal relations to support systematic evidence expansion and controllable QA fuzzification.
  • Agentic event-graph expansion uses cross-modal retrieval and external tools to discover next-hop evidence needed for tightly coupled multi-hop questions.

2. OmniDPO: Fine-Grained Error Correction

The task-generation pipeline expands multimodal event graphs into difficult, verifiable QA tasks by masking information along long reasoning paths and applying automated and human quality checks.

  • Event-Graph Task Construction: Tools autonomously expand the event graph’s information boundary, producing QA pairs enriched with next-hop evidence.The prompting interface embeds tools such as web search, browsing, and code execution.
  • Event-Graph Task Construction: Event fuzzification masks or abstracts entities and attributes along long reasoning paths, forcing multi-source, multimodal evidence integration.This converts direct fact lookup into tasks requiring traversal of the full logical path.
  • Quality Control: LLM screening filters QA pairs for clarity, indispensable omni-modal perception and tool use, answer correctness, and uniqueness.DeepSeek-V3.2 and Gemini-3-Pro form the automated review committee.
  • Quality Control: Difficulty expansion can link additional sources, mine deeper evidence, or introduce more complex computation steps.This step is optional and occurs after preliminary qualification.
  • Quality Control: Three graduate-level reviewers verify each QA pair against the underlying media and correct minor issues.They check question soundness, annotation correctness, and answer correctness and uniqueness.
  • Benchmark Scope: OmniGAIA contains 360 tasks across 9 real-world domains that stress long-horizon perception and tool-integrated reasoning.Tasks may require vision-and-audio grounding over minutes-long media, multistep planning, and external verification.

4 OmniAtlas: Omni-Modal Foundation Agent

OmniAtlas combines vision, audio, and language with long-horizon reasoning, autonomous tool use, and active perception. Its training recipe synthesizes and filters trajectories, masks tool observations during supervised learning, and applies fine-grained preference correction.

  • Agent Design: OmniAtlas unifies vision, audio, and language perception with long-horizon reasoning and autonomous tool use.The agent is designed to address weaknesses in open-source omni-modal perception and tool-integrated reasoning.
  • Autonomous Tool-Integrated Reasoning: The agent switches autonomously between internal reasoning and tools such as web search, page browsing, and code execution.Tool observations are appended to the context so generation can continue coherently.
  • Autonomous Tool-Integrated Reasoning: An agent trajectory records thoughts, actions, and tool observations across interaction steps, conditioned on the user instruction and omni-modal inputs.Actions may be tool calls or final responses.
  • Active Omni-Modal Perception: Active perception lets OmniAtlas request specific video segments, audio intervals, or image regions instead of ingesting all media indiscriminately.This targets token cost and information loss from aggressive downsampling.
  • Trajectory Synthesis: Hindsight-guided tree exploration samples candidate reasoning-and-tool continuations and retains successful trajectories after verifier-based pruning.At each step, k = 3 candidate continuations are sampled and incorrect or redundant branches are removed.
  • Trajectory-Level Supervised Fine-Tuning: Masked trajectory-level SFT trains reasoning and tool-call tokens while excluding tool observations from the loss.This avoids memorizing environment feedback and emphasizes agent-generated behavior.
  • Evaluation: Table 2 reports Pass@1 for all OmniGAIA tasks, with best and second-best scores distinguished separately for proprietary and open-source models.The table’s visual emphasis identifies the leading scores within each model category.
  • OmniDPO: Fine-Grained Error Correction: OmniDPO corrects fine-grained perception, reasoning, and tool-use errors by contrasting failed prefixes with corrected prefixes at the first erroneous step.The objective focuses log-probabilities on agent-generated tokens and the module containing the error.

5 Experiments

Experiments show that OmniGAIA remains difficult for current models, with tool use and reasoning as major failure points. OmniAtlas improves open-model performance and tool policies, while hard-task reasoning and perception remain challenging.

  • 5.2 Main Results: 62.5 Pass@1 for Gemini-3-Pro versus 13.3 for Qwen-3-Omni demonstrates a substantial proprietary–open-source performance gap.The comparison is reported under the unified tool setting.
  • 5.2 Main Results: 20.8 Pass@1 from 13.3 shows OmniAtlas improves Qwen-3-Omni by +7.5 absolute.Qwen-2.5-Omni-7B improves approximately 3.7×, from 3.6 to 13.3.
  • 5.3 Fine-Grained Error Analysis: Tool-use and reasoning failures are the most prevalent error types, reaching 35.3%–91.9% and 15.8%–79.7%, respectively.On hard tasks, open-source models show approximately 90%–96% tool misuse and 80%–90% reasoning errors.
  • 5.3 Fine-Grained Error Analysis: OmniAtlas reduces tool misuse from 81.1% to 59.4% and reasoning errors from 79.7% to 64.4%, but visual and audio perception errors remain approximately 30%–50%.These results identify perception as a persistent bottleneck after training improvements.
  • 5.4 Tool Call Distribution Analysis: More tool calls do not guarantee success: runs with long tails exceeding 10–20 calls can still fail, indicating inefficient exploration or thrashing.OmniAtlas shifts Qwen-3-Omni-30B toward a higher and broader tool-call distribution while improving tool-use errors and Pass@1.
  • 5.5 Native Perception vs. Tool-based Perception: Native perception gives Gemini-3-Flash the best Avg. score, 51.7, with 4.4 tool calls, whereas perception tools lower accuracy and increase interaction cost.For Qwen-3-Omni, perception tools help Easy and Medium tasks but reduce Hard performance; tool calls also increase across settings.

6 Conclusion and Future Work

The paper concludes that OmniGAIA evaluates long-horizon omni-modal reasoning with multi-turn tool use, while OmniAtlas improves tool-integrated behavior in open models. Future work targets reinforcement learning, scalable omni-modal services, and embodied-agent benchmarks.

  • 6 Conclusion and Future Work: OmniGAIA requires multi-hop reasoning and multi-turn tool use over video-with-audio and image+audio inputs.Its event-graph pipeline aligns cross-modal evidence with tools and synthesizes verifiable multi-hop questions.
  • 6 Conclusion and Future Work: OmniAtlas uses active perception, tool-integrated reasoning, hindsight-guided tree exploration, trajectory-level masked SFT, and OmniDPO.The recipe improves Qwen3-Omni from 13.3 to 20.8 Pass@1 while reducing tool-use and reasoning failures.
  • 6 Conclusion and Future Work: Future directions include omni-modal agentic reinforcement learning, scalable omni-modal MCP services, and embodied-agent benchmarks and foundation models.These directions extend omni-modal agents toward broader tasks and physical-world completion.

7 Impact Statement

The work advances native omni-modal agents through a benchmark for long-horizon reasoning and tool use and a recipe for improving open models. It may support more reliable cross-modal grounding and verification in assistive applications and help standardize evaluation.

  • 7 Impact Statement: OmniGAIA and OmniAtlas may enable more reliable cross-modal grounding and verification in assistive applications such as education and accessibility.The impact statement also connects the benchmark to standardizing evaluation of tool-augmented omni-modal agents.

A Implementation Details

OmniAtlas is trained and evaluated with standardized tool-use prompts, structured perception backends, and targeted evidence retrieval for long or ambiguous media.

  • Training: OmniAtlas is trained for 2 epochs of supervised fine-tuning on 2,156 synthesized trajectories, followed by 2 epochs of OmniDPO.Training covers Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Qwen3-Omni-30B-A3B-Thinking backbones.
  • Evaluation: Evaluation first uses exact matching on extracted <answer> spans, then applies an LLM-as-a-Judge fallback when exact matching is insufficient.The judge receives the question, normalized prediction, and labeled answer, returning a binary correctness label.
  • System Prompts: Base-agent system prompts standardize instruction following, tool usage, step-by-step reasoning, and extractable answer formatting.Final answers must be wrapped in <answer>...</answer>.
  • Active Perception: Active perception lets OmniAtlas request additional evidence for unclear image, audio, or video regions instead of relying on a single media ingestion.The read_video tool retrieves targeted time windows from long videos; this reduces unnecessary context and cost.
  • Perception Backends: Audio and vision QA backends restrict answers to their respective modalities and permit abstention when evidence is insufficient.Structured image and audio reports provide factual, time-aligned intermediate signals for downstream event-graph construction.

C.1 Omni-Modal Foundation Models and Benchmarks

Existing omni-modal models and benchmarks do not yet fully support agentic, cross-modal reasoning, while OmniGAIA exposes failures in grounding and tool-query planning.

  • C.1 Omni-Modal Foundation Models and Benchmarks: Current omni-modal models unify text, vision, and audio, but existing agent research and benchmarks remain largely focused on text, bi-modal perception, or multimodal search.The supplied discussion identifies a gap in native omni-modal agentic reasoning and evaluation.
  • C.1 Omni-Modal Foundation Models and Benchmarks: OmniGAIA case studies require location grounding, targeted query formulation, hypothesis testing, verification, and answer normalization.The bridge instance uses Joliet Iron Works and the Ruby Street Bridge to stress this capability chain.
  • C.1 Omni-Modal Foundation Models and Benchmarks: One failure mode is premature closure: without tool calls, the model follows a movie-driven Chicago prior and outputs an unverified bridge identity.The failure yields an incorrect bridge name and age.
  • C.1 Omni-Modal Foundation Models and Benchmarks: Another failure mode is query drift, where tool retrieval remains tied to an incorrect Chicago hypothesis and produces correct arithmetic over incorrect facts.The model does not issue a corrective, location-specific query for the Joliet bridge.

D.4 Case III: Success via Location-First Disambiguation and Verification

The successful trace resolves the bridge by grounding first in location, verifying each relevant fact with targeted tools, and then computing the age.

  • D.4 Case III: Success via Location-First Disambiguation and Verification: Location-first grounding identifies the nearby Ruby Street Bridge at Joliet Iron Works rather than following the Chicago association suggested by The Blues Brothers.This disambiguation anchors subsequent evidence retrieval.
  • D.4 Case III: Success via Location-First Disambiguation and Verification: Targeted web retrieval verifies the Ruby Street Bridge construction year as 1935 and the filming start date as July 1979.The trace links the bridge identity, construction year, and filming date before computation.
  • D.4 Case III: Success via Location-First Disambiguation and Verification: 44 years is obtained by computing 1979 − 1935 after verifying the bridge and filming facts.The computation tool is used only after the relevant evidence has been grounded and retrieved.
  • D.4 Case III: Success via Location-First Disambiguation and Verification: The practical pattern is location-first grounding, hypothesis testing, verification before computation, and strict answer normalization.This addresses both insufficient evidence acquisition and tool-query drift.
  • D.4 Case III: Success via Location-First Disambiguation and Verification: Semantic equivalence can recover a correct answer even when extra explanation causes strict exact match to fail.The LLM-as-a-Judge protocol marks the response as correct despite formatting differences.
Loading 2602.22897v3…