PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention

Zhangquan Chen, Jiale Tao, Ruihuang Li, Yihao Hu, Ruitao Chen, Zhantao Yang, Xinlei Yu, Haodong Jing, Manyuan Zhang, Shuai Shao, Biao Wang, Qinglin Lu, Ruqi Huang

arXiv:2602.05847v2cs.AIcs.CV

TL;DR

Existing omnimodal models struggle with audio-visual reasoning because modality bias and limited evidence-tracking can lead to unimodal shortcuts. OmniVideo-R1 uses self-supervised query-intensive grounding and contrastive modality-attentive fusion within a reinforced framework. Across multiple benchmarks, it consistently outperforms prior methods while maintaining robust visual-only performance.

  • Problem

    Existing omnimodal models can exhibit modality bias and may ignore decisive audio or visual cues because current post-training methods provide little supervision for intermediate evidence tracking.

  • Method

    OmniVideo-R1 combines self-supervised query-intensive grounding with contrastive modality-attentive fusion in a two-stage RL framework without process-level annotations.

  • Results

    OmniVideo-R1 consistently outperforms prior methods on multiple audio-visual benchmarks while maintaining robust visual-only performance.

  • Takeaways & Limitations

    The framework provides a query-driven approach for extracting intention-relevant cues and integrating audio-visual information throughout reasoning.

  • Takeaways & Limitations

    Existing omnimodal reasoning remains relatively limited, with a lack of training methodologies tailored to the distinctive characteristics of omnimodal models.

Abstract

from arXiv · show

While humans perceive the world through diverse modalities that operate synergistically to support a holistic understanding of their surroundings, existing omnivideo models still face substantial challenges on audio-visual understanding tasks. In this paper, we propose OmniVideo-R1, a novel reinforced framework that improves mixed-modality reasoning. OmniVideo-R1 empowers models to "think with omnimodal cues" by two key strategies: (1) query-intensive grounding based on self-supervised learning paradigms; and (2) modality-attentive fusion built upon contrastive learning paradigms. Extensive experiments on multiple benchmarks demonstrate that OmniVideo-R1 consistently outperforms strong baselines, highlighting its effectiveness and robust generalization capabilities.

1. Introduction

Current omnimodal models can suffer from modality bias and fail to explicitly track or combine decisive audio-visual evidence. OmniVideo-R1 addresses this with query-intensive grounding, modality-attentive fusion, and a two-stage RL framework.

  • Motivation: Current omnimodal models may underperform because modality bias can prevent synergistic audio-visual understanding.The Qwen3-30B-A3B Omni variant drops from 72.1 to 68.5 on MMStar and from 80.1 to 75.9 on MathVista_mini relative to the visual-only variant.
  • Motivation: Existing post-training methods do not explicitly train behaviors for locating and composing evidence across modalities, allowing unimodal shortcuts.Models may ignore decisive audio or visual cues while still answering correctly through dataset biases.
  • Approach: OmniVideo-R1 is an RL-based framework designed to improve mixed-modality reasoning through query-intensive grounding and modality-attentive fusion.The framework turns mixed-modality understanding into a query-driven reasoning process with audio-visual cues.
  • Approach: Query-intensive grounding uses self-supervision over multiple time-caption pairs to localize and validate audio-visual segments relevant to a query.The method avoids costly query-conditioned grounding annotations by generating grounding hypotheses and checking them against textual descriptions.
  • Approach: Modality-attentive fusion uses contrastive learning to encourage higher confidence from mixed audio-visual inputs than from single-modality inputs.This strategy encourages the model to discover relationships between visual and audio events without process-level annotations.
  • Contributions: An 80K cleaned audio-visual corpus supports complex reasoning, and experiments show consistent gains over strong open-source baselines while maintaining visual-only performance.The contribution includes a dedicated data-cleaning pipeline and robust visual-only performance across evaluated settings.

2. Related Work

Omnimodal research has progressed from adding audio to visual–language systems toward deeper cross-modal fusion and reinforcement-learning-based reasoning. However, complex audio–visual understanding remains limited, and existing omnimodal reasoning methods lack training approaches tailored to these models’ distinctive fusion requirements.

  • Omnimodal Large Language Models: Audio–visual integration better reflects real-world recordings, but early systems focused on silent video or treated audio as a text add-on.These designs fragmented omnimodal representations and limited performance.
  • Omnimodal Large Language Models: Recent models extend vision–language foundations with audio, progressively align modalities, concatenate audio–visual tokens, or develop deeper fusion mechanisms.Examples include MiniCPM-o-2.6, Baichuan-Omni-1.5, Ola, Video-LLaMA, and Video-SALMONN.
  • Reinforced Multimodal Reasoning: Current omnimodal models still struggle with tightly integrated audio–visual understanding and sophisticated logical reasoning.Compared with silent-video reasoning, omnimodal exploration remains relatively limited.
  • Reinforced Multimodal Reasoning: Reinforcement learning has become a common strategy for improving large-language-model performance and vision–language reasoning.Related methods include visual chain-of-thought approaches that integrate visual evidence into reasoning trajectories.
  • Reinforced Multimodal Reasoning: Omnimodal reasoning studies include referring segmentation, vanilla-GRPO applications, dual-system long-horizon reasoning, and datasets targeting complex human-intention understanding.These approaches broaden omnimodal reasoning but pursue different task or architectural goals.

3. Methodology

OmniVideo-R1 uses a unified reinforcement-learning pipeline with query-intensive grounding followed by modality-attentive fusion. The method combines self-supervised grounding, contrastive multimodal training, and filtered audio–video data to improve reasoning over relevant cues.

  • Method overview: OmniVideo-R1 optimizes query-intensive grounding and modality-attentive fusion within a unified reinforcement-learning framework.The first stage develops query-intensive reasoning, while the second promotes logically consistent integration of multiple modalities.
  • Data preparation: The data pipeline applies structural validation, quality assessment, heuristic filtering, and categorical balancing before training.The resulting dataset contains 88173 examples, with 12887 high-audio–visual-dependency examples reserved for the second stage.
  • Query-intensive grounding: Query-intensive grounding generates time–caption pairs that localize query-relevant audio–visual segments before response generation.The model produces structured time, caption, thinking, and answer outputs, then evaluates whether grounded segments support the reasoning process.
  • Query-intensive grounding: Without ground-truth target segments, grounding is rewarded through soft assessments of content completeness and precision.Predicted segments are cropped and concatenated, then evaluated for whether their audio–visual information is adequate and accurate for deriving the answer.
  • Modality-attentive fusion: The QI stage can miss decisive sound cues and produce redundant outputs, motivating the subsequent modality-attentive fusion stage.The MA stage targets a subset requiring integrated audio–visual understanding and advances the model toward deeper multimodal reasoning.
  • Modality-attentive fusion: Modality-attentive fusion compares combined audio–visual, silent-video-only, and audio-only rollouts to encourage superior fused performance.A contrastive attention reward discourages reliance on a single modality and uses α = 0.3 in the experiments.

4. Experiments

Experiments show that OmniVideo-R1 improves audio-visual reasoning across benchmarks while preserving visual-only performance. Ablations attribute gains to query-intent grounding and modality-attentive training.

  • Main Results: OmniVideo-R1 outperforms strong open-source baselines across audio-video understanding benchmarks.It surpasses Video-SALMONN 2+-72B by at least 4.3% and Gemini3-Pro by 2.1% on reported benchmarks.
  • Main Results: 21.1% improvement over Qwen3-Omni-30B-A3B on OmniVideoBench demonstrates gains on synergistic audio-visual reasoning.The method surpasses the base model across all reported evaluation dimensions.
  • Visual-only Robustness: OmniVideo-R1 shows no evident degradation on visual-only benchmarks after mixed-modality post-training.It gains 4.4% on Video-MME, loses 1.4% on MLVU, and gains 3.4% on LVBench relative to the base model.
  • Training Strategy Comparison: OmniVideo-R1 outperforms alternative training strategies, exceeding Vanilla RL by 12.0% on Daily-Omni and CoT SFT by 11.1% on WorldSense.These comparisons support the effectiveness of the proposed training paradigm.
  • Case Study: Qualitative analyses show that QI+MA better exploits audio and visual evidence and reduces redundant grounding compared with QI-only training.QI-only training can overlook critical audio cues, whereas MA strengthens utilization of audio-visual cues.
  • Ablation Study: Both modality-attentive and query-intensive stages improve performance, with gains of 12.4% and 17.8% over the base model on OmniVideoBench.Removing rintent or rattn causes performance drops, while input-modality mismatch produces benchmark-dependent effects.

5. Conclusion

OmniVideo-R1 is a two-stage framework for audio-visual reasoning that uses self-supervised grounding and contrastive fusion. Experiments show consistent gains across benchmarks and support future audio-visual reasoning research.

  • Conclusion: OmniVideo-R1 learns query-intensive grounding from intermediate time-caption pairs without process-level annotations.The model is trained to identify relevant cues before producing an answer.
  • Conclusion: Contrastive training between full audio-visual and single-modality inputs improves coherent multimodal representations.This stage explicitly strengthens cross-modal fusion.
  • Conclusion: Experiments show that OmniVideo-R1 consistently outperforms prior methods on multiple benchmarks.The reported results establish the framework as a foundation for future audio-visual reasoning work.
  • Conclusion: Figures 5 and 6 visualize incorrect reasoning, correct reasoning, modality overemphasis, and redundant grounding in QI and QI+MA outputs.The visualizations compare reasoning behavior across the two training configurations.

A.1. Training Dataset

The training dataset is filtered and categorized into a diverse collection spanning 16 audio-video categories with highly uneven sample counts.

  • Training Dataset: The processed training dataset contains 16 categories with sample counts ranging from 35 to 34598.The questions are described as high quality and diverse in content.

A.2.1. AUDIO-VISUAL BENCHMARKS

The evaluation suite spans synergistic audio-visual reasoning, human intention understanding, complex video reasoning, and visual-only video analysis. Supporting prompts and data procedures define training and evaluation configurations.

  • Audio-Visual Benchmarks: OmniVideoBench evaluates synergistic audio-visual reasoning with emphasis on modality complementarity and logical coherence.It contains 1000 question-answer pairs from 628 videos.
  • Audio-Visual Benchmarks: Daily-Omni evaluates audio-visual question answering across six major tasks using 1197 questions from 684 daily-life videos.The videos contain diverse auditory and visual cues.
  • Audio-Visual Benchmarks: WorldSense tests strongly coupled audio-video collaboration across 26 tasks and eight domains.It includes 3172 multiple-choice questions from 1662 synchronized videos.
  • Audio-Visual Benchmarks: IntentBench focuses on complex human intentions and emotions grounded in auditory and visual cues.The benchmark contains 2689 questions from 633 videos.
  • Visual Video Benchmarks: VideoHolmes evaluates complex video reasoning by requiring connections between dispersed visual clues and underlying causal events.It contains 1837 questions from 270 suspense short films across seven tasks.
  • Visual Video Benchmarks: Video-MME, MLVU, and LVBench evaluate broad, long-duration, and ultra-long video understanding, respectively.Their descriptions emphasize visual domains, long-video diversity, long-term memory, and extended comprehension.
  • Evaluation and Preparation: The evaluation pipeline uses exact-match Accuracy for multiple-choice questions and includes prompts for reasoning, categorization, consistency judging, and completeness evaluation.These components support both data preparation and model training procedures.

C. Implementation details

The implementation specifies key training hyperparameters governing frame limits, learning-rate warmup, clipping, KL regularization, and mixture-of-experts load balancing.

  • FPS_MAX_FRAMES is set to 64, capping the number of frames per sample.
  • lr_warmup_fraction is 0.05, gradually ramping up the learning rate at training start.
  • The clipping thresholds are ϵ = 3 × 10−4 and ϵhigh = 4 × 10−4.
  • KL regularization uses β = 0.03 to penalize large deviations from the reference policy.
  • moe_aux_loss_coeff is 10−3, weighting the auxiliary mixture-of-experts load-balancing loss.

D. Limitation & Future Work.

The authors identify reliance on outcome-based ground truth as a limitation and propose strengthening models without ground-truth supervision as future work. They also suggest extending the multimodal paradigm beyond audio–visual inputs, such as to 3D.

  • Limitation: Current methods still rely on outcome-based ground truth for training.The authors identify effective training without ground truth as an important future research direction.
  • Future Work: The proposed multimodal training paradigm could extend beyond audio–visual inputs to modalities such as 3D.The authors associate this extension with query intention and modality attention.
Loading 2602.05847v2…