PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains

Xinyue Cai, Chaoyou Fu, Yi-Fan Zhang, Ran He, Caifeng Shan

arXiv:2606.14702v2cs.CV

TL;DR

Existing automated audio-visual QA pipelines often decouple modalities and generate questions with weak long-term temporal and cross-modal dependencies. This paper introduces an entity-anchored scripting and clue-guided QA pipeline, whose dataset improves fine-tuned models by up to 20.59% on OmniVideo-Test and generalizes across established benchmarks.

  • Problem

    Existing pipelines separately describe short audio-visual clips, weakening cross-segment consistency and producing questions with weaker long-term temporal and cross-modal dependencies.

  • Method

    The pipeline builds entity-anchored structured video scripts, then mines long-span multimodal clues to guide QA generation, producing OmniVideo-100K and OmniVideo-Test.

  • Results

    Fine-tuning yields gains of up to 20.59% on OmniVideo-Test and improves performance across multiple established benchmarks.

  • Takeaways & Limitations

    OmniVideo-100K supports improved synergistic cross-modal comprehension in fine-tuned multimodal large language models.

  • Takeaways & Limitations

    Performance on the sound category drops by 4.21%, partly because coarse non-speech sound descriptions can introduce noise into QA generation.

Abstract

from arXiv · show

Current automated pipelines for audio-visual Question Answering (QA) generally adopt a ``video-caption-QA'' paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning. To address these issues, we propose an automated data engine featuring two mechanisms: (1) \textbf{Entity-Anchored Video Scripting} transforms videos into structured scripts, comprising summaries, main entity lists, and segment-wise audio-visual descriptions. The entity list serves as a global prior to ensure cross-segment referential consistency and reconstruct audio-visual associations. (2) \textbf{Clue-Guided QA Generation} prompts models to first mine cross-segment, multimodal clues from the script, and subsequently generate QA pairs based on these high-value clues. Leveraging this pipeline, we construct the instruction-tuning dataset \textbf{OmniVideo-100K} and a human-verified test set, \textbf{OmniVideo-Test}. Fine-tuning VITA-1.5, Qwen2.5-Omni-7B and Qwen3-Omni-30B on OmniVideo-100K yields performance gains of up to 20.59% on OmniVideo-Test, demonstrating strong generalization (up to 12.64% improvements) across established benchmarks like Daily-Omni and JointAVBench.

1 INTRODUCTION

OmniVideo-100K addresses weaknesses in automated audio-visual QA pipelines by using entity-anchored scripting and clue-guided QA generation to preserve cross-segment associations and elicit deeper reasoning. The resulting datasets and fine-tuned models improve performance on human-verified and established audio-visual benchmarks.

  • Motivation: Automated video-caption-QA pipelines decouple modalities and segments, weakening sound-source associations, referential consistency, long-term temporal dependencies, and cross-modal reasoning.Joint long-text comprehension and QA synthesis in one step tends to produce questions with weaker long-term temporal and cross-modal dependencies.
  • Method: Clue-Guided QA Generation first mines cross-segment, multimodal clues from scripts, then generates QA pairs from these high-value clues.This separates clue extraction from QA synthesis to support questions involving longer temporal and cross-modal dependencies.
  • Method: Entity-Anchored Scripting converts videos into summaries, main-entity lists, and timestamped segment descriptions integrating speech, non-speech sounds, and visual information.Consistent entity identifiers maintain narrative coherence across segments and associate speech with visual entities.
  • Results: 20.59%, 17.82% and 13.86% performance gains on OmniVideo-Test are achieved by VITA-1.5, Qwen2.5-Omni-7B and Qwen3-Omni-30B-A3B-Instruct, respectively, after fine-tuning on OmniVideo-100K.The improvements generalize across Video-MME, Daily-Omni, and JointAVBench.

2 METHOD

The method builds structured audio-visual scripts with globally consistent entity references, then uses explicit cross-segment multimodal clues to generate temporally extended, evidence-grounded QA pairs. It organizes the pipeline into entity-anchored scripting, clue-guided generation, and dataset construction with a three-level task taxonomy.

  • 2 METHOD: The framework comprises Entity-Anchored Video Scripting, Clue-Guided QA Generation, and construction of the OmniVideo-100K and OmniVideo-Test datasets.These modules are presented in Sections 2.1–2.3.
  • Entity-Anchored Video Scripting: Entity-anchored scripting converts videos into summaries, main entity lists, and sequential segments containing timestamps, visual descriptions, and speaker-labeled speech or non-speech audio.Unified entity identifiers provide cross-segment referential consistency and help reconstruct audio-visual associations.
  • Clue-Guided QA Generation: Clue-guided QA generation first mines multimodal clues and causal chains across the full script, then focuses generation on their timestamps and logical descriptions.The two stages are Global Clue Mining and Locally Focused Generation.
  • Task Taxonomy: The task taxonomy defines ten audio-visual QA tasks across Alignment, Understanding, and Reasoning levels, progressing from perception and synchronization to logical deduction.The framework includes basic perception, semantic analysis, and advanced reasoning abilities.
  • Dataset Construction: OmniVideo-100K introduces fine-grained alignment and ordering objectives, anchoring open-ended answers to explicit audio-visual evidence chains with detailed cross-modal reasoning explanations.This extends beyond simple before/after temporal queries and brief conclusions.

3 EXPERIMENTS

The experiments evaluate OmniVideo-100K through full-parameter fine-tuning of three audio-visual models, using model-specific training implementations and reporting detailed settings in Table 13.

  • Experimental Setup: Full-parameter fine-tuning is conducted on VITA-1.5, Qwen2.5-Omni-7B, and Qwen3-Omni-30B-A3B-Instruct.The study uses LLaMA-Factory for the Qwen-series models and VITA-1.5’s official implementation.
  • Experimental Setup: Detailed parameter settings for the experiments are provided in Table 13.

3.1 MAIN RESULTS

OmniVideo-100K fine-tuning improves audio-visual reasoning and alignment, with gains on OmniVideo-Test and established audio-visual benchmarks. The results also show stronger cross-modal dependency, preserved general video performance, and a limitation on non-speech sound categories.

  • Performance Discrepancies Across Tasks: Existing models generally perform better on Understanding than Alignment and Reasoning, revealing limitations in temporal alignment and multimodal logical inference.video-SALMONN 2+ scores 49.22% on Understanding, versus 37.93% on Alignment and 43.51% on Reasoning.
  • Improvements from OmniVideo-100K: 20.59%, 17.82% and 13.86% performance gains over respective baselines demonstrate improved synergetic audio-visual understanding on OmniVideo-Test.These gains are reported for models fine-tuned on OmniVideo-100K.
  • Cross-modal Dependency Analysis: MiniCPM-o 4.5 gains 7% to 9% from receiving both modalities, while Qwen3-Omni improves by about 3%, indicating stronger audiovisual synergy for MiniCPM-o 4.5.The comparison uses audio-only, visual-only, and full audio-visual settings on OmniVideo-Test.
  • Generalization: 0.6%–12% performance increases across various metrics show that VITA-1.5 generalizes across Daily-Omni, OmniVideoBench, JointAVBench, and FutureOmni.The reported benchmark comparisons evaluate fine-tuned models against their baselines.
  • Performance Breakdown: 7.43% is Qwen2.5-Omni’s largest gain on Daily-Omni, driven primarily by a 17.23% increase in AV Event Alignment.The breakdown attributes the overall improvement mainly to cross-modal alignment capabilities.
  • Performance Breakdown: 4.21% performance drops on sound samples because coarse non-speech scripting descriptions can introduce noise into QA generation.The model improves on music and speech samples but declines on the sound category; specialized audio models are suggested for more precise descriptions.

3.2 ABLATION ANALYSIS

Ablations show that clue-guided and script-based generation produce more challenging, broader-context QA while strengthening cross-modal reasoning. Main-entity constraints and larger training volumes further support script quality and model performance.

  • Clue-Guided Generation: Clue-Guided Generation produces more difficult QA pairs than Direct Generation and better separates model capabilities.All evaluated models achieve lower accuracy on Clue-Guided samples, with larger performance gaps between models than on Direct samples.
  • Entity-Anchored Video Scripting: The Main Entity List and speaker labels are separately ablated to test referential consistency and audio-visual association in multi-segment QA generation.Both evaluations use Gemini-3-Pro-generated QA pairs requiring entity integration or matching speech with the corresponding speaker’s visual information.
  • Intermediate Script Representation: Script-based QA covers an average temporal span 37.88s higher than the video-based baseline.The comparison uses the interval from the end timestamp of the first relevant segment to the start timestamp of the last.
  • Intermediate Script Representation: Script-based QA yields a consistent performance drop across modality settings, with larger single-modality degradation than video-based QA.The results indicate stronger reliance on cross-modal information for script-based questions, particularly on 8–14min samples.
  • Data Volume Scaling: 10K OmniVideo samples raise Daily-Omni accuracy from 62.41% to 69.92% over the w/o SFT baseline.The data-volume ablation fine-tunes Qwen2.5-Omni on subsets ranging from 10K to 100K samples under consistent training settings.

4 RELATED WORK

Related work advances audio-visual understanding through multimodal models, temporal-alignment mechanisms, expanded QA datasets, and increasingly structured video scripting. These efforts provide the foundation for modeling synchronized audio-visual content and generating more fine-grained descriptions.

  • Multimodal Large Language Models: Multimodal large language models extend visual-only analysis toward audio-visual understanding by adding audio branches.
  • Multimodal Large Language Models: Temporal-alignment designs model synchronization between audio and visual content in audio-visual videos.Qwen3-Omni interleaves visual and audio tokens according to actual time, while JavisGPT uses SyncFusion to integrate audio into visual features and model spatio-temporal synchrony.
  • Audio-Visual Video Datasets: Audio-visual QA research has progressed from foundational studies to a surge of datasets and benchmarks.
  • Video Structured Scripting: Video scripting research moves beyond one caption per segment toward multi-attribute, multi-dimensional, and script-like descriptions.TimeChat-Captioner introduces Omni Dense Captioning with a six-dimensional schema for comprehensive audio-visual coverage.

5 CONCLUSION

The paper presents an automated data synthesis engine for improving audio-visual understanding in multimodal large language models. Its pipeline combines entity-anchored structured scripting with clue-guided question-answer generation.

  • The proposed engine targets enhanced audio-visual understanding capabilities in multimodal large language models.
  • The first stage extracts audio and visual information from video segments into structured, script-like text.It leverages a main entity list prior to maintain narrative coherence across segments and explicitly reconstruct audio-visual associations.
  • The second stage uses a clue-guided question-answer generation strategy.The supplied passage introduces this strategy as the pipeline’s second stage but does not provide further procedural details.

A AUDIO-VISUAL TASK TAXONOMY

OmniVideo-100K and OmniVideo-Test cover ten audio-visual tasks, defining the scope of the paper’s audio-visual task taxonomy.

  • Ten audio-visual tasks are covered by both OmniVideo-100K and OmniVideo-Test.

B DETAILED EXPERIMENTAL SETTINGS AND RESULTS

This section presents parameter settings for full-parameter fine-tuning and evaluates performance on OmniVideo-Test across diverse audio-visual reasoning capabilities.

  • Experimental Settings: Table 13 specifies the parameter settings used for full-parameter fine-tuning of VITA-1.5, Qwen2.5-Omni-7B, and Qwen3-Omni-30B-A3B-Instruct.The supplied passage identifies the compared models and the fine-tuning setting, but does not provide individual parameter values.
  • Experimental Results: Table 14 compares model performance on OmniVideo-Test using fine-grained perception, scene transformation detection, context understanding, comparison, and sentiment analysis.These are among the evaluation categories listed in the table caption.
  • Experimental Results: The OmniVideo-Test comparison also covers event sequence ordering, summarization, causal reasoning, future prediction, and hypothetical reasoning.The passage provides the metric abbreviations but no numerical results.

C PROMPTS FOR THE OVERALL PIPELINE

The overall pipeline uses prompts to structure video understanding across entities, summaries, speech, sounds, speakers, and visual descriptions. It then mines global clues, generates locally focused QA pairs, and validates them for nontrivial video-grounded solvability.

  • Entity and Summary Prompts: Prompts identify the video’s main active entities and generate a global summary.These outputs establish structured global information before segment-level processing.
  • Audio Processing Prompts: Speech transcription prompts produce timestamped, JSON-formatted segments and begin new segments when the speaker changes.The transcription task requires meaningful sentences or complete phrases with timestamps marking segment beginnings.
  • Audio Processing Prompts: Acoustic-event prompts log non-dialogue sounds chronologically using objective sound categories without inferring source, cause, intent, or context.Music receives more detailed characterization under the prompt’s constraints.
  • Audio-Visual Understanding Prompts: Audio-visual diarization prompts assign speaker names to transcript segments using synthesized evidence from the video’s visual and audio tracks.The pipeline also includes prompts for generating visual descriptions.
  • Clue-Guided Generation Prompts: Clue-guided generation first mines global causal clues, then produces locally focused open-ended or multiple-choice QA pairs.Separate prompts support global clue mining and both QA formats.
  • Validation and Iteration: Validation prompts reject drafts whose answers can be guessed without video evidence or found through keyword matching or a single frame.Failed drafts must be regenerated with new questions and options.
Loading 2606.14702v2…