PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni2Sound: Towards Unified Video-Text-to-Audio Generation

Yusheng Dai, Zehua Chen, Yuxuan Jiang, Baolong Gao, Qiuhong Ke, Jianfei Cai, Jun Zhu

arXiv:2601.02731v3cs.SDcs.CVcs.MM

TL;DR

Unified T2A, V2A, and VT2A generation is constrained by scarce V-A-T-aligned captions and competition among tasks and modalities. The paper introduces SoundAtlas, Omni2Sound, and VGGSound-Omni to address these issues, and reports unified state-of-the-art performance across all three tasks within one model.

  • Problem

    Unified VT2A systems face scarce high-quality V-A-T-aligned captions and cross-task and intra-task competition across T2A, V2A, and VT2A.

  • Method

    The paper combines SoundAtlas, an agent-generated multimodal caption dataset, with Omni2Sound's unified diffusion model and three-stage progressive training schedule, evaluated using VGGSound-Omni.

  • Results

    Omni2Sound achieves unified state-of-the-art performance across T2A, V2A, and VT2A within a single model, including under unseen captioning styles.

  • Takeaways & Limitations

    The approach supports one model across flexible input modalities while maintaining audio-visual synchronization and generation faithfulness, including off-screen audio evaluation.

Abstract

from arXiv · show

Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) generation offers significant application flexibility, yet faces two unexplored foundational challenges: (1) the scarcity of high-quality audio captions with tight V-A-T alignment, leading to severe semantic conflict between multimodal conditions, and (2) cross-task and intra-task competition, manifesting as an adverse V2A-T2A performance trade-off and modality bias in the VT2A task. First, to address data scarcity, we introduce SoundAtlas, a large-scale dataset (470k pairs) that significantly outperforms existing benchmarks and even human experts in quality. Powered by a novel agentic pipeline, it integrates Vision-to-Language Compression to mitigate visual bias of MLLMs, a Junior-Senior Agent Handoff for a 5$\times$ cost reduction, and rigorous Post-hoc Filtering to ensure fidelity. Consequently, SoundAtlas delivers semantically rich and temporally detailed captions with tight V-A-T alignment. Second, we propose Omni2Sound, a unified VT2A diffusion model supporting flexible input modalities. To resolve the inherent cross-task and intra-task competition, we design a three-stage multi-task progressive training schedule that converts cross-task competition into joint optimization and mitigates modality bias in the VT2A task, maintaining both audio-visual alignment and off-screen audio generation faithfulness. Finally, we construct VGGSound-Omni, a comprehensive benchmark for unified evaluation, including challenging off-screen tracks. With a standard DiT backbone, Omni2Sound achieves unified SOTA performance across all three tasks within a single model, demonstrating strong generalization across benchmarks with heterogeneous input conditions.

1. Introduction

Unified VT2A models aim to combine semantic understanding, temporal alignment, and flexible input handling, but they face V-A-T data scarcity and competition between tasks and modalities. Omni2Sound addresses these challenges with SoundAtlas and progressive multi-task training, achieving unified performance across T2A, V2A, and VT2A.

  • VT2A combines video and text conditioning for semantic understanding and temporal alignment, but conventional systems degrade under missing-modality conditions.
  • High-quality captions aligned with video and audio remain scarce, while audio-only captions can create semantic conflicts with visual content.
  • Unified training introduces cross-task competition, producing an adverse T2A-V2A trade-off when models jointly optimize heterogeneous modalities.
  • SoundAtlas supplies large-scale multimodal captions with rich semantic and temporal detail, tight V-A-T alignment, and quality reported to surpass human-expert annotations.
  • Omni2Sound uses a three-stage progressive schedule to address cross-task and intra-task competition while supporting flexible input modalities.
  • Omni2Sound achieves unified state-of-the-art performance across T2A, V2A, and VT2A with a standard DiT backbone.

2. Related Works

Prior audio-caption and unified audio-generation approaches expand scale and modality coverage, but they remain limited by V-A-T misalignment, hallucination, visual bias, and overlooked task competition. SoundAtlas targets these gaps through multimodal routing, agent handoff, compression, and verification.

  • Human-annotated datasets provide high-quality alignment but limited scale, while automated audio-caption pipelines increase volume without fully resolving audio ambiguity and V-A-T misalignment.
  • Unified audio models broaden modality combinations, but prior systems often rely on brute-force scaling and overlook cross-task competition during co-training.

3. SoundAtlas: V-A-T Data Construction

SoundAtlas constructs V-A-T-aligned captions through routed multimodal processing, agent escalation, and two-stage verification. Evaluations report stronger text-audio, semantic, and temporal alignment than existing automated pipelines and human-expert annotations.

  • Raw multimodal captioning is costly and introduces visual bias, motivating a cost-effective agentic pipeline for SoundAtlas.
  • A-V Consistency Routing uses visual cues for high-consistency clips, an audio-only path for medium-consistency clips, and discards low-consistency noise.
  • Vision-to-language compression replaces raw video with textual visual context, reducing cost and mitigating direct visual bias in caption generation.
  • The junior agent captions audio with optional visual context, while flagged complex or low-faithfulness samples are escalated to the senior agent.
  • Post-hoc verification filters captions for text-audio faithfulness and V-A-T plausibility before accepting them into SoundAtlas.
  • SoundAtlas significantly outperforms competing pipelines on LA-CLAP and MS-CLAP, and achieves higher MWR-S and MWR-T than the strongest baseline and human-expert annotations.

4. Omni2Sound: Unified VT2A Generation

Omni2Sound is a unified diffusion model for T2A, V2A, and VT2A generation, using decoupled semantic and temporal conditioning for flexible modality control. A three-stage progressive training schedule addresses cross-task competition and modality bias while preserving temporal alignment and off-screen generation.

  • Omni2Sound is a diffusion-based unified VT2A model supporting collaborative VT2A and unimodal V2A and T2A control.
  • Foundation Model Architecture: The framework combines global semantic conditioning with temporal alignment to support flexible T2A, V2A, and VT2A generation.
  • Foundation Model Architecture: Its decoupled architecture enables extensible multimodal conditions and maintains precise temporal alignment comparable to MMAudio.
  • Three-stage Progressive Multi-task Training: The three-stage progressive schedule addresses cross-task and intra-task competition through staged pretraining, interleaved multi-task optimization, and robustness training.
  • Three-stage Progressive Multi-task Training: Large-scale T2A pretraining establishes a robust generative prior and permits lower subsequent T2A sampling frequency without catastrophic forgetting.
  • Three-stage Progressive Multi-task Training: Interleaved task sampling avoids within-batch loss mixing, while VT2A bridges V2A and T2A optimization instead of leaving them in zero-sum competition.
  • Three-stage Progressive Multi-task Training: Robustness training uses text dropout and off-screen synthesis to balance modality reliance, strengthen synchronization, and address modality bias.

5. VGGSound-Omni: Unified Evaluation

VGGSound-Omni is a comprehensive benchmark for unified and specialized VT2A evaluation, combining human-level captions with a challenging off-screen track. Its main evaluation compares Omni2Sound with state-of-the-art models across T2A, V2A, and VT2A, including robustness to unseen caption styles.

  • VGGSound-Omni addresses the lack of a comprehensive benchmark for unified VT2A models.
  • Table 3 compares Omni2Sound with state-of-the-art models on T2A, V2A, and VT2A tasks using the VGGSound-Omni benchmark.
  • VGGSound-Omni Construction: The primary track covers 14,000+ videos with high-fidelity, human-level captions validated against VGGSounder audio and audio-visual labels.
  • VGGSound-Omni Construction: The off-screen track contains 1,000+ items drawn from natural low-audio-visual-correspondence events and synthetic music mixtures.

6. Experiments

Experiments show that Omni2Sound achieves unified performance across T2A, V2A, and VT2A, generalizes across benchmarks, and benefits from high-alignment data plus progressive training.

  • Main Results: Omni2Sound achieves state-of-the-art performance across T2A, V2A, and VT2A against unified and specialized baselines on VGGSound-Omni.This comparison uses standardized evaluation and matched video-text conditions.
  • Generalization: Omni2Sound remains competitive on Kling-Audio-Eval and achieves top-tier AudioCaps results, including best KL, FD, and CLAP = 0.36 scores.It trails HunyuanVideo-Foley on some Kling metrics but outperforms other unified and specialized baselines across tasks.
  • Subjective Evaluation: Human evaluation shows Omni2Sound outperforming all baselines on VT2A and V2A in acoustic fidelity, semantic consistency, and temporal synchronization.The subjective results are reported as consistent with the objective metrics.
  • Ablation Studies: SoundAtlas data resolves the V2A-T2A trade-off, achieving T2A FAD 0.94, V2A FD 3.61, and VT2A FD 2.83.High V-A-T alignment enables cooperative multi-task optimization, unlike standard-quality audio-only captions.
  • Ablation Studies: High-quality bridge data permits reducing the T2A sampling ratio to πT 2A = 0.1 while retaining SOTA T2A performance.The result mitigates resource contention during multi-task training.
  • Ablation Studies: The full S1 →S2 →S3 schedule achieves V2A FAD 0.51 and improves cross-modal consistency from V2A DS 0.49 to 0.47.Stage 3 also improves off-screen audio quality and text-audio alignment on the off-screen track.

7. Conclusion

The paper addresses data scarcity and competition in unified VT2A generation through SoundAtlas, Omni2Sound, and VGGSound-Omni. Experiments report unified state-of-the-art performance across the three tasks.

  • Conclusion: The paper introduces SoundAtlas, Omni2Sound, and VGGSound-Omni to address data scarcity and cross-task competition in unified VT2A generation.The contributions respectively provide a dataset, unified model, and evaluation benchmark.
  • Conclusion: Experiments demonstrate that Omni2Sound achieves unified state-of-the-art performance across T2A, V2A, and VT2A.
  • Conclusion: The approach effectively resolves cross-task and intra-task competition within the unified model.

Supplementary Material

The supplementary material documents the paper’s cost analysis, captioning prompts, evaluation protocols, off-screen benchmark construction, and extended experimental analyses.

  • Supplementary Material: The supplementary material begins with a cost analysis validating SoundAtlas as a scalable and cost-effective pipeline.
  • Supplementary Material: It provides the exact audio caption prompt instructions used in the agentic annotation pipeline.
  • Supplementary Material: It details evaluation protocols for comparing audio caption datasets and constructing the off-screen benchmark track.

A. Cost Analysis on Audio Captioning

Directly processing raw video and audio with Gemini 2.5 Pro is costly, so SoundAtlas compresses visual information and uses a hybrid pipeline to reduce annotation expense.

  • Cost Analysis: $10,275 USD per 1M samples is the estimated cost of naively processing raw video frames alongside audio with Gemini’s standard API pricing.The estimate is based on token consumption for a 10-second sample.
  • Cost Analysis: Vision-to-Language Compression removes approximately 2,500 visual tokens by replacing raw video with a concise video caption.
  • Cost Analysis: Approximately $2,000 per million samples is the cost of the hybrid SoundAtlas pipeline after optimization.The hybrid design balances quality and efficiency compared with standalone model options.
  • Cost Analysis: Figure 5 presents the audio-captioning system prompt used in the agentic annotation pipeline.

C. Audio Caption Dataset Comparison

The evaluation combines absolute scoring, pairwise win-rate analysis, and human expert assessment, while VGGSound-Omni adds a dedicated off-screen audio track with natural and synthetic subsets.

  • Evaluation methodology: The evaluation uses absolute linguistic scoring followed by comparative win-rate calculation from pairwise outcomes.Mean Win Rate aggregates wins, ties, and total pairwise comparisons, with ties receiving half credit.
  • Evaluation methodology: Semantic alignment scores factuality and descriptive detail, while temporal alignment scores whether event order matches the audio stream.Semantic Alignment uses a 1–4 scale; Temporal Alignment uses a 1–3 scale and excludes stationary sounds.
  • Human evaluation: Human validation samples 100 instances and uses five expert annotators with professional audio-visual backgrounds.The annotators assess the samples independently under the standardized protocol.
  • Captioning efficiency: $10,275 to $1,026: progressively removing raw video, applying vision-to-language compression, and switching to Flash reduces captioning cost for one million samples.The comparison concerns one million 10-second samples processed with Gemini 2.5.
  • Off-screen evaluation: The off-screen benchmark contains natural off-screen events and synthetic background-music generation tracks.Natural events are filtered for acoustic purity and modality conditions, while synthetic music mixes aligned MusicCaps clips into high-fidelity videos.
  • Off-screen evaluation: The benchmark extends prior synthetic robustness evaluation through greater realism, scale, and reliance on real-world off-screen events.The comparison with VinTAGe-Bench identifies these as three critical dimensions of extension.

E. Generalization on Third-Party Benchmarks.

Omni2Sound is evaluated across subjective, objective, and third-party settings, including heterogeneous benchmarks and human studies. The reported evaluations position it as competitive across unified audio-generation tasks while using a DiT-based implementation and SoundAtlas fine-tuning data.

  • Third-party benchmark generalization: Omni2Sound remains highly competitive on Kling-Audio-Eval despite a substantial data-scale and distribution gap.The comparison involves YouTube-sourced SoundAtlas and professionally sourced Kling video/Foley data.
  • Third-party benchmark generalization: Omni2Sound consistently outperforms strong baselines across V2A and VT2A tasks on the reported third-party evaluation.The cited baselines include MMAudio, AudioX, and ThinkSound, while HunyuanVideo-Foley leads on several metrics.
  • Subjective evaluation: The user study assigns 16 expert evaluators to independent VT2A and V2A tracks to reduce evaluator fatigue.Each participant evaluates 20 random samples and 80 comparisons within the assigned track.
  • Subjective evaluation: Omni2Sound outperforms all baselines on MOS-Q, MOS-S, and MOS-T for both VT2A and V2A in the subjective evaluation.The three dimensions measure acoustic quality, semantic alignment, and temporal alignment on a 1–5 scale.
  • Model configuration: The model uses a 24-layer, 24-head DiT with hidden dimension 1536 inside a latent diffusion model.Cross-attention injects semantic conditions, while AdaLN integrates temporal signals.
  • Training data: SoundAtlas fine-tuning uses 470k V-A-T pairs sourced from 140k VGGSound and 330k AudioSet samples.The AudioSet portion is curated from a preliminary candidate pool after excluding speech- and music-related categories.
  • Objective evaluation: Objective evaluation uses standardized AV-benchmark tooling, common video and text conditions, and 8-second generated clips.Reported dimensions include distribution matching and audio quality, with distances and divergences computed from several audio embeddings and classifiers.
Loading 2601.02731v3…