PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni2Sound: Towards Unified Video-Text-to-Audio Generation

Yusheng Dai, Zehua Chen, Yuxuan Jiang, Baolong Gao, Qiuhong Ke, Jianfei Cai, Jun Zhu

arXiv:2601.02731v3cs.SDcs.CVcs.MM

TL;DR

Unified T2A, V2A, and VT2A generation is constrained by scarce V-A-T-aligned captions and competition among tasks and modalities. The paper introduces SoundAtlas, Omni2Sound, and VGGSound-Omni to address these issues, and reports unified state-of-the-art performance across all three tasks within one model.

  • Problem

    Unified VT2A systems face scarce high-quality V-A-T-aligned captions and cross-task and intra-task competition across T2A, V2A, and VT2A.

    Source: §1 · pp. 1–2

  • Method

    The paper combines SoundAtlas, an agent-generated multimodal caption dataset, with Omni2Sound's unified diffusion model and three-stage progressive training schedule, evaluated using VGGSound-Omni.

    Source: §1 · p. 2

  • Results

    Omni2Sound achieves unified state-of-the-art performance across T2A, V2A, and VT2A within a single model, including under unseen captioning styles.

    Source: §1 · p. 2, §6.2 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    The approach supports one model across flexible input modalities while maintaining audio-visual synchronization and generation faithfulness, including off-screen audio evaluation.

    Source: §1 · p. 2, §5 · p. 5

Abstract

from arXiv · show

Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) generation offers significant application flexibility, yet faces two unexplored foundational challenges: (1) the scarcity of high-quality audio captions with tight V-A-T alignment, leading to severe semantic conflict between multimodal conditions, and (2) cross-task and intra-task competition, manifesting as an adverse V2A-T2A performance trade-off and modality bias in the VT2A task. First, to address data scarcity, we introduce SoundAtlas, a large-scale dataset (470k pairs) that significantly outperforms existing benchmarks and even human experts in quality. Powered by a novel agentic pipeline, it integrates Vision-to-Language Compression to mitigate visual bias of MLLMs, a Junior-Senior Agent Handoff for a 5$\times$ cost reduction, and rigorous Post-hoc Filtering to ensure fidelity. Consequently, SoundAtlas delivers semantically rich and temporally detailed captions with tight V-A-T alignment. Second, we propose Omni2Sound, a unified VT2A diffusion model supporting flexible input modalities. To resolve the inherent cross-task and intra-task competition, we design a three-stage multi-task progressive training schedule that converts cross-task competition into joint optimization and mitigates modality bias in the VT2A task, maintaining both audio-visual alignment and off-screen audio generation faithfulness. Finally, we construct VGGSound-Omni, a comprehensive benchmark for unified evaluation, including challenging off-screen tracks. With a standard DiT backbone, Omni2Sound achieves unified SOTA performance across all three tasks within a single model, demonstrating strong generalization across benchmarks with heterogeneous input conditions.

1. Introduction

Unified VT2A models aim to combine semantic understanding, temporal alignment, and flexible input handling, but they face V-A-T data scarcity and competition between tasks and modalities. Omni2Sound addresses these challenges with SoundAtlas and progressive multi-task training, achieving unified performance across T2A, V2A, and VT2A.

  • VT2A combines video and text conditioning for semantic understanding and temporal alignment, but conventional systems degrade under missing-modality conditions.Source: §1 · p. 1
  • High-quality captions aligned with video and audio remain scarce, while audio-only captions can create semantic conflicts with visual content.Source: §1 · pp. 1–2
  • Unified training introduces cross-task competition, producing an adverse T2A-V2A trade-off when models jointly optimize heterogeneous modalities.Source: §1 · p. 2
  • SoundAtlas supplies large-scale multimodal captions with rich semantic and temporal detail, tight V-A-T alignment, and quality reported to surpass human-expert annotations.Source: §1 · p. 2
  • Omni2Sound uses a three-stage progressive schedule to address cross-task and intra-task competition while supporting flexible input modalities.Source: §1 · p. 2
  • Omni2Sound achieves unified state-of-the-art performance across T2A, V2A, and VT2A with a standard DiT backbone.Source: §1 · p. 2

2. Related Works

Prior audio-caption and unified audio-generation approaches expand scale and modality coverage, but they remain limited by V-A-T misalignment, hallucination, visual bias, and overlooked task competition. SoundAtlas targets these gaps through multimodal routing, agent handoff, compression, and verification.

  • Human-annotated datasets provide high-quality alignment but limited scale, while automated audio-caption pipelines increase volume without fully resolving audio ambiguity and V-A-T misalignment.Source: §2 · pp. 2–3
  • Unified audio models broaden modality combinations, but prior systems often rely on brute-force scaling and overlook cross-task competition during co-training.Source: §2 · p. 3

3. SoundAtlas: V-A-T Data Construction

SoundAtlas constructs V-A-T-aligned captions through routed multimodal processing, agent escalation, and two-stage verification. Evaluations report stronger text-audio, semantic, and temporal alignment than existing automated pipelines and human-expert annotations.

  • Raw multimodal captioning is costly and introduces visual bias, motivating a cost-effective agentic pipeline for SoundAtlas.Source: §3 · p. 3
  • A-V Consistency Routing uses visual cues for high-consistency clips, an audio-only path for medium-consistency clips, and discards low-consistency noise.Source: §3 · pp. 3–4
  • Vision-to-language compression replaces raw video with textual visual context, reducing cost and mitigating direct visual bias in caption generation.Source: §3 · p. 4
  • The junior agent captions audio with optional visual context, while flagged complex or low-faithfulness samples are escalated to the senior agent.Source: §3 · p. 4
  • Post-hoc verification filters captions for text-audio faithfulness and V-A-T plausibility before accepting them into SoundAtlas.Source: §3 · p. 4
  • SoundAtlas significantly outperforms competing pipelines on LA-CLAP and MS-CLAP, and achieves higher MWR-S and MWR-T than the strongest baseline and human-expert annotations.Source: §3.1 · p. 4

4. Omni2Sound: Unified VT2A Generation

Omni2Sound is a unified diffusion model for T2A, V2A, and VT2A generation, using decoupled semantic and temporal conditioning for flexible modality control. A three-stage progressive training schedule addresses cross-task competition and modality bias while preserving temporal alignment and off-screen generation.

  • Omni2Sound is a diffusion-based unified VT2A model supporting collaborative VT2A and unimodal V2A and T2A control.Source: §4 · p. 4

Foundation Model Architecture

  • The framework combines global semantic conditioning with temporal alignment to support flexible T2A, V2A, and VT2A generation.Source: §4.1 · pp. 4–5, Figure 3 · p. 5
  • Its decoupled architecture enables extensible multimodal conditions and maintains precise temporal alignment comparable to MMAudio.Source: §4.1 · p. 5

Three-stage Progressive Multi-task Training

  • The three-stage progressive schedule addresses cross-task and intra-task competition through staged pretraining, interleaved multi-task optimization, and robustness training.Source: §4.2 · p. 5
  • Large-scale T2A pretraining establishes a robust generative prior and permits lower subsequent T2A sampling frequency without catastrophic forgetting.Source: §4.2 · p. 5
  • Interleaved task sampling avoids within-batch loss mixing, while VT2A bridges V2A and T2A optimization instead of leaving them in zero-sum competition.Source: §4.2 · p. 5
  • Robustness training uses text dropout and off-screen synthesis to balance modality reliance, strengthen synchronization, and address modality bias.Source: §4.2 · p. 5

5. VGGSound-Omni: Unified Evaluation

VGGSound-Omni is a comprehensive benchmark for unified and specialized VT2A evaluation, combining human-level captions with a challenging off-screen track. Its main evaluation compares Omni2Sound with state-of-the-art models across T2A, V2A, and VT2A, including robustness to unseen caption styles.

  • VGGSound-Omni addresses the lack of a comprehensive benchmark for unified VT2A models.Source: §5 · p. 5
  • Table 3 compares Omni2Sound with state-of-the-art models on T2A, V2A, and VT2A tasks using the VGGSound-Omni benchmark.Source: Table 3 · p. 6

VGGSound-Omni Construction

  • The primary track covers 14,000+ videos with high-fidelity, human-level captions validated against VGGSounder audio and audio-visual labels.Source: §5 · p. 6
  • The off-screen track contains 1,000+ items drawn from natural low-audio-visual-correspondence events and synthetic music mixtures.Source: §5 · p. 6

6. Experiments

Experiments show that Omni2Sound achieves unified performance across T2A, V2A, and VT2A, generalizes across benchmarks, and benefits from high-alignment data plus progressive training.

Main Results

  • Omni2Sound achieves state-of-the-art performance across T2A, V2A, and VT2A against unified and specialized baselines on VGGSound-Omni.This comparison uses standardized evaluation and matched video-text conditions.Source: §6.2 · pp. 6–7

Generalization

  • Omni2Sound remains competitive on Kling-Audio-Eval and achieves top-tier AudioCaps results, including best KL, FD, and CLAP = 0.36 scores.It trails HunyuanVideo-Foley on some Kling metrics but outperforms other unified and specialized baselines across tasks.Source: §6.2 · p. 7

Subjective Evaluation

  • Human evaluation shows Omni2Sound outperforming all baselines on VT2A and V2A in acoustic fidelity, semantic consistency, and temporal synchronization.The subjective results are reported as consistent with the objective metrics.Source: §6.2 · p. 7

Ablation Studies

  • SoundAtlas data resolves the V2A-T2A trade-off, achieving T2A FAD 0.94, V2A FD 3.61, and VT2A FD 2.83.High V-A-T alignment enables cooperative multi-task optimization, unlike standard-quality audio-only captions.Source: §6.3 · p. 7
  • High-quality bridge data permits reducing the T2A sampling ratio to πT 2A = 0.1 while retaining SOTA T2A performance.The result mitigates resource contention during multi-task training.Source: §6.3 · p. 7
  • The full S1 →S2 →S3 schedule achieves V2A FAD 0.51 and improves cross-modal consistency from V2A DS 0.49 to 0.47.Stage 3 also improves off-screen audio quality and text-audio alignment on the off-screen track.Source: §6.3 · p. 8, Table 7 · p. 8

7. Conclusion

The paper addresses data scarcity and competition in unified VT2A generation through SoundAtlas, Omni2Sound, and VGGSound-Omni. Experiments report unified state-of-the-art performance across the three tasks.

Conclusion

  • The paper introduces SoundAtlas, Omni2Sound, and VGGSound-Omni to address data scarcity and cross-task competition in unified VT2A generation.The contributions respectively provide a dataset, unified model, and evaluation benchmark.Source: §7 · p. 8
  • Experiments demonstrate that Omni2Sound achieves unified state-of-the-art performance across T2A, V2A, and VT2A.Source: §7 · p. 8
  • The approach effectively resolves cross-task and intra-task competition within the unified model.Source: §7 · p. 8

Supplementary Material

The supplementary material documents the paper’s cost analysis, captioning prompts, evaluation protocols, off-screen benchmark construction, and extended experimental analyses.

Supplementary Material

A. Cost Analysis on Audio Captioning

Directly processing raw video and audio with Gemini 2.5 Pro is costly, so SoundAtlas compresses visual information and uses a hybrid pipeline to reduce annotation expense.

Cost Analysis

C. Audio Caption Dataset Comparison

The evaluation combines absolute scoring, pairwise win-rate analysis, and human expert assessment, while VGGSound-Omni adds a dedicated off-screen audio track with natural and synthetic subsets.

Evaluation methodology

  • The evaluation uses absolute linguistic scoring followed by comparative win-rate calculation from pairwise outcomes.Mean Win Rate aggregates wins, ties, and total pairwise comparisons, with ties receiving half credit.Source: C. Audio Caption Dataset Comparison · pp. 12–13
  • Semantic alignment scores factuality and descriptive detail, while temporal alignment scores whether event order matches the audio stream.Semantic Alignment uses a 1–4 scale; Temporal Alignment uses a 1–3 scale and excludes stationary sounds.Source: C. Audio Caption Dataset Comparison · p. 12

Human evaluation

  • Human validation samples 100 instances and uses five expert annotators with professional audio-visual backgrounds.The annotators assess the samples independently under the standardized protocol.Source: C. Audio Caption Dataset Comparison · p. 12

Captioning efficiency

  • $10,275 to $1,026: progressively removing raw video, applying vision-to-language compression, and switching to Flash reduces captioning cost for one million samples.The comparison concerns one million 10-second samples processed with Gemini 2.5.Source: Table 6 · p. 13

Off-screen evaluation

  • The off-screen benchmark contains natural off-screen events and synthetic background-music generation tracks.Natural events are filtered for acoustic purity and modality conditions, while synthetic music mixes aligned MusicCaps clips into high-fidelity videos.Source: D. Off-Screen Track of VGGSound-Omni · p. 13
  • The benchmark extends prior synthetic robustness evaluation through greater realism, scale, and reliance on real-world off-screen events.The comparison with VinTAGe-Bench identifies these as three critical dimensions of extension.Source: D. Off-Screen Track of VGGSound-Omni · p. 13

E. Generalization on Third-Party Benchmarks.

Omni2Sound is evaluated across subjective, objective, and third-party settings, including heterogeneous benchmarks and human studies. The reported evaluations position it as competitive across unified audio-generation tasks while using a DiT-based implementation and SoundAtlas fine-tuning data.

Third-party benchmark generalization

  • Omni2Sound remains highly competitive on Kling-Audio-Eval despite a substantial data-scale and distribution gap.The comparison involves YouTube-sourced SoundAtlas and professionally sourced Kling video/Foley data.Source: E. Generalization on Third-Party… · pp. 13–14
  • Omni2Sound consistently outperforms strong baselines across V2A and VT2A tasks on the reported third-party evaluation.The cited baselines include MMAudio, AudioX, and ThinkSound, while HunyuanVideo-Foley leads on several metrics.Source: E. Generalization on Third-Party… · p. 14

Subjective evaluation

  • The user study assigns 16 expert evaluators to independent VT2A and V2A tracks to reduce evaluator fatigue.Each participant evaluates 20 random samples and 80 comparisons within the assigned track.Source: F. User Study · p. 14
  • Omni2Sound outperforms all baselines on MOS-Q, MOS-S, and MOS-T for both VT2A and V2A in the subjective evaluation.The three dimensions measure acoustic quality, semantic alignment, and temporal alignment on a 1–5 scale.Source: F. User Study · p. 14, Figure 4 · p. 14

Model configuration

  • The model uses a 24-layer, 24-head DiT with hidden dimension 1536 inside a latent diffusion model.Cross-attention injects semantic conditions, while AdaLN integrates temporal signals.Source: G. Implementation Details · pp. 14–15

Training data

  • SoundAtlas fine-tuning uses 470k V-A-T pairs sourced from 140k VGGSound and 330k AudioSet samples.The AudioSet portion is curated from a preliminary candidate pool after excluding speech- and music-related categories.Source: G. Implementation Details · p. 15

Objective evaluation

  • Objective evaluation uses standardized AV-benchmark tooling, common video and text conditions, and 8-second generated clips.Reported dimensions include distribution matching and audio quality, with distances and divergences computed from several audio embeddings and classifiers.Source: H. Objective Evaluation Metrics · p. 15

Related papers

Loading 2601.02731v3…