Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing

Introduction

• Audio-Omni unifies understanding, generation, and editing across audio domains, addressing prior domain-limited unification and editing-only systems.Audio research remains comparatively underexplored, and existing approaches do not cover the full spectrum of general sound, music, and speech.Source: §1 · p. 2
• A frozen MLLM provides reasoning while a trainable DiT handles generation and editing, with semantic cross-attention guidance and signal-level conditioning for temporal control.The hybrid conditioning separates high-level semantic features from low-level mel-spectrogram and video-sync signals.Source: §1 · p. 2
• AudioEdit contains over 1M curated instruction-guided editing samples covering addition, removal, extraction, and style transfer.Its construction pipeline combines real-world data mining with scalable programmatic synthesis to address the scarcity of large-scale public editing data.Source: §1 · p. 2
• Audio-Omni outperforms prior unified models across understanding, generation, and editing, while matching or surpassing specialized experts on several tasks.Its generative module also inherits world-knowledge reasoning, in-context audio-conditioned synthesis, and multilingual cross-lingual control.Source: §1 · p. 2

Specialized Models in the Audio Domain

• Audio understanding models span all three domains, whereas generation remains specialized across general audio, music, and speech.Prior work includes text-to-audio [Evans et al. 2024; Ghosal et al. 2023; Huang et al. 2023; Majumder et al. 2024], text-to-music [Chen et al. 2024; Copet et al. 2024; Deng et al. 2024; He et al. 2024; Melechovsky et al. 2024; Yinghao et al. 2024; Yuan et al. 2025, 2024], TTS [Chen et al. 2025a; Deng et al. 2025; Du et al. 2024], and voice conversion [Peng et al. 2024; Wang et al. 2023a].Source: §2 · pp. 2–3
• Audio editing remains underexplored because large-scale instruction-guided data are scarce, and existing zero-shot and synthetic-data approaches have key limitations.Zero-shot methods can fail to preserve non-target content or follow precise instructions [Manor and Michaeli 2024]; synthetic pipelines can create a domain gap by mixing isolated segments that miss real-world acoustic complexity [Lan et al. 2025; Tao et al. 2025; Wang et al. 2023b].Source: §2 · p. 3

Unified Multimodal Models

• Unified multimodal models have advanced in vision and across modalities, but audio approaches may orchestrate separate experts or cover only speech or music.Tool orchestration lacks end-to-end optimization [Huang et al. 2024b], while integrated audio approaches are limited to speech-only [AI et al. 2025; An et al. 2024] or music-only generation [Liu et al. 2024a].Source: §2 · p. 3
• Audio-Omni is proposed as an end-to-end framework unifying understanding, generation, and editing across general sound, music, and speech.The authors position it as addressing the fragmentation of specialized models and the domain limits of prior unified audio efforts.Source: §2 · p. 3

3 Audio Editing Dataset Construction

• The real-data branch mines VGGSound clips, separates target and residual tracks, and filters pairs for add, remove, and extract tasks.From over 540K category-labeled samples, VAD filtering retains about 347K pairs and CLAP alignment about 50K; human validation achieves approximately 83% agreement.Source: §3 · p. 3
• For style transfer, Gemini-generated related keywords and CLAP filtering yield about 500K pairs, while ZETA changes target style while preserving temporal structure and pitch.The transformed target is mixed back with the residual track to form the edited output.Source: §3 · p. 3
• The synthesis branch uses Scaper to mix ESC-50 foreground events into 10-second AudioCaps backgrounds with randomized timing, SNR, pitch, and time-stretch settings.It generates annotated data for add, remove, and extract tasks; settings include SNR of 0–3 dB, pitch shifts of −3 to +3 semitones, and time-stretch factors of 0.8–1.2.Source: §3 · p. 3
• AudioEdit contains over 1M samples for add, remove, extract, and style transfer tasks, built through a hybrid real-world mining and synthesis pipeline.The dataset is designed for instruction-guided audio editing.Source: §3 · p. 3

4 Method

• AudioEdit pairs real-world editing data mined with MLLM-based category identification and source separation with programmatically generated, annotated scenarios.The real-data branch uses Gemini and SAM-Audio, while the synthesis branch uses Scaper to generate editing scenarios.Source: Figure 2 · p. 4

4.1 Preliminary

• Rectified Flow models a straight trajectory from random noise to data with a constant velocity, and inference solves the ODE backward from noise to data.The velocity predictor is conditioned on the noisy state, timestep, and available conditioning signals.Source: §4.1 · p. 4

4.2 Model Architecture

• A frozen MLLM handles multimodal understanding and supplies conditioning features to a trainable DiT backbone that synthesizes audio.The MLLM processes text, audio, and video; for understanding tasks it generates text responses, and for generation tasks it provides conditioning features from its penultimate layer.Source: §4.2 · p. 4
• The high-level stream combines MLLM features with transcript features for speech tasks, while the low-level stream combines reference-audio mel features with video synchronization features.Transcript features are character-level embeddings processed by a ConvNeXtV2-based encoder; mel features come from reference audio for editing or speech prompts for voice conversion, and synchronization features are extracted from video by Synchformer. The high-level stream enters the DiT through cross-attention, while low-level features are fused with the time embedding and concatenated with the noisy audio latent for frame-level guidance.Source: §4.2 · pp. 4–5

4.3 Training Objective

• Training minimizes mean squared error between the DiT’s predicted velocity and the constant velocity from the target-audio latent toward random noise.Each training sample uses a uniformly sampled timestep to interpolate between the VAE-encoded target audio and noise; conditioning includes the signals available for that sample.Source: §4.3 · p. 5

5.1 Implementation Details

• Audio-Omni has approximately 7.9B total parameters, including 3.05B trainable, and is trained end-to-end with prompt-mel masking; inference generates audio latents using a 100-step numerical ODE solver.Training randomly masks 20% to 75% of the prompt’s mel-spectrogram to support zero-shot voice cloning and content editing.Source: §5.1 · pp. 5–6

5.2 Evaluation Metrics

• Evaluation covers multimodal reasoning, audio-generation quality and distributional similarity, speech WER, and editing fidelity, perceptual quality, and instruction adherence.Metrics include MMSU and MMAU; KL divergence, IS, FAD, and FD on PANNs embeddings; WER; and editing LSD and FAD.Source: §5.2 · p. 6

5.3 Main Results

• Audio-Omni consistently outperforms other unified models in generation, surpasses specialist models on T2M and TTS, and achieves the best editing performance across reported metrics.Editing metrics average FAD and LSD across add, remove, extract, and style-transfer tasks, with CLAP measuring instruction adherence.Source: §5.3.1 · p. 6, Table 3 · p. 6, Table 4 · p. 6
• Audio-Omni performs strongly on MMSU and MMAU, outperforming most unified models and approaching dedicated understanding specialists.MMSU covers 47 spoken-language tasks; MMAU evaluates 27 reasoning skills across sound, music, and speech.Source: §5.3.1 · p. 6

5.3.2 Zero-shot Cross-lingual Text-to-Audio Generation.

• Despite English-only training, Audio-Omni maintains strong generation performance in Chinese, Spanish, German, French, and Japanese, comparable to English-only specialist models.The reported capability is zero-shot cross-lingual generation.Source: §5.3.2 · p. 6

5.3.3 Inherited Abilities and Zero-Shot Capabilities.

• Audio-Omni demonstrates knowledge-augmented and in-context generation, plus zero-shot voice conversion and speech editing enabled by masking-based training.Examples include inferring electric guitar from a prompt about Jimi Hendrix and transferring a piano recording’s timbre to a new piece; Figure 4 showcases all four capabilities.Source: §5.3.3 · p. 7, Figure 4 · p. 7

5.4 Ablation Studies

• Mixed synthetic and real-world training data performs best overall; synthetic data adds editing-operation coverage, while real-world data improves acoustic realism and fidelity.Synthetic-only training is insufficient for robust generalization to real-world acoustic complexity.Source: §5.4 · p. 7, Table 5 · p. 7
• The strongest conditioning strategy supplies high-level multimodal and transcript features through context, while concatenating synchronization and mel features with input noise.The study compares cross-attention context and channel-wise concatenation across four injection strategies.Source: §5.4 · p. 7, Table 6 · p. 7

6 Conclusion

• Audio-Omni matches or surpasses specialized expert models and exhibits inherited knowledge-augmented reasoning and zero-shot cross-lingual control.The conclusion attributes these inherited abilities to the MLLM.Source: §6 · p. 8
• Audio-Omni is presented as the first end-to-end framework unifying audio understanding, generation, and editing across sound, music, and speech.Source: §6 · p. 8
• Its decoupled architecture combines a frozen MLLM for high-level reasoning with a trainable DiT, using hybrid conditioning to separate semantic and signal features.Source: §6 · p. 8
• AudioEdit addresses a data bottleneck with over one million instruction-guided audio editing pairs.Source: §6 · p. 8

1.3 More Ablation Studies

• In T2A and V2A ablations, the unified frozen Qwen-Omni-3B encoder outperforms specialized text and vision encoders across all metrics.The authors attribute the advantage to richer semantic understanding and joint processing of multimodal contexts and cross-modal relationships.Source: Table A1 · p. 11, §1.3 · p. 11

1.1 Dataset Details

• Training covers T2A (1.4k hours), V2A (700), T2M (17k), V2M (16k), speech (6k), and audio editing (3k).Data sources include AudioCaps, WavCaps, AudioSetCaps [Bai et al. 2025], AudioTime [Xie et al. 2025b], VGGSound [Chen et al. 2020], AudioSet Strong [Hershey et al. 2021], V2M [Tian et al. 2025c], MUCaps [Liu et al. 2024a], Audio-FLAN [Xue et al. 2025], and AudioEdit. For style transfer, Gemini 2.5 Pro proposes related but stylistically different keywords, which are filtered by CLAP before ZETA guides transformation.Source: §1.1 · p. 11

1.2 More Comparison Results

• Audio-Omni achieves state-of-the-art performance across four audio-editing tasks and strong zero-shot cross-lingual T2A results, including from non-English prompts despite almost exclusively English training.The cross-lingual evaluation translated AudioCaps prompts into Chinese, Spanish, German, French, and Japanese; Chinese-prompt output was comparable to strong English-only specialists.Source: Table A2 · p. 11, §1.2 · p. 11, Table A3 · p. 12

1.4 Human Evaluation

• Audio-Omni receives competitive or superior human ratings across generation and editing tasks.Twenty audio professionals rated 20 randomly selected samples per task on Overall Quality (OVL) and Relevance (REL), each on a 1–100 scale.Source: §1.4 · pp. 11–12, Table A6 · p. 12

• Voice conversion and speech synthesis could be misused to create deepfakes, impersonate others, or spread misinformation.Source: §2 · p. 12
• Access will require agreement to terms prohibiting malicious use; the authors also encourage robust audio watermarking and detection methods.They state that, with proper safeguards, Audio-Omni can support creative expression and scientific research.Source: §2 · p. 12