Source-linked AI summary
Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
Zeyue Tian, Binxin Yang, Zhaoyang Liu, Jiexuan Zhang, Ruibin Yuan, Hubery Yin, Qifeng Chen, Chen Li, Jing Lyu, Wei Xue, Yike Guo
TL;DR
Unified audio understanding, generation, and editing remain difficult because prior systems are specialized or limited in domain coverage, while editing data is scarce. Audio-Omni combines a frozen MLLM with a trainable DiT and introduces AudioEdit, achieving performance that surpasses prior unified approaches and matches or exceeds specialized models across several tasks.
Problem
Prior audio systems typically specialize in separate tasks or selected domains, and instruction-guided editing lacks large-scale paired data.
Method
Audio-Omni uses a frozen MLLM for multimodal reasoning and a trainable DiT for audio generation and editing, supported by AudioEdit’s hybrid data-construction pipeline.
Results
Audio-Omni outperforms prior unified models and matches or surpasses specialized expert models across understanding, generation, and editing tasks.
Takeaways & Limitations
The unified model also exhibits inherited knowledge-augmented reasoning, in-context generation, and zero-shot cross-lingual control capabilities.
Abstract
from arXiv · showhide
Recent progress in multimodal models has spurred rapid advances in audio understanding, generation, and editing. However, these capabilities are typically addressed by specialized models, leaving the development of a truly unified framework that can seamlessly integrate all three tasks underexplored. While some pioneering works have explored unifying audio understanding and generation, they often remain confined to specific domains. To address this, we introduce Audio-Omni, the first end-to-end framework to unify generation and editing across general sound, music, and speech domains, with integrated multi-modal understanding capabilities. Our architecture synergizes a frozen Multimodal Large Language Model for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis. To overcome the critical data scarcity in audio editing, we construct AudioEdit, a new large-scale dataset comprising over one million meticulously curated editing pairs. Extensive experiments demonstrate that Audio-Omni achieves state-of-the-art performance across a suite of benchmarks, outperforming prior unified approaches while achieving performance on par with or superior to specialized expert models. Beyond its core capabilities, Audio-Omni exhibits remarkable inherited capabilities, including knowledge-augmented reasoning generation, in-context generation, and zero-shot cross-lingual control for audio generation, highlighting a promising direction toward universal generative audio intelligence. The code, model, and dataset will be publicly released on https://zeyuet.github.io/Audio-Omni.
1 Introduction
Audio-Omni addresses the underexplored problem of unifying audio understanding, generation, and editing across general sound, music, and speech. It combines a decoupled MLLM–DiT architecture with AudioEdit and reports broad performance gains across unified and specialized models.
- Audio remains comparatively underexplored in unified multimodal understanding and generation, especially across its distinct sound, music, and speech domains.
- Prior audio-unification efforts cover only selected domains or rely on tool-based integration rather than end-to-end optimization.
- Audio-Omni uses a frozen Multimodal Large Language Model for reasoning and a trainable Diffusion Transformer for generation and editing.
- AudioEdit provides over 1M curated editing samples spanning addition, removal, extraction, and style transfer tasks.
- Experiments show Audio-Omni outperforms prior unified models across understanding, generation, and editing while matching or surpassing specialized experts on several tasks.
- The model inherits knowledge-augmented reasoning, in-context audio-conditioned synthesis, and multilingual cross-lingual control from the frozen MLLM.
2 Related Work
Prior audio research includes strong specialized understanding and generative systems, but unified coverage and instruction-guided editing remain limited. Large-scale paired editing data is a central unresolved need.
- Audio understanding has advanced across general sound, music, and speech, whereas generative models provide less comprehensive domain coverage.
- Recent systems support multimodal conditions or multiple audio domains, but instruction-guided audio editing remains underexplored.
- The field lacks large-scale paired datasets for instruction-guided audio editing.
3 Audio Editing Dataset Construction
AudioEdit addresses the scarcity and realism limitations of paired instruction-guided audio-editing data through a hybrid construction pipeline. The resulting dataset contains over 1M samples across four editing task types.
- Large-scale paired datasets for instruction-guided audio editing are scarce, limiting progress in the area.
- Prior synthetic pipelines can create a domain gap because mixing isolated audio segments does not reflect integrated real-world audio.
- AudioEdit combines real-world data mining with scalable synthesis to provide both acoustic fidelity and dataset diversity.
- AudioEdit contains over 1M samples covering add, remove, extract, and style transfer editing tasks.
4 Method
Audio-Omni combines a frozen multimodal reasoning module with a trainable diffusion-based generator through separate semantic and signal-level conditioning streams. Its method also uses a hybrid data-construction pipeline and a unified Rectified Flow training objective.
- Dataset Construction: AudioEdit uses parallel real-data mining and programmatic synthesis branches to combine acoustic fidelity with large-scale diversity.The real-data branch uses MLLM-based category identification and SAM-Audio source separation, while Scaper generates annotated editing scenarios.
- Training Objective: Rectified Flow training interpolates VAE-encoded target audio and random noise, then minimizes mean squared error on the predicted constant velocity.The model predicts v = x1 − x0 from the noisy state, timestep, and available conditioning signals.
- Model Architecture: Audio-Omni pairs a frozen MLLM understanding core with a trainable DiT backbone for versatile audio generation and editing.The MLLM processes textual, audio, and video inputs, directly supporting understanding tasks and producing conditioning features for generation.
- Model Architecture: The high-level conditioning stream concatenates MLLM multimodal features with transcript features for instructional semantic control.Transcript features are character-level encodings produced by a dedicated Transcript Encoder for speech-related tasks.
- Model Architecture: The low-level conditioning stream combines reference-audio or speech-prompt features with video synchronization features for temporally aligned editing and synchronization.Mel Encoder features and Synchformer features are concatenated into the low-level signal representation.
- Model Architecture: Distinct injection mechanisms route semantic features through cross-attention and fuse low-level features with time embeddings and noisy audio inputs.This decouples abstract instructional guidance from precise temporal signal guidance within the DiT backbone.
5 Experiments
Audio-Omni is evaluated as a unified system across understanding, generation, and editing, with additional tests of inherited and zero-shot capabilities. Results show strong general performance, while ablations examine data composition, conditioning, and feature-source choices.
- Main Results: Audio-Omni evaluates understanding, generation, and editing against specialized and unified models using task-specific benchmark metrics.Understanding uses MMSU and MMAU; generation uses KL divergence, IS, FAD, FD, and WER; editing uses LSD and FAD.
- Main Results: Audio-Omni outperforms most unified models on multi-domain understanding and approaches dedicated understanding specialists.MMSU covers 47 spoken-language tasks, while MMAU evaluates 27 reasoning skills across sound, music, and speech.
- Overall Performance: Audio-Omni achieves state-of-the-art or highly competitive generation and editing performance across the evaluated benchmarks.The evaluation includes AudioCaps, Musicaps, VGGSound, V2M-bench, and Seed-TTS for generation, alongside audio-editing benchmarks.
- Overall Performance: Audio-Omni maintains strong performance in CN, ES, DE, FR, and JP despite English-only training, with quality comparable to English-only specialist models.This zero-shot cross-lingual capability is attributed to multilingual understanding inherited from the frozen MLLM.
- Inherited Abilities and Zero-Shot Capabilities: Figure 4 showcases knowledge-augmented generation, in-context generation, zero-shot voice conversion, and zero-shot speech editing.The model infers an electric guitar from a world-knowledge prompt, transfers piano timbre to a new piece, and performs voice conversion and speech editing without task-specific supervision.
- Ablation Studies: Mixed real-world and synthetic data yields the best overall editing performance, while synthetic data alone does not robustly generalize to real-world acoustic complexity.Synthetic data provides broad editing-operation coverage, whereas real-world data contributes acoustic realism and fidelity.
- Ablation Studies: Penultimate-layer features provide the most effective conditioning for T2A and T2M, outperforming final-layer and query-based alternatives.The penultimate layer retains richer semantic and acoustic detail, while complex query mechanisms introduce detrimental information bottlenecks.
6 Conclusion
Audio-Omni unifies audio understanding, generation, and editing across sound, music, and speech through a frozen MLLM, a trainable DiT, and hybrid conditioning. Its experiments support strong generalist performance and inherited capabilities, including knowledge-augmented reasoning and zero-shot cross-lingual control.
- Conclusion: Audio-Omni unifies audio understanding, generation, and editing across general sound, music, and speech in an end-to-end framework.Its architecture combines a frozen MLLM for high-level reasoning with a trainable DiT and separates semantic from signal features through hybrid conditioning.
- Conclusion: AudioEdit provides over one million instruction-guided editing pairs to address a critical data bottleneck in audio editing.The dataset supports the unified model's editing capabilities.
- Conclusion: Audio-Omni matches or surpasses specialized expert models while exhibiting knowledge-augmented reasoning and zero-shot cross-lingual control inherited from the MLLM.The paper presents the framework as a scalable baseline toward universal generative audio intelligence.
1 Appendix
The appendix reports ablations, multilingual generation, detailed benchmark results, and human evaluations for Audio-Omni. Across these evaluations, the model shows strong editing, generation, and perceptual performance.
- Cross-lingual Generation: Audio-Omni maintains strong zero-shot text-to-audio performance across Chinese, Spanish, German, French, and Japanese despite predominantly English training.The cross-lingual evaluation uses translated AudioCaps test prompts.
- Audio Editing Results: Audio-Omni consistently achieves state-of-the-art performance across the four individual audio editing tasks.The AudioEdit benchmark reports FAD and LSD for each task and their average.
- Generation Results: Generation results are reported for T2A, T2M, V2A, V2M, and TTS using KL divergence, IS, FD, and FAD.The detailed breakdown appears in the multimodal generation benchmark results.
- Ablation Studies: Audio-Omni is evaluated with encoder ablations for text-to-audio and video-to-audio tasks.The comparison includes T5 and CLAP for text-to-audio, and CLIP and VideoMAE for video-to-audio.
- Human Evaluation: Audio-Omni receives competitive or superior human ratings across all evaluated generation and editing tasks.Twenty audio professionals rated randomly selected samples, with metrics scored from 1 to 100 and higher scores indicating better performance.
2 Ethics Statement
The ethics statement identifies misuse risks from generative audio, especially voice conversion and speech synthesis. It proposes responsible-use terms and encourages watermarking and detection safeguards.
- Risks: Voice conversion and speech synthesis could be misused for deepfakes, impersonation, and misinformation.The paper identifies these as ethical risks associated with generative audio technologies.
- Mitigations: Access will require responsible-use terms that explicitly prohibit malicious applications.The authors also encourage robust audio watermarking and detection methods.