Source-linked AI summary
MMAE: A Massive Multitask Audio Editing Benchmark
Ziyang Ma, Ruiqi Yan, Ruiyang Xu, Jie Fang, Zhikang Niu, Yi-Wen Chao, Wenming Tu, Tianrui Wang, Auden, Qi Chen, Wenxi Chen, Jiaying Chi, Yanru Huo, Zixuan Jiang, Xiquan Li, Yalin Li, Junxi Liu, Minghao Liu, Binghao Qiang, Yijia Shan, Zheshu Song, Tian Tan, Zixiang Wang, Zeyu Xie, Zhifei Xie, Xiaoyu Xing, Qixiang Xu, Chen Yang, Guanrou Yang, Shan Yang, Yifan Yang, Steve Yves, Haotian Zhang, Haina Zhu, Kai Yu, Liefeng Bo, Eng-Siong Chng, Xie Chen
TL;DR
Audio-editing evaluation remains fragmented across domains and basic operations, limiting comprehensive assessment of general-purpose instruction-based systems. MMAE introduces a broad, rubric-based benchmark, finding that current models achieve EMRs below 5% overall and 0% on complex mixed-modality tasks.
Problem
Audio-editing evaluation remains fragmented across domains and basic operations, lacking comprehensive coverage of modalities and complex instruction-based tasks.
Method
MMAE combines a taxonomy across modalities, complexity, granularity, and operations with rubric-based evaluation of instruction following and content consistency.
Results
Exact Match Rate remains below 5% across evaluated models and reaches 0% on complex mixed-modality tasks.
Takeaways & Limitations
Current audio-editing models remain far from the reliable and precise editing demanded by real-world applications.
Abstract
from arXiv · showhide
We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation, interactive editing has rapidly expanded from visual domains, pioneered by models like Nano-banana 2 for images and Gemini-Omni for video, into audio. However, the current evaluation infrastructure lags severely, remaining highly fragmented and restricted to specific subdomains or basic operations. Unlike existing benchmarks that are limited in scope, MMAE extends to a broad spectrum of real-world scenarios, encompassing 7 distinct audio modalities, including sound, speech, music, and their mixtures. Furthermore, we establish a comprehensive taxonomy spanning 6 levels of task complexity, from basic modifications to multi-hop reasoning and multi-round editing, 2 levels of granularity, and 8 distinct operation types. Meticulously curated through human-agent collaboration, MMAE comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework. By decomposing free-form tasks into 17,741 verifiable criteria, this robust rubric-based paradigm enables a precise, multi-dimensional assessment of both instruction following and context consistency. Our extensive evaluation of leading models reveals that current systems remain far from achieving reliable edits. Strikingly, the Exact Match Rate (EMR) consistently falls below 5% and plummets to an absolute 0% in complex, mixed-modality tasks, exposing critical bottlenecks in precise execution and structural robustness. We hope MMAE will serve as a catalyst for future advances in the intelligent creation community, providing a clear diagnostic roadmap and establishing a standardized, long-lasting evaluation paradigm for next-generation audio editing systems.
1 Introduction
MMAE addresses fragmented audio-editing evaluation by introducing a comprehensive benchmark for general-purpose instruction-based editing. Its taxonomy, rubric-based assessment, and model evaluation expose substantial limitations in current systems.
- Motivation: Existing audio-editing benchmarks are fragmented across subdomains and basic operations, leaving evaluation infrastructure substantially behind rapidly evolving systems.
- Benchmark: MMAE is introduced as the first comprehensive benchmark for general-purpose instruction-based audio editing.
- Benchmark: MMAE organizes evaluation across 7 audio modalities and 6 task-complexity levels, including multi-hop reasoning and context-aware multi-round editing.
- Evaluation: MMAE pairs audio-instruction instances with tailored rubrics and systematic metrics to quantify editing outcomes and rigorously assess performance.
- Results: EMR remains below 5% across all evaluated models and reaches 0% in complex, mixed-modality scenarios, revealing that reliable, flawless editing remains unsolved.
2 Related Work
Prior audio-editing research progressed from constrained modality- and operation-specific systems toward instruction-guided, compositional, and unified editing across audio domains. However, evaluation remained fragmented and domain-specific, motivating MMAE as a comprehensive benchmark for universal audio-editing evaluation.
- Prior Audio Editing: Earlier audio-editing works were generally restricted to specific modalities or predefined operation types.Recent research broadened the field toward natural-language instruction guidance, cross-modal and multi-domain generalization, and open-ended compositional editing.
- Prior Audio Editing: AUDIT introduced text-guided sound-event addition, removal, and replacement using a latent diffusion model trained on synthetic triplets.AudioEditor and AudioMorphix later achieved comparable editing capabilities through training-free methods.
- Speech Editing: Speech-editing systems addressed zero-shot content editing, lightweight post-training, and nuanced control over emotional and paralinguistic attributes.VoiceCraft used neural codec token rearrangement, CosyEdit adapted pretrained zero-shot TTS models, and Step-Audio-EditX extended expressive control.
- Unified Audio Editing: Recent unified systems combined understanding, generation, and editing across sound, music, speech, and audio-visual domains.Audio-Omni and AudioChat pursued unified audio capabilities, while InstructAV2AV and SpongeBob added synchronized joint audio-visual editing.
- Evaluation Gap: A comprehensive audio-editing benchmark remained absent because existing evaluations were fragmented and strictly domain-specific.Speech benchmarks such as RealEdit focused on localized insertion, deletion, and substitution evaluated with WER and speaker similarity.
- MMAE: MMAE was introduced as the first comprehensive benchmark for universal audio-editing evaluation across sound, music, speech, and mixed audio.It establishes a systematic taxonomy spanning modality, complexity, and operation dimensions, and pairs diverse real-world audio samples with human-annotated natural-language instructions.
3 MMAE
MMAE is a comprehensive benchmark for instruction-based audio editing that tests perception, reasoning, and high-fidelity generation across diverse task configurations. Its rubric-based evaluation decomposes open-ended edits into verifiable criteria measuring instruction following and context consistency.
- Benchmark Scope: MMAE evaluates instruction-based audio editing through integrated perception, reasoning, and high-fidelity generation.Models must understand source-audio context, interpret complex or implicit intent, and execute the requested edits.
- Task Taxonomy: Seven modality categories combine sound, music, and speech, while six complexity levels span single, multi-part, multi-instruction, multi-audio, multi-round, and multi-hop tasks.The taxonomy also organizes operations into local edits—addition, removal, replacement, extraction, and alteration—and global edits such as background change, foreground change, and global alteration.
- Evaluation Framework: Rubric-based evaluation uses atomic, independent, observable, and verifiable multiple-choice criteria to diagnose multifaceted editing performance.Instruction Following measures whether requested modifications are executed precisely, whereas Consistency measures whether irrelevant acoustic elements remain unaltered.
- Benchmark Statistics: 2,000 samples and 17,741 rubrics form the benchmark, averaging 8.87 rubrics per sample, including 3.58 Instruction Following and 5.29 Consistency rubrics.Samples average 14.46 seconds, 1.22 editing operations, and 14-word instructions; rubric questions average 25.45 words with 3.53 choices.
- Data Curation: MMAE is built through a systematic five-stage pipeline combining expert brainstorming, taxonomy and paradigm construction, instruction-centric collection, and human-agent collaborative annotation.Instruction-centric collection dynamically balances coverage across modality, complexity, and operation dimensions.
4 Experimental Setup
The experiments evaluate five recent end-to-end audio editing models on MMAE, with SmartDJ additionally tested using an external planner. Evaluation also includes identity and noise baselines and a majority-vote rubric judge based on Qwen3-Omni.
- Compared models: Five models—Step-Audio-EditX, Ming-UniAudio, MMEdit, Audio-Omni, and SmartDJ—are evaluated on MMAE.All are end-to-end systems.
- Compared models: SmartDJ is evaluated both without a planner and with Gemini 2.0 Flash, which decomposes complex instructions into sequential atomic edits.These settings are denoted SmartDJ w/o planner and SmartDJ w/ planner.
- Baselines: Identity and Noise provide complementary baselines: unchanged input audio tests consistency, while matching-duration Gaussian noise preserves no meaningful content.Identity is an upper bound on consistency and a lower bound on instruction following.
- Evaluation protocol: Three independent Qwen3-Omni judgments are aggregated by majority vote, assigning rubric score 1 when at least 2/3 judgments match ground truth and 0 otherwise.Option positions are randomly shuffled per query to mitigate positional effects.
5 Experimental Results
MMAE remains highly challenging for current audio editing systems: EMR stays below 5% across models and reaches 0% in complex mixed-modality settings. Performance worsens with complexity and modality mixing, while average instruction-following and consistency do not reliably translate into flawless edits.
- Main results: EMR remains below 5% across evaluated models and reaches 0% in complex mixed-modality settings.These results indicate that current systems struggle with precise, flawless audio editing.
- Main results: 44.86% IFR and 58.88% CR make Step-Audio-EditX the strongest full-set baseline, while Ming-UniAudio records 29.82% IFR and 52.71% CR.IFR denotes Instruction Following Rate, and CR denotes Consistency Rate.
- Complexity and modality: 58.43% to 41.70% Audio-Omni IFR and 64.57% to 47.94% CR show degradation from single to multiple complexity tasks.All models exhibit a clear performance drop as complexity increases, and mixed-modality tasks are generally harder, with Sound-Music-Speech consistently lowest.
- IFR–CR trade-off: 94.13% CR but 27.37% IFR for Identity and 32.08% IFR but 15.68% CR for Noise demonstrate a fundamental IFR–CR trade-off.Identity can preserve content without following requested edits, whereas Noise can accidentally satisfy deletion-verification rubrics.
- Average versus flawless editing: 44.86% versus 29.82% IFR and 58.88% versus 52.71% CR for Step-Audio-EditX versus Ming-UniAudio do not guarantee superior EMR.The results reveal a divergence between average dimension scores and perfect editing rates.
- Agent-guided planning: 42.26% versus 38.20% IFR and 48.33% versus 55.41% CR show that SmartDJ’s planner raises IFR but lowers CR without improving overall EMR.Error analysis attributes the limited benefit to bottlenecks in understanding and generation.
6 Conclusion
MMAE is presented as the first comprehensive benchmark for instruction-guided audio editing across sound, music, speech, and mixtures, combining a systematic taxonomy with rubric-based evaluation. Results show current models remain unreliable and imprecise, motivating research on atomic fidelity, universal modality support, and robust agent-guided compositional editing.
- Benchmark contribution: MMAE is the first comprehensive benchmark for evaluating instruction-guided audio editing across sound, music, speech, and their mixtures.It addresses the lack of a unified and rigorous evaluation framework by introducing systematic taxonomy and rubric-based evaluation.
- Evaluation findings: Exact match rates below 5% show that current audio editing systems remain far from reliable and precise edits.Performance is low across all evaluated models, with further degradation on complex multi-operation and mixed-modality tasks.
- Future directions: MMAE identifies atomic editing fidelity, universal modality support, and robust agent-guided compositional editing as key research directions.The benchmark is intended to help the community track progress, identify bottlenecks, and guide future audio editing research.
Appendices · A Demo Examples
The appendix demonstrates MMAE’s task diversity through six audio-editing cases spanning modalities, complexity levels, granularities, and operation types. Their rubrics verify both requested edits and context consistency using targeted perceptual, linguistic, and structural checks.
- A Demo Examples: Six examples span multi-audio, multi-round, multi-hop, multi-part, multi-instruction, and single-edit settings across music, speech, sound, and mixed modalities.The cases cover global and local edits, including foreground change, replacement, removal, background change, alteration, and extraction.
- A Demo Examples: A multi-audio music case replaces all lyrics with “Hachimi” while transferring the human-voice timbre from audio1.Its rubric checks sung-word content and whether the output voice timbre resembles the reference audio.
- A Demo Examples: A multi-round speech case sequentially swaps the order of the first–second and second–third authors mentioned in a commit.The rubric compares output segments with input segments to verify the first three names and subsequent name identities.
- A Demo Examples: A local sound-removal case removes barks from younger dogs while preserving audio quality and the intended remaining bark characteristics.Rubrics check for residual barking, degradation, and whether the output contains one or multiple bark types consistent with the input.
- A Demo Examples: A mixed sound-music-speech case changes accented Chinese dialogue to Standard Mandarin pronunciation.The rubric compares pronunciation against the input and evaluates additional output content properties.
- A Demo Examples: A music-speech case replaces background music with a guitar preserving the exact melody, deepens vocals, and retains spoken content.Rubrics assess relative vocal pitch, recognition of two spoken sentences, and consistency of the clean simultaneous ending.
- A Demo Examples: A local extraction case isolates dog-produced sounds while suppressing human speech and environmental background noise.Together with the other cases, its rubric-oriented presentation illustrates how demonstrations encode both target edits and preservation constraints.
B Final Meta-Data Format
Each MMAE sample is released as a JSON object containing task metadata, audio references, textual instructions, and rubric-based evaluation annotations. The example encodes a single sound-speech extraction task at local granularity and evaluates instruction following through targeted audio questions.
- Data structure: Each MMAE sample is stored as a JSON object in the final released format.The format is illustrated with a complete sample annotation.
- Data structure: The metadata specifies an identifier, complexity, modality, granularity, and operation, exemplified by single, sound-speech, local, and extraction.The example also includes an audio URL linking to the source file.
- Task specification: The example instruction asks the system to isolate dog sounds while suppressing human speech and environmental background noise.Its associated semantic labels include acoustic event detection, canine vocalizations, non-human audio signal, and background-noise and speech suppression.
- Evaluation annotations: Instruction-following rubrics test whether human speech is absent, whether human vocal timbre is absent in a specified interval, and whether canine vocalizations remain audible.The example uses questions over the full output and timed segments, with predefined right choices and distractors.
C Evaluation Prompt
The evaluation prompt instructs Qwen3-Omni to answer rubric questions using only evidence from the provided audio, with explicit clip-slicing conventions and a constrained JSON response format.
- Prompt role: The system prompt defines Qwen3-Omni as an audio analysis assistant that must answer rubric questions from the factual content of provided clips.It requires careful listening and analysis before answering.
- Audio reference notation: Audio references support full clips and Python-style time slices with optional boundaries and negative offsets counted from the clip end.Examples include <audio output[0.0s:3s]>, <audio input1[:2.0s]>, <audio output[1.5s:]>, and <audio input2[-2.5s:]>.
- Answering guidelines: The guidelines require evidence-based, concise answers, prohibit unsupported details, and require explicitly acknowledging insufficient audio information.They also encourage references to relevant events, sounds, speech content, and timing.
- Response format: For each multiple-choice rubric question, the judger must select exactly one option and return only JSON with “reason” first and an uppercase-letter “choice” last.The required keys are “reason” and “choice”.
D Data Curation Platform
Data annotation and quality inspection were conducted on a professional platform supporting structured editing, version control, and multi-stage review. The system also handled annotation, review, and correction of all metadata and rubrics.
- Platform workflow: A professional platform supported structured editing, version control, and multi-stage review for annotating, reviewing, and correcting all metadata and rubrics.The platform was used for both data annotation and quality inspection.