Source-linked AI summary
SMEBENCH: Benchmarking Neural Music Editing with Synthetized MIDI Edits
Matthieu Cervera, Francesco Paissan, Micael Antunes, Mirco Ravanelli, Ricard Marxer, Cem Subakan
TL;DR
Music-editing evaluations often lack paired targets, making preservation and intended-edit agreement difficult to assess separately. SMEBench constructs paired references through controlled symbolic MIDI edits and uses them to evaluate metrics and editing systems. Its findings show that source-, target-, and text-based metrics can assess the same output differently.
Problem
Method
SMEBench applies known edits to symbolic performances, renders matched source and target audio, and evaluates outputs against source, target, and text references.
Source: §1 · p. 2, §4.2 · p. 7Results
Paired references show that source preservation, target agreement, and textual compatibility can yield different assessments of the same output.
Source: §1 · p. 2Takeaways & Limitations
Paired targets provide a diagnostic complement to unpaired benchmarks by revealing what evaluation metrics reward.
Source: §5 · p. 9The current taxonomy covers twelve MIDI-expressible operations and excludes changes such as vocals, genre transfer, and production effects that require richer audio representations.
Source: §3.1 · p. 5, §5 · p. 9
Abstract
from arXiv · showhide
Neural music editing requires a system to perform a requested change while preserving the musical content that should remain unchanged. Existing evaluations often lack a paired target edit, making these requirements difficult to measure separately. We introduce SMEBench, a large-scale benchmark of controlled music edits constructed by modifying symbolic MIDI performances and rendering matched source and target audio. Each example contains a source, an instruction, a paired target, edit metadata, and complementary descriptions. The benchmark covers twelve instrumentation, articulation, dynamics, and pitch operations and includes a balanced 3,600-example test set together with a smaller real-audio complement derived from multitrack recordings. The paired targets allow source preservation, target agreement, and textual compatibility to be evaluated independently. We use these references to diagnose common audio-editing metrics and benchmark six representative editing systems, showing that the metric used to rank a system can materially change the conclusion. The dataset is available at our repository.
1 INTRODUCTION
Music-editing evaluation without paired targets can conflate source preservation, text compatibility, and edit correctness. SMEBench provides controlled source–target pairs and uses them to examine both editing systems and the metrics used to assess them.
1 INTRODUCTION
- Without a paired target, preservation, intended-edit agreement, and textual compatibility are difficult to measure separately.Source similarity may reward an unchanged output, while text–audio alignment may miss local musical changes.Source: §1 · pp. 1–2
- SMEBench modifies symbolic performances and renders matched source and target audio, with instructions, edit metadata, and complementary descriptions.The benchmark contains 11,160 paired examples and a balanced 3,600-example test set across twelve edit types.Source: §1 · p. 2
- Paired references show that source preservation, target agreement, and textual compatibility can yield different assessments of the same editing output.This motivates reference-based evaluation as a diagnostic complement to unpaired benchmarks.Source: §1 · p. 2
- The paper benchmarks six representative editing systems and evaluates their outputs separately against the source and intended target.The systems are SDEdit, DDIM inversion, ZETA, Virtual Consistency Inversion, SteerMusic, and MusicGen.Source: §1 · p. 3
2 RELATED WORK
Prior music-editing resources offer paired supervision or realistic recordings, but often do not provide paired targets from controlled interventions on musical content. SMEBench complements these resources with known symbolic edits and reference audio.
2 RELATED WORK
- Existing resources leave a need for paired targets produced through explicit interventions on musical content.Some datasets derive targets through signal-level or stem-level manipulation, while other evaluation sets lack target audio references.Source: §2 · p. 3
- AUDIT and InstructME provide paired editing data, but their targets do not expose SMEBench’s controlled symbolic changes to instrumentation and performance attributes.Their targets are largely constructed through signal-level operations or stem manipulation.Source: §2 · p. 3
- MedleyMDPrompts and ZoME-Bench use real recordings but lack reference-edited audio, so edit execution cannot be measured against a known target.Their designs rely on text alignment or changes relative to the source to infer edit execution.Source: §2 · p. 3
3 SMEBENCH: A RECIPE FOR CONTROLLED PAIRED MUSIC EDITS
SMEBench constructs paired music edits by applying known interventions to symbolic performances, rendering matched audio, and grounding complementary descriptions in structured information. Its recipe supports controlled comparisons while allowing extensions to MIDI-expressible edits.
3 SMEBENCH: A RECIPE FOR CONTROLLED PAIRED MUSIC EDITS
- The construction recipe links an eligible edit to a symbolic intervention, matched source–target rendering, audio and caption checks, and a paired benchmark entry.Figure 1 summarizes the pipeline from edit sampling through rendering and textual grounding.Source: §3 · p. 3, Figure 2 · p. 4
3.1 EDIT CONSTRUCTION
- Twelve operations cover instrumentation, articulation, dynamics, and pitch; edits are selected only when the excerpt contains an eligible target.Eligible instruments are weighted by perceived usage, while MIDI-expressible additions can extend the recipe; attributes such as vocals or genre require complementary audio representations.Source: §3.1 · pp. 4–5, Figure 2 · p. 4
3.2 CAPTIONS
- Four textual views describe source audio, edited audio, the requested intervention, and a normalized target description.Structured musical descriptors and edit metadata ground the captions, which are checked for operation alignment and consistency.Source: §3.2 · p. 5
3.3 BENCHMARK COMPOSITION
- The corpus contains 11,160 paired edits, including a balanced 3,600-example test set with 300 examples per operation.SME-Real adds 416 paired edits from multitrack recordings; it tests whether findings persist with recorded stems but does not reproduce the synthetic benchmark’s coverage.Source: §3.3 · p. 5
- Symbolic construction records the applied operation but does not itself establish audibility, musical coherence, or accurate description.The benchmark entries are validated before they are used to diagnose metrics or compare editing systems.Source: §3.3 · p. 6
4 EXPERIMENTS
The experiments validate the paired targets and captions, examine what source-, target-, and text-based metrics measure, and benchmark editing systems under caption and instruction conditioning. They find that metric rankings can reverse, while human ratings and operation-level analyses qualify how the benchmark should be interpreted.
4.1 VALIDATING TARGETS AND CAPTIONS
- Across 236 edit ratings, targets averaged 4.23 for quality, 4.65 for source alignment, and 3.99 for instruction alignment; retained captions averaged 3.70.Instruction alignment was good or very good in 68.7% of ratings and bad or very bad in 13.2%.Source: §4.1 · p. 6, Table 1 · p. 6
- Three operations—LEGATO, REMOVE_INSTRUMENT, and SWAP_DRUMS—did not reliably exceed the instruction-following midpoint and were excluded from the headline perceptual aggregate.For the remaining nine operations, mean instruction following was 4.30, source consistency 4.69, and overall quality 4.43; source consistency remained at least 4.3 for the flagged operations.Source: §4.1 · p. 6
- The caption comparison assesses retained captions but does not validate the LLM filter’s rejection decisions; shared Qwen model-family use is treated as a correlated-error limitation.The experiments use a balanced 3,600-example test split unless otherwise stated, and report the 416-example SME-Real test set separately.Source: §4 · p. 6, §4.1 · p. 7
4.2 DIAGNOSING MUSIC-EDITING METRICS
- Source-reference metrics assess preservation, target-reference metrics assess agreement with the edited outcome, and text–audio metrics assess compatibility; these axes are related but not interchangeable.Identity is the unchanged source and a preservation endpoint, while Ground Truth is the paired target; text similarity is useful for editing insofar as it distinguishes them when the instruction describes their difference.Source: §4.2 · pp. 7–8
- On SMEBench, source and target audio averaged 0.873 CLAP-Audio similarity and 1.513 LPAPS distance; on SME-Real, the corresponding values were 0.804 and 2.167.Extraction alters the global scene while articulation and pitch edits are more localized, so distances should not be compared on the same absolute scale.Source: §4.2 · p. 7, Table 2 · p. 7
- Metric values vary by operation: extraction has the largest LPAPS distances and lowest CLAP-Audio similarities, while normalized-caption scores vary less across groups than instruction and edited-caption scores.Figure 3 groups the twelve operations into eight edit groups; the relative stability of normalized-caption scores is not by itself evidence of a better metric.Source: Figure 3 · p. 7, §4.2 · p. 8
4.3 BENCHMARKING MUSIC-EDITING SYSTEMS
- The benchmark compares five zero-shot editing systems using a shared AudioLDM2 backbone, alongside MusicGen as a text-conditioned generation baseline without source-preserving inversion.Outputs are evaluated against source and target with CLAP-Audio and LPAPS, with CLAP-Text under available textual conditions and Audiobox Aesthetics as a no-reference quality estimate.Source: §4.3 · p. 8
- Caption conditioning favors MusicGen by CLAP-Text but SteerMusic by both target metrics; instruction conditioning favors VCI by CLAP-Text but ZETA by both target metrics.Table 3’s rankings are diagnostic rather than exhaustive because some edits admit multiple valid realizations and the paired target is one controlled realization.Source: §4.3 · p. 8, Table 3 · p. 9
5 CONCLUSIONS AND LIMITATIONS
SMEBench uses paired targets to assess preservation, edit agreement, and textual compatibility separately, and experiments show that source-, target-, and text-based scores can rank systems differently. Its controlled renderings, twelve-operation MIDI taxonomy, and reliance on automatic filtering constrain the benchmark’s scope.
5 CONCLUSIONS AND LIMITATIONS
- Source-, target-, and text-based scores need not produce the same system ranking, making paired targets useful for diagnosing what evaluation metrics reward.The benchmark is designed to reveal these differences rather than only compare editors.Source: §5 · p. 9
- The controlled rendering leaves a gap from recorded music that the smaller real-audio complement only partially addresses.The current taxonomy covers twelve MIDI-expressible operations, excluding vocals, production effects, and genre transfer.Source: §5 · p. 9
- Automatic audio and caption filtering can introduce bias from the pretrained models on which it relies.Some operations are also less reliably audible than others, and synthesized edits represent one plausible solution among multiple valid alternatives.Source: §5 · p. 9
- No single metric captures preservation, edit execution, and perceptual quality, so benchmark results should be interpreted across these axes rather than reduced to one aggregate score.Source: §5 · p. 9
AI USE STATEMENT
The authors describe generative AI assistance in manuscript preparation and experimental work, while distinguishing those uses from formulating the benchmark’s central claims, designing its evaluation, and interpreting results.
AI USE STATEMENT
- Generative AI assisted with drafting, editing, typesetting, references, literature searches, and implementation of experimental methods.Source: AI USE STATEMENT · p. 10
- Qwen3-30B-A3B-Instruct-2507 produced and assessed grounded descriptions, while an independent Mistral checkpoint analyzed judge agreement.Source: AI USE STATEMENT · p. 10
- The authors state that generative AI did not formulate the benchmark’s central claims, design its evaluation methodology, or interpret the experimental results.They report manually verifying AI-assisted text and citations, generated captions, quality control, and experimental code.Source: AI USE STATEMENT · p. 10
REPRODUCIBILITY STATEMENT
The paper documents its benchmark construction, evaluation protocol, and released artifacts, situates audio-editing pipelines, and reviews fidelity and text-consistency metrics. It notes that editing evaluation must balance input fidelity with alignment to the requested transformation, which no single metric jointly captures.
REPRODUCIBILITY STATEMENT
- The benchmark release includes source and target audio, MIDI, instructions, captions, edit metadata, component masks, and generation code, with construction and evaluation details documented in the paper and appendices.Section 3 and Appendices B–D cover construction; Section 4 and Appendices F–G cover evaluation and system results.Source: REPRODUCIBILITY STATEMENT · p. 10
A AUDIO EDITING PIPELINES IN RELATED WORK
- Audio-editing pipelines include closed-set instruction systems, latent-space editing, diffusion inversion, and alternative interfaces such as Recomposer’s event-activity conditioning.The reviewed approaches include AUDIT and InstructME, PPAE and SteerMusic, and ZETA and SDEdit.Source: A AUDIO EDITING PIPELINES IN RELATED… · p. 13
A.1 AUDIO EDITING EVALUATION METRICS
- Editing evaluation must balance fidelity to the input with alignment to the requested transformation, and no single metric jointly captures both aspects.Fidelity metrics include embedding similarity and distributional measures such as FAD; text consistency commonly uses CLAP embeddings.Source: §A.1 · p. 13
B EDIT IMPLEMENTATIONS
The benchmark implements instrument, articulation, dynamics, and pitch edits through explicit MIDI or audio-domain transformations. Eligibility rules select meaningful edits, while rendering and filtering constrain pairs to changes and audio quality within chosen thresholds.
B EDIT IMPLEMENTATIONS
- Instrument edits remove or extract tracks, replace an instrument or drum sound, or double a selected instrument by adding a track playing its notes.Source: B EDIT IMPLEMENTATIONS · p. 14
- Staccato shortens notes to 40% of their onset-cluster IOI, clipped to 40–120 ms; legato extends notes into the next cluster by 15% of the IOI.Monophonic legato selections also receive MIDI control messages to activate legato and monophonic behavior and reinforce glide.Source: B EDIT IMPLEMENTATIONS · p. 14
- Crescendo and decrescendo apply a gain envelope to synthesized audio because editing MIDI velocity alone produced poor perceptual results; pitch correction perturbs source notes by up to 50 cents before using conventionally tuned MIDI as target.The envelope uses f(x)=x for crescendo and f(x)=1−x for decrescendo.Source: B EDIT IMPLEMENTATIONS · p. 14
C EDIT ELIGIBILITY AND SAMPLING
- Edits are excluded when their target is absent or barely active, the change would be imperceptible, or the resulting target would be empty or nearly empty.For example, an excerpt containing only piano excludes drum edits and instrument extraction.Source: C EDIT ELIGIBILITY AND SAMPLING · p. 14
- Instrument selection probabilities prioritize salient instruments while retaining a non-zero chance of editing less prominent ones; perceived usage is estimated from note activity and velocity.Source: C EDIT ELIGIBILITY AND SAMPLING · pp. 14–15
- Table 4 reports instruction-following, source consistency, and overall quality by edit operation; daggered operations are excluded from the headline perceptual aggregate only.Source: Table 4 · p. 15
D AUDIO RENDERING AND FILTERING
- Source and edited MIDI pairs use the same sampled SoundFont, while filtering rejects negligible changes, overly large changes, and audio below a minimum mean Audiobox Aesthetics score.The thresholds use percentile-based rules intended to remove distributional extremes, not define a universal perceptual boundary for a successful edit.Source: D AUDIO RENDERING AND FILTERING · p. 15
E HUMAN-EVALUATION DETAILS
The section describes human-rating criteria and the scale used, alongside global audio and text metric distributions on the test split.
E HUMAN-EVALUATION DETAILS
- Human evaluators rated target quality, source alignment, instruction alignment, and caption quality from 1 (very bad) to 5 (very good).Table 4 reports edit-wise ratings used to identify operations whose instruction-following ratings do not reliably clear the scale midpoint.Source: E HUMAN-EVALUATION DETAILS · p. 15
- Figure 6 presents source–target LPAPS distance and CLAP-Audio similarity distributions for the 11,160-example test split.Source: Figure 6 · p. 16
- Figure 7 presents global CLAP-Text distributions for edit captions, instructions, and normalized captions on the 11,160-example test split.Source: Figure 7 · p. 16
F METRIC BEHAVIOR BY EDIT AND TARGET SALIENCE
Metric distributions differ across edit types, and the section also examines whether target prominence relates to metric sensitivity.
F METRIC BEHAVIOR BY EDIT AND TARGET SALIENCE
- Extraction operations yield larger audio–audio distances than other instrumentation and drum operations because they retain a single component and restructure the mixture.Source: F METRIC BEHAVIOR BY EDIT AND TARGET… · p. 16
- Figure 3 reports CLAP-Audio, LPAPS, and CLAP-Text distributions for all twelve operations, while Figures 6 and 7 show global audio and text distributions.Source: F METRIC BEHAVIOR BY EDIT AND TARGET… · p. 16
- The target-salience analysis groups examples by perceived-usage rank, with lower ranks indicating more prominent targets, to test whether metric sensitivity partly reflects target salience rather than operation correctness.Figure 8 presents the rank grouping for metric distributions, and Figure 9 shows analogous CLAP-Text distributions.Source: F METRIC BEHAVIOR BY EDIT AND TARGET… · p. 16
G COMPLETE SYSTEM BENCHMARK
This section presents complete system-level benchmark results, framing paired synthetic edits as a controlled complement to real, unpaired evaluations and describing how metadata and LLM instructions support excerpt captions and minimal-difference edit descriptions. It also specifies a binary LLM evaluation protocol for edited captions and a Qwen–Mistral judge-agreement analysis over 8,461 generated captions.
H SIM-TO-REAL PERSPECTIVE.
- Paired synthetic edits let researchers test metric behavior when the intended change is known, complementing rather than replacing real benchmarks with unpaired targets.Weak CLAP-Text sensitivity to edit correctness and disagreement between source- and target-reference metrics mean real-world scores should be interpreted with care.Source: H SIM-TO-REAL PERSPECTIVE · p. 17
I FEATURE EXTRACTION AND CAPTIONING
- Caption fact sheets combine MIDI and audio features, with confident excerpt-level key estimates taking precedence over inherited whole-MIDI keys when they disagree.They include instruments, tempo, time signature, note density, dynamics, edit tags, and excerpt-level MAEST genre predictions; the captioner reuses source wording except for the minimal edited span.Source: I FEATURE EXTRACTION AND CAPTIONING · pp. 17–18
J INSTRUCTIONS TO LLM
- The appendix provides captioning-LLM instructions that require edit descriptions to preserve operation semantics and edited captions to retain unaffected source wording.The edited caption must explicitly describe the target’s resulting state while omitting disappeared instruments rather than narrating their removal.Source: J INSTRUCTIONS TO LLM · pp. 18–20
G COMPLETE SYSTEM BENCHMARK
- Tables 5–6 and Figure 10 report six systems across SMEBench and SME-Real, comparing edit-caption and edit-instruction conditioning on audio and text fidelity.Audio fidelity is measured against source and paired-target references; text fidelity is measured against the edit caption, instruction, and normalized caption when available.Source: G COMPLETE SYSTEM BENCHMARK · p. 16, Figure 10 · p. 18, Table 5 · p. 19
J INSTRUCTIONS TO LLM
- Caption changes are judged by their underlying claims: rewording or reordering is compliant, while new or missing entities and materially changed detail can violate minimality.A restatement or added mood or genre adjective is not itself a new entity.Source: J INSTRUCTIONS TO LLM · p. 22
- The evaluator independently judges paraphrase correctness, instruction coherence, edit application, hallucination, minimal change, consistency, and level of detail.It also lists caption differences concisely and requires a binary YES/NO response for each criterion.Source: J INSTRUCTIONS TO LLM · p. 22
- Table 7 compares the original Qwen judge with the Mistral judge on 8,461 generated captions.The caption says its agreement-matrix entries are counts with percentages in parentheses; no entry values are supplied here.Source: Table 7 · p. 23
K LLM-AS-A-JUDGE QWEN CHECKPOINT ANALYSIS
The checkpoint comparison finds partial agreement between caption judges and treats judge-dependent filtering as a limitation. The ethics discussion describes licensing, intended uses, and representation risks.
K LLM-AS-A-JUDGE QWEN CHECKPOINT ANALYSIS
- The comparison uses the same rubric and per-criterion prompts to judge 8,461 generated captions with Qwen and an independent Mistral checkpoint.It reports raw agreement and Cohen’s κ, which accounts for chance agreement under the judges’ marginal acceptance rates.Source: K LLM-AS-A-JUDGE QWEN CHECKPOINT… · p. 23
- The Qwen and independent Mistral judges agree on 69.8% of captions (κ = 0.41); Mistral accepts more items overall, including 80% of discordant pairs.The disagreement does not support the hypothesis that the Qwen judge simply favors outputs from its own checkpoint family.Source: K LLM-AS-A-JUDGE QWEN CHECKPOINT… · p. 23
- Caption acceptance depends on judge choice, so the authors identify automatic caption filtering as a limitation and use the more conservative Qwen decisions.They prefer rejecting some valid captions to retaining descriptions with unsupported content or inadequate isolation of the requested change.Source: K LLM-AS-A-JUDGE QWEN CHECKPOINT… · p. 23
L LICENSING AND ETHICS
- SME-Bench is licensed under CC BY-NC-SA 4.0, while incorporated third-party resources remain subject to their original licenses.Source: L LICENSING AND ETHICS · p. 23
- The MIDI data and Western-optimized soundfonts underrepresent non-Western styles, instruments, and cultural traditions.Source: L LICENSING AND ETHICS · p. 23
- The dataset is intended for training and evaluating neural music-processing pipelines, including music editing and understanding and instruction-based editing benchmarks.Source: L LICENSING AND ETHICS · p. 24
- The authors say the dataset could support interpretability and controllability research, while noting risks of reinforcing Western-centric biases or enabling low-fidelity or misleading generation.Source: L LICENSING AND ETHICS · p. 24