Source-linked AI summary
AURA: Unified Multimodal Framework for Conversational Music Editing
Quoc-Huy Trinh, Minh-Van Nguyen, Debesh Jha
TL;DR
Instruction-guided music editors typically process requests independently, limiting support for workflows that refine a track over multiple turns. AURA interprets dialogue and other modalities through concept tokens, and improves edit correctness and preservation across in- and out-of-domain benchmarks.
Problem
Instruction-guided music editors generally process requests independently and cannot resolve editing instructions that depend on earlier dialogue turns.
Source: §1 · p. 1Method
AURA converts dialogue history, optional images, and reference audio into concept tokens and combines semantic edit control with frame-aligned reference conditioning.
Source: §1 · p. 1, §2.2 · p. 2Results
AURA substantially improves edit correctness and content preservation across in-domain and out-of-domain benchmarks, reducing MoisesDB FAD 4–5× for add and remove.
Source: §1 · p. 1, §3.2 · p. 4Takeaways & Limitations
AURA brings conversational control to instruction-guided music editing.
Source: §4 · p. 4Stem extraction remains limited because a generative decoder cannot match dedicated separation models.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Instruction-guided music editors typically process each request independently, limiting their ability to support workflows in which users progressively refine a track. We introduce AURA, a unified multimodal framework for conversational music editing. AURA uses a multimodal large language model to interpret the complete dialogue history, an optional image, and reference audio, distilling the editing intent into compact concept tokens. A concept-to-audio module injects these tokens and frame-aligned reference features into a frozen MusicGen backbone, enabling precise edits while preserving unaffected content. AURA optimizes only 91M parameters while retaining 1.9B frozen backbone parameters. Experiments on Slakh2100 and MoisesDB demonstrate substantial improvements in edit correctness and content preservation over existing instruction-guided methods, including a 4-5 times reduction in FAD for out-of-domain addition and removal.
1. INTRODUCTION
Existing instruction-guided editors handle requests independently, while AURA jointly reasons over dialogue, images, and reference audio for multi-turn editing; it improves edit correctness and content preservation across in- and out-of-domain settings.
1. INTRODUCTION
- The field has shifted from text-to-music generation toward editing existing tracks through natural-language or visual instructions while preserving unaffected content.Source: §1 · p. 1
- Existing instruction-guided editors generally process each request independently, limiting workflows that depend on earlier dialogue turns.Users progressively refine tracks and expect edits to remain consistent with the evolving conversation.Source: §1 · p. 1
- AURA interprets dialogue history, optional images, and reference audio as concept tokens, then combines semantic control with frame-aligned reference conditioning in a frozen MusicGen decoder.Only LoRA adapters, projectors, and concept-to-audio modules are trainable.Source: §1 · p. 1
- AURA substantially improves edit correctness and content preservation across in-domain and out-of-domain benchmarks.Experiments use Slakh2100-test as the in-domain benchmark and MoisesDB as the out-of-domain benchmark.Source: §1 · p. 1
2. METHOD
AURA routes multimodal dialogue context into compact edit concepts, then combines those concepts with frame-aligned reference features in a frozen MusicGen decoder. Its training jointly supervises assistant responses and music-code prediction.
2.1. Overall framework
- AURA maps an optional image, multi-turn history, and reference audio to an edited signal by separating intent understanding from generation.The MLLM interprets the inputs, while the MusicGen-based decoder applies the edit and preserves unaffected reference content.Source: §2.1 · pp. 1–2, Figure 1 · p. 2
2.2. Multimodal large language model
- The MLLM processes the complete conversation, image, and reference audio; concept-token states encode the operation, target instrument, attributes, and conversational constraints.When no audio edit is required, the MLLM emits no concept tokens.Source: §2.2 · p. 2
2.3. Concept and audio projectors
- LayerNorm and an MLP project concept states into MusicGen space, while frozen EnCodec features are projected framewise into that same space.Concept vectors serve as cross-attention keys and values; concepts specify what changes, while frame-aligned reference features specify content to preserve.Source: §2.3 · p. 2
2.4. Music decoder
- MusicGen-medium’s frozen decoder integrates aligned reference features through BiFAM and shared concept cross-attention to generate edits.BiFAM’s zero-initialized FiLM preserves identity initially and reference structure without extra autoregressive tokens; concept attention uses rank-64 LoRA on key/value projections. AURA trains 91M parameters while freezing 1.9B backbone parameters; reference states can be cached, and classifier-free guidance zeros concept memory while retaining reference features.Source: §2.4 · p. 3
2.5. Objective function
- A joint objective combines assistant-response language modeling with the decoder’s codebook cross-entropy.Language loss supervises multi-turn reasoning and concept-token placement; music loss shapes token representations through the projector.Source: §2.5 · p. 3
3. EXPERIMENTAL DETAILS
Experiments evaluate AURA on Slakh2100 and MoisesDB in single- and multi-turn editing, with an ablation comparing concept-token fusion to full-text conditioning.
3.1. Experimental setup
- The study trains on 66,539 Slakh2100-training-subset dialogues and evaluates on 1,000 Slakh test-split and 1,000 MoisesDB samples; SI-SDR and SI-SDRi apply only to removal and extraction.Evaluation covers audio quality, instruction adherence, edit correctness, and content preservation in single-turn and multi-turn editing.Source: §3.1 · p. 3
3.2. Quantitative results
- On Slakh2100, add SSIM is 0.78 vs. 0.37 for the best baseline and remove SI-SDR is +11.32 dB; MoisesDB FAD falls 4–5× for add and remove.Table 1 compares methods on in-domain Slakh2100-test and out-of-domain MoisesDB.Source: Table 1 · p. 4, §3.2 · p. 4
- Across MoisesDB turns, AURA maintains positive SI-SDR (+21.3 to +5.1 dB) while Instruct-MG falls from −0.9 to −19.3 dB; the final-edit-only control scores −5.2 dB.The control applies only the final instruction to the original mixture; dialogue history is necessary to recover the intended editing state.Source: Table 2 · p. 4, §3.2 · p. 4
3.3. Ablation study
- Concept-token fusion improves audio quality over full-text conditioning, with compact semantic representations less susceptible to irrelevant variation in full-sentence hidden states.Table 3 compares the two conditioning approaches on MoisesDB.Source: Table 3 · p. 4, §3.3 · p. 4
4. CONCLUSION
AURA brings conversational control to instruction-guided music editing by routing multimodal and dialogue context through concept tokens and frame-aligned reference audio. Its generative decoder improves editing outcomes, but stem extraction remains a limitation.
4. CONCLUSION
- AURA routes multimodal and dialogue context through compact concept tokens and injects frame-aligned reference audio into a frozen decoder to preserve unaffected content.This separates understanding what to change from producing the result.Source: §4 · p. 4
- Experiments on in-domain and out-of-domain benchmarks show large gains in content preservation and edit correctness over prior editors.Source: §4 · p. 4
- Stem extraction is the main limitation: the generative decoder cannot match dedicated separation models, and an explicit separation front-end is a proposed next step.Source: §4 · p. 4