PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Music Post-Production

Jaywon Koo, Junghyun Koo, Marco A. Martinez-Ramirez, Wei-Hsiang Liao, Seungheon Doh, Qiyu Wu, Vicente Ordonez, Yuki Mitsufuji

arXiv:2610.29093v1cs.SDcs.AI

TL;DR

Music post-production requires translating a reference song into editable effect settings across interacting tracks and mix levels, while prompting multimodal models to revise mixes does not consistently improve them. MixAgent combines hierarchical planning with learned numerical correction trained on repair critiques. On MedleyDB, learned refinement reduces reconstruction error, and MixAgent receives higher quality ratings than tested multimodal LLM baselines in a listening test.

  • Problem

    Mixing requires choosing and ordering effects and settings across interacting tracks, subgroups, and the full mix, while audio-language models prompted to revise do not consistently improve mixes.

    Source: §1 · p. 1

  • Method

    MixAgent creates editable multilevel effect plans using retrieved examples and trains a detector and critic for bottom-up correction, using 45,657 MixCritic critiques and preference optimization on rendered outcomes.

    Source: §1 · p. 2

  • Results

    MixAgent achieves the lowest reconstruction error and higher predicted quality than tested multimodal LLM baselines on held-out MedleyDB songs, and higher quality ratings in an 11-participant listening test.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest that targeted, executable feedback trained on outcomes improves mixing, while explicit effect names and values let users inspect and edit the plan.

    Source: §7 · p. 8

    Reconstruction error against one professional mix does not fully capture mixing quality, and the evaluation sets are small.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We introduce MixAgent, an agentic pipeline for music post-production that follows the workflow of audio engineers to build interpretable and editable mixing plans guided by a reference song. A grouping agent first routes related input tracks, such as drum recordings, into subgroups. A planning agent then uses retrieved examples to choose and order audio effects, such as volume control and reverb, for tracks, subgroups, and the full mix. To refine the mix, a detector identifies what needs correction, and a critic proposes numerical parameter adjustments from tracks to the full mix. In our experiments, prompting recent multimodal models to listen and revise often worsens the mix, making reliable correction a central challenge. We therefore build MixCritic, a dataset of 45,657 repair critiques derived from parameters fitted to professional mixes, and train the critic with supervised fine-tuning followed by preference optimization on rendered audio outcomes. On MedleyDB, learned refinement reduces error by 19% relative to the initial mix, with preference optimization reducing it by 8.4% over supervised training alone. Retrieved examples further improve effects selection, and a human listening test shows that MixAgent gets higher quality ratings than mixes produced by Qwen3-Omni, Gemini 3.8 Flash and GPT-4o-audio.

1 Introduction

Music mixing requires translating a reference style into editable effect settings across interacting tracks, subgroups, and the full mix, but existing systems address only parts of this workflow. MixAgent combines hierarchical planning with learned correction; its analyses find gains from retrieved examples and numerical refinement, and held-out evaluations favor it over tested multimodal LLM baselines.

1 Introduction

  • Existing systems expose only parts of the workflow: neural and generative models do not expose effect settings, while prior LLM systems focus on individual-track chains or instruction-based edits.Diff-MST predicts editable parameters for fixed effect sequences.Source: §1 · p. 1
  • Mixing requires choosing and ordering effects and settings across interacting tracks and mix levels, while prompting audio-language models to listen and revise does not consistently improve the mix.A reference song provides an example of the desired sound when that sound is difficult to describe in words.Source: §1 · pp. 1–2
  • MixAgent builds an editable plan across tracks, subgroups, and the full mix, using retrieved examples for effect selection and a learned detector and critic for bottom-up parameter refinement.The critic is trained on MixCritic’s 45,657 repair critiques using supervised fine-tuning followed by preference optimization on rendered audio outcomes.Source: §1 · p. 2
  • Retrieved examples improve effects selection, while learned numerical refinement improves parameters; with numerical feedback, DPO reduces reconstruction error by a further 8.4% over SFT.Selecting correction targets and refining tracks before subgroups and the full mix further reduces error.Source: §1 · p. 2
  • On held-out MedleyDB songs, MixAgent achieves the lowest reconstruction error and higher predicted quality than tested multimodal LLM baselines, with higher quality ratings in an 11-participant listening test.The listening test uses references from other songs within the same genre.Source: §1 · p. 2

2 Preliminaries

The task is reference-guided multitrack mixing: infer editable effect chains and settings for tracks, subgroups, and the full mix. MixAgent keeps the plan explicit and processes corrections hierarchically.

2 Preliminaries

  • Mixing transforms input recordings through effects at track, subgroup, and full-mix levels; subgroup and full-mix processing correspond to subgroup buses and the master bus.An effects chain is an ordered list of effects, such as EQ followed by compression.Source: §2 · p. 2
  • A mixing plan assigns tracks to subgroups and assigns an effect chain to each track, subgroup, and the full mix, with parameters for each chain.The input includes unprocessed tracks with instrument labels, an effect library, and a finished mix from another song as the style reference.Source: §2 · p. 2
  • MixAgent groups related tracks, plans effect chains sequentially at track, subgroup, and full-mix levels, then detects and corrects issues from tracks upward.The critic can provide perceptual, tool-specific, or numerical feedback.Source: Figure 2 · p. 3
  • The system must match the reference’s mixing style rather than copy its audio, account for interactions across levels, and use two non-overlapping excerpts of the same song for training and main evaluation.The listening test instead uses references from other songs within the same genre; effects and settings remain inspectable and editable.Source: §2 · p. 3

3 MixAgent

MixAgent combines retrieved-example effects planning with learned, track-to-full-mix correction; MixCritic supplies repair critiques and preference pairs for training the critic.

3.1 Planning Audio effects Chains with Retrieved Examples

  • Retrieved plans guide a frozen planner to select and order fixed audio-effect chains for tracks, subgroups, and the full mix.Three text-retrieved and one audio-retrieved plan provide effect choices and order, not parameter values; each level is planned in sequence.Source: §3.1 · p. 3

3.2 Hierarchical Estimation

  • MixAgent revises parameters bottom-up, using a detector to select elements for correction and a critic to propose numerical changes.The critic inspects elements above a detector threshold, while the full mix is always inspected; each pass re-renders before the next stage.Source: §3.2 · p. 4

3.3 Learning from Repairs with MixCritic

  • MixCritic contains 45,657 critiques for repairing corrupted mixing states, built from fitted parameters and teacher-generated feedback.For each state, a target track’s fitted parameters define the repair; critiques are retained when the target repair exceeds a score threshold.Source: §3.3 · pp. 4–5
  • The critic is trained with supervised fine-tuning, then preference optimization that favors corrections with better rendered-audio repair scores.Preference pairs compare candidate critiques applied to the same starting parameters, ranked by normalized repair scores.Source: §3.3 · pp. 4–5

4 Experiments

Experiments compare MixAgent with multimodal LLM baselines on MedleyDB and AlbumDB, using reconstruction, predicted-quality, and style-similarity metrics. MixAgent leads on several MedleyDB measures and receives higher human quality ratings than the tested LLM baselines, though results vary by dataset and metric.

4.1 Setup

  • Evaluation uses 185 MedleyDB clips, with 20 artist-disjoint clips held out and separate 15-second windows for input/target and style reference.The baselines use the same input tracks, style reference, and available effects; evaluation metrics cover reconstruction, predicted quality, and mixing-style similarity.Source: §4.1 · p. 6

4.2 Main Results

  • On MedleyDB, MixAgent achieves the lowest MSS (2.83 vs. 3.67 for single-pass GPT-4o-audio) and the highest TuneJury and Audiobox PQ scores.On AlbumDB, MixAgent has the highest Audiobox PQ, ties single-pass Gemini on TuneJury, and has second-best MSS (3.81 vs. 3.77 for GPT-4o-audio with Self-Refine).Source: §4.2 · p. 6
  • Prompted revision does not consistently improve on single-pass mixing, while learned refinement lowers MixAgent’s MSS from 3.70 to 2.83 with fixed audio-effect chains.Self-Critic has higher MSS than Self-Refine for every tested model on both datasets.Source: §4.2 · p. 6

4.3 Human Listening Test

  • In the listening test, MixAgent receives significantly higher quality scores than all three LLM baselines, but remains below the original human mixes.Eleven participants rated mixes on a 0–100 scale across 10 trials each; the baseline score distributions largely overlap.Source: §4.3 · p. 6, Figure 4 · p. 6

5 What Makes Feedback Useful?

MixAgent’s refinement depends on trained, specific feedback: preference optimization improves numerical corrections, while retrieved examples improve effect selection. Its ablations support bottom-up correction and targeted inspection, and its named edits remain inspectable.

5.1 Critic Training and Feedback Form

  • Untrained prompted Qwen3-Omni corrections increase error on average across all three feedback forms, with only 42–52% improving the mix; trained A3 refinement reaches 1.75 MRS.The planner alone has 2.17 MRS; final MRS for the untrained critic ranges from 2.18 to 2.44.Source: §5.1 · p. 7
  • 8.4%: preference training lowers numerical-feedback MRS from 1.91 to 1.75, nearly doubles mean improvement per correction (+0.022 to +0.040), and raises success from 64.7% to 66.8%.Under the same detector, planner, and execution.Source: §5.1 · p. 7
  • After SFT+DPO, MRS falls from 2.09 for perceptual advice to 1.98 for tool-specific advice and 1.75 for numerical corrections; GT-fitted plans reach 0.90 MRS.Trained A3 without GT access also outperforms oracle A1/A2 advice on MRS and TuneJury, leaving substantial room for better parameter estimation.Source: §5.1 · p. 7

5.2 Choosing Audio effects with Retrieved Examples

  • 2.13 to 1.08 MRS: retrieved examples improve fitted effect plans over no examples, while random examples reach 1.71; text retrieval provides most of the gain.Plans use three text-retrieved and one audio-retrieved example; retrieval also raises tools per track from 1.6 to 3.8, accounting for part of the gain. Table 4 covers 100 randomly selected MedleyDB clips.Source: §5.2 · p. 7, Table 4 · p. 8

5.3 Component Ablations

  • Retrieval lowers MRS by 0.64 and refinement by 0.42 in either condition; bottom-up refinement reaches 1.75 MRS versus 1.82 top-down and 1.88 for simultaneous corrections.Detector-selected targets reach 1.75 MRS, compared with 1.87 for inspecting all tracks and subgroups and 1.81 for a random subset; retrieval improves effect choice while refinement improves parameters.Source: §5.3 · pp. 7–8

5.4 An Inspectable Output

  • The critic applies 241 named-parameter corrections on the evaluation set, about 12 per clip, and two thirds reduce full-mix error; users can trace, revert, or adjust them.These edits are exposed in the plan, unlike audio-only outputs.Source: §5.4 · p. 8

6 Related Work

Earlier music systems address individual parts of mixing, including fixed-chain parameter estimation, generated audio, or narrower LLM tasks. MixAgent instead builds mixes from unprocessed tracks with editable plans and correction trained on rendered outcomes.

6 Related Work

  • Earlier LLM music tools predict single-track effects, recover a single-track chain, give conversational mixing advice, or edit finished mixes from synthetic instructions.MixAgent instead starts from unprocessed input tracks, training its detector and critic on whether each correction improves rendered audio.Source: §6 · p. 8
  • Prior explicit-processing methods estimate parameters or chains, while most use fixed consoles without selecting effects or grouping tracks; neural approaches also generate or transform audio.GRAFX fits graphs to known targets and supplies MixAgent’s retrieval plans, training data, and oracles.Source: §6 · p. 8
  • Direct audio generation can introduce artifacts and obscure effect settings, whereas MixAgent combines open-ended chains, subgroups, editable parameters, and learned correction.Source: §6 · p. 8

7 Conclusion

MixAgent builds explicit effect plans across tracks, subgroups, and the full mix, then refines them bottom-up with a learned detector and critic. Its logged decisions let users inspect, trace, and edit the plan.

7 Conclusion

  • MixAgent builds an explicit audio-effects plan across tracks, subgroups, and the full mix, then refines it bottom-up with a learned detector and critic.Source: §7 · p. 8
  • Each logged decision names an audio effect and value, allowing users to inspect results, trace corrections, and edit the plan.Source: §7 · p. 8

Limitations

The evaluation and system have several scope limits: reconstruction against one professional mix is incomplete as a quality measure, and the test sets are small. Supervision, tools, editable plans, and computational efficiency also constrain the conclusions.

Limitations

  • Reconstruction error against one professional mix does not fully capture quality because multiple mixes can be valid for the same tracks.Source: Limitations · p. 9
  • Evaluation sets are small, limiting generalizability and precision of perceptual comparisons; AlbumDB results are mixed, and MST uses separated references rather than identical input tracks.Source: Limitations · p. 9
  • GRAFX-fitted supervision and oracles approximate rather than recover engineer settings; the 1.75 versus 0.90 MRS gap shows parameter setting remains unsolved.Source: Limitations · p. 9
  • The audio-effects library covers only a small subset of professional tools, and the critic cannot add or remove effects after planning.Source: Limitations · p. 9
  • User interpretation of editable plans has not been evaluated, and producing a mix requires many large audio-language-model calls, limiting computational efficiency.Source: Limitations · p. 9

Ethical Considerations

The section describes dataset handling, audio-effect constraints, and baseline evaluation, while framing MixAgent as an editable tool intended to support rather than replace human judgment.

Ethical Considerations

  • MixAgent is designed to expose and accept edits to any decision, supporting rather than replacing human judgment, and may also serve as an educational tool for audio engineers.The work uses MedleyDB and AlbumDB in accordance with their terms of use.Source: Ethical Considerations · p. 9

A.1 Data

  • MedleyDB evaluation uses 20 clips from 17 songs after filtering, split by artist from 165 training clips drawn from 106 songs.The 14 AlbumDB evaluation clips are built in the same way; MedleyDB clips are sampled as non-overlapping 15-second windows from song sections longer than 35 seconds.Source: §A.1 · p. 12

A.4 Metrics

These sections specify the evaluation metric and the retrieval, refinement, detection, and critic-training components used to implement MixAgent.

A.4 Metrics

  • TuneJury ranks professional mixes above random processing 70.5% of the time, but above dry sums only 48% and random track gains 41%, limiting its use to checks for severe problems.Scores are means over clips after normalization to −12 LUFS.Source: §A.4 · p. 12

B.1 Retrieval and Audio effects Chain Planning

  • Retrieval matches input-track caption text and reference-audio embeddings against stored GRAFX plans and renders, excluding all excerpts from the query song.The library includes effects such as filtered-noise reverb and a limiter with threshold, attack, and release controls.Source: §B.1 · p. 12

B.2 Hierarchical Refinement

  • MixAgent makes one refinement pass per stage, progressing from tracks through subgroups to the full mix.Appendix D.6 studies using additional passes.Source: §B.2 · p. 12

B.3 Detector

  • The detector combines fixed audio embeddings with the plan and current parameters to predict element error probability and identify effects that may need changes.It is trained on 40,264 states; a positive label requires a replaced parameter to change that track’s MRS by at least 1.0.Source: §B.3 · pp. 12, 14

B.4 Critic Architecture and Training

  • The critic combines a frozen RLAT audio-effects encoder and Qwen3.5-4B through a trained adapter, then uses SFT followed by DPO on preference pairs.Its inputs include the selected element, current mix, reference, detector output, and the selected element’s current effects chain and parameters.Source: §B.4 · p. 14

C The MixCritic Dataset

The section outlines MixCritic’s critique and validation setup, compares MixAgent with dedicated mixing systems, and reports the retrieval configuration’s effect on chain agreement.

C The MixCritic Dataset

  • Preference pairs are built by independently applying each valid SFT-sampled critique to the same corrupted parameters; malformed feedback receives a score of 0.The 2,200 validation examples come from held-out states built on validation clips, and evaluation clips are reserved for testing.Source: C The MixCritic Dataset · p. 14

D.1 Comparison with Dedicated Mixing Models

  • The comparison is not strictly fair: Diff-MST training songs may overlap MedleyDB evaluation clips, while MST and StemFX start from the professionally mixed target and thus have privileged access to ground truth.MST outputs processed audio without exposing effects parameters; Diff-MST predicts parameters for a fixed console but cannot choose effects or group tracks.Source: §D.1 · p. 15
  • MixAgent has the lowest MSS among four systems on MedleyDB, while MST scores better on Audiobox PQ, FxEnc++, and AF, and StemFX on TuneJury.On AlbumDB, MixAgent improves on Diff-MST in reconstruction (3.81 vs. 4.72 MSS), while MST and StemFX score higher on most metrics.Source: §D.1 · p. 15

D.2 Audio effects Chain Planning

  • Text (3) + audio (1) retrieval raises track-level audio-effects-chain F1 from 0.33 to 0.57 on 100 MedleyDB clips.The study excludes all excerpts of each query song from retrieval and involves no training.Source: §D.2 · p. 15

D.3 Example Plans: MixAgent and GRAFX

The appendix illustrates how MixAgent’s plans differ from GRAFX and reports detector, refinement, and prompt details across the system’s evaluation variants.

D.3 Example Plans: MixAgent and GRAFX

  • MixAgent adapts grouping, effect order, and effect choice per element, unlike GRAFX’s fixed console order and uniform processing approach.Examples include gating before compression on drums and vocals, omitting delay or reverb on bass, and applying reverb at subgroup level; chain F1 can count such departures from GRAFX as errors.Source: §D.3 · p. 15

D.4 Detector Accuracy

  • The detector reaches 0.81 F1 for tracks and 0.77 F1 for audio effects at threshold 0.5 on 3,130 held-out corrupted states.That threshold is used only for validation; inference uses τ = 0.15.Source: §D.4 · p. 17, Table 11 · p. 17

D.5 Error at Each Stage

  • Across processing stages, numerical-feedback DPO has lower MRS than SFT at every stage.The stage targets are GT-fitted outputs for tracks, subgroups, and the full mix, followed by the real mix.Source: §D.5 · p. 17, Table 12 · p. 17

D.6 Number of Refinement Passes

  • With A3, MRS improves from 1.75 to 1.69 with three passes, then returns to 1.73 with five; A1 and A2 perform best with one pass.Additional passes therefore do not consistently help.Source: §D.6 · p. 17, Figure 9 · p. 18

E.1 Critic Training and Feedback Form

  • SFT and SFT+DPO comparisons share the detector, planner, plan, initial state, prompts, and decoding within each feedback form, while later states may diverge.Oracle A1 and A2 substitute a ground-truth critique; oracle A3 directly fits the same plan’s parameters to ground truth.Source: §E.1 · p. 18

E.2 Target Selection

  • Target-selection conditions share the plan, starting state, DPO A3 critic, and effect-level diagnoses; only the selected tracks and subgroups vary, while the full mix is always inspected.Inspect-all averages 13.1 turns and 56.6 minutes per clip; random selection averages 12.1 turns.Source: §E.2 · p. 18

E.3 Refinement Order

  • Top-down refinement visits the full mix, then subgroups, then tracks, rendering between levels; the non-hierarchical variant computes corrections from initial audio and applies them together.The variants use 11.9 and 11.8 turns, respectively.Source: §E.3 · p. 18

F Prompts

  • The prompt appendix marks runtime-filled text with braces and omitted lines with “[...]”.It presents prompts for grouping, planning, parameter initialization, and critic calls, as well as the shared prompt used to build MixCritic.Source: F Prompts · p. 18
Loading 2610.29093v1…