Source-linked AI summary
MixJudge: Learning a Language-Queryable Critic for Vocal Mixing
Kuan-Hsun Ho, Yuan-Chiao Cheng, Yen-Tung Yeh, Yi-Hsuan Yang
TL;DR
Mixing research and audio-language evaluations have focused more on producing mixes or conversational advice than on judging specific issues by severity. MixJudge trains a language-queryable critic on a curriculum of synthetic vocal-mix issues, and its judgments show promise both on issue tests and in selecting processing outcomes closer to a held-out professional version.
Problem
Existing evaluations do not separately test whether audio-language models can make issue-specific judgments that vary consistently with issue severity.
Source: §1 · p. 1Method
Results
MixJudge recognizes queried issues, discriminates opposite issues, orders severity, and its issue evidence remains informative for ordering processing outcomes relative to a held-out professional version.
Source: §5.1 · p. 4, §5.2 · p. 4Takeaways & Limitations
Explicit mixing judgment and comparison provide a complementary form of language-assisted mixing, with menu selection testing their use in a downstream decision.
Source: §1 · p. 1, §6 · p. 4The opposite-issue loss omits masking because masking has no opposite label.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
Mixing requires not only applying processing but also assessing the resulting mix. Audio-language models can provide conversa- tional mixing advice, but existing evaluations do not separately test whether they can judge specific mixing issues. Since professional supervision for such judgments is scarce, we construct a curriculum of 11 vocal-mix issues at multiple severity levels and use it to train MixJudge, a language-queryable audio critic. MixJudge shows its ability to retrieve mixing issues, distinguish opposite issues, and order the severity of an issue. Moreover, its judgments transfer to menu selection: given a set of processing candidates, MixJudge often favors versions acoustically closer to a held-out professional version, demonstrating how explicit mixing judgment can support comparison and downstream mixing decisions.
1. INTRODUCTION
The introduction frames mixing as both producing a treatment and assessing the resulting mix, while noting that existing systems and evaluations emphasize production or conversational advice. MixJudge addresses the judgment gap with paired synthetic vocal-mix issues and evaluates whether those judgments support downstream comparisons.
1. INTRODUCTION
- Existing evaluations do not separately test whether audio-language models make issue-specific judgments that vary consistently with issue severity.MixAssist generates audio-grounded conversational mixing advice, but its advice is evaluated as a whole.Source: §1 · p. 1
- Unlike production-focused automatic mixing systems, the proposed critic assesses a mix state against a language-specified issue, enabling comparisons between processing alternatives.The introduction distinguishes reference-free mixing and reference-conditioned style transfer as systems that produce outcomes rather than separately assess them.Source: §1 · p. 1
- MixJudge trains on paired audio-text examples of clean vocal clips and versions degraded by known issues at multiple severity levels.The curriculum holds musical content fixed and focuses on vocals with their backing because professional supervision for these judgments is scarce.Source: §1 · p. 1
- The paper evaluates MixJudge first on mixing judgments and then on menu selection as a downstream mixing decision.Its stated contributions include an 11-issue vocal-mix curriculum, a language-queryable critic, and comparison-based assistance.Source: §1 · p. 1
2. VOCAL-MIX CURRICULUM
The vocal-mix curriculum uses issue labels and clean states as a teaching vocabulary, with controlled recipes that generate multiple severity levels and template-based captions. Its construction is bounded by mono audio, recipe eligibility and acceptance checks, and the available source material.
2. VOCAL-MIX CURRICULUM
- A clip is a 10-second vocal-and-backing excerpt, and a state is one processed version of that clip.A recipe is signal processing that instantiates an issue; an issue names an audible problem.Source: §2 · p. 1
- The curriculum covers 11 issue labels plus clean, with the first 10 issues forming five opposite pairs and masking lacking a named opposite.The labels span level, tone, space, dynamics, and source interaction; clean means no injected problem, not a universally optimal mix.Source: §2 · pp. 1–2
- Each recipe checks clip eligibility, applies mild, medium, or severe degradation, and retains states that pass recipe-specific acoustic, proxy, and safety checks.Most non-level recipes normalize the affected source; the curriculum uses 18 heuristic recipes, and the checks are implementation-level filtering rather than perceptual validation.Source: §2 · p. 2
- The curriculum uses mono audio, excluding stereo-width issues, and omits phase problems because the available public stems lack required multimicrophone relationships.Source: §2 · p. 2
- Captions are template-generated from fixed lexicons of subjects, degree words, issue phrases, and scopes after issue and severity are chosen.For muddy tone, the example recipe boosts low-mid energy according to severity after checking eligibility, then normalizes the affected source.Source: §2 · p. 2
3. MIXJUDGE
MixJudge compares an audio state with a text description of a mixing issue using role-marked audio features and a relational fusion network. Training combines same-clip groups for issue discrimination and severity ordering with audio-text matching and auxiliary margin losses.
3. MIXJUDGE
- MixJudge fuses backing, vocal, blend, and their differences into a state embedding, then scores its cosine similarity with the issue-description embedding.Learnable role embeddings mark audio slots, exposing how the vocal and resulting blend relate to the backing.Source: §3 · p. 2
3.1. Training
- Same-clip severity groups pair clean audio with each available recipe severity, while comparison groups pair clean audio with up to seven recipes at one nominal severity.These groups provide severity-order supervision and issue discrimination while discouraging song or singer identity shortcuts.Source: §3.1 · p. 2
- Training combines pairwise sigmoid state matching with issue-selectivity and opposite-issue margin losses, plus severity ranking over ordered pairs in each severity group.Matching treats states with the same recipe and nominal severity as positives, including across clips; other pairs are negative.Source: §3.1 · p. 2
- Fixed issue anchors provide stable comparison axes for the auxiliary margin losses because sampled captions do not guarantee coverage of every issue.The opposite-issue loss compares an issue with its named opposite; masking is omitted because it lacks an opposite label.Source: §3.1 · p. 2
4. EXPERIMENTAL SETUP
The evaluation tests MixJudge in two stages: issue-level judgments and downstream menu selection. It compares the model with frozen and generative audio-language baselines using synthetic examples and professionally processed vocal pairs.
4. EXPERIMENTAL SETUP
- Evaluation uses 4,557 synthetic examples and 88 raw/professionally processed vocal pairs from 23 selected songs, with training and evaluation tracks separated.Training uses 78,740 state-caption examples from 236 MoisesDB and 134 MUSDB18-HQ songs.Source: §4 · p. 3
4.1. Baselines
- Frozen CLAP and three generative audio-language models serve as baselines; the generative models are scored by their YES–NO logit margins.Baselines receive only the blend, while MixJudge receives the vocal, backing, and blend; all systems use identical candidates and pools.Source: §4.1 · p. 3
4.2. The critic’s exam
- The critic’s exam measures issue retrieval, opposite-issue discrimination, and severity ordering, using retrieval and severity metrics in both retrieval directions and across ordered sequences.Opposite-issue accuracy requires both queries to be correct; severity metrics include correlation, full-order accuracy, and selectivity.Source: §4.2 · p. 3
4.3. Menu selection
- Menu selection evaluates 31 processing candidates per raw vocal, selecting by mean or maximum issue evidence and comparing the choice with a held-out professional version.Recovery R is normalized between the no-edit distance and the minimum candidate distance; Fig. 3 marks issue evidence in gray and candidate distance in red.Source: §4.3 · p. 3, Figure 3 · p. 3
- Table 2 reports an objective ablation alongside the evaluation setup.The table caption identifies its scope as objective ablation.Source: Table 2 · p. 4
5. RESULTS
MixJudge outperforms the baselines on the critic’s exam and shows that its issue evidence can inform menu selection relative to a held-out professional version. Ablations indicate that training objectives contribute complementary properties rather than improving all metrics uniformly.
5.1. What MixJudge learns
- Adapted MixJudge achieves the highest values across every reported critic-exam metric, while baseline opposite-issue discrimination and severity ordering remain well below MixJudge.Generative audio-language baselines show stronger text-to-audio than audio-to-text retrieval, an asymmetry consistent with text bias but not causally isolated by this evaluation.Source: §5.1 · p. 4
5.2. From known issues to mixing assistance
- Menu-selection results show strong performance on closer-rate, candidate-ranking, and recovery criteria, with issue evidence remaining informative for ordering outcomes against a held-out professional version.The results extend beyond recognizing synthetic degradations to comparing processing outcomes relative to one professional realization.Source: §5.2 · p. 4
5.3. What each objective contributes
- The full objective combines complementary gains: Lrank improves severity ordering, while Lsel and Lopp improve critic-exam metrics and restore strong menu correlation.Improvements in critic-exam measures do not translate monotonically to menu selection, and weaker selectivity can make downstream performance difficult to attribute to intended issue semantics.Source: §5.3 · p. 4
6. CONCLUSION
A vocal-mix curriculum trains MixJudge to make explicit language-queryable judgments, and menu selection shows that its issue evidence can transfer beyond synthetic degradations to support decisions. The paper frames explicit judgment and comparison as a complementary form of language-assisted mixing.
6. CONCLUSION
- The conclusion presents explicit mixing judgment and comparison as complementary language-assisted mixing, with future directions including broader issues, interactivity, and iterative treatment refinement.The reported scope is a vocal-mix curriculum and menu-selection evidence that judgments transfer beyond synthetic degradations.Source: §6 · p. 4