PaperlayerView exact revision on arXiv

Source-linked AI summary

SPHERE: Automatic Music Upmixing via Audio Language Model Post-Training with Spatial Heuristic Rewards

Zixun Guo, Calvin Murdock, Sanjeel Parekh, W Owen Brimijoin, Simon Dixon, Joshua Reiss, Ishwarya Ananthabhotla

arXiv:2608.30559v1cs.SD

TL;DR

Automatic music upmixing requires predicting spatial mixing parameters from multitrack stems, but existing approaches often rely on task-specific encoders. The paper post-trains audio language models with rejection-sampling SFT and GRPO-based RL guided by six-component SPHERE rewards. High-reward mixes were preferred in 76.8% of trials, and the post-trained model surpassed frontier ALMs on SPHERE, supporting reward-based distillation of mixing expertise.

  • Problem

    Existing task-specific encoders have limited ability to represent musical semantics, while general-purpose ALMs show inconsistent zero-shot performance on spatial mixing.

  • Method

    The paper uses rejection-sampling SFT followed by GRPO reinforcement learning, guided by SPHERE, a deterministic suite of six spatial and balance rewards.

  • Results

    76.8% of subjective A/B trials preferred high-reward mixes, and the post-trained model surpassed frontier ALMs on the SPHERE evaluation suite.

  • Takeaways & Limitations

    The results suggest that expert musical domain knowledge can be encoded as verifiable rewards and distilled into language models through post-training.

  • Takeaways & Limitations

    Context length and limited audio-modality support constrain usable multitrack inputs, excluding 23.5% of the dataset and increasing training cost.

Abstract

from arXiv · show

In this paper, we study the task of automatic music upmixing, wherein a system predicts spatial mixing parameters from a multi-stem recording. Different from existing methods that rely on task-specific music encoders, we approach this task via audio language model (ALM) post-training, leveraging rich representations from existing ALMs, which encode both music semantics and mixing knowledge. Specifically, we propose a post-training recipe that first employs rejection sampling SFT, followed by reinforcement learning (RL) with verifiable rewards (RLVR) via GRPO. We propose Sphere (Spatial Heuristic Rewards), a deterministic reward suite inspired by music mixing conventions, to guide our post-training. It consists of 6 perceptually-motivated sub-rewards and encourages the output mix to be centered, balanced and spacious. More broadly, our results suggest that expert domain knowledge can be encoded as verifiable rewards and distilled into language models, without task-specific architectures.

1 Introduction

The paper frames automatic binaural upmixing as predicting spatial parameters for isolated stems, and addresses it through audio language model post-training guided by deterministic spatial rewards. The resulting pipeline uses SFT and GRPO-based RL, with experiments showing human-aligned rewards and improved performance over frontier ALMs.

  • Motivation: Automatic binaural upmixing extends music mixing by predicting each stem’s azimuth and elevation alongside conventional parameters such as gain and panning.The goal is a three-dimensional, immersive musical presentation.
  • Motivation: Existing music-mixing systems commonly use task-specific encoders, while ALMs offer richer representations of musical semantics and possible mixing knowledge.The paper highlights instrument grouping as a semantic capability relevant to mixing decisions.
  • Method: The proposed pipeline filters teacher outputs for high-quality examples, performs rejection-sampling SFT, then applies GRPO reinforcement learning against SPHERE.SPHERE combines six perceptually motivated metrics into continuous rewards for centering, spread, and level balance.
  • Results: 76.8% of A/B preference trials favored high-reward mixes, while the post-trained model surpassed frontier ALMs on the SPHERE evaluation suite.The listening test included 44 participants, and the reward suite also served as the objective evaluation metric.
  • Method: SPHERE is inspired by music-mixing conventions and is designed to encourage mixes that are centered, balanced, and spatially wide.A subjective listening test evaluates whether the deterministic reward reflects human spatial-audio preferences.
  • Implications: The results suggest that expert domain knowledge can be encoded as verifiable rewards and distilled into language models through post-training.The paper presents this as a practical recipe potentially applicable beyond music mixing and data-rich task-specific architectures.

2 Related Work

Prior automatic music-mixing systems generally use encoder-decoder architectures with task-specific encoders, while binaural approaches also use modality conversion or direct spatial prediction. ALMs provide broader audio and music understanding but remain inconsistent for spatial mixing without task-specific post-training.

  • Automatic Music Mixing: Automatic music-mixing systems typically encode latent music representations and decode either interpretable mixing parameters or fully mixed audio.For binaural upmixing, prior work also studies mono-or-stereo-to-binaural conversion and direct spatial prediction from stereo inputs.
  • Audio Language Models: ALMs can process multiple audio files and produce descriptive text, making them suitable for multitrack music-mixing applications.Their audio and music understanding capabilities distinguish them from conventional task-specific encoders.
  • Audio Language Models: General-purpose ALMs are not trained on spatial mixing data, resulting in inconsistent zero-shot performance for this task.The related-work discussion motivates post-training with SFT followed by RL to align outputs with desired behavior.

3 Method

The method formulates binaural upmixing as an ALM mapping multitrack stems and instructions to spatial parameters, then post-trains a student with filtered teacher outputs and SPHERE-guided RL. SPHERE combines six convention-inspired rewards measuring centering, spread, and level balance through data-driven continuous shaping.

  • 3.1 Task Formulation: The ALM maps mono stems and text instructions to per-stem azimuth, elevation, gain, and JSON spatial outputs for binaural rendering.A spatializer applies gains and HRTFs to produce a two-channel headphone mix.
  • 3.2 Post-training: Teacher-generated spatial configurations are filtered by an RSPHERE threshold, used for student SFT, and further optimized with RL to exceed teacher imitation.The pipeline comprises teacher generation, quality filtering, SFT, and RSPHERE-based RL.
  • 3.3 SPHERE: SPHERE encodes six perceptually motivated sub-rewards for centered, spacious, and balanced mixes, using centering, spread, spotlight, and role-level metrics.Centering uses gain-weighted means, while spread uses unweighted standard deviations so gains cannot inflate spatial spread alone.
  • 3.3.2 Data-driven Thresholds and Reward Shaping: Data-driven thresholds are derived from teacher-output percentiles instead of hand-selected metric cutoffs.For centering, the threshold covers the top p% near zero; for spread, it excludes the narrowest 1−p% of outputs.
  • 3.3.2 Data-driven Thresholds and Reward Shaping: Continuous Gaussian and Sigmoid shaping replaces sparse step rewards for RL, with thresholds set at half reward and role-level rewards aggregated geometrically.The geometric mean acts as a soft “and,” penalizing optimization that sacrifices one musical role for others.
  • 3.3.2 Data-driven Thresholds and Reward Shaping: The six-reward combination creates a non-linear landscape in which optimizing centering alone conflicts with spread, making trivial reward maximization difficult.The paper reports that removing any sub-reward pair leads to reward hacking.

4 Experiments

Experiments use MedleyDB segments, teacher-generated candidates, and held-out objective evaluation across proprietary and open-source ALMs. The study also varies SFT filtering quality and evaluates SPHERE alignment with human preferences through controlled listening tests.

  • 4.1 Dataset: MedleyDB provides 193 full-length songs, segmented into 1650 aligned 30-second windows with up to 29 mono stems per example.Each song includes a stereo reference mix, isolated mono stems, and metadata.
  • 4.2 Training: Gemini 2.5 Pro generates five candidate spatial configurations per segment, yielding 8243 data points after seven failures, while Qwen2.5-Omni 3B serves as student.The prompt includes stems, metadata, a stereo reference, and detailed mixing instructions.
  • 4.2 Training: SFT uses LoRA for 20 epochs, and GRPO-based RL runs for up to 800 steps with long-sequence generation and scoring.An 800-step run takes approximately 24 hours on eight H100 GPUs.
  • 4.3 Objective Evaluation: Evaluation covers 618 held-out examples, reporting RSPHERE, all sub-rewards, and JSON format pass rate across eight compatible baseline models.Baselines are evaluated zero-shot with identical prompts.
  • 4.5 Ablations: SFT filtering thresholds of 0.2, 0.3, 0.4, and 0.5 produce 4074, 3152, 2283, and 1408 accepted data points, respectively.A sample is accepted only when all six sub-rewards meet the threshold.
  • 4.4 Subjective Evaluation: The listening study screens 89 recruits to 44 valid participants, each completing 10 trials, and uses binomial and Wilcoxon tests for preference and reward comparisons.The post-training comparison additionally evaluates student, teacher, SFT, and SFT+RL systems with 10 expert listeners.

5 Results

SPHERE aligns with human spatial-audio preferences, and post-training substantially improves a small student model through SFT and RL. Ablations show that high-quality filtered data and a complete, smoothly shaped reward suite are important for avoiding reward hacking.

  • 5.1 Reward-Human Preference Alignment: 76.8% of listening-test trials favored high-reward mixes, with preference significantly above chance.Listeners preferred high-reward mixes in 338 of 440 trials; the preference-rate test reported p = 6.65 × 10−31.
  • 5.2 Post-Training Effectiveness: RSPHERE was 4.981 after RL, a +1.407 gain over the untrained student and the highest score across models.SFT reached 4.657 and surpassed the teacher, while RL further improved the total reward.
  • 5.2 Post-Training Effectiveness: SFT raised format pass rate from 80.1% to 98.8% and reduced degenerate center placement by improving centering, spread, and level-balance metrics.Centering decreased from 0.918 to 0.754 and from 0.956 to 0.798, while spread increased from 0.177 to 0.682 and from 0.182 to 0.825; level balance rose from 0.539 to 0.824.
  • 5.2 Post-Training Effectiveness: RL further improved spotlight prominence from 0.774 to 0.879 and level balance from 0.824 to 0.903 after SFT.Spread metrics also increased from 0.682 to 0.759 and from 0.825 to 0.890, at the cost of a slight decrease in elevation centering from 0.798 to 0.734.
  • 5.3 Ablation Studies: Stricter SFT filtering achieved RSPHERE = 4.657 with 1408 examples, outperforming κ = 0.2 with nearly three times more data and RSPHERE = 4.517.RSPHERE increased monotonically from 4.517 at κ = 0.2 to 4.657 at κ = 0.5.
  • 5.3 Ablation Studies: Removing spatial-spread or centering sub-rewards caused reward hacking, whereas all six sub-rewards produced the highest total without metric collapse.Removing spread yielded RSPHERE = 4.388; removing centering produced centering = 0.681 and spread = 0.787.
  • 5.4 Subjective Evaluation of Post-Trained Models: Post-training improved subjective ratings from 3.24 to 5.71 after SFT and 5.73 after RL, exceeding the teacher’s 5.55.The listening test used 10 expert listeners and 10 tracks, with ratings on a 0-to-10 preference scale.

6 Limitations and Future Directions

The approach is limited by audio-modality compatibility and context length, while RL training can be vulnerable to reward hacking when sub-rewards are removed.

  • Limitations: Audio-modality compatibility limits the frontier models available for automatic binaural upmixing.Among major frontier model families, only Gemini, Qwen, Gemma, and GPT-4o support audio input.
  • Limitations: Removing individual sub-reward pairs can cause reward hacking on the remaining metrics during RL.The severity depends on the removed components, with spatial-spread sub-rewards producing the most severe case.
  • Limitations: Context length constrains multitrack recordings because long stems consume many audio tokens.A 32k-token window accommodates at most 29 30-second stems, and this limitation caused 23.5% of the dataset to be excluded.
  • Future Directions: Large audio contexts also make both SFT and RL training expensive.The paper suggests context-length scaling techniques or efficient tokenizers as future directions.

7 Conclusion

The paper presents ALM post-training for automatic binaural upmixing, guided by SPHERE, and reports human-preference alignment, improved performance over frontier ALMs, and the necessity of all six sub-rewards.

  • Conclusion: The approach applies ALM post-training to automatic music binaural upmixing using SPHERE rewards inspired by music mixing conventions.The conclusion frames musical domain expertise as a source of reward functions for guiding post-training.
  • Conclusion: SPHERE aligns with human preference, while the post-trained student model surpasses frontier ALMs.The conclusion also reports that SFT data quality matters more than quantity.
  • Conclusion: All six SPHERE sub-rewards are necessary to prevent reward hacking during RL.The paper presents this as evidence that musical domain expertise can be distilled into rewards for optimizing ALM outputs against common practices.

A Prompt Template

The prompt produces per-stem spatial and mixing configurations, with gain prediction organized into normalization, aesthetic balancing, and compensation stages.

  • Prompt Structure: The prompt outputs per-stem JSON containing azimuth, elevation, gain, musical role, and a reasoning trace.Role assignment is performed by the model because metadata may not specify musical function.
  • Prompt Structure: Gain prediction follows gain staging normalization, aesthetic balance, and compensation.Spatial placement principles are drawn from music mixing conventions.
  • Prompt Structure: The model assigns musical roles rather than receiving them as input.This design addresses cases where metadata does not indicate a stem’s musical function.

B SFT Training Dynamics

SFT rapidly corrects degenerate zero-shot spatial outputs, and stricter reward-based filtering achieves the highest total reward despite using fewer training samples.

  • Training Dynamics: During the first ∼100 steps, SFT learns output formatting and spreads azimuth and elevation away from collapsed baselines.Centering metrics decrease as stems become distributed across the panorama instead of clustered at center.
  • Training Dynamics: The κ = 0.5 model achieves the highest total reward despite using the fewest training samples.This supports the paper’s finding that data quality is more important than quantity during SFT.
  • Training Dynamics: All SFT configurations surpass the teacher on total reward after reward filtering.The result indicates that even basic reward filtering enables the student to exceed teacher performance.

C Label-to-Role Assignment Analysis

The analysis partially validates ALM-predicted musical roles against MedleyDB’s melody and bass labels, while exposing a mismatch between static annotations and context-dependent roles.

  • Direct evaluation is difficult because no ground truth exists for the complete musical role taxonomy.
  • Melody-labeled stems are primarily assigned spotlight or accompanying_parts roles by the ALMs.
  • Bass-labeled stems are assigned the foundation role most of the time by all evaluated ALMs.
  • MedleyDB’s song-level labels cannot fully represent roles that change across segments and depend on musical context.In the example Oil, both piano and guitar are labeled melody, although the guitar solos while the piano comps in a given segment.
  • The annotation gap motivates future work in multimodal music representation learning.
Loading 2608.30559v1…