PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Source Diffusion Models for Simultaneous Music Generation and Separation

Giorgio Mariani, Irene Tallini, Emilian Postolache, Michele Mancusi, Luca Cosmo, Emanuele Rodolà

arXiv:2302.02257v4cs.SDcs.LGeess.AS

TL;DR

Existing approaches generally handle music generation and source separation separately, despite the importance of relationships among musical sources. This paper learns a joint-source diffusion model for generation, separation, and partial source imputation, and reports competitive separation results alongside generation evaluations.

  • Problem

    Prior generation models learn mixture distributions without the source information needed for separation, while separation models condition on mixtures or model sources independently and cannot generate coherent mixtures.

    Source: §1 · p. 2

  • Method

    A single denoising score-matching diffusion model learns the joint prior over contextual sources and supports total generation, partial generation, and source separation at inference.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    The Dirac-likelihood method with one correction step gives source separation results comparable to state-of-the-art models and superior overall to standard Demucs.

    Source: §5.2 · p. 9

  • Takeaways & Limitations

    The model handles total and partial music generation as well as source separation, a step toward general audio models.

    Source: §6 · p. 9

    Performance is constrained by the amount of available contextual data.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

In this work, we define a diffusion-based generative model capable of both music synthesis and source separation by learning the score of the joint probability density of sources sharing a context. Alongside the classic total inference tasks (i.e., generating a mixture, separating the sources), we also introduce and experiment on the partial generation task of source imputation, where we generate a subset of the sources given the others (e.g., play a piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task based on Dirac likelihood functions. We train our model on Slakh2100, a standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the source separation setting. Our method is the first example of a single model that can handle both generation and separation tasks, thus representing a step toward general audio models.

1 INTRODUCTION

Musical sources share context, so a useful composition model must represent both their relationships and the mixture they form. The paper proposes one joint-source diffusion model for generation and separation, including generating missing sources from those that are fixed.

1 INTRODUCTION

  • Because musical stems are interdependent, modeling their joint distribution retains information that separate source distributions do not capture.The bass follows the drums’ rhythm, while the guitar harmonizes with the melody.Source: §1 · pp. 1–2
  • The authors argue that waveform music generation and source separation are connected because composition requires both inventing coherent sources and isolating sources for control.Such control lets a composer choose which parts of a composition to modify or retain.Source: §1 · p. 2
  • The paper proposes a single diffusion model of the joint distribution of contextual sources, using prior sampling for generation and mixture conditioning for separation.This bridges the tasks that prior generation and separation models handled separately.Source: §1 · p. 2
  • The authors introduce source imputation, generating a subset of sources given the others, such as a piano accompaniment for drums.They also propose a Dirac-delta-based posterior score procedure for separation and report competitive results against state-of-the-art regressors on Slakh2100.Source: §1 · p. 3

2 RELATED WORK

Prior audio generation work includes autoregressive and diffusion approaches, while source separation includes deterministic regressors and generative models. The paper distinguishes its approach by learning the joint distribution of contextual sources without architecturally conditioning on a mixture.

2.1 GENERATIVE MODELS FOR AUDIO

  • Latent autoregressive music models support longer contexts and more coherent outputs, while text-conditioned MusicLM guides generation and SingSong generates mixture accompaniments from vocals.Unlike SingSong’s single-mixture output, the paper describes its accompaniment generation as composable at the stem level.Source: §2.1 · p. 3
  • Audio diffusion work spans speech synthesis, task-specific applications, general sound generation, and music models such as Riffusion and Moûsai.The proposed score network follows Moûsai’s U-Net design but uses waveform data.Source: §2.1 · p. 3

2.2 AUDIO SOURCE SEPARATION

  • Source separation includes deterministic regression models and generative Bayesian models that learn source priors and use a mixture likelihood during inference.Generative priors have included GANs, normalizing flows, and autoregressive models.Source: §2.2 · pp. 3–4
  • Unlike mixture-conditioned separation approaches, the proposed model learns the full joint source distribution and can therefore also generate mixtures unconditionally.The paper identifies NCSN-BASIS as a nearby method and reports that its Gaussian likelihood is sub-optimal to the proposed Dirac-based likelihood.Source: §2.2 · p. 4

3 BACKGROUND

The background describes score-based diffusion as learning a distribution’s score by denoising data perturbed with controlled Gaussian noise. Sampling starts from a high-noise Gaussian and follows a backward probability-flow ODE.

3 BACKGROUND

  • Score matching estimates the score function, ∇x log p(x), rather than modeling the target distribution directly.Denoising diffusion adds controlled Gaussian noise to help estimate scores in sparse data regions.Source: §3 · p. 4
  • With the noise schedule σ(t) = t, the forward probability-flow ODE evolves data under the negative score field.The cited formulation follows Karras et al. (2022).Source: §3 · p. 5
  • At large T, the noisy sample is approximately Gaussian; reversing the ODE describes denoising, and integrating it from a Gaussian start produces samples.The score in the sampling procedure is approximated by a neural network trained with score-matching loss.Source: §3 · p. 5

4 METHOD

The method trains a single diffusion model to learn the joint distribution of contextual sources, then uses its prior score to generate mixtures, impute sources, or separate sources given a mixture. For separation, it introduces a Dirac-likelihood approach that constrains one source to make the sources sum to the observed mixture.

4.1 MULTI-SOURCE AUDIO DIFFUSION MODELS

  • The Multi-Source Diffusion Model learns the joint prior over contextual sources by approximating their noisy score function with a neural network.Its inputs are perturbed source waveforms and a noise scale.Source: §4.1 · p. 5

4.2 INFERENCE

  • All three tasks are solved at inference by discretizing the backward diffusion equation, using score functions derived from the same prior score.Source: §4.2 · p. 5

4.2.1 TOTAL GENERATION

  • For total generation, the model samples sources from the prior and sums them to form a mixture.Source: §4.2.1 · p. 6

4.2.2 PARTIAL GENERATION

  • For partial generation, fixed sources condition the generation of the remaining sources; the method estimates the conditional gradient using an inpainting approach.The approach follows Song et al. (2021).Source: §4.2.2 · p. 6

4.2.3 SOURCE SEPARATION

  • For source separation, the method approximates the posterior score with a Dirac likelihood, constrains one source to the mixture residual, and models the limit as γ(t) → 0.The method treats separation as conditional generation given the mixture.Source: §4.2.3 · pp. 6–7
  • For weakly supervised separation without contextual source sets, the method assumes source independence and trains a separate Dirac-likelihood model for each source class.The resulting model is called ISDM Dirac.Source: §4.2.3 · p. 7

5 EXPERIMENTAL RESULTS

Experiments assess music generation and source separation on Slakh2100. MSDM’s Dirac-based separation is competitive with state-of-the-art regressors, while generation results indicate minimal difference from a mixture-trained model and non-trivial partial-generation quality.

5 EXPERIMENTAL RESULTS

  • Slakh2100 provides 145 hours of data, compared with 10 hours in MUSDB18-HQ, and was chosen for its larger quantity of training data.The paper also studies data efficiency on MUSDB18-HQ in Appendix E.Source: §5 · p. 7

5.1 MUSIC GENERATION

  • Listening tests rate total-generation quality and instrument coherence, and partial-generation quality and accompaniment density, on a 1-to-10 scale.Total generation is compared with a model trained on mixtures using the same diffusion architecture.Source: §5.1 · p. 8
  • The objective generation evaluation generalizes FAD to mixtures of fixed conditioning sources and generated sources, calling the measure sub-FAD.It uses VGGish embeddings and compares generated mixtures with ground-truth mixture chunks.Source: §5.1 · p. 8
  • Total-generation results show minimal difference between MSDM and an equivalent mixture-trained model, while partial generation achieves non-trivial subjective quality.The authors report no competitor baseline for partial-generation objective metrics and provide sub-FAD results as baselines for future research.Source: §5.1 · p. 8, Table 1 · p. 8, Table 2 · p. 8

5.2 SOURCE SEPARATION

  • Source separation is evaluated with SI-SDR improvement against the mixture baseline on filtered Slakh2100 test chunks, using 150 inference steps.The evaluation uses four-second chunks with two-second overlap, excludes silent and single-source chunks, and compares Dirac and Gaussian posterior scores with Demucs baselines.Source: §5.2 · pp. 8–9
  • MSDM with Dirac likelihood and one correction step gives source-separation results comparable to state of the art and superior to standard Demucs overall.The Dirac likelihood improves overall results, and ISDM with Dirac likelihood outperforms the state of the art on Bass and Drums.Source: §5.2 · p. 9, Table 3 · p. 9

6 CONCLUSIONS

The paper presents one denoising score-matching model for source separation and total or partial music generation. It reports competitive separation results and identifies contextual-data availability as a performance constraint.

6 CONCLUSIONS

  • A single neural network trained once handles source separation, mixture generation, and accompaniment generation, with tasks differentiated during inference.The authors report quantitative separation results comparable to state-of-the-art regressor models and qualitative and quantitative tests of total and partial generation.Source: §6 · p. 9

6.1 LIMITATIONS AND FUTURE WORK

  • The amount of available contextual data constrains model performance; pre-separating mixtures for training may prove beneficial.The authors also propose exploring known non-additive relations among sub-signals and jointly modeling MIDI information for further control.Source: §6.1 · p. 9

A DERIVATION OF MSDM DIRAC POSTERIOR SCORE FOR SOURCE SEPARATION

The derivation obtains a source posterior conditioned on the observed mixture using a Dirac likelihood. It parameterizes the posterior so the sources sum to the mixture and derives its score for inference.

A DERIVATION OF MSDM DIRAC POSTERIOR SCORE FOR SOURCE SEPARATION

  • The separation posterior conditions the generative model on the observed mixture and substitutes a Dirac likelihood through Bayes’ theorem.The derivation first notes that estimating the posterior requires the mixture density, which it does not model.Source: A DERIVATION OF MSDM DIRAC POSTERIOR… · pp. 15–16
  • Because the mixture density is unknown, the method approximates the required expectation with Monte Carlo at the mean mixture trajectory y(0).The cited prose identifies y(0) as the mean of p(y(t) | y(0)).Source: A DERIVATION OF MSDM DIRAC POSTERIOR… · p. 16
  • The posterior is parameterized by the first N−1 sources, with the last source constrained to make their sum equal the observed mixture.The derivation defines a mapping from the free sources to the full source tuple, then differentiates the parameterized posterior.Source: A DERIVATION OF MSDM DIRAC POSTERIOR… · p. 16
  • The resulting gradient uses the score-network entries for the free source and the constrained final source, applying the chain rule through the parameterization.The derivation also invokes the forward diffusion’s Markov property when simplifying the conditional terms.Source: A DERIVATION OF MSDM DIRAC POSTERIOR… · pp. 15–17

B DERIVATION OF GAUSSIAN AND WEAKLY-SUPERVISED POSTERIOR SCORES FOR SOURCE SEPARATION

The appendix derives Gaussian and weakly supervised posterior scores alongside the Dirac approach. It also reports that inference is slower than Demucs but faster than long-run Demucs-plus-Gibbs baselines.

B DERIVATION OF GAUSSIAN AND WEAKLY-SUPERVISED POSTERIOR SCORES FOR SOURCE SEPARATION

  • The Gaussian posterior adapts Jayaram and Thickstun’s likelihood to continuous-time score-based diffusion, then derives the supervised MSDM score by replacing the contextual prior with the score network.The inference procedure keeps y(t) fixed at y(0) and uses that value for Monte Carlo.Source: B DERIVATION OF GAUSSIAN AND… · p. 17
  • The weakly supervised derivations approximate the posterior score and factorize the prior to obtain the ISDM Dirac score.In these cases, each source has an individual score function rather than a slice of the joint score network.Source: B DERIVATION OF GAUSSIAN AND… · pp. 17–18
  • For a 12-second separation, ISDM and MSDM are less time-efficient than Demucs but more time-efficient than Demucs + Gibbs with 256 or 512 steps.Demucs + Gibbs at 256 steps is included because it is the minimum step count whose SI-SDRI over all instruments exceeds ISDM’s 17.59 dB.Source: Table 4 · p. 18

C.1 DATASET

The experiments compare Slakh2100 and MUSDB18-HQ data, describe the model and sampler setup, and report source-separation hyperparameter findings. Dirac likelihoods achieve higher SI-SDRI than Gaussian likelihoods for both separator types, while ISDM separators outperform contextual MSDM separators but cannot generate.

C.1 DATASET

  • Slakh2100 contains 2,100 multitrack waveform tracks synthesized from MIDI, with 31 stem classes; experiments use its four most abundant classes.The selected classes are Bass, Drums, Guitar, and Piano.Source: §C.1 · p. 18
  • MUSDB18-HQ contains 150 tracks split into 100 training and 50 test tracks, with Bass, Drums, Vocals, and Other stems.The dataset provides roughly 10 hours of professional-grade audio.Source: §C.1 · p. 18

C.2 ARCHITECTURE AND TRAINING

  • The score network uses a non-latent, unconditional Moûsai basis and trains MSDM with four stacked mono stem channels on 22-kHz audio in roughly 12-second contexts.The U-Net has six encoder layers and a symmetric decoder.Source: §C.2 · pp. 18–19

C.3 THE SAMPLER

  • Sampling uses a first-order Euler ODE integrator with added stochasticity, iterative noise-and-reoptimization correction, and a nonlinear schedule emphasizing lower noise levels.The correction iterates for R steps after each prediction, holding the score network fixed at σ_i.Source: §C.3 · p. 19

D HYPERPARAMETER SEARCH FOR SOURCE SEPARATION

  • Table 6 compares MSDM with Demucs v2 using SI-SDRI, with the network trained at 44 kHz on 6-second chunks.The caption identifies the comparison but supplies no outcome values.Source: Table 6 · p. 20
  • Stochasticity benefits all separators, Dirac likelihoods yield higher SI-SDRI than Gaussian likelihoods for both model types, and ISDM outperforms MSDM but cannot generate.The reported SI-SDRI results are averaged across Bass, Drums, Piano, and Guitar.Source: Table 5 · p. 19, D HYPERPARAMETER SEARCH FOR SOURCE… · p. 20

E DATA EFFICIENCY STUDY: RESULTS ON MUSDB

The MUSDB18-HQ experiments compare MSDM with Demucs v2 through direct transfer, fine-tuning, and training on MUSDB18-HQ. Because only Bass and Drums are shared with Slakh2100, the first two strategies are limited to those stems.

E DATA EFFICIENCY STUDY: RESULTS ON MUSDB

F SUBJECTIVE EVALUATION

The subjective evaluation uses separate listening-test forms for total and partial generation. The forms assess generated-song quality and instrument coherence or density, with different song counts and participant groups.

F SUBJECTIVE EVALUATION

  • The total-generation form asks subjects to rate quality and instrument coherence for 30 songs, split evenly between MSDM and a mixture-trained model.Thirty-two people answered the survey.Source: Figure 3 · p. 21
  • The partial-generation form evaluates 15 songs with a fixed subset of sources, while MSDM generates the remaining sources.The requested generated sources are stated above each song; 21 subjects answered.Source: Figure 3 · p. 21
  • The evaluation form rates song quality and instrument coherence for total generation, and quality and density for partial generation.Source: Figure 3 · p. 21
Loading 2302.02257v4…