Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation

1 Introduction

• The pipeline applies a separator and diffusion model to each mixture, then learns weights to combine their estimates linearly in the frequency domain.The separator produces deterministic estimates; the diffusion model generates corresponding estimates, and a shallow CNN predicts the mixing weights.Source: §1 · p. 1
• Frequency-domain combination addresses possible phase differences between estimates produced by separate processes.The reconstructed phase in each segment can be arbitrary, so phase compensation is needed.Source: §1 · p. 1
• Only the weight-prediction network is trained; the separator and diffusion model are taken from published models.The separator is Gated-LSTM or SepFormer, while the vocoder is DiffWave.Source: §1 · p. 1
• Across multiple deep architectures and methods, pretrained diffusion post-processing improves source-separation results by a significant gap.The paper also derives bounds for mutual information and SDR error for such combinations.Source: §1 · pp. 1–2
• The mutual information of the best combined estimate with its ground-truth source is bounded by twice the mixture–source mutual information, and its SDR-error bound depends on separator quality and that mutual information.The bounds are stated for combinations of deterministic and generative estimates.Source: §1 · p. 2

2 Related work

• Earlier speech-separation systems progressed from embedding clustering and permutation-invariant training to encoder-separator-decoder, convolutional, ensemble, recurrent, and MulCat designs.The cited approaches include PIT [32], uPIT, Conv-Tasnet [21], and MulCat [24].Source: §2 · p. 2
• PIT-based methods can be difficult to scale to many speakers because their assignment step has O(C!) time complexity; the Hungarian algorithm reduces this to O(C^3).The passage presents the Hungarian algorithm as an alternative permutation-invariant training method [5].Source: §2 · p. 2
• The prior upper bound applies to deterministic separation models, leaving open whether it also holds for generative models.The bound depends on mixture–source mutual information and decreases as the number of sources increases.Source: §2 · p. 2
• Diffusion models have been used for speech generation, with score-matching approaches reporting state-of-the-art speech-generation results against established methods.The passage cites Wavernn [10], Wavenet [25], and GAN-TTS as comparison methods.Source: §2 · p. 2
• Prior generative source-separation work included image-mixture experiments and DiffSep for audio, but neither established state-of-the-art results or a bound for generative methods.DiffSep’s reported single-channel separation lagged deterministic models by a large margin [27].Source: §2 · p. 2
• The authors report no similar diffusion-model generalization bounds and suggest that analogous analysis could extend to other domains with suitable data distributions.They give image denoising as an example and leave the corresponding derivation for future work.Source: §2 · pp. 2–3

3 Analysis

• The analysis uses the mutual-information chain rule to bound the combined estimates, models generative noise, and updates the earlier deterministic bound.The SDR error metric is defined as 10log10 of source variance divided by error variance.Source: §3 · p. 3

3.1 Generative Bound

• I(vr; vgr) ≤ 2 · I(vr; mr) bounds the mutual information between the source and the combined estimates by twice that between the source and mixture.The authors model generative-model error as additive noise and consider Gaussian noise for the tightness analysis.Source: §3.1 · p. 4
• ρ is approximately one when σ² ≤ 10⁻², so the generative estimate retains nearly the maximum mutual information described for the mixture.The passage relates this range to reconstruction errors below 10⁻² for well-functioning pretrained generative models [14] [2] [3].Source: §3.1 · pp. 4–5
• Under the stated assumptions, the derived bound is tight, and deterministic and generative approximations have similar error magnitudes.The analysis models inherent generative-model reconstruction error as additive noise.Source: §3.1 · pp. 4–5
• 3 dB is the maximum addition to the upper bound when generative and deterministic results are combined.The bound is described as tight when worst-case independent additive noise is taken into account.Source: §3.1 · p. 5

4 Method

• The deterministic backbone estimates each source, and pretrained DiffWave produces a generative estimate from each backbone output.The generative vocoder is not given phase information, so its output can have a time-varying phase shift.Source: §4 · pp. 5–6
• The method transforms both estimates to the frequency domain and uses learned phase alignment before combining them.The alignment features include the deterministic phase and the relative phase between deterministic and generative estimates.Source: §4 · p. 6
• A dual six-layer convolutional network predicts magnitude and phase coefficients used to combine the two spectrograms, which are then converted back to a time-domain signal.The coefficient heads share hyperparameters, and the inverse short-time Fourier transform produces the final signal.Source: §4 · p. 6
• Scale-invariant SDR is the training objective, and for C ≥ 10 the network uses Hungarian assignments rather than permutation-invariant loss.The Hungarian algorithm explicitly assigns estimated sources to ground-truth sources.Source: §4 · p. 6

Experimental setup

• Only the alignment network F is trained; B and the diffusion model GM are taken from published models, with DiffWave trained unsupervised on LibriMix and WSJ0.The same six-layer CNN alignment architecture is used across datasets, and DiffWave is trained on unmixed training sources for LibriMix and WSJ0.Source: §5 · p. 7

Baselines and ablations

• The experiments compare against DiffSep, SepFormer, and SepIt using pretrained DiffWave, and test both removing F with heuristic phase alignment and averaging and substituting deterministic HiFiGAN.The heuristic equal-mixing ablation performs considerably better than SepFormer but only slightly better than SepIt; deterministic HiFiGAN yields little to no improvement.Source: §5 · pp. 7, 9

Alignment

• The alignment procedure uses segment-wise cross-correlation to estimate time delays, applies complex phase shifts, and returns the signal to the time domain with inverse STFT.Cross-correlation is computed by conjugate multiplication in the frequency domain; the maximum of the inverse-transformed correlation identifies each segment's delay.Source: §5 · p. 7

Qualitative results

• The deterministic and generative estimates can differ in phase, while the generative estimate sometimes repairs bursts that the deterministic estimate misses, making their combination track the source more closely.The generative estimate has magnitude information but not synchronized absolute phase; supplementary examples also show it recovering temporal parts omitted by the deterministic estimate, despite retaining more background noise.Source: §5 · pp. 7–8

Error comparison

• Across 20 ms segments, generative estimates have a slight tendency toward greater MSE but remain in the same error range as deterministic estimates, with neither markedly better overall.The comparison uses MSE to aligned sources and supports combining the two estimates to improve overall source estimation.Source: §5 · p. 8

Source separation results

• On WSJ0, the method improves SI-SDR by 1.5 dB for two speakers and 1 dB for three, reaching state of the art and surpassing the deterministic-model bound for two speakers.On LibriSpeech, it improves over the state of the art for all reported speaker counts; the five-speaker result is 0.3 dB short of the classical bound, with larger gaps for other mixtures.Source: §5 · pp. 8–9

• The bound applies to any non-deterministic model, rather than being tailored to diffusion models.Source: §6 · p. 9
• Its non-specificity provides no practical method-development guidance beyond encouraging non-deterministic post-processing.Source: §6 · p. 9
• Because the bound assumes algorithms operating on limited-size chunks, it may not hold for future methods using large contexts.Source: §6 · p. 9

7 Conclusions

• For two sources, the method surpasses the previous upper bound, while estimates improve across the tested numbers of speakers.Source: §7 · p. 9
• The proposed general upper bound for generative source separation indicates that a pretrained generative model may achieve at most 3 dB improvement over the previous deterministic bound.Source: §7 · p. 9
• The method combines deterministic and generated signals in the frequency domain using learned weights, and outperforms classical alignment.Source: §7 · p. 9
• Extending the bound to other data domains by replacing the underlying distributions is left for future work.Source: §7 · p. 9

A Proof of Lemma 3.2

• The proof invokes the definition of mutual information and the entropy chain rule.Source: A Proof of Lemma 3.2 · p. 12
• It rewrites the entropy differences as I(X; Z) + I(X; Y | Z).Source: A Proof of Lemma 3.2 · p. 12

B Sample spectrograms

• The combined estimate uses frequency-domain combination coefficients produced by F.Source: B Sample spectrograms · p. 12
• Figure 5 presents four spectrograms: the source, deterministic estimate, generated voice, and combined estimate.Source: B Sample spectrograms · p. 12, Figure 5 · p. 13
• The panels allow comparison of the deterministic and generated estimates with their combined estimate and the source.Source: B Sample spectrograms · p. 12, Figure 5 · p. 13