Source-linked AI summary
Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation
Qiuqiang Kong, Yin Cao, Haohe Liu, Keunwoo Choi, Yuxuan Wang
TL;DR
The paper addresses limitations in music source separation from inaccurate phase reconstruction, mask magnitudes bounded by 1, and insufficiently explored depth. It decouples cIRM magnitude and phase estimation, combines bounded masks with direct magnitude prediction, and uses a 143-layer residual UNet. On MUSDB18, the proposed system achieves 8.98 dB vocals SDR versus the previous best 7.24 dB, while the ablation shows only a slight average gain from unbounded cIRMs.
Problem
Neural MSS systems can suffer from inaccurate phase reconstruction, mask magnitudes limited to 1, and underexplored very deep architectures.
Source: §1 · pp. 1–2Method
The system decouples cIRM magnitude and phase estimation, combines bounded mask and direct magnitude predictions, and uses a 143-layer residual UNet.
Source: §1 · pp. 1–2Results
8.98 dB vocals SDR versus the previous best 7.24 dB is reported on MUSDB18 for the proposed system.
Source: §4.3 · p. 6, §5 · p. 6Takeaways & Limitations
Decoupled magnitude–phase estimation, flexible magnitude prediction, and deep residual architecture improve the reported MSS system performance.
Source: §4.4 · p. 6, §5 · p. 6Unbounded cIRM magnitude modeling produced only a 0.082 dB average improvement over UNetPhase and did not improve bass or drums.
Source: §4.4 · p. 6
Abstract
from arXiv · showhide
Deep neural network based methods have been successfully applied to music source separation. They typically learn a mapping from a mixture spectrogram to a set of source spectrograms, all with magnitudes only. This approach has several limitations: 1) its incorrect phase reconstruction degrades the performance, 2) it limits the magnitude of masks between 0 and 1 while we observe that 22% of time-frequency bins have ideal ratio mask values of over~1 in a popular dataset, MUSDB18, 3) its potential on very deep architectures is under-explored. Our proposed system is designed to overcome these. First, we propose to estimate phases by estimating complex ideal ratio masks (cIRMs) where we decouple the estimation of cIRMs into magnitude and phase estimations. Second, we extend the separation method to effectively allow the magnitude of the mask to be larger than 1. Finally, we propose a residual UNet architecture with up to 143 layers. Our proposed system achieves a state-of-the-art MSS result on the MUSDB18 dataset, especially, a SDR of 8.98~dB on vocals, outperforming the previous best performance of 7.24~dB. The source code is available at: https://github.com/bytedance/music_source_separation
1. INTRODUCTION
The paper targets limitations in neural music source separation caused by inaccurate phase reconstruction, bounded mask magnitudes, and shallow architectures. It addresses these through decoupled magnitude–phase cIRM estimation, residual magnitude prediction, and a 143-layer residual UNet.
1. INTRODUCTION
- MSS supports downstream MIR tasks including melody extraction, pitch estimation, music transcription, and music remixing.It also has direct applications such as Karaoke and music remixing.Source: §1 · p. 1
- Neural MSS systems are limited by inaccurate phase reconstruction, mask magnitudes constrained to 1, and underexplored very deep architectures.These limitations motivate the paper’s focus on phase estimation, unbounded magnitude modeling, and network depth.Source: §1 · pp. 1–2
- The proposed system decouples cIRM magnitude and phase estimation to improve phase-aware source separation.The method estimates complex ideal ratio masks while separating magnitude and phase predictions.Source: §1 · pp. 1–2
- 22% of cIRM time-frequency bins have magnitudes larger than 1, motivating a residual combination of bounded mask and direct spectrogram predictions.The spectrogram term acts as a residual component, allowing predicted mask magnitudes to exceed 1.Source: §1 · pp. 1–2
- A 143-layer residual UNet uses residual encoder, intermediate, and decoder blocks to extend deep architectures for MSS.The paper argues that deeper architectures can increase separation performance.Source: §1 · p. 2
2. BACKGROUNDS
Magnitude-based MSS methods estimate source magnitudes while commonly reusing mixture phase, whereas cIRM methods additionally model magnitude scaling and phase rotation. Empirical MUSDB18 analysis shows that correct phase information and unbounded mask magnitudes substantially raise separation upper bounds.
2.1 Approach 1: Direct Magnitude Prediction
- Direct magnitude prediction estimates |S| from |X|, then reconstructs the separated STFT using the mixture phase.Magnitude mask-based methods similarly apply predicted masks element-wise and approximate separated-source phase with mixture phase.Source: §2.1 · p. 2, §2.2 · p. 2
2.3 Complex Mask-Based Approach
- cIRM separation applies magnitude scaling and phase rotation, with the cIRM magnitude controlling scaling and its angle controlling phase rotation.The separated waveform is obtained after applying an inverse STFT.Source: §2.4 · p. 2, §2.1 · p. 2
2.5 Empirical analysis of the effect of bounded magnitude mask
- Correct phase information yields substantially higher MUSDB18 upper bounds than IBM and IRM, with cIRM(1) about 10 dB above IRM(1).Increasing the cIRM magnitude limit from 1 to 2 raises upper bounds by more than 9.89 dB for every instrument; unbounded cIRM has theoretical infinite SDR and observed SDR above 50 dB.Source: §2.5 · p. 3, Table 1 · p. 3
2.4 Out-of-Phase and Masks
- cIRM phase distributions span all directions, while spectrogram-based methods assume cIRM phases are zero, supporting explicit phase prediction.Figure 2 plots real and imaginary cIRM parts for vocals, accompaniment, bass, drums, other instruments, and all sources against unit circles.Source: §2.6 · p. 3, Figure 2 · p. 3
3. PROPOSED SYSTEM
The proposed MSS system decouples cIRM magnitude and phase estimation, extends mask magnitudes beyond one through residual direct-magnitude prediction, and uses a 143-layer residual UNet.
3. PROPOSED SYSTEM
- The complete system combines cIRM phase estimation, magnitude-mask prediction with direct magnitude residuals, and a 143-layer residual UNet trained with waveform-domain L1 loss.Its four outputs are M̂mag, Q̂, P̂r, and P̂i, which share a backbone and use individual linear output layers.Source: §3 · p. 4, §3.2 · p. 4
3.1 Decoupling Magnitude and Phase for cIRM Estimation
- Decoupled cIRM estimation predicts a bounded magnitude mask plus real and imaginary phase components, which are converted into a complex mask for magnitude scaling and phase rotation.The separated waveform is obtained after applying an inverse STFT.Source: §3.1 · p. 4
3.2 Combination of Bounded Mask Estimation and Direct Magnitude Prediction
- The magnitude estimator combines a bounded mask with direct magnitude prediction as a residual, allowing the resulting source magnitude to exceed the mask’s 0-to-1 bound.A ReLU ensures the predicted magnitude remains nonnegative, while phase estimation uses the predicted real and imaginary components.Source: §3.2 · p. 4
3.3 Residual UNet
- The residual UNet extends prior UNet designs to 143 layers using residual encoder, intermediate, decoder, and convolutional blocks.The baseline UNet has 33 convolutional layers; the residual design uses six REBs and six RDBs, with four intermediate convolutional blocks.Source: §3.3 · pp. 4–5
- Residual encoder and decoder blocks use residual convolutional blocks with shortcut connections, while decoder blocks upsample feature maps with transposed convolution.Figure 3 presents the overall residual-block system, and Figure 4 details the REB, RDB, and RCB components.Source: §3.3 · p. 4, Figure 3 · p. 5, Figure 4 · p. 5
4. EXPERIMENTS
Experiments on MUSDB18 compare the proposed residual UNet system with prior MSS methods and ablate phase decoupling, magnitude prediction, and network depth. The proposed modifications improve separation, although unbounded masks show only a slight average gain in the tested systems.
Ablation Study
- 0.57 dB average improvement results from predicting phase compared with magnitude-mask-only UNet models.UNet performed slightly worse than UNetPhase, which incorporates phase estimation.Source: §4.4 · p. 6, Table 3 · p. 6
- 0.35 dB and 0.45 dB average gains result from decoupling magnitude and phase in UNet and ResUNet, respectively.Combining bounded magnitude masks with direct magnitude prediction adds a further 0.2 dB for UNet and 0.196 dB for ResUNet on average.Source: §4.4 · p. 6
- 0.7 dB average improvement is achieved by ResUNet over UNet when other conditions are fixed, with ResUNet outperforming for every source instrument.The comparison supports the effectiveness of the very deep architecture in MSS.Source: §4.4 · p. 6, Table 3 · p. 6
- Unbounded cIRM masks provide only a 0.082 dB average advantage over UNetPhase and do not improve bass or drums.The reported cIRM upper bounds exceed UNetPhase performance by more than 10 dB for all listed sources, so the tested systems do not clearly reach that theoretical limit.Source: §4.4 · p. 6
5. CONCLUSION
The paper concludes that accurate phase estimation, unbounded complex ideal ratio masks, and a 143-layer residual UNet improve music source separation. On MUSDB18, the proposed modifications achieve 8.98 dB vocal SDR.
Conclusion
- The proposed system decouples cIRM magnitude and phase estimation, combines bounded masks with direct prediction, and uses a 143-layer residual UNet.These modifications address phase estimation, mask-magnitude flexibility, and deep-architecture design in MSS.Source: §5 · p. 6
- 22% of cIRMs have magnitude larger than one, motivating magnitude estimation beyond the usual bounded mask range.The paper reports that the proposed modifications improve performance on MUSDB18.Source: §5 · p. 6
- 8.98 dB vocal SDR is achieved on MUSDB18 with the proposed modifications.This is the principal reported experimental outcome in the conclusion.Source: §5 · p. 6