Source-linked AI summary
Deep Clustering and Conventional Networks for Music Separation: Stronger Together
Yi Luo, Zhuo Chen, John R. Hershey, Jonathan Le Roux, Nima Mesgarani
TL;DR
Music separation has limited evidence for deep clustering despite its success in speech, especially under dataset mismatch. The paper combines deep clustering and mask inference in a hybrid Chimera network and reports that the hybrid outperforms either component, with robust mismatched-condition performance.
Problem
Deep clustering had been successful for speaker-independent speech separation, but its effectiveness for challenging music source separation and mismatched datasets was not well established.
Method
The paper combines a deep clustering embedding head and a conventional mask-inference head in a shared two-headed Chimera network trained with complementary objectives.
Results
The hybrid network outperformed both the plain deep clustering and mask-inference networks, with experimental results confirming robustness under mismatched conditions.
Takeaways & Limitations
Deep clustering is suitable for music source separation, and jointly training it with mask inference yields stronger performance than either approach alone.
Abstract
from arXiv · showhide
Deep clustering is the first method to handle general audio separation scenarios with multiple sources of the same type and an arbitrary number of sources, performing impressively in speaker-independent speech separation tasks. However, little is known about its effectiveness in other challenging situations such as music source separation. Contrary to conventional networks that directly estimate the source signals, deep clustering generates an embedding for each time-frequency bin, and separates sources by clustering the bins in the embedding space. We show that deep clustering outperforms conventional networks on a singing voice separation task, in both matched and mismatched conditions, even though conventional networks have the advantage of end-to-end training for best signal approximation, presumably because its more flexible objective engenders better regularization. Since the strengths of deep clustering and conventional network architectures appear complementary, we explore combining them in a single hybrid network trained via an approach akin to multi-task learning. Remarkably, the combination significantly outperforms either of its components.
1. INTRODUCTION
Music source separation seeks to recover vocals or instruments from mixtures, but variability across recordings limits dataset generalization. The study examines whether deep clustering can improve singing voice separation and combines it with mask inference.
- 1. INTRODUCTION: Music source separation separates recordings into individual instrument tracks or vocals and accompaniment.Applications include predominant pitch tracking, Karaoke accompaniment generation, and singer identification.
- 1. INTRODUCTION: Generalization across music datasets remains difficult because recordings vary greatly in genre and instrumentation.The passage also identifies processing-speed and generalization problems in several traditional supervised approaches.
- 1. INTRODUCTION: Conventional regression networks estimate source signals, often through time-frequency masks applied to mixture representations.These networks have produced superior singing voice separation results compared with traditional approaches.
- 1. INTRODUCTION: Deep clustering uses pairwise affinities, allowing separation of multiple same-type sources and an arbitrary number of sources.The framework had previously been applied successfully to single-channel, speaker-independent speech separation.
- 1. INTRODUCTION: The study combines deep clustering and mask inference in a two-headed Chimera network for singing voice and accompaniment separation.The deep clustering objective also reduces within-source variance in the internal representation, potentially benefiting generalization.
2. MODEL DESCRIPTION
The model maps time-frequency features to embeddings and masks, combining deep clustering with signal-approximation training. A shared Chimera body supports complementary heads and test-time separation procedures.
- 2.1. Deep clustering: Deep clustering assigns each time-frequency bin a D-dimensional embedding, with same-source bins encouraged to be similar and different-source bins dissimilar.Source identity depends on the task, such as vocals versus all other instruments in singing voice separation.
- 2.1. Deep clustering: The label indicator matrix produces a binary affinity target, while the network estimates an embedding matrix whose pairwise products form estimated affinities.Low-rank structure reduces the computational cost of these large affinity matrices.
- 2.1. Deep clustering: At test time, clustering the embeddings produces assignments used as binary time-frequency masks for source estimation.K-means is given as an example of the clustering algorithm.
- 2.2. Multi-task learning and Chimera networks: Deep clustering and mask inference address the same separation task while combining representation learning with direct signal-recovery objectives.Unlike mask inference, deep clustering requires a post-clustering step that is outside its original training objective.
- 2.2. Multi-task learning and Chimera networks: The Chimera network shares a body and branches into an embedding head and a soft-mask head.The embedding head follows deep clustering, while the mask head uses frequency-specific fully connected layers to output source masks.
- 2.2. Multi-task learning and Chimera networks: The mask-inference head can use MSA or mMSA objectives, with mMSA addressing cases where destructive interference makes mixture magnitude smaller than source magnitude.The alternative objective approximates each source magnitude by masking the mixture with a reference mask.
- 2.2. Multi-task learning and Chimera networks: The global Chimera objective weights deep clustering and mask inference through α, with α = 1 yielding deep clustering and α = 0 yielding mask inference.The deep clustering loss is divided by TF to account for pairwise-bin losses versus time-frequency-bin spectrum losses.
- 2.2. Multi-task learning and Chimera networks: Either trained head can be used at test time: mask inference directly outputs masks, while deep clustering outputs embeddings for clustering.This preserves distinct separation procedures within the same architecture.
3. EVALUATION AND DISCUSSION
The evaluation compares deep clustering, mask-inference, and hybrid systems on matched DSD100-remix and mismatched public iKala data, including MIREX results. Deep clustering generalizes better under mismatch, while the hybrid Chimera system achieves the best reported performance in the proposed experiments.
- Datasets and setup: The evaluation uses DSD100-remix for training and testing, plus hidden and public iKala datasets for evaluation.The proposed Chimera models are evaluated on matched DSD100-remix and mismatched public iKala data.
- MIREX submission: The MIREX system trained on DSD100-remix won first place in 2016 and outperformed the best systems from previous years on hidden iKala.The hidden iKala dataset differs from DSD100, so this result reflects mismatched-data performance.
- Hybrid-system results: MI performs competitively with DC on DSD100-remix, whereas DC performs significantly better on public iKala.The comparison uses SDRi, defined as the improvement in scale-invariant SDR relative to the mixture.
- Hybrid-system results: CHIα-MI achieves the best performance, combining a deep-clustering objective with a mask-inference head.The authors suggest that joint training produces a stronger representation that the mask-inference head exploits through signal approximation.
- Objective and embedding analysis: Using the mMSA objective slightly improves MI performance overall, while reducing embedding dimension D from 20 to 10 yields further improvements.The lower embedding dimension also leaves room to increase the number of frequency bins.
- Input-feature analysis: Higher sampling rate, larger STFT window size, and more mel-frequency bins produce better performance.These comparisons use hybrid models with the mMSA objective and D = 10.
4. CONCLUSION
The paper shows that deep clustering is suitable for singing voice separation and that combining it with mask inference improves performance. The hybrid approach also remains robust under mismatched conditions.
- Conclusion: Deep clustering is suitable for separating sources in music signals, despite being originally designed for speech mixtures.The study evaluates the framework on singing voice separation.
- Conclusion: Jointly optimizing deep clustering with mask inference produces a hybrid network that outperforms both component networks.The conclusion identifies the hybrid network as stronger than plain deep clustering and mask inference.
- Conclusion: Experimental results confirm the hybrid approach’s robustness in mismatched conditions.This conclusion is supported by the reported evaluation of the hybrid system.