Source-linked AI summary
Deep clustering: Discriminative embeddings for segmentation and separation
John R. Hershey, Zhuo Chen, Jonathan Le Roux, Shinji Watanabe
TL;DR
Deep clustering addresses class-independent acoustic source separation by learning discriminative spectrogram embeddings whose partitions can be recovered with simple clustering. Preliminary speech experiments outperform an oracle NMF baseline, generalize to unknown speakers and three-speaker mixtures, and achieve 6.15 SDR improvement in one three-speaker setting.
Problem
Class-based separation requires known output classes, while flexible spectral clustering does not readily leverage deep networks' learning power and speed.
Method
A deep network learns unit-norm, K-dimensional embeddings for input elements so simple clustering recovers target partitions without class labels.
Results
6.15 SDR improvement was achieved for mixtures of three fixed speakers, while the system also generalized to unknown speakers and separated three-speaker mixtures after two-speaker training.
Takeaways & Limitations
Deep clustering provides a class-independent framework that can generalize to unknown source identities and source counts beyond those used in training.
Takeaways & Limitations
The results are preliminary, and the authors expect further model refinement and alternative architectures to improve performance.
Abstract
from arXiv · showhide
We address the problem of acoustic source separation in a deep learning framework we call "deep clustering." Rather than directly estimating signals or masking functions, we train a deep network to produce spectrogram embeddings that are discriminative for partition labels given in training data. Previous deep network approaches provide great advantages in terms of learning power and speed, but previously it has been unclear how to use them to separate signals in a class-independent way. In contrast, spectral clustering approaches are flexible with respect to the classes and number of items to be segmented, but it has been unclear how to leverage the learning power and speed of deep networks. To obtain the best of both worlds, we use an objective function that to train embeddings that yield a low-rank approximation to an ideal pairwise affinity matrix, in a class-independent way. This avoids the high cost of spectral factorization and instead produces compact clusters that are amenable to simple clustering methods. The segmentations are therefore implicitly encoded in the embeddings, and can be "decoded" by clustering. Preliminary experiments show that the proposed method can separate speech: when trained on spectrogram features containing mixtures of two speakers, and tested on mixtures of a held-out set of speakers, it can infer masking functions that improve signal quality by around 6dB. We show that the model can generalize to three-speaker mixtures despite training only on two-speaker mixtures. The framework can be used without class labels, and therefore has the potential to be trained on a diverse set of sound types, and to generalize to novel sources. We hope that future work will lead to segmentation of arbitrary sounds, with extensions to microphone array methods as well as image segmentation and other domains.
1 Introduction
The paper frames audio and vision segmentation as partitioning signal elements into regions without requiring known object classes. It proposes learned embeddings that make partition labels recoverable by simple clustering, targeting flexible speech separation and broader class-independent segmentation.
- Problem: Segmentation partitions indexed signal elements, such as pixels or time-frequency coordinates, into regions associated with individual objects or sound sources.In audio source separation, the resulting regions can define masks that extract target signals from mixtures.
- Problem: Partition-based segmentation learns from partition labels rather than object classes, allowing unknown objects to be segmented.This differs from class-based methods that learn labels for known object categories.
- Motivation: Class-based approaches require explicit output classes or instances, while generative approaches may not scale computationally to general segmentation tasks.These limitations motivate a more flexible partition-based approach.
- Motivation: Spectral clustering handles clusters of arbitrary topology but relies on a computationally expensive eigenspace transformation.Its local affinity structure can be sparse and block-diagonal, making the transformation useful before central clustering.
- Approach: Deep clustering learns embeddings that bring elements from the same partition together and separate elements from different partitions for efficient methods such as k-means.The approach combines partition-based training with deep learning rather than relying on specially designed features and spectral clustering.
- Results and scope: Preliminary experiments separate speech with a speaker-independent model, generalize to an open set of test speakers, and separate three-speaker mixtures after training on two-speaker mixtures.The broader goal is class-independent segmentation of arbitrary sounds, with possible applications to images and other domains.
2 Learning deep embeddings for clustering
The method learns deep, unit-norm embeddings whose clustering recovers input partitions, while directly optimizing a low-rank affinity representation for efficient training and inference.
- The network maps each input element to a K-dimensional embedding that can be clustered into the target partition.The embedding is a global function of the full input and provides a permutation- and cardinality-independent encoding.
- Training uses partition indicators and an objective invariant to the number and permutation of partition labels.This allows the model to learn partitions without requiring semantic class labels.
- The objective pulls same-partition embeddings together and different-partition embeddings apart without allowing trivial collapse.Its first term corresponds to the k-means objective for labeled assignments, while the second term is constant in that formulation.
- The model approximates the ideal pairwise affinity matrix with the low-rank matrix V V^T.The ideal affinity is defined as Y Y^T, while the learned affinity is compared with it using a weighted Frobenius objective.
- The low-rank formulation avoids explicitly constructing the N × N affinity matrix and enables efficient derivative computation.Because N is typically much larger than K, this produces a significant speedup for deep-network optimization.
- At test time, rows of the learned embedding are clustered with k-means, optionally after spectral-clustering-style dimensionality reduction.The optional reduction begins with an SVD and clusters normalized rows of principal left singular vectors.
3 Speech separation experiments
The experiments evaluate deep clustering for speech separation using two- and three-speaker mixtures, with embeddings clustered into speaker masks. Results examine clustering methods, embedding settings, mixture types, and generalization to three speakers.
- Experimental setting: Speech separation is tested on challenging two- and three-speaker mixtures, including same-gender speakers.Same-gender mixtures are particularly difficult because the speakers’ pitches overlap; three-speaker mixtures always include at least two speakers of the same gender.
- Training procedure: The system uses log short-time Fourier magnitudes, processes 100-frame segments, and trains against ideal-binary-mask-derived affinity targets.A binary weight excludes time-frequency bins below the specified source-magnitude threshold during training.
- Speech separation procedure: At test time, row embeddings are clustered into as many groups as speakers, and the resulting assignments produce time-frequency masks for waveform reconstruction.The experiments compare k-means variants and spectral clustering on embeddings produced for each segment.
- Generalization: The model separates three-speaker mixtures despite training only on two-speaker mixtures by changing the clustering step from two to three clusters.The authors present this as evidence that deep clustering can scale to more sources without changing the training procedure.
- Evaluation: Performance is measured by averaged signal-to-distortion ratio (SDR), with initial averages of 0.16 dB for two-speaker mixtures and −2.95 dB for three-speaker mixtures.A supervised sparse non-negative matrix factorization system with oracle speaker bases serves as the baseline.
4 Results and discussion
The proposed system generalizes across speaker conditions and embedding configurations, including three-speaker mixtures despite training only on two-speaker mixtures. Preliminary analyses also identify performance differences across gender mixtures and motivate further architectural and domain expansion.
- Speaker generalization: The system generalizes well to unknown speakers, with open-speaker performance similar to closed-speaker results.Both oracle and non-oracle clustering methods significantly outperform the oracle NMF baseline.
- Parameter regimes: K = 5 completely fails, whereas K = 20, K = 40, and K = 60 perform similarly across a wider parameter range.The authors suggest either optimization failure or insufficient embedding dimensionality as possible explanations for the K = 5 result.
- Gender mixtures: Female-male mixtures perform significantly better than same-gender mixtures across all evaluated situations.The paper attributes this difference to the intrinsic difficulty of the segmentation problem and notes a trade-off from more balanced training data.
- Embedding analysis: Embedding dimensions are sensitive to different aspects of the signal in both female-female and male-female mixtures.Figure 2 plots several embedding dimensions for each time-frequency bin alongside the spectrogram and ideal binary mask.
- Three-speaker separation: The proposed system separates three-speaker mixtures despite training only on two-speaker mixtures.The paper reports an open-speaker three-speaker example and an SDR improvement of 6.15 for fixed-speaker three-speaker mixtures.
- Future work: The authors identify joint clustering, alternative network architectures, and broader audio or image datasets as directions for future improvement.The reported results are preliminary, and proposed extensions include deep unfolding, convolutional architectures, recursive embedding networks, and other domains.