Source-linked AI summary
Wavesplit: End-to-End Speech Separation by Speaker Clustering
Neil Zeghidour, David Grangier
TL;DR
Speech separation from a single mixture is difficult because same-class sources create permutation ambiguity and inconsistent channel assignments. Wavesplit jointly infers clustered, sequence-wide source representations and reconstructs separated signals from the raw waveform. It reports strong results across clean, noisy, and reverberated speech benchmarks and also separates maternal and fetal heart rates from abdominal ECG.
Problem
Separating multiple same-class sources from one mixture is ill-posed because predicted channels can swap source assignments over time.
Method
Wavesplit jointly trains source-representation inference and signal reconstruction, clustering local speaker vectors into sequence-wide representations that condition separation.
Results
Wavesplit reports state-of-the-art performance across clean, noisy, and reverberated speech benchmarks, including 21.0 ∆SI-SDR on WSJ0-2mix versus 20.1 for gated dual path RNN.
Takeaways & Limitations
The approach extends beyond speech by separating maternal and fetal heart-rate signals from a single abdominal electrode.
Abstract
from arXiv · showhide
We introduce Wavesplit, an end-to-end source separation system. From a single mixture, the model infers a representation for each source and then estimates each source signal given the inferred representations. The model is trained to jointly perform both tasks from the raw waveform. Wavesplit infers a set of source representations via clustering, which addresses the fundamental permutation problem of separation. For speech separation, our sequence-wide speaker representations provide a more robust separation of long, challenging recordings compared to prior work. Wavesplit redefines the state-of-the-art on clean mixtures of 2 or 3 speakers (WSJ0-2/3mix), as well as in noisy and reverberated settings (WHAM/WHAMR). We also set a new benchmark on the recent LibriMix dataset. Finally, we show that Wavesplit is also applicable to other domains, by separating fetal and maternal heart rates from a single abdominal electrocardiogram.
I. INTRODUCTION
Wavesplit targets the ill-posed problem of separating same-class sources from one mixture, with speech separation as its primary application. It uses jointly learned, clustered source representations to address permutation ambiguity and evaluates generality across speech and heart-rate signals.
- Motivation: Separating multiple same-class sources from one mixture is difficult because source assignments can permute across time.This permutation problem can produce well-separated channels with inconsistent source identities.
- Generality: Wavesplit is also applied to separating maternal and fetal heart-rate signals from a single abdominal electrode.The application is motivated by noninvasive fetal-health monitoring using abdominal ECGs.
- Approach: Wavesplit jointly learns speaker identification and speech separation from raw waveforms, using clustered representations to form long-term source identities.The method leverages speaker identities during training while inferring representations directly from the test mixture.
- Approach: Clustering produces order-agnostic source representations, while aggregation over the whole mixture limits channel-swap errors.This design differs from PIT by maintaining a persistent representation for each source.
- Evaluation: The paper reports state-of-the-art results across clean and noisy speech-separation benchmarks, including WSJ0-2/3mix, Libri2/3mix, WHAM, and WHAMR.The contribution list also identifies empirical analysis of the method's advantages and drawbacks.
III. WAVESPLIT
Wavesplit separates sources by jointly inferring source representations and reconstructing signals, using sequence-level representations to resolve arbitrary channel ordering. Its design avoids test-speaker enrollment and contrasts with PIT and mask-based separation.
- Wavesplit: Wavesplit jointly trains a speaker stack and separation stack, conditioning separated-signal reconstruction on inferred speaker representations.The speaker stack is trained with speaker labels while the separation stack reconstructs the speech.
- Wavesplit: Unlike enrollment-based separation, Wavesplit infers all test-speaker representations directly from the mixture without enrollment sequences.Joint training optimizes the representations for both speaker identification and separated-speech reconstruction.
- Wavesplit: Unlike PIT, Wavesplit conditions decoding on a speaker representation valid for the whole sequence and resolves permutation ambiguity at the representation level during training.The separation stack receives speaker vectors ordered consistently with the labels during training.
- Problem Setting & Notations: Given a mixture of N source waveforms, Wavesplit reconstructs each source while evaluating predictions up to an arbitrary permutation.The reconstruction quality uses a single-channel metric and searches over source orderings because channel order is arbitrary.
- Evaluation: Scale-invariant SDR treats prediction scale as irrelevant by searching over gains, while related metrics use richer signal transformations.SDR is described as a reconstruction-quality measure based on normalized log squared error.
B. Model Architecture
The architecture uses a speaker stack to produce local speaker vectors and sequence-level centroids, then a separation stack conditioned on those centroids to generate isolated channels. Clustering aggregates unordered local vectors into persistent source representations.
- Model Architecture: Wavesplit uses two residual convolutional subnetworks: one extracts speaker representations and the other produces isolated recordings conditioned on them.Both stacks transform the input mixture, with the separation stack using the speaker representation as conditioning.
- Speaker Stack: The speaker stack maps a mixture to N same-length sequences of d-dimensional latent vectors, with N as the maximum number of simultaneous speakers.Local vectors need not maintain a consistent speaker ordering across time.
- Speaker Stack: K-means clusters local speaker vectors across the sequence and returns N speaker centroids as whole-sequence summaries.During training, centroids are formed by grouping vectors according to speaker identity rather than applying clustering.
- Separation Stack: The separation stack maps the mixture and speaker centroids into an N-channel separated signal.Its residual blocks are conditioned by the concatenated centroids using FiLM.
- Speaker Stack: Speaker-stack residual blocks use dilated convolutions, nonlinearities, and layer normalization, followed by Euclidean normalization of speaker vectors.The residual update is xl+1 = xl + lnorm(nl(dconv(xl))).
C. Model Training Objective
Wavesplit jointly learns speaker representations and signal reconstruction, using clustering to produce order-agnostic source estimates without permutation search. Training combines speaker-discrimination and reconstruction objectives, with regularization for generalization to unseen speakers.
- Joint objectives: Training optimizes speaker-vector clustering and reconstruction of separated signals as two complementary objectives.Speaker vectors should form well-separated identity clusters, while aggregated vectors support signal reconstruction.
- Speaker supervision: Speaker supervision is used during training, but test-time separation requires only the mixture recording, not test-speaker annotations.The model assumes identities from a finite training-speaker set and exploits them to build internal source representations.
- Speaker objective: The speaker loss encourages correct identities at each time step while allowing arbitrary output ordering and reordering vectors consistently with training labels.This enables averaging vectors from the same speaker during training and avoids the training-time k-means optimization used in prior work.
- Speaker objective: Distance, local-classifier, and global-classifier objectives are explored for learning speaker representations.The distance objective pulls vectors toward corresponding speaker embeddings and separates different speakers by a margin of 1; the local classifier discriminates among speakers present in a sequence.
- Reconstruction objective: Clustering-based centroids yield order-agnostic predictions and remove the need to search over output permutations, unlike PIT.The reconstruction loss uses clipped negative SDR and is averaged across separation-stack layers.
- Regularization: Gaussian centroid noise, speaker dropout, and speaker mixup regularize the model for generalization to new speakers.Speaker dropout removes at most one centroid regardless of the number of speakers in a sequence.
D. Training Algorithm
Wavesplit is trained with Adam on shuffled mini-batches of fixed-size windows sampled uniformly from each sequence. It performs well across window sizes beginning at 750ms, whereas most cited PIT systems require approximately 4-second segments.
- Optimization: Adam optimizes the weighted sum of the speaker and reconstruction losses using mini-batches of fixed-size windows.Training windows are sampled after shuffling the training set each epoch.
- Window sampling: 750ms is the shortest reported window size at which Wavesplit performs well across a wide range of windows.The passage contrasts this with most PIT approaches, which require longer segments of approximately 4s.
- Window sampling: Uniformly sampling a window start each time a sequence is visited gives sequences equal importance regardless of their length.This sampling strategy is applied during each training epoch.
E. Data Augmentation with Dynamic Mixing
The dynamic-mixing augmentation creates an unlimited stream of training mixtures by sampling random recording windows and combining them with random gains. This augments the finite mixtures commonly used in separation benchmarks.
- Dynamic mixing: Dynamic mixing samples random windows from training recordings and sums them after applying random gains.The procedure creates new examples indefinitely rather than reusing a finite set of pre-generated mixtures.
- Motivation: The augmentation is presented as orthogonal to the Wavesplit model and expands benchmark training data beyond fixed mixture lists.Standard benchmarks generate finite mixtures by summing specified clean signals with specified gains.
IV. EXPERIMENTS & RESULTS
Experiments evaluate Wavesplit on clean, noisy, reverberated, and larger-scale speech-separation benchmarks, alongside a fetal/maternal heart-rate task. Results are reported with SDR-based improvements, with additional analyses of errors and long sequences.
- Datasets and tasks: The evaluation covers WSJ0-2/3mix, WHAM!, WHAMR!, LibriMix, and fetal/maternal heart-rate separation.Speech datasets include clean and noisy 2- or 3-speaker settings, while WHAMR! additionally includes reverberation.
- Error analysis: Error analysis is performed on WSJ0-2mix to examine sequences with a strong negative impact on overall performance.The analysis is described as covering a small fraction of sequences.
- Long sequences: Long-sequence performance is evaluated using ΔSDR in a dedicated benchmark table.The supplied caption identifies the metric and long-sequence setting but does not provide the table values.
- Noisy and reverberated speech: WHAM! and WHAMR! results are summarized using SI-SDR and SDR improvements in decibels.WHAM! represents noisy mixtures, while WHAMR! represents mixtures with noise and reverberation.
- Metrics: SDR and SI-SDR improvements are the primary evaluation measures, computed relative to using the input mixture as the prediction.SDR is measured with the standard MIR-eval library.
A. Hyperparameter Selection
The experiments selected architectural and training hyperparameters for Wavesplit, including stack dimensions, dilation, learning rates, loss weights, and regularization settings.
- Architecture: 512-dimensional latent representations are used in both the speaker and separation stacks.Dilated convolutions use kernel size 3 without striding, preserving the input temporal resolution and avoiding output upsampling.
- Architecture: The speaker stack has 14 layers, while the separation stack has 40 layers.The speaker-stack dilation grows exponentially with depth.
- Training: The learning rate was selected as 1e−3, and the speaker loss weight as 2.These values were validated from the reported candidate ranges.
- Regularization: Regularization uses distance weight 0.3, Gaussian noise standard deviation 0.2, speaker dropout 0.4, and speaker mixup 0.5.Each setting was selected from a validated candidate range.
B. Clean Settings
Wavesplit performs strongly on clean speech mixtures, while long-sequence tests expose channel-assignment errors associated with confusing speaker identities.
- Clean benchmarks: 21.0 ∆SI-SDR on WSJ0-2mix exceeds the gated dual path RNN baseline at 20.1.Dynamic augmentation further improves Wavesplit to 22.2 ∆SI-SDR.
- Clean benchmarks: On WSJ0-2mix, dynamic mixing increases Wavesplit’s advantage over alternatives.The passage reports this pattern for both two- and three-speaker settings.
- Error analysis: Test sequences contain more poor-∆SDR examples than validation sequences, at 5.6% versus 0.9%.Low-∆SDR test examples occur when both speakers are close to the same learned training-speaker identity.
- Error analysis: The oracle permutation results indicate that most observed errors are channel-assignment errors.The oracle reports the best permutation of predicted samples across channels.
- Noisy and reverberated settings: WHAM! and WHAMR! require clean-source prediction under noise, with WHAMR! additionally requiring dereverberation.Dynamic remixing produces an even larger improvement over previous work on both datasets.
D. Large scale experiments on LibriMix
LibriMix evaluates Wavesplit across clean and noisy two- and three-speaker mixtures, while additional experiments test its robustness and cross-domain applicability.
- LibriMix: Wavesplit significantly outperforms Conv-TasNet in all four LibriMix conditions and consistently exceeds ideal-mask oracles.The conditions cover two or three speakers with clean or noisy mixtures.
- LibriMix: LibriMix training uses 921 speakers in train-360, compared with 101 speakers in WSJ0-2/3mix training.The authors identify the larger speaker population as improving speaker-stack robustness.
- Ablation study: Global classifier loss outperforms distance loss, while local classifier loss trains more slowly and generalizes worse.Multiplicative FiLM conditioning improves SDRs and enables a higher learning rate with faster training.
- Cross-domain evaluation: On FECGSYNDB, Wavesplit separates maternal and fetal ECG from a single noisy electrode recording.The model is trained and evaluated alongside Dual-Path RNN and Conv-TasNet using simulated pregnancies and electrode recordings.
- Conclusions: The conclusion reports state-of-the-art results on clean and noisy WSJ0-mix and LibriMix benchmarks, plus fetal/maternal heart-rate separation.The paper presents these results as evidence for application beyond speech separation.
Appendix
The appendix documents dataset statistics and visualizes Wavesplit’s maternal/fetal heart-rate separation example.
- Dataset statistics: The appendix reports split statistics for the WSJ0-mix and LibriMix datasets.The LibriMix experiments use the train-360 training-set version.
- Heart-rate separation: Figure 2 shows the input mixture alongside separated maternal and fetal heart-rate signals.The example comes from a simulated abdominal electrode on the FECGSYNDB test set.