Source-linked AI summary
Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed
Alexandre Défossez, Nicolas Usunier, Léon Bottou, Francis Bach
TL;DR
Music source separation seeks to recover individual stems, but spectrogram-mask methods reuse mixture phase and waveform methods have lagged. Demucs combines a waveform architecture with remixing of unlabeled music; on MusDB, it narrows the performance gap between waveform and spectrogram methods, with its augmented model reported almost on par with spectrogram state of the art.
Problem
Spectrogram-mask methods reuse mixture phase and may produce artifacts, while waveform methods have lagged behind spectrogram approaches.
Source: §1 · p. 2Method
Demucs combines a convolutional-recurrent waveform model with a scheme that detects silent-source excerpts in unlabeled songs and remixes them with isolated supervised sources.
Source: §1 · p. 2, §4.2 · p. 6Results
2.2 to 0.5 points of median SDR: Demucs reduced the gap between spectrogram and waveform methods when trained on standard MusDB, and its model with remixed unlabeled data and 100 extra examples was almost on par with spectrogram state of the art.
Source: Conclusion · p. 8Takeaways & Limitations
The authors present their approach as the first semi-supervised source-separation method that does not rely on adversarial losses.
Source: Conclusion · p. 8Only a few minutes of audio with source other silent could be extracted because such training examples were rare and led to unreliable prediction.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
We study the problem of source separation for music using deep learning with four known sources: drums, bass, vocals and other accompaniments. State-of-the-art approaches predict soft masks over mixture spectrograms while methods working on the waveform are lagging behind as measured on the standard MusDB benchmark. Our contribution is two fold. (i) We introduce a simple convolutional and recurrent model that outperforms the state-of-the-art model on waveforms, that is, Wave-U-Net, by 1.6 points of SDR (signal to distortion ratio). (ii) We propose a new scheme to leverage unlabeled music. We train a first model to extract parts with at least one source silent in unlabeled tracks, for instance without bass. We remix this extract with a bass line taken from the supervised dataset to form a new weakly supervised training example. Combining our architecture and scheme, we show that waveform methods can play in the same ballpark as spectrogram ones.
1 Introduction
Music source separation recovers four broad stem categories from a coordinated mixture. Spectrogram-mask methods perform well but have limitations that motivate waveform modeling, and the paper introduces a waveform architecture and unlabeled-data augmentation scheme.
1 Introduction
- Music source separation aims to recover drums, bass, other, and vocals from a mastered mixture of coordinated instruments.These four broad categories regroup the individual stems used in the SiSec Mus evaluation campaign.Source: §1 · p. 1
- Spectrogram-mask methods reuse mixture phase and can create artifacts because the inverse transform projects masks that need not be real spectrograms.The paper also notes that reusing mixture phase can transfer a singer’s vibrato to a guitar at the same pitch.Source: §1 · p. 2
- Waveform modeling makes the training loss end-to-end without an extra synthesis step, while waveform separation has lagged partly amid limited training data.The paper says the MusDB dataset is only a few GB, compared with 30GB for NSynth, and links dataset scale at least partially to weaker waveform-separation performance.Source: §1 · p. 2
- The paper proposes a simple waveform architecture and a semi-supervised augmentation scheme that leverages 2,000 unlabeled songs.The scheme extracts parts with at least one silent source and remixes them with a source from supervised data.Source: §1 · p. 2
2 Related Work
Prior music-separation work includes supervised spectrogram models and waveform models, with the latter performing worse in evaluation. The paper distinguishes its unlabeled-song remixing approach from earlier semi-supervised and blind-separation setups.
2 Related Work
- Supervised spectrogram methods predict each source’s power spectrogram and reuse mixture phase; deep models include fully connected, LSTM, and multiscale convolutional or recurrent approaches.Earlier approaches also used NMF, ICA, HMM-based prediction, and segmentation techniques.Source: §2 · p. 2
- Waveform separation developed with deep learning, including Wave-U-Net, but waveform methods performed significantly worse than spectrogram methods in the SiSec Mus evaluation.The paper reports that its architecture alone outperforms Wave-U-Net by a large margin.Source: §2 · p. 3
- Earlier semi-supervised and blind approaches used adversarial losses and assumed missing isolated sources; the paper instead targets unlabeled songs with remixing and direct training-loss supervision.The authors state that, to their knowledge, prior work had not used unlabeled songs to improve supervised source separation.Source: §2 · p. 3
3 Model Architecture
Demucs combines a U-Net waveform structure with convolutional encoding and decoding around an LSTM. Its large-stride direct synthesis, skip connections, and weight rescaling are design choices intended to support efficient, stable training.
3 Model Architecture
- Demucs blends SING’s large strides, many channels, and LSTM-convolution combination with Wave-U-Net’s U-Net structure.The model uses a convolutional encoder and decoder linked by skip connections, with an LSTM between them.Source: §3 · p. 3
- The authors omit batch normalization because early experiments found it detrimental to model performance.Source: §3 · p. 3
- The encoder uses six stride-4 layers, GLU activations, and exponentially increasing channels, followed by a two-layer bidirectional LSTM.Channels grow from 48 to 1,536; the LSTM output is reduced back to 1,536 channels by a 1x1 convolution.Source: §3 · p. 4
- The decoder concatenates its intermediate representation with the matching encoder output and uses transposed convolutions to synthesize the output.The final layer outputs four sources, while skip connections inject higher-frequency information.Source: §3 · p. 4
- Weight rescaling is equivalent to using specific learning rates per layer; the authors observed faster loss decay and convergence to a better optimum with reference scale 0.1.Rescaling by 1/α instead of 1/√α made the training loss diverge.Source: §3 · p. 4
- Unlike Wave-U-Net’s iterative upsampling and filtering, Demucs uses direct synthesis, allowing larger decoder strides and faster computation.The authors believe the larger number of channels enabled by this design is one reason for better performance.Source: §3 · p. 4
4 Unlabeled Data Remixing
The remixing scheme finds unlabeled excerpts with a silent source, adds a matching labeled stem, and uses the resulting example for strong supervision of that source and weak supervision of the remaining sources. It relies on a detector and extraction thresholds, and yields little usable data for excerpts where “other” is silent.
4 Unlabeled Data Remixing
- Detected excerpts with at least five seconds of source silence are mixed with a matching labeled stem, giving strong supervision for that source and weak supervision for the other sources’ sum.For a silent bass excerpt, the pipeline adds one bass sample from the training set.Source: §4 · pp. 4–5, Figure 2 · p. 5
- Unsupervised reconstruction regularization proved too weak, while passages with a single source present were too rare; source-silent passages were more frequent, including 13% for bass and 32% for vocals.In MusDB, “other” was alone 2.6% of the time, while the other sources were alone less than 0.5% of the time.Source: §4 · p. 5
4.1 Silent source detector
- Source silence is identified by relative volume; a source 20 dB below the mixture is described as almost silent, and one below 30 dB as perceptually zero.The paper sets the detector’s silence threshold at −13 dB to increase positive samples, then selects a high predicted-silence probability to retain truly silent excerpts.Source: §4.1 · p. 5
- A classifier estimates each source’s silence probability from second-order wavelet-scattering features, producing estimates every 0.64 seconds with a 64 ms stride.The model uses convolutional layers, max pooling, batch normalization, and a final LSTM; downsampled mono input further helps prevent overfitting.Source: §4.1 · p. 5
4.2 Extraction procedure
- Extraction retains five-second excerpts whose predicted silence probability exceeds a source-specific threshold, calibrated so at least 95% of selected stem-set samples have relative volume at or below −20 dB.This filtering reduces extracted data by roughly 50% while lowering the 95th-percentile relative volume from −20 to −30 dB.Source: §4.2 · p. 6
- Only a few minutes of audio with “other” silent were extracted, because that source is frequently present and the available examples did not support reliable absence prediction.Source: §4.2 · p. 6
5 Experimental results
On MusDB, Demucs benefits from unlabeled remixed data, while MMDenseNet and MMDenseLSTM retain the best overall performance. The evaluation uses supervised and unlabeled music data, and the reported extraction pipeline yields silent-source excerpts for drums, bass, and vocals, but not “other.”
5.1 Evaluation framework
- MusDB provides 150 fully supervised stereo songs at 44.1 kHz, split into 100 training and 50 test songs; the study also uses 2,000 unlabeled tracks and 100 additional labeled stem tracks.The unlabeled set comprises roughly 4.5 days of audio; the additional labeled stems come from the same distribution and do not overlap with it.Source: §5.1 · p. 7
- Extraction from the 2,000 unlabeled songs yielded about 1.5 days of excerpts with drums, bass, or vocals silent, but no significant dataset with “other” silent.The authors attribute the latter constraint to the source’s frequent presence and the difficulty of reliably predicting its absence with the available data.Source: §5.1 · p. 7
- Performance is reported as SDR, using the median across tracks of each track’s median metric; baselines include MMDenseNet, MMDenseLSTM, and Wave-U-Net.The spectrogram baselines include versions trained with 804 extra labeled songs; Wave-U-Net is the waveform baseline.Source: §5.1 · p. 7
5.2 Training procedure
- Training uses a 400-epoch Demucs run with L1 source loss and data augmentation; an extra remix-training step is sampled with probability 0.25 after each main batch.That step combines a silent-source excerpt with a single-source training example; it uses a separate Adam instance with a learning rate ten times smaller, and weak supervision acts as a regularizer.Source: §5.2 · p. 8
5.3 Evaluation results
- Nearly 0.3 median SDR comes from unlabeled remixing; 100 labeled tracks add 0.6, and unlabeled data adds another 0.2 with extra tracks.MMDenseNet and MMDenseLSTM still perform best overall, while Demucs with unlabeled data achieves state-of-the-art bass separation using 100 extra labeled songs versus 804 for those baselines.Source: §5.3 · p. 8
Conclusion
Demucs narrows the waveform-versus-spectrogram separation gap and uses remixed unlabeled music to bring waveform performance close to spectrogram methods.
Conclusion
- The waveform-versus-spectrogram median SDR gap falls from 2.2 to 0.5 points with Demucs trained on standard MusDB.The conclusion attributes this reduction to the Demucs architecture.Source: Conclusion · p. 8
- The remixing scheme uses a detector to find unlabeled excerpts with at least one silent source, then combines them with an isolated source from supervised training data.The approach uses 2,000 unlabeled MP3s and does not rely on adversarial losses.Source: Conclusion · p. 8
- With remixed data and 100 extra training examples, Demucs performs almost on par with state-of-the-art spectrogram methods and better for bass.The authors conclude waveform methods are a legitimate contender for supervised source separation.Source: Conclusion · p. 8
A Architecture of the silent sources detector
The silent-sources detector applies a second-order scattering transform, reorganizes its coefficients for 2D convolutions, and uses recurrent processing to produce four outputs.
A Architecture of the silent sources detector
- The input is a second-order scattering transform computed with Kymatio using 8 wavelets per octave.Order-one coefficients are indexed by frequency f1; order-two coefficients by f1 and f2.Source: A Architecture of the silent sources… · p. 11
- Scattering coefficients are arranged as a (C, F, T) tensor, with order-one coefficients in the first channel and order-two coefficients organized by f2 in the following channels.This reorganization enables 2D convolutions over the scattering output.Source: A Architecture of the silent sources… · p. 11
- After convolutional blocks and frequency-axis elimination, a two-layer bidirectional LSTM feeds 1D convolutions that produce four outputs.The BiLSTM has hidden size 1024 and dropout 0.18.Source: A Architecture of the silent sources… · p. 11
B Results for all metrics with boxplots
The appendix presents box plots for SDR, SIR, and SAR across targets, using the SiSec Mus evaluation notebook and marking extra and remixed training data.
B Results for all metrics with boxplots
- SDR compares target-source energy with the combined interference and artifact errors, while SIR compares target-source energy with interference error.The metrics are expressed in decibels.Source: B Results for all metrics with boxplots · p. 11
- SAR compares target-source energy with artifact error; the metrics are shown for each target in box plots.The displayed plots were generated using the SiSec Mus organizers’ evaluation notebook.Source: B Results for all metrics with boxplots · p. 11
- An “Extra” suffix indicates extra training data, while “Remixed” indicates use of the unlabeled-data remixing scheme.The plots include extra invariants from the museval package.Source: B Results for all metrics with boxplots · p. 11