Source-linked AI summary
Music Source Separation in the Waveform Domain
Alexandre Défossez, Nicolas Usunier, Léon Bottou, Francis Bach
TL;DR
Music source separation seeks to recover individual instrument stems from mixtures, while leading systems traditionally operate on spectrogram masks. This paper compares an adapted waveform-domain Conv-Tasnet with Demucs, a U-Net waveform-to-waveform model, and reports that Demucs surpasses existing architectures in SDR with proper augmentation, despite leakage and model-size limitations.
Problem
Music source separation must recover drums, bass, other, and vocals from a coordinated mixture, while leading methods operate on spectrogram masks and reuse mixture phase.
Method
The paper adapts Conv-Tasnet to stereophonic music and introduces Demucs, a raw-waveform U-Net with a bidirectional LSTM and convolutional encoder-decoder.
Results
Demucs surpasses waveform- and spectrogram-domain state-of-the-art by at least 0.3 dB SDR, reaching 6.3 SDR without extra data and up to 6.8 with extra data.
Takeaways & Limitations
Waveform-domain separation performs especially well for bass and drums, whereas spectrogram-domain models perform best for vocals and other sources.
Takeaways & Limitations
Demucs has a model size exceeding 1014MB before reduction and still suffers from larger leakage, especially between vocals and other sources.
Abstract
from arXiv · showhide
Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song. Such components include voice, bass, drums and any other accompaniments.Contrarily to many audio synthesis tasks where the best performances are achieved by models that directly generate the waveform, the state-of-the-art in source separation for music is to compute masks on the magnitude spectrum. In this paper, we compare two waveform domain architectures. We first adapt Conv-Tasnet, initially developed for speech source separation,to the task of music source separation. While Conv-Tasnet beats many existing spectrogram-domain methods, it suffersfrom significant artifacts, as shown by human evaluations. We propose instead Demucs, a novel waveform-to-waveform model,with a U-Net structure and bidirectional LSTM.Experiments on the MusDB dataset show that, with proper data augmentation, Demucs beats allexisting state-of-the-art architectures, including Conv-Tasnet, with 6.3 SDR on average, (and up to 6.8 with 150 extra training songs, even surpassing the IRM oracle for the bass source).Using recent development in model quantization, Demucs can be compressed down to 120MBwithout any loss of accuracy.We also provide human evaluations, showing that Demucs benefit from a large advantagein terms of the naturalness of the audio. However, it suffers from some bleeding,especially between the vocals and other source.
1 Introduction
Music source separation recovers instrument stems from a coordinated mixture, but leading spectrogram methods reuse mixture phase. This paper adapts Conv-Tasnet and introduces Demucs, a waveform-to-waveform alternative that achieves strong SDR results while reducing artifacts, though leakage remains.
- Music source separation generates separate waveforms for drums, bass, other, and vocals from a mixed track using supervised ground-truth stems.
- Spectrogram-domain methods estimate magnitude masks for each source and reconstruct audio with the mixture phase through inverse STFT.
- Conv-Tasnet was adapted from monophonic speech at 8 kHz to stereophonic music at 44.1 kHz, but produced audible artifacts including noise, hollow attacks, and missing parts.
- Demucs directly maps raw mixtures to source waveforms with a U-Net, convolutional encoder-decoder, bidirectional LSTM, gated linear units, and a new initialization scheme.
- 6.3 points of SDR were achieved by Demucs without extra training data, compared with 6.0 for D3Net; extra data increased performance to 6.8.
- Demucs surpassed the IRM oracle for bass with 7.6 SDR against 7.1, while its reported artifacts and source leakage remained important trade-offs.
2 Related Work
Related work spans spectrogram-based supervised separation and newer waveform-domain models. Spectrogram systems historically led music benchmarks, while waveform approaches developed from speech separation and initially underperformed in music.
- Spectrogram methods predict source power spectrograms and reuse mixture phase, with Wiener filtering becoming common post-processing for leading systems.
- Deep supervised music separation progressed from fully connected networks and LSTMs to multiscale convolutional-recurrent architectures.
- Open Unmix matched top submissions trained only on MusDB, while MMDenseLSTM held the absolute SiSec SDR record using 807 unreleased songs.
- Wave-U-Net and other early waveform-domain music models performed substantially worse than most spectrogram-domain systems in the SiSec 2018 evaluation.
- Tasnet established waveform-domain masking for speech separation, and Conv-Tasnet later replaced recurrent separation with stacked dilated convolutions.
- The paper reports that Conv-Tasnet surpassed many spectrogram methods but reached 5.7 SDR, produced artifacts, and had less contamination than Demucs.
3 Adapting Conv-Tasnet for music source separation
The paper formulates Conv-Tasnet separation as waveform prediction and adapts its speech architecture to stereo MusDB music. Track-level evaluation uses chunking because training and evaluation normalization differ for short extracts versus entire songs.
- Each source is a C-channel waveform with T samples, and the mixture is defined as the sum of all source waveforms.
- The model g predicts one waveform per source from the mixture and is trained to minimize a reconstruction error over the dataset.
- The adapted system uses a simple L1 loss instead of Conv-Tasnet’s original SI-SNR loss.
- Conv-Tasnet encodes the waveform into a high-dimensional representation, masks it with a separation network, and decodes source waveforms.
- For stereo 44.1 kHz music, the adaptation increased encoder-decoder kernel size and stride and used 4 × N blocks with 256 channels.
- Entire tracks were split into 8-second chunks because global layer normalization behaves differently on short training extracts and songs containing quiet and loud sections.
4 The Demucs Architecture
Demucs is a waveform-to-waveform U-Net that encodes stereo mixtures, processes them with a bidirectional LSTM, and decodes source waveforms using skip connections. Its design targets expressive waveform synthesis while preserving input phase information and uses waveform reconstruction losses for separation.
- Architecture: Demucs takes a stereo mixture as input and outputs a stereo estimate for each source through a convolutional encoder, bidirectional LSTM, and convolutional decoder.The encoder and decoder are linked by skip U-Net connections, and batch normalization is omitted because early experiments found it detrimental.
- Encoder: Six stacked convolutional encoder blocks downsample the representation with stride 4 and use ReLU followed by gated linear units.Each block uses kernel size 8 for the strided convolution and a kernel-1 convolution producing twice as many channels before GLU gating.
- Decoder and U-network: The decoder reverses the encoder hierarchy with contextual convolutions and transposed convolutions, synthesizing all sources only at the final layer.Its transposed convolutions use stride 4, while skip connections link encoder and decoder blocks with matching indices.
- Motivation: synthesis vs masking: U-Net skip connections provide direct access to the original signal and can transfer the input phase to the separated outputs.GLU gating and large-stride transposed convolutions also make the model expressive enough to represent masks on a learned input representation.
- Resampling: 2x input upsampling followed by output downsampling improves performance when included in the end-to-end training loss.The operation uses a sinc resampling filter.
- Loss function: Waveform L1 or L2 reconstruction losses are valid for separation because the mixture supplies the original phase, which U-Net skip connections can recover.This differs from unconditional audio generation, where phase shifts can produce large waveform losses despite perceptual similarity.
- Weight rescaling at initialization: The proposed initialization rescales convolution weights so layerwise feature magnitudes remain comparable, using reference scale a = 0.1.Without initial rescaling, the final layer output was observed to have 20 times smaller magnitude than the first.
5 Experimental setup
The experiments evaluate waveform and spectrogram baselines on the fully supervised stereo MusDB benchmark using source-separation metrics and extensive augmentation. They also assess training with additional stems, model compression, and perceptual audio quality.
- Evaluation framework: MusDB contains 150 fully supervised stereo songs at 44.1 kHz, split into 84 training, 16 validation, and 50 test songs.Each song provides exact waveforms for drums, bass, other, and vocals; an additional stem set supplies raw stems from 150 tracks.
- Training procedure: Adding pitch/tempo augmentation gives Demucs a 0.4-point SDR gain, particularly for large models, while it can be detrimental to Conv-Tasnet.The experiments also compare models trained on MusDB alone with models using 150 additional training songs.
- Evaluation framework: Performance is reported primarily with SDR, the log ratio between projected estimated-source volume and residual contamination or artifacts.The evaluation uses the museval implementation and reports the median across tracks of the per-track median metric.
- Baselines: The comparison includes spectral baselines such as Open Unmix, MMDenseLSTM, D3Net, and Spleeter, waveform models including Wave-U-Net and Tasnet variants, and the IRM oracle.Several spectral baselines use Wiener filtering as post-processing, while the IRM uses ground-truth sources to compute an ideal mask.
- Training procedure: Training augmentation combines source shuffling, channel swapping, random scaling from 0.25 to 1.25, sign inversion, and optional pitch/tempo shifts.Pitch changes are sampled from -2, -1, 0, +1, or +2 semitones, while tempo varies uniformly from 0.88 to 1.12; it can harm Conv-Tasnet with extra data.
- Quantization: DiffQ quantization adds differentiable weight noise and a model-size penalty of 0.0003 to the reconstruction loss.The noise scale depends on learnable parameters controlling the number of bits used per group of 8 weight entries.
- Results reporting: Table 1 reports median SDR summaries across MusDB test tracks, with an All column averaging sources and Demucs values averaged over 3 runs.Bold entries identify values statistically state-of-the-art with or without extra training data.
- Human evaluation: Table 2 uses mean opinion scores from 38 listeners rating 20 randomly sampled 8-second excerpts, where 5 denotes perfect quality without artifacts.The ratings compare three models with ground-truth audio.
6 Experimental results
Experiments on MusDB compare waveform and spectrogram-domain separation methods, quantify Demucs’s quality and efficiency trade-offs, and evaluate artifacts through human ratings and ablations.
- Quality of the separation: 0.3 dB of SDR separates Demucs from D3Net when averaging over all sources, although D3Net remains stronger for other and vocals.Without extra data, D3Net also beats Demucs for drums.
- Human evaluations: 2.85±.08 versus 3.22±.09 MOS shows that Conv-Tasnet’s artifacts degrade user-rated audio quality relative to Demucs.Conv-Tasnet artifacts include static noise, hollow attacks, and missing notes, especially for drums and bass.
- Human evaluations: 3.42±.09 versus 3.30±.10 and 3.27±.11 indicates slightly less source contamination for Conv-Tasnet than Demucs and Open-Unmix.The MOS evaluates contamination by other sources, with higher ratings indicating less contamination.
- Ablation study for Demucs: 0.4 dB extra SDR comes from initial weight rescaling, while the LSTM and L1 loss also make substantial contributions.Extra encoder and decoder convolutions improve expressivity, especially with GLU activation; decoder kernel size 3 further improves performance.
- Ablation study for Demucs: Almost 0.5 dB extra SDR from pitch/tempo augmentation highlights the importance of strong augmentation when training data is limited.The same augmentation can be detrimental to Conv-Tasnet, while Demucs consistently benefits from it.
- Quantization of Demucs: 120MB is achievable through DiffQ quantization without SDR loss, compared with more than 1014MB for uncompressed Demucs.Reducing the initial channels to 32 decreases performance by 0.2 dB, while 48 channels avoid a large SDR loss.
Conclusion
With data augmentation, Demucs surpasses state-of-the-art separation architectures, while waveform and spectrogram models retain source-dependent strengths and Demucs still exhibits leakage.
- 0.3 dB of SDR separates Demucs from all state-of-the-art waveform- and spectrogram-domain architectures with proper augmentation.
- Waveform models perform best on bass and drums, whereas spectrogram models lead on vocals and other sources under objective and human evaluation.
- Pitch/tempo-shift augmentation, LSTM layers, and powerful 1x1-convolution and GLU encoding and decoding layers are important design choices.
- Demucs surpasses the IRM oracle for bass with extra data but retains greater leakage, especially for vocals and other sources.
A Results for all metrics
The evaluation defines SDR, SIR, and SAR through projected target, interference, and artifact components, with museval adding extra invariants to the reported metrics.
- SDR, SIR, and SAR are defined in decibels using target, interference, and artifact components derived from source projections.
- The source decomposition uses orthogonal projections of each estimated source onto the target source and the span of all sources.
- The reported museval metrics include extra invariants, while baseline results are only provided for some systems and otherwise taken from original papers.
- The metrics table organizes results by architecture, waveform usage, extra training data, and overall, drums, bass, other, and vocals scores.