Source-linked AI summary
Hybrid Spectrogram and Waveform Source Separation
Alexandre Défossez
TL;DR
Source separation must handle the differing strengths and artifacts of spectrogram and waveform representations. The paper develops an end-to-end hybrid Demucs with parallel domains and architectural improvements, achieving leading challenge and MusDB results while increasing model complexity.
Problem
Finite training data gives spectrogram and waveform representations different inductive biases and source-dependent artifacts, motivating hybrid separation.
Method
The paper extends Demucs with parallel temporal and spectral branches, compressed residual branches, local attention, biLSTMs, and stabilization techniques.
Results
Hybrid Demucs improved MusDB HQ SDR by around 1.4 dB and achieved first rank in the 2021 Sony Music Demixing Challenge.
Takeaways & Limitations
The hybrid architecture improved overall separation quality and reduced bleeding between sources in human evaluations.
Takeaways & Limitations
The approach has increased U-Net encoder-decoder complexity, and the competition submission used a difficult-to-reproduce heterogeneous four-model bag.
Abstract
from arXiv · showhide
Source separation models either work on the spectrogram or waveform domain. In this work, we show how to perform end-to-end hybrid source separation, letting the model decide which domain is best suited for each source, and even combining both. The proposed hybrid version of the Demucs architecture won the Music Demixing Challenge 2021 organized by Sony. This architecture also comes with additional improvements, such as compressed residual branches, local attention or singular value regularization. Overall, a 1.4 dB improvement of the Signal-To-Distortion (SDR) was observed across all sources as measured on the MusDB HQ dataset, an improvement confirmed by human subjective evaluation, with an overall quality rated at 2.83 out of 5 (2.36 for the non hybrid Demucs), and absence of contamination at 3.04 (against 2.37 for the non hybrid Demucs and 2.44 for the second ranking model submitted at the competition).
Introduction
Music source separation commonly targets four instruments using either spectrogram-based or waveform-based models. This work motivates a hybrid approach because finite training data allows domain-specific inductive biases and produces different source-dependent artifacts.
- Music source separation is commonly evaluated on drums, bass, vocals, and other accompaniments using the MusDB benchmark.
- Spectrogram models operate on amplitude or complex spectrogram representations, whereas waveform models directly map raw audio to separated sources.
- With constrained training data, spectrogram and waveform models retain different inductive biases despite theoretically equivalent input-output representations.
- Domain-specific artifacts differ by source: spectrogram phase inconsistency can hollow drum and bass attacks, while Demucs vocals can contain crunchy static noise.
- Hybrid Demucs adds parallel temporal and spectral branches to Demucs and combines compressed residual branches, LSTM, and local attention improvements.
- Hybrid Demucs ranked first in the MDX competition when trained only on MusDB, reaching 7.32 dB SDR.
Related work
Prior source-separation systems largely specialized in either spectrogram or waveform processing. The paper identifies true end-to-end hybrid separation as an unstudied alternative to post-training model blending.
- Spectrogram systems include Open-Unmix, D3Net, and related architectures using recurrent, dense-skip, or complex-modulation processing.
- Earlier waveform methods often lagged spectrogram competitors, while Demucs improved waveform separation using strided convolutions, GLUs, and a biLSTM.
- No other work had studied true end-to-end hybrid source separation, although competition teams used post-training blending across domains.
Architecture
Hybrid Demucs extends a U-Net with parallel temporal and spectral branches that merge into shared processing and recombine at the output. Additional residual, attention, normalization, and alignment mechanisms support the hybrid design and training stability.
- Architecture: Hybrid Demucs extends the original U-Net encoder-decoder with temporal, spectral, and shared layers.
- Architecture: The temporal branch processes waveform input, while the spectral branch processes an STFT and reduces frequency dimensions through frequency-wise convolutions.
- Architecture: The spectral output is converted by ISTFT and summed with the temporal output, allowing representation choice and information sharing within a source.
- Padding for easy alignment: Padding with P = (K − S)/2 makes convolution outputs have length L/S, so matching overall strides aligns temporal and spectral representations.
- Compressed residual branches: Compressed residual branches use dilated convolutions, normalization, GELU activations, LayerScale, and GLUs, with two branches per encoder layer.
- Local attention: The fifth and sixth encoder layers add local attention and a two-layer biLSTM with a maximum span of 200 steps for long-range context.
- Local attention: Local attention replaces positional embedding with a learned distance penalty whose βi,k values control the receptive field.
- Stabilizing training: Training became unstable when deeper models and additional data caused large eigenvalues in innermost encoder and decoder layers.
Experimental Results
The experiments evaluate challenge-track settings, realistic remix augmentation, and alignment constraints. The supplied passages describe the training data and remix construction, but do not report the requested result tables themselves.
- The MDX challenge used Track A with MusDB HQ only and Track B with unrestricted training data.
- MusDB HQ contains 150 tracks: 86 for training, 14 for validation, and 50 for testing.
- Additional gains came from fine-tuning on realistic remixes and using 30-second training samples instead of 10-second samples.
- Realistic remixing constrained stem combinations to less than 3 semitones of pitch shift and 15% of tempo change.
Metrics
The MDX challenge introduced a simpler, faster Signal-To-Distortion measure, distinguished here as nSDR from the earlier SDR definition. Evaluation aggregates nSDR across songs on MDX, while MusDB uses a median-based segment aggregation.
- nSDR is the new Signal-To-Distortion measure introduced by the MDX challenge for simpler and faster evaluation.
- The paper uses nSDR for the new definition and SDR for the older Vincent et al. definition.
- On the MDX test set, nSDR is averaged across all songs.
- On MusDB, evaluation uses the traditional median across songs of the median over each song’s 1-second segments.
Models
Competition submissions used ensembles of four models, with model composition differing between tracks. Baseline comparisons emphasize KUIELAB-MDX-Net, which combines spectrogram separation with original Demucs predictions for drums and bass.
- Competition submissions were bags of four models, with Track A mixing hybrid and non-hybrid Demucs models.Hybrid models performed worse on bass for Track A.
- Track B used only hybrid models because extra training data improved performance for all sources.
- Track submissions mixed Hybrid Demucs models using CaC or real masking because reevaluating every model was too costly.
- KUIELAB-MDX-Net, the second-place MDX model, combines a pure spectrogram model with original Demucs predictions for drums and bass.
Results on MDX
On the MDX test set, Hybrid Demucs is particularly strong for drums and bass, while its gains and remaining weaknesses differ across the Other and Vocals sources. Results are reported separately for MusDB-only training and training with extra data.
- Hybrid Demucs especially improves nSDR for the Other and Vocals sources on MDX Track A.Despite these gains, it remains below KUIELAB-MDX-Net on both sources.
- Track B results show Hybrid Demucs is very strong on Drums and Bass but lags behind on Other and Vocals.
- Table 1 reports Hybrid Demucs results on the MDX test set after training only on MusDB for Track A using nSDR.
- Table 2 reports Hybrid Demucs results on the MDX test set after training with extra data for Track B using nSDR.
Results on MusDB
On MusDB, Hybrid Demucs achieves the best performance for Drums and Bass and substantially improves over waveform-only Demucs for Other and Vocals, though it remains below spectrogram-based competitors on those sources.
- Hybrid Demucs achieves the best MusDB performance for the Drums and Bass sources.
- Hybrid Demucs improves substantially over waveform-only Demucs for Other and Vocals on MusDB.
- Hybrid Demucs does not surpass KUIELAB-MDX-Net for Other and Vocals, where KUIELAB-MDX-Net is purely spectrogram based.
- ResUNetDecouple+ achieves the best Vocals performance and uses complex modulation of the input spectrogram.
Human evaluations
Human evaluations measured generated samples for quality, artifact absence, and bleeding between sources using experienced music listeners and 1–5 ratings.
- Human subjects rated 25 randomly drawn 12-second samples from MusDB’s 50 test tracks on artifact absence and source bleeding.Subjects included amateur and professional musicians, sound engineers, and others with strong music experience.
- Mean Opinion Scores used a 1–5 scale, where 5 represented the best grade for quality, artifact absence, or bleeding absence.The reported standard deviation was around 0.15.
- The evaluation protocol separately assessed perceived quality and absence of artifacts from absence of bleeding between sources.Tables 4 and 5 report these as distinct Mean Opinion Score outcomes.
Reproducibility and Ablation
The authors describe a reproducible multi-model setup and ablate training and architectural choices, finding benefits from hybrid combinations, fine-tuning, averaging, EMA, and selected residual-branch components.
- Reproducibility: The competition submission used four heterogeneous models, complicating straightforward reproduction of its performance.Different models were sufficiently trained at different times and were used for different sources.
- Reproducibility: Reproduction trained two time-domain and two Complex-As-Channels hybrid Demucs models for 600 epochs, then averaged their predictions with equal source weights.All models used the SVD penalty, exponential moving average, and fine-tuning on realistic track remixes.
- Reproducibility: 7.64 dB overall SDR was achieved, within 0.04 dB of the MDX Track A submission.Combining time-only and hybrid model bags surpassed either bag individually on sources including drums or vocals.
- Ablation: 0.3 dB overall SDR was gained from realistic-remix fine-tuning, while bagging added further gains and EMA improved SDR by 0.2 dB.These effects were measured in the ablation study against the improved time-only Demucs base.
- Ablation: 0.1 dB overall SDR was gained from the SVD penalty, mainly through vocals (+0.7 dB), while drums deteriorated (-0.4 dB).The ablation therefore found source-specific effects rather than a uniform improvement.
- Ablation: Removing LSTM or local attention from residual branches strongly decreased SDR, whereas replacing ReLU with GELU produced no real gain.The ablation identified local attention as especially important despite its lack of positional embedding.
Conclusion
The paper reports substantial source-separation improvements from architectural changes centered on a hybrid temporal/spectrogram U-Net, while identifying increased encoder–decoder complexity as a limitation.
- The hybrid temporal/spectrogram U-Net uses parallel branches that merge into a common core, alongside compressed residual branches, local attention, and chunked biLSTM.These changes extend the Demucs architecture and were reported as central to the improved system.
- Around 1.4 dB SDR gain was achieved on the MusDB HQ benchmark, with the system attaining first rank in Sony’s 2021 Music Demixing Challenge.Human evaluations also showed strong improvements in overall quality and absence of bleeding between sources.
- The approach’s increased U-Net encoder/decoder complexity requires careful alignment of temporal and spectral signals through well-shaped convolutions.This is identified as a limitation despite the reported quality gains.