Source-linked AI summary
Hybrid Transformers for Music Source Separation
Simon Rouard, Francisco Massa, Alexandre Défossez
TL;DR
Music source separation must balance local acoustic features against potentially useful long-range context, but the benchmark dataset is small for training Transformers. The paper introduces HT Demucs, a temporal/spectral Hybrid Demucs variant with cross-domain attention, and finds that extra data, sparse attention, and fine-tuning produce stronger results, reaching 9.20 dB SDR.
Problem
The paper asks whether long-range contextual information improves music source separation beyond local acoustic features, given MUSDB18’s small training corpus.
Method
HT Demucs replaces Hybrid Demucs’s innermost layers with a cross-domain Transformer Encoder using self-attention within temporal or spectral domains and cross-attention between them.
Results
9.20 dB SDR was achieved on the MUSDB test set using extra training data, sparse attention, and per-source fine-tuning.
Takeaways & Limitations
HT Demucs outperforms Hybrid Demucs by 0.45 dB and gains a further 0.4 dB from scaling input length with sparse attention.
Abstract
from arXiv · showhide
A natural question arising in Music Source Separation (MSS) is whether long range contextual information is useful, or whether local acoustic features are sufficient. In other fields, attention based Transformers have shown their ability to integrate information over long sequences. In this work, we introduce Hybrid Transformer Demucs (HT Demucs), an hybrid temporal/spectral bi-U-Net based on Hybrid Demucs, where the innermost layers are replaced by a cross-domain Transformer Encoder, using self-attention within one domain, and cross-attention across domains. While it performs poorly when trained only on MUSDB, we show that it outperforms Hybrid Demucs (trained on the same data) by 0.45 dB of SDR when using 800 extra training songs. Using sparse attention kernels to extend its receptive field, and per source fine-tuning, we achieve state-of-the-art results on MUSDB with extra training data, with 9.20 dB of SDR.
1. INTRODUCTION
The paper studies whether long-range context and Transformer architectures improve music source separation, given MUSDB18’s small supervised training set. It introduces HT Demucs, adds 800 songs, and evaluates context, sparse attention, and fine-tuning strategies.
- MUSDB18 contains only 87 training songs, making its corpus relatively small compared with Transformer applications in vision and language.
- HT Demucs replaces Hybrid Demucs’s innermost layers with Transformer layers operating in temporal and spectral representations.Self-attention operates within each domain, while cross-attention connects the domains.
- 0.35 dB improvement over Hybrid Demucs was obtained when both models were retrained on the same expanded data.
- 9.20 dB SDR was achieved on MUSDB’s test set using sparse attention, longer context, and per-source fine-tuning.
- The training code, pretrained models, and audio samples were released on GitHub.
2. RELATED WORK
Related work distinguishes spectrogram- and waveform-based separation, highlights the value of large datasets, and identifies memory constraints for long-context attention models.
- Spectrogram-based methods include Open-Unmix, D3Net, and Band-Split RNN, while waveform-based methods form the other major MSS family.Band-Split RNN combines complex spectrograms with multiple dual-path RNNs.
- Table 1 compares baseline SDR results on MUSDB HQ and identifies extra-song counts, mix-only training, and per-source fine-tuning.
- Large datasets benefit MSS, with Spleeter trained on 25,000 song extracts and prior systems benefiting from extra training data.Band-Split RNN improved by 0.7 dB SDR using an unsupervised augmentation technique requiring only mixes.
- SepFormer applies short- and long-range attention to speech separation but requires almost 11GB of memory for 5 seconds of 8 kHz audio.This memory requirement makes it unsuitable for studying longer 44.1 kHz inputs.
3. ARCHITECTURE
HT Demucs preserves Hybrid Demucs’s outer U-Net layers while replacing inner processing with a cross-domain Transformer Encoder for temporal and spectral signals. Sparse attention addresses the memory cost of longer sequences.
- Hybrid Demucs uses separate temporal and spectral U-Nets with shared representations at the deepest encoder and first decoder layers.Each U-Net has five encoder and five decoder layers.
- HT Demucs keeps the outermost four layers and replaces two innermost encoder and decoder layers with a cross-domain Transformer Encoder.
- The cross-domain encoder processes 2D spectral and 1D waveform signals in parallel using self-attention within domains and cross-attention across domains.
- The depth-5 cross-domain Transformer interleaves self-attention and cross-attention layers, with sinusoidal encodings for spectral and waveform inputs.
- Sparse HT Demucs uses xformer sparse kernels and Locally Sensitive Hashing to dynamically remove 90% of softmax elements.The sparsity pattern is selected from 32 LSH rounds with four buckets each.
4. DATASET
The authors curate and filter an internal dataset of songs and stems to obtain 800 tracks with usable four-source labels and controlled separation quality.
- The internal dataset contains 3,500 songs from 200 artists spanning diverse genres, with stems assigned to drums, bass, other, or vocals.Automated stem-name labeling is noisy because producer names can be subjective or ambiguous.
- 150 tracks were manually checked, and ambiguous stems were discarded before further filtering.
- The filtering procedure retains songs whose four sources are non-silent for at least 30% of one-second segments.A segment is silent when its volume is below -40dB.
- Songs are retained when diagonal P_i,i values exceed 70% and off-diagonal P_i,j values remain below 30%.P_i,j measures the proportion of present one-second segments where separated output volume differs from source volume by more than -10 dB.
- 800 songs satisfy the final source-quality criteria.
5. EXPERIMENTS AND RESULTS
Experiments evaluate HT Demucs under varied training, architecture, augmentation, sparse-kernel, and fine-tuning settings. The results show gains over Hybrid Demucs and identify longer context, remixing, and source-specific fine-tuning as important factors.
- Experimental setup: Experiments use MUSDB18-HQ together with 800 curated songs, with waveform L1 loss and model selection on a validation set.Training uses 8 Nvidia V100 GPUs, 44.1 kHz stereo audio, and 1200 epochs unless otherwise stated.
- Experimental setup: Test-time inference uses overlapping chunks with linear transitions, and SDR is reported using the SiSEC18 definition.Real Time Factor is also measured on a single Intel Xeon CPU core.
- Comparison with baselines: 0.45 dB higher SDR than Hybrid Demucs is obtained with improved transformer sequence modeling in the simple HT Demucs version.The sparse variant with fine-tuning reaches almost 0.9 dB improvement over Hybrid Demucs.
- Architectural hyper-parameters: Longer training excerpts improve performance by almost 0.6 dB at depth 5 and dimension 384.For 3.4-second excerpts, increasing depth from 5 to 7 helps, while increasing dimension from 384 to 512 lowers SDR by 0.05 dB.
- Data augmentation: Remixing augmentation is highly important, with a 0.7 dB SDR loss when it is removed.Disabling augmentations generally deteriorates final SDR, while repitching has limited impact.
- Sparse kernels and fine-tuning: Sparse kernels enable 12.2-second training segments and add 0.14 dB SDR, reaching 8.94 dB before source-specific fine-tuning.Fine-tuning per source adds 0.25 dB and reaches 9.20 dB; extending context to 15 seconds during fine-tuning produces the same 9.20 dB.
Conclusion
HT Demucs replaces Hybrid Demucs’ innermost convolutional layers with a cross-domain Transformer Encoder operating over temporal and spectral representations. With extra data, sparse attention, and longer training context, it improves SDR and reaches 9.20 dB after source-specific fine-tuning.
- Conclusion: HT Demucs replaces innermost convolutional layers with a cross-domain Transformer Encoder using self-attention and cross-attention.The encoder processes spectral and temporal information.
- Conclusion: 0.45 dB improvement over Hybrid Demucs is obtained with the larger training dataset.The conclusion attributes the benefit to the Transformer-based variant and extra training data.
- Conclusion: 12.2-second training context enabled by sparse attention provides a supplementary 0.4 dB gain.The conclusion identifies sparse attention as enabling the longer input length during training.
- Conclusion: The authors identify spectrogram subband splitting as a possible direction for further exploration.This proposal is connected to prior processing of subbands differently.