Source-linked AI summary
The 2018 Signal Separation Evaluation Campaign
Fabian-Robert Stöter, Antoine Liutkus, Nobutaka Ito
TL;DR
Audio source separation requires recovering individual sources from mixtures, and SiSEC 2018 addressed the need for scalable evaluation and substantial training data. The campaign released MUSDB18, new oracle and evaluation software, and reported participant results, showing strong performance from data-rich DNN systems while identifying training-data dependence as an open issue.
Problem
SiSEC needed scalable audio-separation evaluation and broader datasets because deep-learning systems require substantial learning data.
Method
The campaign released MUSDB18, oracle implementations, BSS Eval v4, and standardized evaluation resources for audio separation.
Results
Participant analyses found that DNN methods using additional training data performed best and were not significantly different from oracles for accompaniment separation.
Takeaways & Limitations
MUSDB18 and standardized tools support comparable separation evaluation and research, while participant performance highlights the importance of training-data scale.
Takeaways & Limitations
Performance gaps appeared to depend more on additional training data than on system architecture, leaving reduced-data performance an open challenge.
Abstract
from arXiv · showhide
This paper reports the organization and results for the 2018 community-based Signal Separation Evaluation Campaign (SiSEC 2018). This year's edition was focused on audio and pursued the effort towards scaling up and making it easier to prototype audio separation software in an era of machine-learning based systems. For this purpose, we prepared a new music separation database: MUSDB18, featuring close to 10h of audio. Additionally, open-source software was released to automatically load, process and report performance on MUSDB18. Furthermore, a new official Python version for the BSSEval toolbox was released, along with reference implementations for three oracle separation methods: ideal binary mask, ideal ratio mask, and multichannel Wiener filter. We finally report the results obtained by the participants.
1 Introduction
SiSEC 2018 shifted emphasis toward scalable, community-oriented audio separation evaluation and research resources, especially for machine-learning-based music separation. The campaign introduced new data, oracle methods, and tooling while retaining asynchronous separation.
- SiSEC compares separation systems through community-defined tasks, datasets, and metrics, with a longstanding emphasis on audio source separation.
- Speech separation and denoising tasks were dropped because other campaigns had become their core focus, while asynchronous separation was retained for its originality and fit with SiSEC’s objectives.
- SiSEC data is made public both to report standardized community progress and to support source-separation research beyond the campaign.
- The MUS data scope expanded from evaluation toward system design because deep-learning systems require substantial learning data.
- The 2018 edition added IRM and MWF oracle evaluations, released MUSDB18, and introduced BSS Eval v4 with faster time-invariant distortion-filter handling.
2 Oracle performance for audio separation
The paper frames oracle separation as an upper-bound analysis of filtering strategies, covering binary and ratio masks alongside multichannel Wiener filtering. These methods estimate source images from mixtures using time-frequency masks and, for MWF, spatial covariance information.
- Filtering methods estimate source images linearly from the mixture using method-specific parameters and complex time-frequency masks.
- Oracle evaluation uses true sources to assess upper bounds for three filtering strategies: IBM, IRM, and MWF.
- IBM assigns each time-frequency-channel element to one source using binary masks based on magnitude or power comparisons.
- IRM relaxes binary masking through fractional-power spectrogram ratios, including IRM1 for magnitude ratios and IRM2 for power ratios.
- MWF exploits multichannel information through covariance matrices, with spatial covariance encoding average interchannel correlations and power spectral density encoding time-frequency power.
- IBM1, IBM2, IRM1, IRM2, and MWF were implemented in Python and released under an open-source license.
3 Data and metrics
SiSEC 2018 released MUSDB18 as a realistic, structured music-separation dataset and introduced BSS Eval v4 with fixed track-level distortion matching. The changes support data-driven development while reducing evaluation cost and instability.
- MUSDB18 contains 150 full-length tracks totaling approximately 10 h of audio.
- The corpus provides stereo professionally mixed signals, four source categories, diverse genres, and predefined training and test splits.
- The dataset and Python development tools are freely available online.
- BSS Eval measures separation using SDR, SAR, SIR, and, in v3, ISR after optimally matching estimates to sources through linear distortion filters.
- BSS Eval v4 fixes distortion filters over each track instead of allowing them to vary over time, reducing computation and making matching more stable.
- The Python BSS Eval v4 implementation remains compatible with earlier MATLAB versions within 10−12 dB when time-varying filters are selected.
4 Separation results
The SiSEC 2018 separation results compare oracle strategies, evaluation-toolbox versions, and submitted systems, revealing strong benefits from soft masks, data-driven methods, and additional training data.
- Oracle performance: IRM2 and MWF produced the best overall soft-mask performance, while IBM underperformed soft masks on SDR, SIR, and especially SAR.IBM’s SAR loss reached around 5 dB for accompaniment; its strong ISR scores may be artificially boosted by zero estimates.
- Oracle performance: BSS Eval v4 was 8 times faster than v3, enabling 1 s frames while yielding statistically indistinguishable whole-dataset score distributions.Time-invariant distortion filters also exposed lower performance for lower-energy true sources and produced more dynamic scores.
- Submitted systems: Data-driven methods clearly outperformed model-based approaches across most targets and metrics in the MUSDB18 evaluation.The submitted methods largely reflected a shift toward deep learning, with model-based systems using prior spectrogram knowledge instead of MUSDB18 training data.
- Submitted systems: Methods using additional training data performed comparably to oracles on approximately half the tracks, with remaining difficulty concentrated in distorted tracks.Tracks with easy accompaniment separation often had challenging vocals estimation because vocals were rarely active, making vocals-presence detection useful.
- Submitted systems: DNN methods using additional data performed best in pairwise significance tests and were not significantly different from oracles for accompaniment, whereas vocals remained improvable.TAK1 and UHL2 also compared well with additional-data methods for vocals, while model-based methods generally performed worse.
- Submitted systems: The evaluation leaves open whether architecture or training-data quantity matters more, while supporting standardized MUSDB18 training and test sets.The reported gap appeared to come from additional training data, exemplified by the difference between UHL2 and UHL3; reduced-data performance remains challenging.
5 Conclusion
SiSEC 2018 delivered a Python BSS Eval version, the MUSDB18 corpus, and open-source oracle implementations, then analyzed their evaluation impact and participant results.
- Contributions: SiSEC 2018 released MUSDB18, a 150-track corpus totaling almost 10 hours of music with isolated stems.The corpus was designed for music separation evaluation and includes full-length tracks from multiple sources.
- Contributions: The campaign provided open-source implementations of three oracle methods to establish performance upper bounds.The released oracle systems include IBM1, IBM2, IRM1, IRM2, and MWF implementations.
- Evaluation: BSS Eval version 4 added time-invariant distortion filters, significantly reducing computational load compared with earlier MATLAB versions.The campaign also examined how this choice affected evaluation and oracle-score discrepancies relative to BSS Eval v3.
- Results: The paper reports overall participant scores and provides detailed metric, target, method, and asynchronous-task results through figures and tables.More detailed analyses and sound excerpts were made available on the SiSEC webpage.