Source-linked AI summary
SDR - half-baked or well done?
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, John R. Hershey
TL;DR
The paper examines misleading behavior in BSS_eval SDR, especially for single-channel separation. It proposes SI-SDR and shows that it overcomes multiple SDR failure cases.
Problem
BSS_eval SDR can produce misleading evaluations, particularly in single-channel separation, despite widespread use for comparing methods.
Method
The paper proposes scale-invariant SDR, based on orthogonal projection of the estimate onto the target, and related SI-SIR and SI-SAR measures.
Results
SI-SDR overcomes multiple critical SDR failure cases, including destructive filtering that yields -4.7 dB SI-SDR but 11.6 dB BSS_eval SDR.
Takeaways & Limitations
SI-SDR provides a simpler alternative for evaluating speech enhancement and source separation quality than the BSS_eval SDR measure.
Takeaways & Limitations
SI-SDR does not treat scaling as an error, while global projections may not provide proper insight for highly non-stationary processing.
Abstract
from arXiv · showhide
In speech enhancement and source separation, signal-to-noise ratio is a ubiquitous objective measure of denoising/separation quality. A decade ago, the BSS_eval toolkit was developed to give researchers worldwide a way to evaluate the quality of their algorithms in a simple, fair, and hopefully insightful way: it attempted to account for channel variations, and to not only evaluate the total distortion in the estimated signal but also split it in terms of various factors such as remaining interference, newly added artifacts, and channel errors. In recent years, hundreds of papers have been relying on this toolkit to evaluate their proposed methods and compare them to previous works, often arguing that differences on the order of 0.1 dB proved the effectiveness of a method over others. We argue here that the signal-to-distortion ratio (SDR) implemented in the BSS_eval toolkit has generally been improperly used and abused, especially in the case of single-channel separation, resulting in misleading results. We propose to use a slightly modified definition, resulting in a simpler, more robust measure, called scale-invariant SDR (SI-SDR). We present various examples of critical failure of the original SDR that SI-SDR overcomes.
1. INTRODUCTION
The paper argues that BSS_eval’s SDR can produce misleading evaluations, particularly for single-channel separation, because its treatment of channel and scaling errors creates loopholes. It motivates SI-SDR as a simpler alternative and contrasts it with existing BSS_eval versions and decompositions.
- Problems with BSS_eval SDR: BSS_eval’s SDR can forgive severe signal modifications when a time-invariant 512-tap filter makes the reference fit the estimate.This can yield near-infinite SDR even when some signal frequencies are completely removed.
- Problems with BSS_eval SDR: BSS_eval’s image-based SDR does not allow global rescaling and takes the estimate’s scaling at face value, creating another potential loophole.The toolkit reports channel errors, including gain errors, through ISR rather than SDR.
- BSS_eval usage: Version 3.0 is recommended for reverberated or diffuse convolutive mixtures, while many single-channel separation papers appear to use the sources-based function instead.Version 2.1 provides a decomposition allowing only constant gain through bss decomp gain.
- Proposed alternative: SI-SDR offers a simpler alternative to BSS_eval SDR, and prior comparisons found significant differences between the two measures.The paper reviews the measures, presents SDR failure cases, and numerically compares them on speech separation.
2. PROPOSED MEASURES
The paper proposes SI-SDR to replace BSS_eval’s filter-based SDR, using a single optimal scaling coefficient so the residual is orthogonal to the target. It also introduces scale-dependent and scale-invariant decompositions that simplify interpretation while exposing trade-offs around rescaling and global projections.
- BSS_eval limitations: BSS_eval sources permits time-invariant filter modifications of the reference, potentially treating severe spectral changes as negligible channel errors.The paper motivates this assumption by distinguishing source references from microphone spatial images, which may include room-response filtering.
- SI-SDR: SI-SDR rescales the target or estimate so the residual is orthogonal to the target, yielding equivalent projection-based definitions.The optimal target scaling is α = ˆs^T s/||s||^2, after which the estimate is decomposed into a scaled target and residual.
- SI-SDR: SI-SDR uses one scaling coefficient rather than BSS_eval’s 512-tap FIR filter, making computation simpler and faster.The paper relates SI-SDR to the gain-only decomposition available in BSS_eval Version 2.1.
- Scale-aware SDR: SI-SDR is invariant to estimate rescaling, whereas SD-SDR adds a separate rescaling error to penalize scale changes.SD-SDR preserves the residual term while also accounting for the difference between the original and rescaled target.
- Scale-aware SDR: SD-SDR peaks at µ = 1 and properly discourages downscaling, but large upscaling reduces it only to −3 dB; combining SNR and SD-SDR is suggested when both matter.The proposed combination is intended for applications where downscaling and upscaling errors are both important.
- SI-SIR and SI-SAR: The proposed SI-SIR and SI-SAR use an orthogonal decomposition, creating a direct relationship among interference, artifact, and distortion measures.The decomposition separates the residual into interference and artifact components, unlike BSS_eval’s less intuitive source-to-artifacts definition.
3. EXAMPLES OF EXTREME FAILURE CASES
The paper presents extreme cases where BSS_eval SDR remains high despite severe signal degradation, while SI-SDR and related metrics better track quality loss.
- 3.1. Optimizing a filter to minimize SI-SDR: -4.7 dB SI-SDR accompanies destructive filtering, yet BSS_eval SDR remains 11.6 dB because it filters the reference similarly.The optimized filter removes most of the speech spectrum, passing only a couple of narrow bands.
- 3. EXAMPLES OF EXTREME FAILURE CASES: The failure cases show that SDR can reward removing signal frequencies or applying matching filtering to the reference instead of measuring preserved speech quality.The paper introduces these examples as failure modes that SI-SDR overcomes.
- 3.2. Progressive deletion of frequency bins: 10–15 dB SDR persists until nearly all frequency bins are deleted, and SDR even increases when the masking proportion reaches 0.4.SNR, SI-SDR, and SD-SDR instead decrease monotonically as frequency bins are removed.
- 3.3. Varying band-stop filter gain for speech corrupted with band-pass noise: SDR monotonically increases as bandstop gain decreases, whereas SNR and SI-SDR peak near the expected gain of 0.5.The example uses bandpass noise with 0 dB local SNR and varies the bandstop gain from 0 to 1.
4. COMPARISON ON A SPEECH SEPARATION TASK
The paper compares SI-SDR with BSS_eval SDR on single-channel speech separation systems using wsj0-2mix test-set results. The reported score differences are modest but systematic, and may understate SDR’s loopholes in less favorable settings.
- 4. COMPARISON ON A SPEECH SEPARATION TASK: The comparison collects SI-SDR and BSS_eval SDR improvements on the wsj0-2mix test set, mainly from prior work and including a recent state-of-the-art score.The table compares various speech separation systems on the dataset test set.
- 4. COMPARISON ON A SPEECH SEPARATION TASK: 0.3–0.6 dB separates SI-SDR and BSS_eval SDR improvements across the evaluated wsj0-2mix systems.The typical difference is around 0.5 dB, and the systems produce signals considered to have good perceptual quality.
- 4. COMPARISON ON A SPEECH SEPARATION TASK: More varied score differences could arise for poorer-quality algorithms or more stationary targets and interferences, where short-FIR-filter loopholes may be easier to exploit.The paper identifies speech enhancement scenarios as a likely setting for such loopholes.
5. CONCLUSION
The paper identifies problems in BSS eval’s SDR use, especially for single-channel scenarios, and presents SI-SDR as a simpler scale-invariant alternative. It also reports that SI-SDR overcomes multiple SDR failure cases.
- SI-SDR is presented as a simpler, scale-invariant alternative to BSS eval’s SDR for single-channel scenarios.
- SI-SDR overcomes multiple failure cases identified for BSS eval’s SDR.