SDR - half-baked or well done?

Introduction

Source separation and speech enhancement have been an intense focus of research in the signal processing community for several decades, and interest has gotten even stronger with the recent advent of powerful new techniques based on deep learning . An important area of research has focused on single-channel methods, which can denoise speech or separate one or more sources from a mixture recorded using a single microphone. Many new methods are proposed, and their relevance is generally justified by their outperforming some previous method according to some objective measure.

While the merits of various objective measures such as PESQ , Loizou’s composite measure , PEMO-Q , PEASS , or STOI , could be debated and compared, we are concerned here with an issue with the way the widely relied upon BSS_eval toolbox has been used. We focus here on the single-channel setting. The BSS_eval toolbox reports objective measures related to the signal-to-noise ratio (SNR), attempting to account for channel variations, and to report a decomposition of the overall error, referred to as signal-to-distortion ratio (SDR), into components indicating the type of error: source image to spatial distortion ratio (ISR), signal to interference ratio (SIR), and signal to artifacts ratio (SAR). In version 3.0, BSS_eval featured two main functions, bss_eval_images and bss_eval_sources.

bss_eval_sources completely forgives channel errors that can be accounted for by a time-invariant 512-tap filter, modifying the reference to best fit each estimate. This includes very strong modifications of the signal, including low-pass or high-pass filters. Thus, obliterating some frequencies of a signal by setting them to 0 could absurdly still result in near infinite SDR.

bss_eval_images reports channel errors (including gain errors) as errors in the ISR measure, but its SDR is nothing else than vanilla SNR. While not as fatal as the modification of the reference in bss_eval_sources, bss_eval_images suffers from some issues. First, it does not even allow for a global rescaling factor, which may occur when one tries to avoid clipping in the reconstructed signal. Second, as does SNR, it takes the scaling of the estimate at face value, a loophole that algorithms could (potentially unwittingly) exploit, as explained in section 2.2.

An earlier version (2.1) of the toolbox does provide, among other functions, a decomposition which only allows a constant gain via the function bss_decomp_gain. Performance criteria such as SDR can then be computed from this decomposition, but most papers on single-channel separation appear to be using bss_eval_sources. The BSS_eval websitehttp://bass-db.gforge.inria.fr/bss_eval/ actually displays a warning about which version should be used. Version 3.0 “is recommended for mixtures of reverberated or diffuse sources (aka convolutive mixtures), due to longer decomposition filters enabling better correlation with subjective ratings. It [is] also recommended for instantaneous mixtures when the results are to be compared with SiSEC.” On the other hand, version 2.1 “is practically restricted to instantaneous mixtures of point sources. It is recommended for such mixtures, except when the results are to be compared with SiSEC.” It appears that this warning has not been understood, and most papers use Version 3.0 without further consideration. The desire to compare results to (early editions of) SiSEC should also not be a justification for using a flawed measure. The same issues apply to an early Python version of BSS_eval, bss_evalhttp://github.com/craffel/mir_eval/ . Recently, BSS_eval v4 was released as a Python implementationhttps://sigsep.github.io/sigsep-mus-eval/museval.metrics.html : the authors of Version 4 acknowledged the issue with the original bss_eval_sources, and recommended using bss_eval_images instead. This however does not address the scaling issue.

These problems shed doubt on many results, including some in our own older papers, especially in cases where algorithms differ by a few tenths of a dB in SDR. This paper is intended both to illustrate and propagate this message more broadly, and also to encourage the use, for single-channel separation evaluation, of simpler, scale-aware, versions of SDR: scale-invariant SDR (SI-SDR) and scale-dependent SDR (SD-SDR). We also propose a definition of SIR and SAR in which there is a direct relationship between SDR, SIR, and SAR, which we believe is more intuitive than that in BSS_eval. The scale-invariant SDR (SI-SDR) measure was used in . Comparisons in showed that there is a significant difference between SI-SDR and the SDR as implemented in BSS_eval’s bss_eval_sources function. We review the proposed measures, show some critical failure cases of SDR, and give a numerical comparison on a speech separation task.

Proposed measures

A critical assumption in bss_eval_sources, as it is implemented in the publicly released toolkit up to Version 3.0, is that time-invariant filters are considered allowed deformations of the target/reference. One potential justification for this is that a reference may be available for a source signal instead of the spatial image at the microphone which recorded the noisy mixture, and that spatial image is likely to be close to the result of the convolution of the source signal with a short FIR filter, as an approximation to its convolution with the actual room impulse response (RIR). This however leads to a major problem, because the space of signals achievable by convolving the source signal with any short FIR filter is extremely large and includes perceptually widely different signals from the spatial image. Note that the original BSS_eval paper also considered time-varying gains and time-varying filters as allowed deformations. Taken to an extreme, this creates the situation where the target can be deformed to match pretty much any estimate.

Modifying the target/reference when comparing algorithms is deeply problematic when the modification depends on the outputs of each algorithm. In effect, bss_eval_sources chooses a different frequency weighting of the error function depending on the spectrum of the estimated signal: frequencies that match the reference are emphasized, and those that do not are discarded. Since this weighting is different for each algorithm, bss_eval_sources cannot provide a fair comparison between algorithms.

2 The problem with not changing anything

As is illustrated in Fig. 1, where for simplicity we consider the case where the estimate is in the subspace spanned by speech and noise (i.e., no artifact), what is considered as the noise in such a context is the residual $s-\hat{s}$, which is not guaranteed to be orthogonal to the target $s$. A tempting mistake is to artificially boost the SNR value without changing anything perceptually by rescaling the estimate, for example to the orthogonal projection of $s$ on the line spanned by $\hat{s}$: this leads to a right triangle whose hypotenuse is $s$, so SNR could always be made positive. In particular, starting from a mixture $x$ where $s$ and $n$ are orthogonal signals with equal power, so with an SNR of 0 dB, projecting $s$ orthogonally onto the line spanned by $x$ corresponds to rescaling the mixture to $x/2$: this “improves” SNR by $3$ dB. Interestingly, bss_eval_images’s ISR is sensitive to the rescaling, so the ISR of $x$ will be higher than that of $x/2$, while its SDR (equal to SNR for bss_eval_images) is lower.

3 Scale-aware SDR

To ensure that the residual is indeed orthogonal to the target, we can either rescale the target or rescale the estimate. Rescaling the target such that the residual is orthogonal to it corresponds to finding the orthogonal projection of the estimate $\hat{s}$ on the line spanned by the target $s$, or equivalently finding the closest point to $\hat{s}$ along that line. This leads to two equivalent definitions for what we call the scale-invariant signal-to-distortion ratio (SDR):

The optimal scaling factor for the target is obtained as $\alpha=\hat{s}^{T}s/||s||^{2}$, and the scaled reference is defined as $e_{\text{target}}=\alpha s$. We then decompose the estimate $\hat{s}$ as $\hat{s}=e_{\text{target}}+e_{\text{res}}$, leading to the expanded formula:

Instead of a full 512-tap FIR filter as in BSS_eval, SI-SDR uses a single coefficient to account for scaling discrepancies. As an extra advantage, computation of SI-SDR is thus straightforward and much faster than that of SDR. Note that SI-SDR corresponds to the SDR obtained from bss_decomp_gain in BSS_eval Version 2.1. SI-SDR has recently been used as an objective measure in the time domain to train deep learning models for source separation, outperforming least-squares on some tasks (it is referred to as SDR in and as SI-SNR in ).

A potential drawback of SI-SDR is that it does not consider scaling as an error. In situations where this is not desirable, one may be interested in designing a measure that does penalize rescaling. Doing so turns out not to be straightforward. As we saw in the example in Section 2.2 of a mixture $x$ of two orthogonal signals $s$ and $n$ with equal power, considering the rescaled mixture $\hat{s}=\mu x$ as the estimate, SNR does not peak at $\mu=1$ but instead encourages a down-scaling of $\mu=1/2$. It does however properly discourage large up-scaling factors. As an alternative measure that properly discourages down-scalings, we propose a scale-dependent SDR (SD-SDR), where we consider the rescaled $s$ as the target $e_{\text{target}}=\alpha s$, but consider the total error as the sum of two terms, $||\alpha s-\hat{s}||^{2}$ accounting for the residual energy, and $||s-\alpha s||^{2}$ accounting for the rescaling error. Because of orthogonality, $||\alpha s-\hat{s}||^{2}+||s-\alpha s||^{2}=||s-\hat{s}||^{2}$, and we obtain:

Going back to the example in Section 2.2, SI-SDR is independent of the rescaling of $x$, while SD-SDR for $\hat{s}=\mu x$ is equal to

which does peak at $\mu=1$. While this measure properly accounts for down-scaling errors where $\mu<1$, it only decreases to $-3$ dB for large up-scaling factors $\mu\gg 1$. For those applications where both down-scaling and up-scaling are critical, one could consider the minimum of SNR and SD-SDR as a relevant measure.

4 SI-SIR and SI-SAR

In the original BSS_eval toolkit, the split of SDR into SIR and SAR is done in a mathematically non intuitive way: in the original paper, the SAR is defined as the “sources to artifacts ratio,” not the “source to artifacts ratio,” where “sources” refers to all sources, including the noise. That is, if the estimate contains more noise, yet everything else stays the same, then the SAR actually goes up. There is also no simple relationship between SDR, SIR, and SAR.

Similarly to BSS_eval, we can further decompose $e_{\text{res}}$ as $e_{\text{res}}=e_{\text{interf}}+e_{\text{artif}}$, where $e_{\text{interf}}$ is defined as the orthogonal projection of $e_{\text{res}}$ onto the subspace spanned by both $s$ and $n$. But differently from BSS_eval, we define the scale-invariant signal to interference ratio (SI-SIR) and the scale-invariant signal to artifacts ratio (SI-SAR) as follows:

These definitions have the advantage over those of BSS_eval that they verify

because the orthogonal decomposition leads to $||e_{\text{res}}||^{2}=||e_{\text{interf}}||^{2}+||e_{\text{artif}}||^{2}$. There is thus a direct relationship between the three measures. Scale-dependent versions can be defined similarly.

That being said, we feel compelled to note that, whether it is still relevant to split SDR into SIR and SAR is a matter of debate: machine-learning based methods tend to perform a highly non-stationary type of processing, and using a global projection on the whole signal may thus not be guaranteed to provide the proper insight.

Examples of extreme failure cases

We present some failure modes of SDR that SI-SDR overcomes.

For this example, we optimize an STFT-domain, time-invariant filter to minimize SI-SDR. We will show that despite SI-SDR being minimized by the filter, SDR performance remains relatively high since it is allowed to apply filtering to the reference signal.

An example of a learned filter and resulting spectrograms for a single male utterance from CHiME2 is shown in Fig. 2. To minimize SI-SDR, the filter learns to remove most of the signal’s spectrum, only passing a couple of narrow bands. This filter achieves -4.7 dB SI-SDR, removing much of the speech content. However, despite this destructive filtering, we have the paradoxical result that the SDR of this signal is still high at 11.6 dB, since BSS_eval is able to find a filter to be applied to the reference signal that removes similar frequency regions. This filter is shown in red in the top part of Fig. 2, somewhat matching the filter minimizing SI-SDR in blue.

2 Progressive deletion of frequency bins

The previous example illustrated that SDR can yield high scores despite large regions of a signal’s spectrum being deleted. Now we examine how various metrics perform when frequency bins are progressively deleted from a signal.

We add white noise at 15 dB SNR to the same speech signal used in Section 3.1. Then time-invariant STFT-domain masking is used to remove varying proportions of frequency bins, where the mask is bandpass with a center frequency at the location of median spectral energy of the speech signal averaged across STFT frames. We measure four metrics: SDR, SNR, SI-SDR, and SD-SDR. The results are shown in Fig. 3. Despite more and more frequency bins being deleted, SDR (blue) remains between 10 dB and 15 dB, until nearly all frequencies are removed. In fact, SDR even increases for a masking proportion of 0.4. In contrast, the other metrics more appropriately measure signal degradation since they monotonically decrease.

An important practical scenario in which such behavior would be fatal is that of bandwidth extension: it is not possible to properly assess the baseline performance, where upper frequency bins are silent, using SDR.

3 Varying band-stop filter gain for speech corrupted with band-pass noise

We consider the performance of such filters when varying the bandstop gain from 0 to 1 in steps of 0.025, again for SDR, SNR, SI-SDR, and SD-SDR. The results are shown in Fig. 4. Notice that SNR, SI-SDR have a peak around a gain of 0.5 as expected. However, SDR monotonically increases as gain decreases. This is an undesirable behavior, as SDR becomes more and more optimistic about signal quality as more of the signal’s spectrum is suppressed, because it is all too happy to see the noisy part of the spectrum being suppressed and modify the reference to focus only on the remaining regions. SD-SDR peaks slightly above 0.5, because it penalizes the down-scaling of the speech signal within the noisy band.

Comparison on a speech separation task

Both SI-SDR and BSS_eval’s SDR have recently been used by various studies in the context of single-channel speaker-independent speech separation on the wsj0-2mix dataset , some of these studies reporting both figures . We gather in Table 1 various SI-SDR and BSS_eval SDR improvements (in dB) on the test set of the wsj0-2mix dataset mainly from , to which we add the recent state-of-the-art score of . The difference between the SI-SDR and the SDR scores for the algorithms considered are around 0.5 dB, but vary from 0.3 dB to 0.6 dB. Note furthermore that the algorithms considered here all result in signals that can be considered of good perceptual quality: much more varied results could be obtained with algorithms that give worse results. If the targets and interferences in the dataset were more stationary, such as in some speech enhancement scenarios, it is also likely there could be loopholes for SDR to exploit, where a drastic distortion that can be well approximated by a short FIR filter happens to lead to similar results on the mixture and the reference signals.

Conclusion

We discussed issues that pertain to the way BSS_eval’s SDR measure has been used, in particular in single-channel scenarios, and presented a simpler scale-invariant alternative called SI-SDR. We also showed multiple failure cases for SDR that SI-SDR overcomes.

Acknowledgements: The authors would like to thank Dr. Shinji Watanabe (JHU) and Dr. Antoine Liutkus and Dr. Fabian Stöter (Inria) for fruitful discussions.

References