PaperlayerView exact revision on arXiv

Source-linked AI summary

On Loss Functions for Supervised Monaural Time-Domain Speech Enhancement

Morten Kolbæk, Zheng-Hua Tan, Søren Holdt Jensen, Jesper Jensen

arXiv:1909.01019v2cs.SDcs.LGeess.AS

TL;DR

The paper addresses limited evidence about how loss functions affect time-domain deep-learning speech enhancement, where MSE optimization may not ensure speech quality or intelligibility. It evaluates popular losses in an end-to-end FCNN and finds that learning rate materially affects comparisons, waveform metrics can fail under misalignment, and SI-SDR is a strong general-purpose candidate.

  • Problem

    The paper examines how loss-function choice affects time-domain speech-enhancement systems because MSE optimization does not necessarily guarantee high speech quality or intelligibility.

  • Method

    The study trains an end-to-end time-domain FCNN with multiple losses and evaluates the resulting systems using popular speech-enhancement metrics.

  • Results

    SI-SDR achieves good general performance across a range of popular speech-enhancement evaluation metrics, while learning rate critically affects results and SI-SDR and SDR can fail when signals are misaligned.

  • Takeaways & Limitations

    SI-SDR is suggested as a general-purpose loss candidate for supervised monaural time-domain speech enhancement, with learning rate requiring careful identification for each loss.

Abstract

from arXiv · show

Many deep learning-based speech enhancement algorithms are designed to minimize the mean-square error (MSE) in some transform domain between a predicted and a target speech signal. However, optimizing for MSE does not necessarily guarantee high speech quality or intelligibility, which is the ultimate goal of many speech enhancement algorithms. Additionally, only little is known about the impact of the loss function on the emerging class of time-domain deep learning-based speech enhancement systems. We study how popular loss functions influence the performance of deep learning-based speech enhancement systems. First, we demonstrate that perceptually inspired loss functions might be advantageous if the receiver is the human auditory system. Furthermore, we show that the learning rate is a crucial design parameter even for adaptive gradient-based optimizers, which has been generally overlooked in the literature. Also, we found that waveform matching performance metrics must be used with caution as they in certain situations can fail completely. Finally, we show that a loss function based on scale-invariant signal-to-distortion ratio (SI-SDR) achieves good general performance across a range of popular speech enhancement evaluation metrics, which suggests that SI-SDR is a good candidate as a general-purpose loss function for speech enhancement systems.

I. INTRODUCTION

The paper studies how loss-function choice affects supervised monaural time-domain speech enhancement, using an end-to-end FCNN and several complementary losses. It aims to compare their speech-enhancement performance and identify guidance for selecting loss functions.

  • Monaural speech enhancement is important for improving noisy-speech quality and intelligibility in applications including hearing aids, mobile communications, and human-machine interfaces.
  • Time-domain end-to-end FCNN systems can estimate enhanced speech waveforms directly from noisy single-channel waveforms, avoiding exclusive reliance on STSA-domain processing.
  • The FCNN uses supervised pairs of clean and noisy speech to learn parameters that minimize a selected loss while producing an enhanced waveform.
  • The study compares six losses: time-domain MSE, STSA-MSE, STOI, ESTOI, SI-SDR, and PMSQE.
  • The experiments test whether systems optimized for different losses differ in speech-enhancement performance across commonly used evaluation metrics.

A. Time-Domain Mean Square Error

This section introduces time-domain MSE and STSA-MSE as baseline losses, emphasizing their mathematical simplicity and differing sensitivity to waveform versus spectral-amplitude errors.

  • A. Time-Domain Mean Square Error: Time-domain MSE is included because it is computationally simple and widely used in machine learning and signal processing.
  • A. Time-Domain Mean Square Error: Time-domain MSE measures squared error directly between the estimated and clean time-domain speech signals.
  • B. Short-Time Spectral Amplitude Mean Square Error: STSA-MSE is a classical and widely used speech-enhancement loss in both deep-learning and traditional algorithms.
  • B. Short-Time Spectral Amplitude Mean Square Error: STSA-MSE computes MSE between the single-sided amplitude spectra of the clean target and estimated speech.
  • B. Short-Time Spectral Amplitude Mean Square Error: Unlike time-domain MSE, STSA-MSE is insensitive to variations in the signals’ short-time phase spectrum.

C. Short-Time Objective Intelligibility

The STOI loss adapts an objective speech-intelligibility estimator for differentiable training by comparing clean and enhanced temporal envelopes across one-third-octave bands.

  • C. Short-Time Objective Intelligibility: STOI is a widely used speech-intelligibility estimator whose predictions correspond well with measured intelligibility across many acoustic scenarios.
  • C. Short-Time Objective Intelligibility: STOI compares clean and enhanced short-time temporal envelope vectors in one-third-octave frequency bands using sample linear correlation.
  • C. Short-Time Objective Intelligibility: The STOI score averages these intermediate correlations over 15 one-third-octave bands and the available temporal envelope vectors.
  • C. Short-Time Objective Intelligibility: With N = 30, STOI uses approximately 384 ms temporal-envelope vectors and is sensitive to modulation frequencies of 2.6 Hz and higher.
  • C. Short-Time Objective Intelligibility: The STOI loss is suitable for gradient-based training because it is almost entirely differentiable, with the clipping operator requiring only two subgradients.

D. Extended Short-Time Objective Intelligibility

ESTOI extends STOI to address reduced intelligibility-estimation performance in highly fluctuating noise by adding spectro-temporal normalization to envelope correlations.

  • D. Extended Short-Time Objective Intelligibility: ESTOI was proposed because STOI correlates less well with listening tests when noise components are highly fluctuating, such as competing-talker noise.
  • D. Extended Short-Time Objective Intelligibility: ESTOI significantly outperformed STOI, with ρ > 0.90, in highly fluctuating-noise conditions while performing similarly in less fluctuating noise.
  • D. Extended Short-Time Objective Intelligibility: Like STOI, ESTOI uses average correlations between one-third-octave-band short-time temporal-envelope vectors.
  • D. Extended Short-Time Objective Intelligibility: ESTOI constructs row- and column-normalized spectrogram matrices for clean and enhanced speech before defining its intelligibility index.
  • D. Extended Short-Time Objective Intelligibility: The ESTOI loss omits STOI’s clipping step and is therefore fully differentiable; it also omits the usual VAD because VAD is applied before training.

E. Scale-Invariant Signal-to-Distortion Ratio

SI-SDR measures distortion using a scale-adjusted reference, making it invariant to processed-signal scale. The paper uses its negative as a loss and relates it to correlation and minimum-energy reconstruction.

  • SI-SDR was introduced as an alternative to the SDR measure from the BSS eval toolbox.
  • Unlike SDR, SI-SDR is invariant to processed-signal scale but remains sensitive to finite-impulse-response filtering.
  • SI-SDR compares weighted clean speech with residual noise formed from the estimate and the scaled reference.
  • Scaling the reference makes SI-SDR invariant to the estimate’s scale, which suits DNN systems that may not guarantee proper output scaling.
  • Maximizing SI-SDR is equivalent to maximizing sample correlation while producing the minimum-energy solution; the loss is defined as LSI-SDR = −SI-SDR.

F. Perceptual Metric for Speech Quality Evaluation

PMSQE is designed as a differentiable approximation to PESQ, linking a perceptually inspired training loss to speech-quality evaluation. Across SSN and BBL corruption, PMSQE and PESQ vary monotonically with SNR and approximately inversely with each other.

  • PMSQE approximates the nondifferentiable PESQ speech-quality estimator, which predicts mean opinion scores for specified degradations.
  • PESQ scores range from 1 for extremely poor quality to 4.5 for no distortion.
  • For SSN and BBL corruption, PESQ and PMSQE have an approximately inverse, monotonic relationship with SNR.
  • The PMSQE loss is a log-domain STSA-MSE loss augmented with terms inspired by human perception.

III. EXPERIMENTAL DESIGN

The experiments examine how alternative loss functions affect fully convolutional, time-domain speech enhancement systems under realistic acoustic conditions. The study uses a noisy-speech dataset designed for substantial speaker and noise variability.

  • The study trains multiple FCNN-based enhancement systems with different loss functions on a large noisy-speech dataset.
  • The experimental dataset is designed to provide high speaker and noise variability for realistic acoustic evaluation.

A. Noise-free Speech Mixtures

The speech-mixture design combines WSJ0 utterances with varied synthetic and real-world noises, separate speaker splits, and matched or unseen test conditions. Preprocessing removes long silent regions and adapts sampling and duration to the loss functions.

  • Training uses 30,000 randomly selected WSJ0 utterances, validation uses 3,000, and testing uses 1,000 utterances from specified WSJ0 subsets.
  • Repeated utterance selection increases speech-material variability because each instance is mixed with a unique noise signal.
  • A STOI/ESTOI voice activity detector removes 25 ms segments more than 40 dB below each waveform’s maximum-energy segment.
  • STOI, ESTOI, time-domain MSE, and SI-SDR use 10 kHz signals truncated or padded to four seconds, whereas PMSQE uses 8 kHz signals of about five seconds.
  • The training data contains stationary speech-shaped noise, six-speaker babble, and two real-life natural-sound recordings.
  • Each noise type contributes 40 minutes for training, five minutes for validation, and five minutes for testing, with unique realizations across splits.
  • Evaluation also includes bus and pedestrian recordings whose noise statistics differ from the training noise types.

C. Noisy Speech Mixtures

The study constructs noisy speech mixtures and trains an encoder/decoder FCNN for time-domain enhancement using adaptive optimization and multiple loss functions. The architecture introduces 256 ms latency at 10 kHz, which may constrain hard real-time applications.

  • C. Noisy Speech Mixtures: Noisy training and validation mixtures combine clean utterances with equal-length noise, scaling noise to uniformly sampled SNRs from −10 to 10 dB.This range produces noisy waveforms spanning poor to perfectly intelligible conditions.
  • C. Noisy Speech Mixtures: The enhancement model is an 18-layer encoder/decoder FCNN with PReLU activations, input dimension L = 38656, and a bottleneck dimension of L/256.
  • C. Noisy Speech Mixtures: The FCNN contains approximately 6.8 million parameters across convolutional layers using 11-sample filters.
  • C. Noisy Speech Mixtures: The encoder/decoder receptive field is 2561 samples, yielding 256 ms latency at 10 kHz and potentially requiring architectural changes for hard real-time use.
  • C. Noisy Speech Mixtures: Training uses ADAM with β1 = 0.9 and β2 = 0.999, while the learning rate is reduced by a factor of two after two validation-loss plateaus.Training stops after five validation-loss epochs without improvement or 200 epochs.
  • C. Noisy Speech Mixtures: The experiments compare how six loss functions affect enhancement performance in the time-domain FCNN system.

A. Learning Rate vs. Performance Metric

The experiments show that learning-rate selection strongly affects comparisons among loss functions. Middle-range rates perform best overall, while using one shared non-optimal rate can produce misleading conclusions.

  • A. Learning Rate vs. Performance Metric: 10^-2 is too large and 10^-5 is generally too small for all evaluated loss functions, whereas 5·10^-4 and 10^-4 achieve particularly large scores.
  • A. Learning Rate vs. Performance Metric: LTIME-MSE, LSI-SDR, LSTSA-MSE, and LPMSQE reach their largest overall performance scores at 5·10^-4, while LSTOI and LESTOI peak at 10^-4.
  • A. Learning Rate vs. Performance Metric: At 10^-3, LTIME-MSE and LESTOI both achieve an ESTOI score of 0.79, but at 5·10^-4 their scores are 0.79 and 0.83, respectively.
  • A. Learning Rate vs. Performance Metric: At 10^-5, SI-SDR scores range from −22.36 dB for LESTOI systems to 10.15 dB for LSI-SDR systems despite comparable STOI and ESTOI results.
  • A. Learning Rate vs. Performance Metric: Selecting a non-optimal learning rate can lead to incorrect conclusions when systems trained with different loss functions are compared at the same rate.
  • A. Learning Rate vs. Performance Metric: The study selects separate learning rates for later experiments by maximizing the metric most similar to each system’s training loss.

B. Signal Integrity vs. Performance Metric

Time-domain comparisons reveal that loss functions can trade waveform fidelity for perceptually relevant enhancement, while waveform-matching metrics may misrepresent performance when signals are misaligned. Across matched and unmatched noise conditions, SI-SDR training provides strong general performance across evaluation metrics.

  • Waveform comparisons: All six systems reduce noise, but time-domain MSE produces the most per-sample-accurate waveform estimate.The other systems can produce less sample-accurate waveforms while still improving speech enhancement outcomes.
  • Waveform comparisons: STOI- and ESTOI-based losses can invert waveform polarity yet achieve similar or higher intelligibility scores because they match short-time energy in one-third-octave bands.These losses are invariant to signal polarity, so waveform inversion does not necessarily imply lower perceived intelligibility.
  • Waveform comparisons: Magnitude-spectrum-based losses may permit sub-1 ms time shifts, below the 10–30 ms range usually required for perception in low-latency applications.Such shifts affect waveform alignment without substantially changing the estimated amplitude spectrum.
  • Metric interpretation: A few samples of delay can cause SI-SDR and SDR to fail or vary greatly, despite stable STOI and ESTOI behavior and successful perceptual enhancement.These waveform-matching metrics require exact time alignment and should therefore be interpreted cautiously for phase-modifying time-domain systems.
  • Metric interpretation: Metric-specific losses generally maximize their corresponding metrics, whereas SI-SDR training achieves high improvements across metrics in matched and unmatched noise conditions.This general behavior holds across a wide range of SNRs, unseen speakers, and matched or unmatched noise types.

V. CONCLUSION

The study evaluates six loss functions for time-domain monaural speech enhancement across common performance metrics. It finds that metric-matched losses can optimize their corresponding metrics, learning rate strongly affects comparisons, waveform metrics require caution, and SI-SDR is a strong general-purpose candidate.

  • Study design: Six loss functions were evaluated in time-domain convolutional neural-network speech enhancement systems using five commonly used performance metrics.The study included time-domain MSE, STSA-MSE, STOI, ESTOI, SI-SDR, and PMSQE-based losses.
  • Metric-specific optimization: Optimizing a loss function designed to resemble a target performance metric is generally advantageous when optimizing that metric.This is particularly relevant for STOI and ESTOI, which predict speech intelligibility.
  • Training considerations: The learning rate can critically affect performance, so each loss function requires identification of an appropriate learning rate before comparisons are made.A sub-optimal learning rate can produce sub-optimal results and erroneous conclusions when comparing systems.
  • Metric limitations: SDR and SI-SDR must be used cautiously with systems that can modify signal phase because both failed completely when reference and processed signals were not perfectly aligned.This caveat concerns waveform-matching metrics in phase-capable time-domain enhancement systems.
  • Conclusion: SI-SDR achieves good general performance across popular speech-enhancement evaluation metrics and is suggested as a general-purpose loss candidate.The conclusion applies to supervised monaural time-domain speech enhancement.
Loading 1909.01019v2…