PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervised Speech Separation Based on Deep Learning: An Overview

DeLiang Wang, Jitong Chen

arXiv:1708.07524v2cs.CLcs.LGcs.NEcs.SD

TL;DR

Speech separation seeks to extract target speech from interference, and the paper overviews its formulation as supervised learning. It synthesizes learning machines, training targets, features, algorithms, and generalization across monaural and array-based methods. The overview reports that deep-learning-based separation rapidly elevated performance across a wide range of speech-separation tasks.

  • Problem

    Speech separation must extract target speech from interference, while supervised methods must also generalize across unseen noise, speakers, and SNR conditions.

  • Method

    The paper provides a comprehensive overview organized around learning machines, training targets, acoustic features, representative monaural and array-based algorithms, and generalization.

  • Results

    Deep-learning-based separation greatly elevated the state of the art across monaural enhancement, dereverberation, speaker separation, and array speech separation.

  • Takeaways & Limitations

    Future advances are expected from tighter integration of domain knowledge with data-driven frameworks and continued deep-learning progress.

Abstract

from arXiv · show

Speech separation is the task of separating target speech from background interference. Traditionally, speech separation is studied as a signal processing problem. A more recent approach formulates speech separation as a supervised learning problem, where the discriminative patterns of speech, speakers, and background noise are learned from training data. Over the past decade, many supervised separation algorithms have been put forward. In particular, the recent introduction of deep learning to supervised speech separation has dramatically accelerated progress and boosted separation performance. This article provides a comprehensive overview of the research on deep learning based supervised speech separation in the last several years. We first introduce the background of speech separation and the formulation of supervised separation. Then we discuss three main components of supervised separation: learning machines, training targets, and acoustic features. Much of the overview is on separation algorithms where we review monaural methods, including speech enhancement (speech-nonspeech separation), speaker separation (multi-talker separation), and speech dereverberation, as well as multi-microphone techniques. The important issue of generalization, unique to supervised learning, is discussed. This overview provides a historical perspective on how advances are made. In addition, we discuss a number of conceptual issues, including what constitutes the target source.

I.INTRODUCTION

Speech separation aims to extract target speech from interference, a difficult but important signal-processing problem with applications in communication and recognition. This overview frames it as supervised learning and surveys its learning machines, targets, features, algorithms, and generalization.

  • Speech separation extracts target speech from nonspeech noise, interfering speech, or reverberation, supporting hearing prostheses, telecommunications, and robust recognition.
  • Human speech segregation varies strongly with interference: speech reception thresholds are about 2 dB for broadband noise, –10 dB for one talker, and –2 dB for two talkers.
  • Automatic speech separation remains challenging despite humans’ apparent ease in following a speaker amid other voices and background sounds.
  • Traditional monaural enhancement estimates clean speech using speech and noise statistics, while array methods use spatial filtering to attenuate interference from other directions.
  • Supervised separation learns discriminative patterns from data and is organized around learning machines, training targets, acoustic features, algorithms, and generalization.
  • Deep networks gain representational power with depth, but random initialization makes them difficult to train because gradients can vanish in lower layers.

III.TRAINING TARGETS

Supervised speech separation uses training targets to specify either time-frequency relationships or clean-speech spectral representations. Evaluation combines signal-level measures with perception-level measures of intelligibility and quality.

  • Training targets fall into masking-based targets, which describe clean speech relative to interference, and mapping-based targets, which represent clean-speech spectra.
  • Speech-separation metrics are divided into signal-level measures of enhancement or interference reduction and perception-level measures of listener-relevant outcomes.
  • STOI measures correlation between the short-time temporal envelopes of clean reference and separated utterances to estimate intelligibility.
  • PESQ is the standard speech-quality metric and is recommended by the International Telecommunication Union.

A. Ideal Binary Mask

Binary and ratio masks provide alternative supervised targets on time-frequency representations. Binary masks classify units, whereas the ideal ratio mask provides a soft, energy-based weighting.

  • The ideal binary mask labels each time-frequency unit according to whether local target-to-interference SNR exceeds a criterion.
  • IBM estimation naturally becomes supervised classification because every time-frequency unit receives a target-dominant or interference-dominant label.
  • The target binary mask compares target energy with fixed speech-shaped noise rather than the actual interference, while retaining binary time-frequency labels.
  • The ideal ratio mask is a soft IBM variant whose weighting depends on speech and noise energy within each time-frequency unit.
  • With β commonly set to 0.5, the square-root IRM preserves speech energy under uncorrelated speech and noise; this assumption is weaker for convolutive interference.

D. Spectral Magnitude Mask

The spectral magnitude mask (SMM) is defined from clean and noisy speech STFT magnitudes and applied to noisy magnitudes to reconstruct separated speech. The phase-sensitive mask (PSM) extends this approach with phase-difference information, while the complex ideal ratio mask operates directly in the complex domain.

  • Spectral Magnitude Mask: The SMM is defined as a ratio involving clean-speech and noisy-speech STFT magnitudes.It is not upper-bounded by 1.
  • Spectral Magnitude Mask: Applying the SMM or its estimate to noisy magnitudes and resynthesizing with noisy or estimated clean phases produces separated speech.
  • Phase-Sensitive Mask: The PSM extends the SMM by incorporating the phase difference between clean and noisy speech within each time-frequency unit.The added phase information leads to higher SNR and tends to improve clean-speech estimates relative to the SMM.
  • Complex Ideal Ratio Mask: The cIRM reconstructs clean speech through complex multiplication of the cIRM and noisy-speech STFTs.Its real and imaginary components can be estimated separately, but mask values are unbounded and require compression.
  • Complex Ideal Ratio Mask: Cartesian cIRM components contain structure in both real and imaginary parts, whereas polar representation lacks clear phase structure for direct supervised phase estimation.

G. Target Magnitude Spectrum

The target magnitude spectrum (TMS) estimates clean-speech spectral magnitude from noisy-speech magnitude. Related targets include power, mel, and gammatone-filterbank spectra, with the latter converted back to waveform through cochleagram inversion.

  • Target Magnitude Spectrum: The TMS maps noisy-speech magnitude spectrograms to the magnitude spectrum of clean speech.Power, mel, or log-compressed spectral forms may be used instead of magnitude spectrum.
  • Target Comparison: Figure 3 compares training targets using STOI and PESQ for clean speech mixed with factory noise at -5 dB, 0 dB, and 5 dB SNR.At 0 dB mixtures, speech enhancement and NMF benchmarks are also highlighted.
  • Gammatone Frequency Target Power Spectrum: The GF-TPS represents clean speech as the power spectrum of a gammatone-filterbank cochleagram.An estimate can be converted to separated speech through cochleagram inversion.

I. Signal Approximation

Signal approximation (SA) trains a ratio-mask estimator to minimize the spectral-magnitude difference between clean speech and estimated speech. Two-stage training first estimates the SMM and then fine-tunes the model with the SA loss.

  • Signal Approximation: SA trains a ratio-mask estimator by minimizing the difference between clean and estimated speech spectral magnitudes.
  • Signal Approximation: SA combines ratio masking and spectral mapping with the goal of maximizing SNR.A related earlier target also sought maximal SNR through IBM estimation.
  • Training Procedure: Two-stage training first uses the SMM as the target and then fine-tunes the learning machine with the SA loss.
  • Target Comparison: A fixed three-hidden-layer feedforward DNN was used to compare training targets with identical input features, using STOI and PESQ for evaluation.Speech enhancement and supervised NMF served as benchmarks.
  • Scope and Alternatives: The reviewed target list is not exhaustive, and direct waveform mapping has been reported to perform poorly even with a DNN.

IV.FEATURES

Acoustic features and learning machines play complementary roles in supervised speech separation. Studies find that gammatone-domain features perform strongly at low SNRs, while feature choice substantially affects separation performance and raw waveforms can perform poorly with feedforward DNNs.

  • Feature–Model Relationship: Feature quality and learning-machine power place complementary demands on supervised separation systems.
  • Feature Development: Early systems used binaural ITD and IID, monaural pitch and AMS, and later expanded to MFCC, GFCC, and other acoustic features.
  • Feature Evaluation: The evaluated feature set spans mel, linear-prediction, gammatone, zero-crossing, autocorrelation, filtering, modulation, and pitch-based features.
  • Feature Evaluation: Gammatone-domain features, especially MRCG, consistently outperform other features in classification accuracy and HIT−FA rate across six noises at -5 dB SNR.Comparisons indicate that DCT cepstral compaction and modulation extraction are not effective, while inaccurate low-SNR pitch estimation harms pitch-feature performance.
  • Reverberant Separation: A separate study evaluated LOG-MAG, LOG-MEL, raw waveform, and other features for denoising and speaker separation under simulated and recorded reverberation.
  • Feature–Model Relationship: Large performance differences across feature conditions demonstrate feature importance, while poor raw-waveform results may reflect mismatch with feedforward DNNs.CNNs and RNNs may be better suited to end-to-end waveform separation.

V.MONAURAL SEPARATION ALGORITHMS

This section reviews monaural speech enhancement, speech dereverberation, dereverberation plus denoising, speaker separation, and generalization in supervised speech separation.

  • Monaural algorithms cover speech enhancement, speech dereverberation, dereverberation plus denoising, and speaker separation.
  • The overview explains representative monaural algorithms and discusses how supervised speech separation generalizes.
  • Generalization is treated alongside the representative algorithms reviewed for monaural separation.

A. Speech Enhancement

Speech enhancement research uses deep learning for masking, spectral mapping, and end-to-end temporal mapping, while addressing generalization across noise, speakers, and SNR. Reviewed systems include DNNs, recurrent models, GANs, autoencoders, and NMF-based approaches.

  • Deep learning approaches: Deep learning was introduced to speech separation through DNN subband classification for estimating the ideal binary mask (IBM).Later work used DNN-learned features with linear SVMs for efficient subband IBM estimation.
  • Spectral mapping: DNN speech enhancement estimates clean-speech spectra from noisy inputs using spectral mapping, with about 0.4 to 0.5 PESQ gains on untrained noises.These gains exceeded those of a representative traditional enhancement method.
  • Enhancement methods: Speech enhancement also includes time-frequency masking, spectral mapping, end-to-end temporal mapping, GANs, deep NMF, and DNN-assisted NMF.End-to-end methods avoid a time-frequency representation and may avoid reliance on noisy-speech phase during waveform reconstruction.
  • Generalization: Supervised enhancement must generalize across noise, speakers, and SNR, making untrained conditions a central evaluation issue.This generalization burden differs from traditional enhancement and CASA methods, which use minimal supervised training.
  • Adaptation: A two-stage DNN combines spectral mapping with an autoencoder that performs unsupervised adaptation during testing to address training–test mismatch.The autoencoder is trained to reconstruct clean-utterance magnitude spectra without labeled data.
  • Generalization: LSTM models trained on 3,200,000 mixtures with 10,000 noises and up to 77 speakers substantially outperform DNNs in STOI for untrained speakers.The LSTM benefits from additional training speakers, whereas DNN performance degrades as more training speakers are added.

C. Speech Dereverberation and Denoising

Speech dereverberation addresses convolutional distortion from room reflections, often alongside background noise. Reviewed deep-learning methods use spectral mapping, reverberation-time-conditioned processing, dynamic feature prediction, and staged denoising plus dereverberation.

  • Motivation: Room reverberation convolves direct speech with a room impulse response and distorts signals across time and frequency.The challenge is especially important when reverberation is combined with background noise.
  • Spectral mapping: DNN spectral mapping predicts anechoic speech from reverberant frames and can reconstruct anechoic cochleagrams with high quality.Later work extended spectral mapping from cochleagrams to spectrograms and combined dereverberation with denoising.
  • Reverberation-aware processing: Dereverberation performance improves when frame length and shift are selected according to reverberation time, T60.A reverberation-time-aware system estimates T60 and uses it to choose feature-extraction parameters.
  • Combined denoising and dereverberation: A two-stage DNN performs ratio-mask denoising first and spectral-mapping dereverberation second, reflecting their reported task-specific strengths.The approach also extends time-domain reconstruction to reduce adverse effects from reverberant-noisy phase.

D. Speaker Separation

Speaker separation extracts multiple speech signals from mixtures containing two or more voices. The overview covers DNN masking and mapping methods, speaker-dependent and speaker-independent settings, and clustering-based approaches.

  • DNN speaker separation extracts multiple speech signals from mixtures containing two or more voices.
  • Speaker-dependent separation: Masking layers enforce mixture-consistent outputs, while discriminative training separates one speaker from the estimated other.Both masking layers and discriminative training improve speaker separation.
  • Speaker-dependence: Speaker-dependent, target-dependent, and speaker-independent settings differ in whether training and testing speakers may change.
  • Speaker-dependent separation: 42.5, 49.2, and 58.7 percentage points are the reported intelligibility improvements at -3, -6, and -9 dB TIR for HI listeners.At -6 and -9 dB TIR, HI listeners performed equivalently to unaided NH listeners.
  • Speaker-independent separation: Deep clustering embeds time-frequency units, concatenates segment embeddings, and applies K-means to form speaker clusters.Its cost matches estimated embedding affinities to indicator-matrix affinities; the method produces high-quality speaker separation.
  • Speaker-independent separation: Permutation-invariant training outputs masks without tying outputs to speaker identities, while matching separation quality obtained with deep clustering.
  • Speaker-independent separation: Speaker-independent models generalize to unseen speaker pairs at the frame level, but sequential organization across frames or segments remains unresolved.The overview identifies integration of traditional CASA techniques and deep learning as a future research direction.

VI. ARRAY SEPARATION ALGORITHMS

Array separation exploits spatial information from multiple microphones, increasingly integrating it with supervised deep learning and monaural features. These methods support binaural separation, beamforming, and iterative mask-based enhancement, with gains reported for separation and ASR.

  • Spatial information: Microphone arrays provide spatial cues that enable source localization and extraction from a target direction, complementing traditional beamforming and independent component analysis.Beamforming boosts signals from a chosen direction while attenuating interference from others.
  • Feature integration: Combining monaural and spatial features provides a natural framework for integrating spectral and spatial information in deep-learning source separation.Later binaural systems used features such as IPD, ILD, mixing vectors, and enhanced representations alongside monaural inputs.
  • Binaural separation: Binaural DNN systems estimate ideal binary masks from interaural time and level differences, often combined with monaural features.The first DNN binaural study used ITD, ILD, and GFCC features for frequency-channel mask estimation.
  • Binaural separation: Binaural DNN separation generalizes to untrained spatial configurations and reverberation conditions, while monaural features particularly help when sources are co-located or nearby.The reported generalization is important because training data cannot enumerate all source and sensor configurations.
  • Beamforming: 5.05 WER on CHiME-3 real evaluation data represented a 13.34% relative improvement over the previous best when iterative ratio masking and beamforming refined one another.Adaptive local noise-covariance estimation and repeated mask-beamforming refinement were used in the reported system.
  • Beamforming: DNN mask estimation and clustering improved CHiME-3 results over RNN- or cGMM-generated masks, while another system outperformed monaural DNN and array NMF baselines for separation and ASR.The reviewed studies compared different mask-estimation and beamforming designs.

VII.DISCUSSION AND CONCLUSION

The overview argues that deep learning does not eliminate the value of domain-informed features and that time-domain methods offer an alternative to dominant time-frequency approaches. It also highlights target definition as a central conceptual issue in multi-speaker separation.

  • A. Features vs. Learning Machines: Feature extraction remains useful because domain knowledge can improve learning, and no learning algorithm achieves superior performance across all tasks.The authors use the no-free-lunch theorem and pitch as examples of why engineered cues can remain valuable.
  • A. Features vs. Learning Machines: Deep networks can learn useful features, but discovering known discriminative cues may be computationally inefficient compared with incorporating them directly.Pitch is identified as a salient cue for auditory organization and voiced-speech separation.
  • B. Time-frequency Domain vs. Time Domain: Time-domain separation provides an alternative to time-frequency processing and can potentially enhance magnitude and phase jointly through temporal mapping.The overview presents temporal mapping and end-to-end separation as emerging approaches.
  • C. What’s the Target?: The definition of an ideal mask presumes a known target source, which is straightforward for speech enhancement but problematic for multi-speaker separation.Speech enhancement treats speech as target and nonspeech as interference; selecting a target becomes ambiguous with multiple speakers.

D. What Does a Solution to the Cocktail Party Problem Look Like?

The paper questions how machine separation should be compared with human cocktail-party performance and proposes a listener-centered criterion. It concludes that current DNN enhancement meets this criterion only in limited conditions, leaving versatility as a major challenge.

  • Evaluation criterion: CASA defines a cocktail-party solution as human-level separation performance in all listening conditions, but comparing machine output with human perception is difficult.ASR and human intelligibility scores are possible comparison routes, though realistic-condition ASR remains imperfect.
  • Evaluation criterion: ASR-based evaluation is limited by dependence on the idiosyncrasies of the recognition system.The paper identifies this dependence as a drawback of using ASR scores to evaluate separation.
  • Proposed criterion: The paper proposes matching normal-hearing listeners’ speech intelligibility for hearing-impaired listeners in all listening situations as a concrete alternative criterion.This measure is narrower than CASA’s definition and directly linked to improving speech understanding for listeners with impaired hearing.
  • Current status: The reviewed DNN speech-enhancement system met the proposed criterion only in limited conditions, not across all listening situations.The paper identifies versatility as the primary challenge facing supervised speech separation research.
  • Human comparison: Human performance in noisy speech is associated with extensive exposure to varied or polyphonic auditory environments, including experience reflected in age and musicianship differences.The overview cites poorer speech-in-noise recognition in children and better noisy-speech perception in musicians than non-musicians.
Loading 1708.07524v2…