PaperlayerView exact revision on arXiv

Source-linked AI summary

TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation

Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, Shinji Watanabe

arXiv:2211.12433v2cs.SDeess.AS

TL;DR

Speech separation needs effective modeling of complex time-frequency structure across anechoic, noisy, reverberant, and multichannel conditions. The paper introduces TF-GridNet with integrated full- and sub-band modeling and combines it with complex spectral mapping and multi-frame Wiener filtering. It reports state-of-the-art performance across the evaluated separation, dereverberation, and enhancement tasks, including 23.5 dB SI-SDRi on WSJ0-2mix.

  • Problem

    Complex time-frequency modeling had been under-explored for anechoic speaker separation, while the preliminary TF-GridNet study lacked noisy-reverberant evaluation and a multichannel extension.

  • Method

    TF-GridNet integrates full-band, sub-band temporal, and cross-frame modeling in the complex T-F domain, with multi-microphone extension through a two-DNN system and multi-frame Wiener filtering.

  • Results

    The proposed systems achieve state-of-the-art performance across four major separation, dereverberation, and enhancement tasks, including 23.5 dB SI-SDRi on WSJ0-2mix.

  • Takeaways & Limitations

    Integrated full- and sub-band modeling supports effective single- and multichannel speech separation in the evaluated conditions.

Abstract

from arXiv · show

We propose TF-GridNet for speech separation. The model is a novel deep neural network (DNN) integrating full- and sub-band modeling in the time-frequency (T-F) domain. It stacks several blocks, each consisting of an intra-frame full-band module, a sub-band temporal module, and a cross-frame self-attention module. It is trained to perform complex spectral mapping, where the real and imaginary (RI) components of input signals are stacked as features to predict target RI components. We first evaluate it on monaural anechoic speaker separation. Without using data augmentation and dynamic mixing, it obtains a state-of-the-art 23.5 dB improvement in scale-invariant signal-to-distortion ratio (SI-SDR) on WSJ0-2mix, a standard dataset for two-speaker separation. To show its robustness to noise and reverberation, we evaluate it on monaural reverberant speaker separation using the SMS-WSJ dataset and on noisy-reverberant speaker separation using WHAMR!, and obtain state-of-the-art performance on both datasets. We then extend TF-GridNet to multi-microphone conditions through multi-microphone complex spectral mapping, and integrate it into a two-DNN system with a beamformer in between (named as MISO-BF-MISO in earlier studies), where the beamformer proposed in this paper is a novel multi-frame Wiener filter computed based on the outputs of the first DNN. State-of-the-art performance is obtained on the multi-channel tasks of SMS-WSJ and WHAMR!. Besides speaker separation, we apply the proposed algorithms to speech dereverberation and noisy-reverberant speech enhancement. State-of-the-art performance is obtained on a dereverberation dataset and on the dataset of the recent L3DAS22 multi-channel speech enhancement challenge.

I. INTRODUCTION

The paper develops TF-GridNet to address limitations of complex time-frequency speaker-separation models and extends the approach across reverberant, noisy, and multichannel settings. It combines complex spectral mapping, full- and sub-band modeling, and beamforming-based processing, achieving strong benchmark results.

  • Proposed approach: Complex spectral mapping predicts target real and imaginary components from input components, a direction previously unexplored for monaural anechoic speaker separation.The paper presents this as the first study of complex spectral mapping for monaural, anechoic speaker separation.
  • Proposed approach: TF-GridNet models complex T-F spectrograms with a novel architecture that adds cross-frame self-attention to capture global information across frames.The architecture is based on an improved TFPSNet and models speech spectrograms in a grid-like manner.
  • Results: 23.5 dB SI-SDRi on WSJ0-2mix exceeds the previous best result of 22.1 dB without data augmentation or dynamic mixing.The reported 23.5 dB result is obtained by the best model using the proposed mixture-constraint loss term.
  • Evaluation scope: The study addresses gaps in the preliminary work by evaluating monaural reverberant and noisy-reverberant separation and extending TF-GridNet to multichannel conditions.The multichannel extension uses a MISO-BF-MISO system with a beamformer between two multi-channel-input single-channel-output DNNs.
  • Results: State-of-the-art performance is reported across reverberant separation, noisy-reverberant separation, dereverberation, and noisy-reverberant enhancement tasks.The reported tasks span both single-channel and multichannel separation settings.

B. Approach Outline

TF-GridNet uses a two-DNN complex T-F-domain system in which DNN1 estimates sources, a multi-frame Wiener filter refines them, and DNN2 performs post-filtering. Its stacked blocks combine full-band, sub-band, and cross-frame processing, with LN-Unfold reducing parameters and memory while slightly improving separation.

  • System architecture: DNN1 estimates each target source, a multi-frame Wiener filter performs sub-band linear filtering, and DNN2 post-filters the mixture and intermediate outputs.
  • System architecture: Both DNNs perform complex spectral mapping by stacking input real and imaginary components to predict each speaker’s target components.
  • TF-GridNet blocks: Each TF-GridNet block contains intra-frame full-band, sub-band temporal, and cross-frame self-attention modules that progressively transform T-F embeddings.
  • Intra-frame full-band module: The intra-frame module models full-band spectral and spatial information within each frame using frequency-wise stacked embeddings and a BLSTM.
  • Intra-frame full-band module: LN-Unfold uses fewer parameters and less memory than Unfold-LN when I/J > 1, while producing slightly better separation.

B. Sub-Band Temporal Module

The sub-band temporal module models each frequency independently across time with shared BLSTM parameters. This design is motivated by frequency-local spatial and reverberation patterns and exchanges information iteratively with full-band modules.

  • Module operation: The sub-band temporal module views the tensor as frequency-specific sequences and uses one parameter-shared BLSTM to model temporal information within each frequency.
  • Motivation: In multi-channel conditions, sub-band modeling leverages inter-microphone phase patterns that remain stable over time but vary with frequency.
  • Motivation: The full-band module complements sub-band processing by modeling harmonic structure and gradual frequency changes in inter-microphone phase patterns.
  • Motivation: Frequency-specific sub-band modules are also motivated by reverberation times and patterns that vary across frequency.
  • Relation to prior work: Unlike earlier approaches that omit full-band modeling or use one-way sequencing, stacked TF-GridNet blocks enable iterative information flow between full- and sub-band modules.

D. Cross-Frame Self-Attention Module

Cross-frame self-attention gives each time frame direct access to long-range utterance context, complementing the multi-step information flow of full-band and sub-band BLSTMs. Its frame-level formulation substantially lowers attention-memory requirements relative to TFPSNet.

  • Module operation: Cross-frame self-attention forms frame embeddings from T-F units and applies full-utterance attention so frames can directly access relevant long-range context.
  • Module operation: The module computes multi-head query, key, and value representations from T-F embeddings, aggregates attention outputs, and adds them through a residual connection.
  • Efficiency: The attention matrices require O(B × L × T^2) memory, compared with TFPSNet’s O(B × L × F × T^2) + O(B × L × T × F^2).
  • SI-SDR loss: The SI-SDR-SE loss scales the estimate rather than the source and achieves similar performance with faster convergence than the alternative scaling convention.
  • SI-SDR loss: The mixture-constraint loss penalizes disagreement between summed target sources and summed scaled estimates, without enforcing equality at runtime.
  • SI-SDR loss: The combined loss omits a weighting term because the mixture-constraint term falls below 0.01 at convergence, leaving SI-SDR as the dominant objective.

2) Wav+Mag Loss:

Wav+Mag combines time-domain and magnitude-domain losses on re-synthesized predictions. The magnitude term can improve perceptual and intelligibility-oriented metrics, while the two components are similarly scaled.

  • Loss definition: Wav+Mag defines loss on the re-synthesized signal and its magnitude, combining time-domain and frequency-domain objectives.
  • Metric trade-off: Magnitude loss can improve PESQ, STOI, and WER while degrading time-domain metrics such as SI-SDR.
  • Loss scaling: The time-domain and frequency-domain losses use no weighting term because Parseval’s theorem places them on similar scales.
  • System context: The paper presents its loss design alongside a novel DNN-supported beamformer connecting integrated sub-band and full-band modeling.

A. DNN-Supported Multi-Frame Wiener Filter

The paper develops a DNN-supported multi-frame Wiener filter for multi-microphone separation and relates it to full- and sub-band modeling. The filter uses nearby time-frequency frames and DNN-estimated target speech, while TF-GridNet supplies sub-band temporal modeling and can be followed by beamforming and post-filtering.

  • DNN-Supported Multi-Frame Wiener Filter: A multi-frame Wiener filter is computed from DNN1 outputs to support beamforming between two DNNs.The filter is time-invariant per frequency and uses estimated target speech from the first DNN.
  • DNN-Supported Multi-Frame Wiener Filter: Nearby past and future frames are stacked as filter input, with Δl and Δr controlling the temporal context.When both are zero, the method reduces to a single-frame Wiener filter; otherwise it is multi-frame.
  • DNN-Supported Multi-Frame Wiener Filter: MCMFWF uses multi-microphone complex spectral mapping outputs containing DNN-estimated magnitude and phase.This differs from an earlier system that used monaural real-valued magnitude masking and mixture phase.
  • DNN-Supported Multi-Frame Wiener Filter: In monaural conditions, SCMFWF exploits correlations among nearby frames to reduce reverberation, including correlations with future frames.The paper contrasts this with an approach filtering only past frames.
  • Discussion on Beamforming and Sub-Band Modelling: TF-GridNet models each frequency with a shared BLSTM sub-band temporal module, while a second TF-GridNet can post-filter MCMFWF output.This integrated system improves over using one TF-GridNet alone, although less than two DNNs using full-band modeling.
  • Discussion on Beamforming and Sub-Band Modelling: STFT enables per-frequency modeling and access to intra- and cross-frequency information, unlike the usually non-narrow-band learned bases of time-domain models.The paper presents this frequency structure as an advantage of STFT-domain approaches.

A. Setup for Monaural, Anechoic Speaker Separation

The experiments cover monaural, multichannel, reverberant, noisy-reverberant, dereverberation, and enhancement settings using simulated or public datasets. Evaluation targets and conditions vary by task, with direct-path speech used in several noisy-reverberant setups.

  • Monaural, Anechoic Speaker Separation: WSJ0-2mix benchmarks monaural two-speaker separation in anechoic conditions with fully overlapped mixtures at 8 kHz.Its mixtures use separate speakers across training and testing and relative energy levels from −5 to 5 dB.
  • Reverberant Speaker Separation: SMS-WSJ evaluates two-speaker separation in reverberant conditions with six microphones arranged on a 20 cm diameter circle.Speaker-to-array distance ranges from 1.0 to 2.0 m and T60 from 0.2 to 0.5 s.
  • Noisy-Reverberant Processing: The systems perform joint denoising, dereverberation, and separation in one-, two-, and six-channel tasks, using direct-path signals as training targets.For two-channel processing, microphones 1 and 4 are used; microphone 1 is the reference.
  • Noisy-Reverberant Speaker Separation: WHAMR! evaluates noisy-reverberant speaker separation by reverberating WSJ0-2mix sources and adding non-stationary noise.Its simulated conditions include T60 values from 0.2 to 1.0 s and SNR from −6 to 3 dB.
  • Speech Dereverberation: Speech dereverberation uses a simulated noisy-reverberant dataset because no well-designed popular dereverberation dataset was available.The authors note that REVERB training data is based on only 24 eight-channel room impulse responses.
  • Noisy-Reverberant Speech Enhancement: L3DAS22 evaluates reconstruction of dry speech from far-field mixtures formed with two four-channel Ambisonic arrays in a noisy-reverberant office environment.The task uses real room impulse responses and fixed array geometry.

F. Baselines

The baseline evaluation compares MFWF with established linear filters and uses published strong systems as external reference points. The proposed MCMFWF is presented as simpler to compute than a convolutional beamformer.

  • Baselines: MFWF is compared with convolutional beamformers for multichannel processing and WPE filters for monaural processing.These filters replace the MFWF module between DNN1 and DNN2 in the comparison system.
  • Baselines: Published results from competing approaches and prior MISO-BF-MISO studies provide reference points for system-level comparisons.The prior studies used TCN-DenseUNet and other sub-band linear filters.
  • Baseline for MCMFWF: Convolutional beamformer inputs stack current and delayed past multichannel frames, with prediction delay and filter taps defining the context.The filter coefficients map these stacked observations to the current-frame output.
  • Baseline for MCMFWF: The convolutional beamformer uses DNN-estimated target speech to compute target power spectral density and a principal-eigenvector steering estimate.The resulting filter output is then used as the beamforming result.
  • Baseline for MCMFWF: The DNN-supported MCMFWF is simpler to compute than the convolutional beamformer.This is stated as a computational distinction between the proposed filter and the multichannel baseline.

3) Baseline for SCMFWF:

The monaural SCMFWF baseline is implemented within an evaluation setup spanning multiple tasks, metrics, and model comparisons. The experiments also examine computational cost and compare masking and mapping configurations on WSJ0-2mix.

  • Baseline for SCMFWF: In the single-microphone case, the convolutional beamformer becomes a WPE filter driven by DNN1’s estimated target magnitude.The WPE filter uses a prediction delay and past-frame taps to produce its output.
  • Miscellaneous Setup: Default STFT settings use a 32 ms window, 8 ms hop, and 257-dimensional complex spectra at 16 kHz or 129-dimensional spectra at 8 kHz.These dimensions come from 512-point and 256-point DFTs, respectively.
  • Miscellaneous Setup: MFWF filter contexts vary by channel count, using larger past and future contexts as the number of channels decreases.The specified contexts are 4/3 for eight channels, 5/4 for six, 15/14 for two, and 20/19 for one channel.
  • Miscellaneous Setup: Training samples are four-second mixture segments normalized to unit variance and optimized with Adam and gradient clipping.The learning rate starts at 0.001 and is halved after three epochs without validation improvement.
  • Miscellaneous Setup: The experiments omit dynamic mixing and data augmentation and include computational comparisons across TF-GridNet configurations.The paper evaluates separation effectiveness first, then computational requirements and WSJ0-2mix performance.
  • Evaluation Metrics: The study reports SI-SDR, SDR, PESQ, STOI, eSTOI, and WER according to task requirements.These metrics assess signal quality, speech quality, intelligibility, and recognition performance.
  • Ablation Results: Table III studies model hyperparameters, including embedding dimension, kernel size, Unfold and Deconv1D use, and the mixture-consistency loss.The ablation identifies configurations that reduce memory without decreasing SI-SDRi relative to a larger embedding setup.
  • Comparison with DPRNN: 21.2 vs. 18.8 dB: complex spectral mapping outperforms the compared masking configuration under matched WSJ0-2mix model settings.The comparison uses TF-GridNet and DPRNN configurations with closely matched parameter counts and computation.

2) Ablation Results with Different Hyper-Parameters:

The ablations show that larger models, full-band self-attention, and the chosen normalization and embedding configurations improve WSJ0-2mix separation. The broader evaluations also report strong monaural and multichannel performance, including without dynamic mixing.

  • 2) Ablation Results with Different Hyper-Parameters:: 23.5 dB SI-SDRi is obtained after enlarging the model, following gains from LN+Unfold and other hyper-parameter changes.LN+Unfold improves SI-SDRi from 23.0 to 23.2 dB, and further model enlargement reaches 23.5 dB.
  • Comparison in Single-DNN Setup:: TF-GridNet substantially outperforms competing single-DNN systems on monaural SMS-WSJ and WHAMR! separation.On SMS-WSJ, it achieves 15.7 dB SI-SDR versus 6.5 dB for DPRNN-TasNet and 5.7 dB for SISO1.
  • Comparison in Single-DNN Setup:: 10.6 dB SI-SDR is achieved on monaural WHAMR! without dynamic mixing, exceeding previous dynamic-mixing results.Dynamic mixing provides only slight improvements for previous models, whose final performance remains below TF-GridNet.

3) Comparison in Multi-Channel, Single-DNN Setup:

In multichannel settings, TF-GridNet outperforms competing single-DNN approaches across SMS-WSJ and WHAMR!, while beamforming and post-filtering provide additional gains. The same complex spectral-mapping approach also performs strongly for dereverberation and enhancement.

  • 3) Comparison in Multi-Channel, Single-DNN Setup:: 19.9 dB SI-SDR is obtained by TF-GridNet on six-channel SMS-WSJ, versus 8.6, 10.8, 10.2, and 13.2 dB for four competing systems.The comparison is against FasNet+TAC, MC-Conv-TasNet, TCN-DenseUNet, and LBT, respectively.
  • 3) Comparison in Multi-Channel, Single-DNN Setup:: Using MCMFWF between DNN1 and DNN2 produces clear improvements over a single TF-GridNet and over stacking two TF-GridNets without beamforming.Both past and future context outperform past-only or no-context MCMFWF and a convolutional beamformer.
  • C. Results on WSJ0CAM-DEREVERB: 16.6 dB SI-SDR on one-channel dereverberation and 19.9 dB on eight-channel dereverberation exceed the corresponding SISO1 and MISO1 results.The comparisons are 16.6 versus 8.4 dB and 19.9 versus 11.3 dB, respectively.
  • C. Results on WSJ0CAM-DEREVERB: 21.2 dB SI-SDR with beamforming and post-filtering exceeds the competing 18.2 dB result on eight-channel dereverberation.The competing system uses two TCN-DenseUNets with a composition of linear filters.
  • D. Results on L3DAS22: A single TF-GridNet outperforms the winning solution and the other 16 submissions on the L3DAS22 3D speech enhancement task.Adding beamforming and post-filtering yields further improvement.

E. Computation Cost vs. Separation Performance

TF-GridNet offers a configurable trade-off between computation and separation performance. Reduced-cost configurations retain competitive accuracy and can outperform comparable T-F models at similar computational budgets.

  • E. Computation Cost vs. Separation Performance: 231.1 GMAC/s is required by the configuration obtaining 23.5 dB SI-SDRi, while reducing model dimensions lowers cost to 131.1 GMAC/s with 23.2 dB.The lower-cost configuration reduces BLSTM hidden units from 256 to 196 and embedding dimension from 64 to 48.
  • E. Computation Cost vs. Separation Performance: 66.0 GMAC/s is reached by reducing the window size from 32 to 16 ms, nearly halving computation as the number of frequencies is reduced.The passage states that this configuration reduces computation by almost half.
  • E. Computation Cost vs. Separation Performance: 22.0 dB SI-SDRi at 29.8 GMAC/s exceeds TFPSNet’s 21.1 dB at 29.6 GMAC/s, while using less memory and faster CPU inference.The comparison uses similar computation cost.
  • E. Computation Cost vs. Separation Performance: TF-GridNet can be configured to use a reasonable amount of computation while achieving reasonable separation performance.Training-time measurements show that configurations beyond the first two have reasonable epoch times.
  • VII. CONCLUSION: Real-time, online speech separation remains a stated direction for future work.The conclusion points to extending TF-GridNet for real-time, online operation.
Loading 2211.12433v2…