Source-linked AI summary
TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separation
Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, Shinji Watanabe
TL;DR
Monaural anechoic speaker separation has been dominated by time-domain models, leaving complex time-frequency approaches comparatively under-explored. The paper proposes TF-GridNet, a multi-path complex spectral-mapping network with a mixture-related loss, and reports 23.4 dB SI-SDRi on WSJ0-2mix without dynamic mixing.
Problem
Complex time-frequency models have been under-explored and under-represented for anechoic speaker separation compared with dominant time-domain approaches.
Method
TF-GridNet operates in the complex time-frequency domain with multi-path processing, full-band self-attention, complex spectral mapping, and a loss term encouraging estimated sources to sum toward the mixture.
Results
23.4 dB SI-SDRi on WSJ0-2mix surpasses the previous best of 22.1 dB without data augmentation or dynamic mixing.
Takeaways & Limitations
The findings indicate that complex time-frequency models are not sub-optimal compared with time-domain approaches for anechoic speaker separation.
Takeaways & Limitations
The mixture-constraint loss does not strictly enforce the separated sources to sum to the mixture at run time.
Abstract
from arXiv · showhide
We propose TF-GridNet, a novel multi-path deep neural network (DNN) operating in the time-frequency (T-F) domain, for monaural talker-independent speaker separation in anechoic conditions. The model stacks several multi-path blocks, each consisting of an intra-frame spectral module, a sub-band temporal module, and a full-band self-attention module, to leverage local and global spectro-temporal information for separation. The model is trained to perform complex spectral mapping, where the real and imaginary (RI) components of the input mixture are stacked as input features to predict target RI components. Besides using the scale-invariant signal-to-distortion ratio (SI-SDR) loss for model training, we include a novel loss term to encourage separated sources to add up to the input mixture. Without using dynamic mixing, we obtain 23.4 dB SI-SDR improvement (SI-SDRi) on the WSJ0-2mix dataset, outperforming the previous best by a large margin.
1. INTRODUCTION
The paper addresses the under-explored use of complex time-frequency models for anechoic monaural speaker separation by introducing TF-GridNet. It combines complex spectral mapping with a multi-path architecture and a mixture-related loss, achieving a strong WSJ0-2mix result.
- Complex time-frequency models have been under-explored for anechoic speaker separation, while TasNet variants have become the dominant approach.
- TFPSNet reached 21.1 dB SI-SDRi on WSJ0-2mix, within the 20.0–22.1 dB range commonly reached by modern time-domain models.
- TF-GridNet introduces complex spectral mapping for anechoic speaker separation, a direction the paper identifies as previously unstudied in this setting.
- The model adds full-band self-attention to dual-path processing, forming a multi-path architecture that captures cross-frame global information.
- 23.4 dB SI-SDRi on WSJ0-2mix surpasses the previous best of 22.1 dB without data augmentation or dynamic mixing.
2. PROPOSED ALGORITHMS
TF-GridNet models anechoic monaural speaker separation in the complex time-frequency domain, combining local spectral and temporal processing with global frame-level self-attention. It predicts source RI components and trains with SI-SDR-related objectives augmented by a mixture constraint.
- Complex spectral mapping: The mixture is represented in the STFT domain, and the model predicts each source from the mixture's stacked real and imaginary components.The predicted RI components are re-synthesized with inverse STFT for time-domain loss computation.
- TF-GridNet architecture: TF-GridNet applies an initial Conv2D and normalization, then stacks blocks that refine D-dimensional embeddings over time-frequency units.Each block contains spectral, temporal, and self-attention processing before a Deconv2D predicts 2C output channels for the source RI components.
- Intra-frame spectral module: Each intra-frame spectral module uses a BLSTM over neighboring frequency embeddings, with unfolding, normalization, deconvolution, and a residual connection.The stride can exceed one to reduce sequence length and computation.
- Sub-band temporal module: Each sub-band temporal module applies an analogous BLSTM procedure across time within each frequency sub-band.Its output is a D × T × F tensor passed to the full-band self-attention module.
- Full-band self-attention module: The full-band self-attention module forms frame-level embeddings so each frame can attend to any other frame and capture long-range information.It uses multi-head attention with residual aggregation; its attention-matrix memory cost is O(B × L × T^2).
- Loss functions: The training objective uses utterance-level PIT with an SI-SDR formulation, plus a mixture-constraint term encouraging scaled estimated sources to approach the mixture.The constraint improves separation experimentally, while the summed scaled estimates do not equal the mixture at run time.
3. EXPERIMENTAL SETUP
The experiments evaluate TF-GridNet on WSJ0-2mix, using SI-SDRi and SDRi alongside dataset, STFT, and model-comparison details.
- Dataset: WSJ0-2mix benchmarks monaural talker-independent speaker separation in anechoic conditions with separate training, validation, and test speakers.The dataset contains 20,000 training, 5,000 validation, and 3,000 test two-speaker mixtures.
- Configuration: The system uses 32 ms STFT windows, 8 ms hops, 129-dimensional complex spectra, six TF-GridNet blocks, and four frame-level embedding groups.Training samples are four-second mixture segments.
- Comparisons: Table 3 compares masking- and mapping-based systems on WSJ0-2mix.
- Metrics: SI-SDRi and SDRi are the evaluation metrics, with mixture SI-SDR of 0 dB and mixture SDR of 0.2 dB.Model parameters are reported in millions.
4. EVALUATION RESULTS
The evaluation compares similarly sized models and reports that TF-domain modeling can outperform modern time-domain systems, while ablations examine configuration and memory trade-offs.
- Main results: DPRNN, TFPSNet, and TF-GridNet are compared with nearly equal parameter counts and computational costs.The comparison standardizes embedding and BLSTM hidden dimensions.
- Ablations: Using Unfold and Deconv1D with embedding dimension D = 16 does not degrade performance relative to D = 128 without nearby T-F stacking.The smaller-embedding configuration noticeably reduces memory consumption when the kernel size is sufficiently large.
- Ablations: Table 4 reports ablation SI-SDRi results in decibels for different WSJ0-2mix model configurations.
- Main results: TF-GridNet surpasses the common modern time-domain SI-SDRi range of 20.0–22.1 dB on WSJ0-2mix by a large margin.The cited evaluation context identifies TF-domain models as previously under-explored for anechoic separation.
5. CONCLUSIONS
The paper concludes that TF-GridNet makes complex T-F-domain modeling highly competitive for anechoic monaural separation, while WSJ0-2mix performance is nearing saturation.
- Conclusion: TF-GridNet combines complex spectral mapping with a mixture-constrained time-domain loss to achieve 23.4 dB SI-SDRi on WSJ0-2mix without dynamic mixing.The paper describes this as state-of-the-art performance.
- Future work: The paper identifies noisy-reverberant and multi-channel separation, along with more advanced DNN architectures, as follow-up directions.
- Conclusion: Complex T-F models that predict target RI components are not sub-optimal relative to time-domain approaches for anechoic speaker separation.The paper attributes earlier performance differences mainly to DNN architecture rather than over-complete learned filterbanks with very short windows.