Source-linked AI summary
Glance and Gaze: A Collaborative Learning Framework for Single-channel Speech Enhancement
Andong Li, Chengshi Zheng, Lu Zhang, Xiaodong Li
TL;DR
Monaural speech enhancement must recover clean speech from a single noisy channel while handling difficult phase estimation. The paper proposes GaGNet, which combines multiscale feature recalibration with parallel magnitude suppression and complex-detail repair, iteratively refined across stages. It reports state-of-the-art performance on WSJ0-SI84 and DNS-Challenge, and competitive performance on Voicebank+Demand.
Problem
Monaural speech enhancement requires recovering target speech from noisy single-channel recordings, while phase estimation remains difficult and time-domain criteria may poorly match human ratings.
Method
GaGNet combines multiscale feature recalibration with stacked Glance-Gaze modules that separately suppress magnitude-domain noise and repair complex spectral detail through stage-wise refinement.
Results
The method achieves state-of-the-art performance on WSJ0-SI84 and DNS-Challenge, and competitive performance on Voicebank+Demand.
Takeaways & Limitations
The experiments support collaborative dual-path and multi-stage complex-spectrum estimation as an effective approach across three speech-enhancement datasets.
Abstract
from arXiv · showhide
The capability of the human to pay attention to both coarse and fine-grained regions has been applied to computer vision tasks. Motivated by that, we propose a collaborative learning framework in the complex domain for monaural noise suppression. The proposed system consists of two principal modules, namely spectral feature extraction module (FEM) and stacked glance-gaze modules (GGMs). In FEM, the UNet-block is introduced after each convolution layer, enabling the feature recalibration from multiple scales. In each GGM, we decompose the multi-target optimization in the complex spectrum into two sub-tasks. Specifically, the glance path aims to suppress the noise in the magnitude domain to obtain a coarse estimation, and meanwhile, the gaze path attempts to compensate for the lost spectral detail in the complex domain. The two paths work collaboratively and facilitate spectral estimation from complementary perspectives. Besides, by repeatedly unfolding the GGMs, the intermediate result can be iteratively refined across stages and lead to the ultimate estimation of the spectrum. The experiments are conducted on the WSJ0-SI84, DNS-Challenge dataset, and Voicebank+Demand dataset. Results show that the proposed approach achieves state-of-the-art performance over previous advanced systems on the WSJ0-SI84 and DNS-Challenge dataset, and meanwhile, competitive performance is achieved on the Voicebank+Demand corpus.
I. INTRODUCTION
Monaural speech enhancement seeks to recover clean speech from a single noisy recording, but phase estimation remains difficult and time-domain criteria may correlate poorly with human ratings. GaGNet addresses this with parallel coarse magnitude estimation and refined complex-domain detail recovery, followed by stage-wise refinement.
- Monaural speech enhancement extracts target speech from a noisy single-channel mixture to improve quality and intelligibility.
- Phase-aware enhancement is challenging because phase lacks clear structural regularity, despite its importance for perception at low SNRs.
- Time-domain loss functions may correlate poorly with human ratings, motivating continued complex-domain speech-enhancement research.
- Stacked modules progressively refine the spectrum through multi-stage optimization rather than mapping the task through one black box.
- GaGNet uses parallel paths for coarse magnitude recovery and refined complex spectral-detail reconstruction.
II. METHODLOGY
GaGNet combines spectral feature extraction with stacked Glance-Gaze modules. Its recalibrate encoder layers replace plain convolutions with multiscale UNet-style processing and residual feature recalibration.
- The proposed architecture contains a spectral feature extraction module and multiple stacked Glance-Gaze modules.The network takes a noisy complex spectrum as input and estimates the clean complex target.
- Plain convolutional encoders capture local context and can lose spectral information during repeated downsampling.
- Recalibrate encoder layers combine 2D-GLU, instance normalization, PReLU, and a residual UNet-Block to enlarge feature scale and mitigate information loss.
- The implementation uses four recalibrate encoder layers.
B. Glance-Focus Module
Each Glance-Gaze module separates coarse magnitude suppression from complex-domain detail repair, then combines both outputs with the previous estimate. Repeated modules update the real-imaginary spectrum stage by stage.
- The Glance Block produces a magnitude-domain gain for coarse noise suppression, while the Gaze Block predicts a complex-domain residual for spectral-detail repair.
- Each module receives extracted features and the real-imaginary estimate from the previous stage, with the original input used at the first stage.
- The Glance Block uses single-path temporal convolutional modules and a sigmoid mask, whereas the Gaze Block uses dual-path modules for real-imaginary spectral details.
- The two block outputs and the previous estimate enter the collaborative reconstruction module to update the current stage's real-imaginary components.
- The reconstruction decouples magnitude and phase, filters magnitude with a gain, and adds a complex residual to restore spectral detail.
C. Loss Function
The loss supervises intermediate complex-spectrum estimates with weighted multi-target MSE, while adding a magnitude constraint and learning gain and residual jointly.
- C. Loss Function: Each GGM stage generates an estimated RI spectrum supervised by a weighted multi-target MSE loss.The final stage receives weight λQ = 1.0, while earlier stages receive λq = 0.1 for q ≠ Q.
- C. Loss Function: The loss combines RI reconstruction with a magnitude constraint intended to improve speech quality.
- C. Loss Function: Gain and residual are supervised only through their sum, allowing them to be learned adaptively in an end-to-end process.
A. Dataset
The WSJ0-SI84 evaluation uses speaker-disjoint utterances mixed with DNS noise at challenging signal-to-noise ratios, with separate training, validation, and testing sets.
- A. Dataset: WSJ0-SI84 contains 7,138 clean utterances from 83 speakers, with 5,428 training and 957 validation utterances drawn from 77 speakers.The test set contains 150 utterances from 6 untrained speakers.
- A. Dataset: Around 150,000 training and 10,000 validation noisy-clean pairs are generated, corresponding to approximately 300 hours of data.Noise is mixed with speech at randomly selected SNRs from -5dB to 0dB.
- A. Dataset: Evaluation uses four SNR conditions: −3dB, 0dB, 3dB, and 6dB, with 300 noisy-clean pairs generated for each condition.
B. Baseline
The study compares GaGNet with advanced complex-domain and time-domain speech-enhancement systems, while specifying the principal architectural settings used in its modules.
- B. Baseline: Five baselines are evaluated: GCRN, DCCRN, PHASEN, AECNN, and ConvTasNet.The first three investigate complex-domain recovery, whereas AECNN and ConvTasNet operate on raw waveforms.
- B. Baseline: GCRN, DCCRN, and PHASEN recover magnitude and phase, while AECNN and ConvTasNet use time-domain inputs and targets.
- B. Baseline: The REL uses 2D-GLU kernel size (2, 3), stride (1, 2), and 64 channels, while its UNet-Block uses kernel size (1, 3) and stride (1, 2).The GGM sets compressed feature size D to 256 after 1D-GLU.
2) Training Details:
Training uses fixed waveform preprocessing and optimization settings, alongside ablations of stage depth, reconstruction design, and parameter trade-offs.
- 2) Training Details:: Utterances are sampled at 16kHz, chunked into 8-second segments, and transformed with a 20ms Hanning window, 50% overlap, and 320-point FFT.The FFT produces 161-dimensional features, and magnitude power compression uses β = 0.5 while preserving phase.
- 2) Training Details:: All models use Adam with learning rate 0.0005, batch size 8, and 60 training epochs.
- 2) Training Details:: The ablation study samples approximately 50,000 training pairs.
- 2) Training Details:: Table II compares spectral reconstruction algorithms, including magnitude-only, complex-from-scratch, and phase-focused variants.
- 2) Training Details:: Increasing TCM groups P or GGM stages Q improves PESQ, ESTOI, and SDR, with (P, Q) = (2, 3) selected to balance performance and parameter burden.The comparison values are averaged across different testing SNRs, and Table I marks the best result in each case in bold.
2) The effect of CRM:
The experiments compare reconstruction variants, baselines, computational cost, and subjective quality. Collaborative magnitude–complex reconstruction with iterative refinement achieves stronger objective and subjective results while retaining lower overall statistics than other time-frequency models.
- Mag-RM performs worst because it ignores phase information, which heavily hampers speech quality.
- Com-RM outperforms Phasen-RM because it jointly couples magnitude and phase in the RI format rather than predicting phase independently.
- GaGNet improves over GaGNet† by 0.11 PESQ, 2.12% ESTOI, and 0.62dB SDR through UNet-Block-based multiscale feature exploitation and recalibration.
- Compared with DCCRN, GaGNet gains 0.21 PESQ, 7.37% ESTOI, and 1.90dB SDR using complementary glance–gaze reconstruction and iterative spectrum refinement.
- The proposed system has overall lower parameters, MACs, processing time, and memory statistics than other time-frequency-domain peers.
- DNSMOS evaluation on 750 noisy utterances gives the proposed approach the highest score, supporting superior subjective speech-quality perception.
C. Comparison on other benchmarks
The study evaluates the system on DNS-Challenge and Voicebank+Demand benchmarks using separate datasets, test sets, and objective metrics.
- DNS-Challenge uses around 3000 hours of noisy-clean training pairs and a 150-pair non-blind test set evaluated with WB-PESQ, PESQ, STOI, and SI-SDR.
- Voicebank+Demand uses 11,572 training pairs and 824 testing pairs, evaluated with WB-PESQ, CSIG, CBAK, and COVL.
V. CONCLUSION
The conclusion presents a complex-domain collaborative framework that separates coarse magnitude estimation from spectral-detail repair and progressively refines the target across stages. Experiments on three datasets show promising performance.
- The framework decouples complex-spectrum optimization into a magnitude-domain glance path and a complex-domain gaze path for complementary target recovery.
- Multi-stage training progressively optimizes the target by repeatedly refining the spectrum.
- Experiments on WSJ0-SI84, DNS-Challenge, and Voicebank+Demand show promising performance across three datasets.