Source-linked AI summary
FullSubNet+: Channel Attention FullSubNet with Complex Spectrograms for Speech Enhancement
Jun Chen, Zilin Wang, Deyi Tuo, Zhiyong Wu, Shiyin Kang, Helen Meng
TL;DR
FullSubNet has limitations in frequency-band processing and input-output information matching. FullSubNet+ addresses them with multi-scale channel attention, magnitude/real/imaginary inputs, and efficient TCN-based full-band extractors, achieving superior DNS Challenge performance while reducing computational cost and meeting real-time processing requirements.
Problem
FullSubNet has coarse frequency-band processing and predicts cIRM from magnitude-only input, creating an input-output information mismatch involving phase.
Method
FullSubNet+ uses MulCA, parallel extractors for magnitude, real, and imaginary spectrograms, and stacked TCN blocks replacing full-band LSTMs.
Results
FullSubNet+ outperforms FullSubNet and other state-of-the-art methods on DNS Challenge noise-reduction tasks, with the full-band extractor reducing parameters by 1.64 M and CPU inference becoming as much as 80% faster.
Takeaways & Limitations
FullSubNet+ provides a more computationally efficient real-time enhancement framework that uses frequency-band attention and noisy-speech phase information.
Takeaways & Limitations
The comparison uses the same experimental settings and cIRM learning target for the evaluated models.
Abstract
from arXiv · showhide
Previously proposed FullSubNet has achieved outstanding performance in Deep Noise Suppression (DNS) Challenge and attracted much attention. However, it still encounters issues such as input-output mismatch and coarse processing for frequency bands. In this paper, we propose an extended single-channel real-time speech enhancement framework called FullSubNet+ with following significant improvements. First, we design a lightweight multi-scale time sensitive channel attention (MulCA) module which adopts multi-scale convolution and channel attention mechanism to help the network focus on more discriminative frequency bands for noise reduction. Then, to make full use of the phase information in noisy speech, our model takes all the magnitude, real and imaginary spectrograms as inputs. Moreover, by replacing the long short-term memory (LSTM) layers in original full-band model with stacked temporal convolutional network (TCN) blocks, we design a more efficient full-band module called full-band extractor. The experimental results in DNS Challenge dataset show the superior performance of our FullSubNet+, which reaches the state-of-the-art (SOTA) performance and outperforms other existing speech enhancement approaches.
1. INTRODUCTION
Frequency-domain speech enhancement dominates state-of-the-art methods because spectral features are more distinguishable for learning. FullSubNet+ addresses FullSubNet’s frequency-band discrimination and input-output information mismatch by weighting frequency bands and incorporating phase-related spectrogram information.
- Frequency-domain approaches dominate state-of-the-art speech enhancement because frequency-domain signals provide more distinguishable features for model learning.
- FullSubNet combines a full-band model for global spectral information and cross-band dependencies with a sub-band model for local patterns and signal stationarity.
- FullSubNet’s whole-frequency-band input degrades discrimination among frequency bands, while magnitude-only input mismatches its cIRM prediction target by omitting phase information.
- FullSubNet+ introduces MulCA, TCN-based full-band extractors, and separate processing of real and imaginary spectrograms to address these limitations.The framework applies channel attention to frequency bands and uses additional extractors for phase-related information.
2. METHODOLOGY
FullSubNet+ processes magnitude, real, and imaginary spectrograms through parallel attention and full-band extraction before sub-band prediction. Its MulCA module weights frequency bands using multi-scale temporal features, while stacked TCN blocks provide efficient global-context extraction.
- FullSubNet+ targets denoising and speech recovery in the STFT domain using three parallel MulCA modules, full-band extractors, and a sub-band model.
- Magnitude, real, and imaginary spectrograms are independently weighted and processed by parallel extractors before their embeddings are concatenated with magnitude-based sub-band units.
- Multi-scale Time Sensitive Channel Attention: MulCA treats frequency bins as channels and assigns them different weights so the model can emphasize bands that matter more for noise reduction.
- Multi-scale Time Sensitive Channel Attention: Multi-scale depthwise convolutions extract small-, medium-, and large-time-scale features for computing frequency-bin attention weights.
- Full-band Extractor: Stacked TCN blocks replace full-band LSTMs to reduce model size and computation while capturing long-range dependencies through exponentially increasing dilation.
- Sub-band Model: The sub-band model uses two unidirectional LSTM layers and a fully connected layer to predict cIRM from weighted local units and full-band embeddings.
3. EXPERIMENTS
Experiments evaluate FullSubNet+ on DNS Challenge data against FullSubNet and other methods, while ablations isolate MulCA and phase-information effects. The proposed changes improve enhancement performance and computational efficiency, including faster processing than FullSubNet.
- Experimental setup: FullSubNet+ was trained and evaluated on a subset of the Interspeech 2021 DNS Challenge dataset.The setup used dynamic mixing to generate noisy speech during training.
- Experimental setup: FullSubNet served as the baseline, using full-band and sub-band models with stacked LSTM layers and cIRM targets for fair comparison.FullSubNet+1 used 8.67 M parameters under the reported configuration.
- Comparison with baseline and SOTA methods: FullSubNet+ significantly improved over FullSubNet on all reported DNS Challenge metrics, with gains on both reverberant and non-reverberant test sets.The comparison metrics were WB-PESQ, NB-PESQ, STOI, and SI-SDR.
- Comparison with baseline and SOTA methods: FullSubNet+ showed superior performance to current SOTA methods, with especially prominent improvement on the reverberant noise-reduction task.The authors attribute the result to improved frequency-band attention and extraction of real and imaginary spectrogram information.
- The effect of the MulCA: The MulCA ablation found gains from weighting either full-band or sub-band inputs, with the best performance when both were weighted.The improvement required only a small increase in parameter count.
- The effect of using phase information: Phase-information ablations improved performance when real and imaginary spectrogram branches were added, and the combination with MulCA produced the best results.Replacing LSTMs with TCN blocks reduced full-band-extractor parameters by 1.64 M; GPU speed increased nearly 18%, while CPU inference reached 18.33 ms per 32 ms STFT frame.
4. CONCLUSIONS
FullSubNet+ is a single-channel real-time speech enhancement framework that combines MulCA with TCN-based full-band extractors for magnitude, real, and imaginary spectrograms. Ablation studies and DNS Challenge comparisons demonstrate its effectiveness and superior performance over other speech enhancement methods.
- FullSubNet+ uses multi-scale time sensitive channel attention to focus on discriminative frequency bands for noise reduction.
- Stacked TCN blocks replace the original FullSubNet full-band LSTMs, extracting features from magnitude, real, and imaginary spectrograms.
- FullSubNet+ outperforms other contemporary speech enhancement methods on the DNS Challenge.