Source-linked AI summary
FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement
Xiang Hao, Xiangdong Su, Radu Horaud, Xiaofei Li
TL;DR
Single-channel real-time speech enhancement needs both global spectral dependencies and local, frequency-wise cues that pure full-band or sub-band models do not simultaneously capture. FullSubNet sequentially fuses and jointly trains these models, and experiments on the DNS Challenge dataset show complementary information and performance exceeding compared top-ranked methods.
Problem
Pure full-band and sub-band models have complementary capabilities: full-band models capture global and cross-band context, while sub-band models capture stationarity and local spectral patterns.
Method
FullSubNet sequentially connects a full-band model to a sub-band model and jointly optimizes them to combine their information.
Results
FullSubNet outperforms the compared sub-band, pure full-band, and top-ranked DNS Challenge methods on reported performance measures.
Takeaways & Limitations
The experiments show that full-band and sub-band information are complementary and can be effectively integrated for real-time single-channel speech enhancement.
Abstract
from arXiv · showhide
This paper proposes a full-band and sub-band fusion model, named as FullSubNet, for single-channel real-time speech enhancement. Full-band and sub-band refer to the models that input full-band and sub-band noisy spectral feature, output full-band and sub-band speech target, respectively. The sub-band model processes each frequency independently. Its input consists of one frequency and several context frequencies. The output is the prediction of the clean speech target for the corresponding frequency. These two types of models have distinct characteristics. The full-band model can capture the global spectral context and the long-distance cross-band dependencies. However, it lacks the ability to modeling signal stationarity and attending the local spectral pattern. The sub-band model is just the opposite. In our proposed FullSubNet, we connect a pure full-band model and a pure sub-band model sequentially and use practical joint training to integrate these two types of models' advantages. We conducted experiments on the DNS challenge (INTERSPEECH 2020) dataset to evaluate the proposed method. Experimental results show that full-band and sub-band information are complementary, and the FullSubNet can effectively integrate them. Besides, the performance of the FullSubNet also exceeds that of the top-ranked methods in the DNS Challenge (INTERSPEECH 2020).
1. INTRODUCTION
FullSubNet addresses complementary weaknesses in full-band and sub-band speech enhancement by connecting them sequentially and jointly training the fusion. On the DNS Challenge dataset, it outperformed the compared methods and showed that full-band and sub-band information are complementary.
- Motivation: Frequency-domain methods dominate single-channel speech enhancement, which this paper studies for real-time operation.They use noisy spectral features to predict clean speech features or masks.
- Limitations of existing models: The sub-band model captures frequency-wise stationarity and local spectral patterns but cannot model global spectral context or long-distance cross-band dependencies.This limitation is especially problematic for sub-bands with extremely low SNR.
- Proposed approach: FullSubNet sequentially connects a full-band model to a sub-band model and jointly optimizes them.The full-band output becomes input to the sub-band model.
- Proposed approach: FullSubNet retains sub-band stationarity and local-pattern modeling while adding full-band global context and long-distance cross-band dependencies.The architecture is intended to integrate complementary information rather than rely on increased parameter count.
- Results: FullSubNet outperformed the compared DNS Challenge methods on objective performance measures.The introduction reports this outcome without providing metric values in the supplied passages.
2. METHOD
The method represents noisy speech in the STFT domain, uses a full-band model for global context, and feeds its output with local sub-band sequences into a sub-band model. It predicts compressed cIRM targets with a low-latency, jointly trained two-stage architecture.
- 2. METHOD: The task suppresses interference noise N(t, f) and recovers the reverberant speech signal S(t, f) in the STFT domain.The system contains pure full-band and sub-band models named Gfull and Gsub.
- 2.1. Input: Gfull processes noisy full-band magnitude sequences to capture global contextual information and produce embeddings for Gsub.Its output has the same sequence size as the full-band input.
- 2.1. Input: Gsub independently processes each frequency using one target frequency and 2N neighboring frequencies plus the full-band model output.Circular Fourier frequencies handle boundary cases.
- 2.1. Input: Temporal evolution provides stationarity cues, while noisy neighboring frequencies provide local spectral patterns that Gsub learns alongside complementary Gfull information.The paper states that these cues are not especially learned by Gfull despite being present in its input.
- 2.2. Learning target: The model predicts a compressed complex Ideal Ratio Mask rather than estimating phase directly.Hyperbolic tangent compression is used during training and its inverse during inference, with K = 10 and C = 0.1.
- Architecture and inference: Both models use two stacked unidirectional LSTM layers and one linear layer, while output delay permits limited future information under real-time inference.The full-band LSTM layers contain 512 hidden units each, and future frames are supplied to infer a delayed output.
3. EXPERIMENTAL SETUP
The evaluation uses dynamically mixed DNS Challenge data and compares FullSubNet with pure sub-band, pure full-band, and top-ranked challenge systems. The setup includes both reverberant and non-reverberant test conditions under a real-time latency constraint.
- Dataset: The DNS training resources include over 500 hours of clean speech from 2150 speakers and over 180 hours of noise from 150 classes.Speech-noise mixtures are simulated with dynamic mixing during training.
- Dataset: The test set contains 150 synthetic noisy clips each for without-reverberation and with-reverberation categories, with SNRs from 0 dB to 20 dB.These two categories are used for evaluation.
- Training and signal processing: The STFT uses a 512-sample Hanning window and 256-sample frame step, with training sequences of T = 192 frames.The optimizer is Adam with learning rate 0.001.
- Training and signal processing: The real-time setup uses τ = 2, exploiting two future frames and a 32 ms look-ahead.This configuration follows the DNS Challenge real-time requirement.
- Baselines: FullSubNet is compared under matched settings and cIRM targets against sub-band and pure full-band baselines, plus NSNet, DTLN, Conv-TasNet, DCCRN, and PoCoNet.The additional systems include methods from the DNS Challenge rankings.
4. RESULTS
FullSubNet combines full-band and sub-band modeling, whose complementary information improves speech-enhancement performance. On the DNS challenge test set, it outperforms the compared baseline and state-of-the-art methods while meeting real-time requirements.
- Comparison with the baselines: Most full-band evaluation scores exceed those of the sub-band model, while the sub-band model is relatively more effective on reverberant data.The authors connect this pattern to full-band wide-band information and sub-band modeling of temporal narrow-band evolution.
- Comparison with the baselines: FullSubNet dramatically improves the sub-band model after integrating a full-band model through joint training.The fusion adds global spectral patterns and long-distance cross-band dependencies while retaining sub-band capabilities.
- Comparison with the baselines: FullSubNet significantly exceeds the full-band model despite using fewer parameters.The authors attribute the gain to complementary sub-band information, including signal stationarity, local spectral patterns, and reverberation modeling.
- Comparison with the state-of-the-art methods: The proposed fusion model greatly surpasses NSNet on every reported objective metric.NSNet is the DNS challenge’s official baseline and uses a compact single-frame-in, single-frame-out RNN.
- Comparison with the state-of-the-art methods: FullSubNet achieves considerably better objective scores than the compared DNS challenge methods on the limited test dataset.The comparison includes DTLN, Conv-TasNet, DCCRN, and PoCoNet; PoCoNet is close but does not satisfy the real-time requirement.
- Comparison with the state-of-the-art methods: 10.32 ms is the proposed model’s one-frame processing time on a virtual quad-core CPU, meeting the real-time requirement.The measurement uses a 32 ms STFT frame and a PyTorch implementation.
5. CONCLUSION
FullSubNet fuses full-band and sub-band information for real-time single-channel speech enhancement. On the DNS challenge test dataset, the information is complementary, and FullSubNet outperforms the compared top-ranked methods.
- 5. CONCLUSION: FullSubNet integrates global spectral information and long-distance cross-band dependencies with signal-stationarity and local-pattern modeling.The model is designed to combine the advantages of full-band and sub-band approaches.
- 5. CONCLUSION: The DNS challenge test results demonstrate that full-band and sub-band information are complementary.FullSubNet effectively integrates both information types in a real-time single-channel enhancement model.
- 5. CONCLUSION: FullSubNet outperforms the top-ranked methods compared on the DNS challenge.The conclusion reports this outcome on the DNS challenge test dataset.