Source-linked AI summary
DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filtering
Hendrik Schröter, Alberto N. Escalante-B., Tobias Rosenkranz, Andreas Maier
TL;DR
Monaural speech enhancement commonly relies on TF masks, but their performance degrades at low frequency resolution and they do not exploit broader local correlations. DeepFilterNet combines ERB-scaled gains with a deep-filtering stage for periodic components, achieving stronger performance than complex masks across frequency resolutions and competitive quality with lower computational complexity than related systems.
Problem
TF-mask speech enhancement can degrade at low frequency resolution, limiting noise removal between speech harmonics.
Method
DeepFilterNet uses ERB-scaled gains to enhance the spectral envelope and deep filtering to enhance periodic speech components.
Results
DeepFilterNet outperforms complex masks across 5–30 ms FFT windows, performs on par with related algorithms, and is computationally more efficient.
Takeaways & Limitations
The framework provides low-complexity speech enhancement suitable for real-time usage while retaining performance at small STFT window sizes.
Abstract
from arXiv · showhide
Complex-valued processing has brought deep learning-based speech enhancement and signal extraction to a new level. Typically, the process is based on a time-frequency (TF) mask which is applied to a noisy spectrogram, while complex masks (CM) are usually preferred over real-valued masks due to their ability to modify the phase. Recent work proposed to use a complex filter instead of a point-wise multiplication with a mask. This allows to incorporate information from previous and future time steps exploiting local correlations within each frequency band. In this work, we propose DeepFilterNet, a two stage speech enhancement framework utilizing deep filtering. First, we enhance the spectral envelope using ERB-scaled gains modeling the human frequency perception. The second stage employs deep filtering to enhance the periodic components of speech. Additionally to taking advantage of perceptual properties of speech, we enforce network sparsity via separable convolutions and extensive grouping in linear and recurrent layers to design a low complexity architecture. We further show that our two stage deep filtering approach outperforms complex masks over a variety of frequency resolutions and latencies and demonstrate convincing performance compared to other state-of-the-art models.
1. INTRODUCTION
DeepFilterNet addresses the limitations of TF-mask enhancement at coarse frequency resolutions by combining perceptual spectral gains with deep filtering. It outperforms complex masks across 5–30 ms FFT windows, including low-latency settings where periodic speech components remain difficult to enhance.
- TF-mask approaches degrade at low frequency resolution because they cannot reliably remove noise between speech harmonics.Conventional approaches use at least 20 ms windows, corresponding to a minimum frequency of 50 Hz.
- DeepFilterNet first models the smooth spectral envelope with real-valued ERB gains, then applies deep filtering to recover periodic speech components.The ERB filter bank reduces the representation to 32 bands, while the second stage addresses the resulting insufficient bandwidth for periodic components.
- Deep filtering exploits correlations across multiple time-frequency bins, enabling noise reduction within a frequency band while preserving speech components.This is particularly useful when multiple speech harmonics occupy one frequency bin or when noise is periodic.
- Deep filtering outperforms complex masks across FFT windows corresponding to 5–30 ms and remains effective at the 5 ms, 250 Hz resolution.The authors report that DF enhances periodic speech components even under low-latency requirements.
2. DEEPFILTERNET
DeepFilterNet enhances noisy speech through a two-stage frequency-domain pipeline: ERB-scaled gains model the spectral envelope, followed by deep filtering of periodic components. Its low-complexity design combines separable convolutions, grouped layers, and sparsity-oriented processing while supporting low latency.
- Deep Filtering: Deep filtering applies complex coefficients across multiple time-frequency bins, optionally using look-ahead and frequency correlations, rather than a pointwise mask.A learned weighting factor combines the deep-filtered output with the gain-enhanced spectrogram.
- Framework Overview: DeepFilterNet transforms noisy audio with an STFT, predicts ERB-scaled gains, and applies them through an inverse ERB filter bank before deep filtering.The framework supports sampling rates up to 48 kHz and STFT windows from 5 ms to 30 ms.
- Deep Filtering: The model restricts deep filtering to lower frequencies, where periodic speech components are assumed to contain most of their energy.The filter coefficients are predicted per band up to a cutoff frequency fDF.
- Framework Overview: The overall latency can reach 5 ms with a 240-point FFT when both DNN and deep-filter look-ahead are zero.The latency is lNFFT + max(lDNN, lDF).
- DNN Model: The architecture uses separable convolutions, pathway skips, transposed convolutions, and grouped linear and recurrent layers to reduce computational complexity while retaining frequency resolution.Grouped layers introduce sparsity, while a global pathway skip preserves a representation of the noisy phase at the output.
- Loss: Training uses a compressed spectral loss with magnitude- and phase-aware terms, plus a local-SNR loss that confines deep filtering to speech-dominant low-frequency segments.The local SNR is computed below fDF over 20 ms windows; the method is designed to avoid unnecessary filtering of noise-only or stochastic speech components.
3. EXPERIMENTS
Experiments evaluate DeepFilterNet on full-band speech enhancement across FFT sizes, overlaps, and related-work baselines. Deep filtering outperforms CRMs across 5–30 ms windows and achieves competitive quality with lower computational complexity.
- 3.1. Training setup: The evaluation trains on the DNS dataset and uses speaker-disjoint VCTK/PTDB splits with a separate VCTK/DEMAND test comparison.The DNS data contains over 750 h of clean speech and 180 h of noise; reported results use full-band 48 kHz signals.
- 3.2. Results: Performance is compared using SI-SDR across multiple FFT sizes, treating CRM as DF with filter order N = 1 and look-ahead l = 0.The DNN look-ahead remains unchanged for CRM models.
- 3.2. Results: DF outperforms CRM across FFT sizes corresponding to 5–30 ms, while CRM performance drops at window sizes ≤20 ms.DF performance remains relatively constant until dropping around 30 ms, where neighboring-frame correlation is smaller.
- 3.2. Results: +0.6 dB SI-SNR at input SNR 0 results from increasing FFT overlap to 75% for both DF and CRM.The increase is attributed to higher intraframe correlation and more RNN hidden-state updates, while computational complexity doubles.
- 3.2. Results: DeepFilterNet outperforms PercepNet and performs on par with DCCRN+ while using much lower computational complexity for real-time usage.The comparison uses WB-PESQ on the VCTK/DEMAND test set at NFFT = 960 (20 ms).
4. CONCLUSION
The conclusion presents DeepFilterNet as a low-complexity speech enhancement framework with competitive performance and greater computational efficiency. It also reports that deep filtering outperforms CRMs particularly for smaller STFT windows.
- 4. CONCLUSION: DeepFilterNet performs on par with other algorithms while being computationally more efficient through a perceptually motivated approach.The framework minimizes model complexity using perceptual properties of speech.
- 4. CONCLUSION: Deep filtering outperforms CRMs particularly for smaller STFT window sizes.
- 4. CONCLUSION: Future work will improve deep filtering of periodic speech components using a correlation-based voiced probability.