Source-linked AI summary
Towards efficient models for real-time deep noise suppression
Sebastian Braun, Hannes Gamper, Chandan K. A. Reddy, Ivan Tashev
TL;DR
Speech enhancement models must deliver quality within tight inference-time compute budgets, while results need evaluation on real-world recordings. This paper trains and tests recurrent and convolutional-recurrent models under those conditions, finding that quality and computational effort trade off across tested models. It reports that proposed CRUSE models with parallel GRUs and optimized skip connections offer efficient tradeoffs.
Problem
State-of-the-art noise suppression networks can be difficult to run on resource-limited devices, motivating the question of how to maximize speech quality under a computational budget.
Source: §1 · p. 1Method
The authors compare recurrent and convolutional-recurrent architectures trained on large-scale data representing reverberation and evaluated on real recordings.
Source: §1 · p. 1Results
Across tested models, average achieved quality increases with computational effort, while proposed CRUSE models with parallel GRUs and optimized skip connections are among the most efficient.
Source: §5 · p. 4Takeaways & Limitations
Speech quality depends on network size, and modified skip connections and disconnected parallel RNNs improve the speech-quality-versus-complexity tradeoff.
Source: §6 · p. 4
Abstract
from arXiv · showhide
With recent research advancements, deep learning models are becoming attractive and powerful choices for speech enhancement in real-time applications. While state-of-the-art models can achieve outstanding results in terms of speech quality and background noise reduction, the main challenge is to obtain compact enough models, which are resource efficient during inference time. An important but often neglected aspect for data-driven methods is that results can be only convincing when tested on real-world data and evaluated with useful metrics. In this work, we investigate reasonably small recurrent and convolutional-recurrent network architectures for speech enhancement, trained on a large dataset considering also reverberation. We show interesting tradeoffs between computational complexity and the achievable speech quality, measured on real recordings using a highly accurate MOS estimator. It is shown that the achievable speech quality is a function of network complexity, and show which models have better tradeoffs.
1. INTRODUCTION
The paper asks how to maximize speech quality under a fixed computational budget, given that state-of-the-art noise suppressors can be impractical on resource-limited devices. It compares recurrent and convolutional-recurrent models and evaluates them using large-scale training data and real recordings.
1. INTRODUCTION
- The central challenge is obtaining the best speech quality within a maximum computational budget on devices where noise suppression shares resources with other tasks.The paper notes that current state-of-the-art models can be challenging to run on resource-limited devices.Source: §1 · p. 1
- Time-domain networks are not used because generalization on real reverberant recordings had not yet been shown; the authors therefore use frequency-domain implementations.The paper also notes that larger convolutional-recurrent and convolutional models can be impractical for edge devices.Source: §1 · p. 1
- The study compares RNN and CRN designs and proposes a CRN with around 4–5 times fewer computational operations and similar quality to previously proposed CRNs.The experiments use large-scale training data representing reverberation, diverse speakers and noises, and varying microphone levels, with results on real recordings evaluated by a DNN-based MOS predictor.Source: §1 · p. 1
2. ENHANCEMENT SYSTEM AND TRAINING OBJECTIVE
The enhancement system predicts time-varying suppression gains from log power spectra and applies them to the complex input spectrum. Training combines STFT consistency with a compressed complex MSE objective and level normalization.
2. ENHANCEMENT SYSTEM AND TRAINING OBJECTIVE
- The network predicts a real-valued suppression gain for each time-frequency bin from log power spectra, applies it to the complex input spectrum, and transforms the result back to the time domain.The algorithmic delay depends only on the STFT window size.Source: §2 · p. 1
- Training enforces STFT consistency by reconstructing the frequency-domain output and applying another STFT, while normalizing sequences by the active target utterance level.The system retains flexibility to integrate with other frequency-domain algorithms and to use optimized STFT implementations.Source: Figure 1 · p. 1, §2 · p. 2
- The complex compressed MSE loss blends a magnitude-only term with a phase-aware term, using a compression factor of 0.3 and a weighting λ = 0.3.The authors report that they found this loss superior to other losses for reverberant speech enhancement.Source: §2 · p. 2
- Models are trained with AdamW on batches of ten 10-second sequences, using a learning rate of 8·10^-5 and weight decay of 0.1.The best model is selected using a validation criterion based on PESQ, siSDR, and cepstral distance.Source: §2 · p. 2
3. NETWORK ARCHITECTURES
The paper describes NSnet2 and CRUSE architectures and modifications aimed at improving efficiency. CRUSE uses a convolutional encoder-decoder around a recurrent bottleneck, with parallel recurrent layers and revised skip connections as efficiency options.
3. NETWORK ARCHITECTURES
- All architectures use the same features, prediction targets, loss, and training strategy for comparisons.Source: §3 · p. 2
3.1. NSnet2
- NSnet2 consists of fully connected and GRU layers, with a sigmoid output predicting a constrained suppression gain.Its standard dimensions are 400-unit GRUs, 600-unit fully connected layers, and K frequency-bin inputs and outputs.Source: §3.1 · p. 2
3.2. CRUSE
- CRUSE is a U-Net with symmetric convolutional encoder and decoder layers around a recurrent bottleneck; replacing two LSTMs with one GRU yields very little performance loss and substantial computational savings.Its convolutions downsample along frequency, and the encoder and decoder channel counts mirror one another.Source: §3.2 · p. 2
- Parallel RNN grouping reshapes the bottleneck into P disconnected recurrent layers while retaining the same forward information flow and allowing parallel execution.P = 1 uses a single GRU; P > 1 divides the encoder output into vectors processed by disconnected GRUs.Source: §3.2 · p. 2
- Additive skip connections are cheaper than concatenation, and adding trainable channel-wise scaling and bias through 1×1 convolutions improves performance at negligible additional cost.The architecture figure contrasts doubled decoder input channels for concatenation with addition skips.Source: §3.2 · p. 2, Figure 3 · p. 2
- The training-data pipeline uses reverberant speech as is, augments non-reverberant speech with RIRs, and creates targets using shaped RIRs.Source: Figure 4 · p. 3
4. EXPERIMENTAL SETUP
The experiments use large-scale synthetic training data with reverberation augmentation and evaluate models on real recordings. They compare quality, inference complexity, and network configurations using a consistent evaluation setup.
4.1. Dataset
- Reverberant impulse responses are shaped with an exponential decay targeting T60 = 0.3 s, and noisy training mixtures use SNRs drawn from N(5, 10) dB.Speech and noise portions are assembled into 10-second clips; reverberant speech is paired with non-reverberant target speech.Source: §4.1 · p. 3
- The roughly 1000-hour augmented training set combines speech and noise, with reverberant mixtures generated using measured and simulated room responses.It starts from 544 hours of MOS-rated LibriVox speech and 247 hours of noise; non-reverberant speech is augmented with room impulse responses, while 20% remains unaugmented. The final test uses 300 real recordings and 300 synthetic mixtures from unseen datasets in the DNS challenge development set.Source: §4.1 · p. 3
4.2. Evaluation metrics
- Models are evaluated with predicted DNSMOS, while rankings and trends also agree with crowd-sourced MOS and objective metrics on synthetic validation data.The authors report coherence with PESQ, CD, and siSDR; computational complexity counts weight and bias MACs, excluding activations and shared signal-processing operations.Source: §4.2 · p. 3
4.3. Algorithmic parameters
- The setup uses 16 kHz audio, 20 ms square-root-Hann STFT windows with 50% overlap, and 161-dimensional log-power-spectrum inputs.NSnet2 variants are indexed by GRU nodes per layer; CRUSE variants specify encoder-decoder depth and width, recurrent layers, and parallel RNNs.Source: §4.3 · p. 3
5. RESULTS
Across tested models, MOS generally rises with computational effort, but optimized CRUSE models with parallel GRUs and skip connections offer a better tradeoff. Results also show how architecture choices, RNN width, and skip connections affect quality and complexity.
5. RESULTS
- 283M MACs make CRN-LSTM9 computationally inefficient despite its significant MOS improvement; the fully convolutional baseline underperforms, while NSnet achieves MOS similar to classicNS.The classic noise suppressor improves MOS by around 0.12 on a test set with many highly nonstationary noise types; its few-thousand-MAC burden is a tiny fraction of the smallest models.Source: §5 · pp. 3–4
- Replacing CRUSE's two LSTM layers with two GRUs and then one GRU improves efficiency; grouping GRUs into P = 4 reduces complexity with very little performance loss.Using 1D convolutions instead of CRUSE's 2D convolutions worsens the complexity-quality tradeoff in the tested models.Source: §5 · p. 4
- Optimized CRUSE models with parallel GRUs and skip connections break the general MAC–MOS trend, while fully recurrent NSnet models are generally cheaper but lower quality.Across tested models, average quality increases monotonically with computational effort; CRUSE encoder-decoder structures, especially with temporal layers, are helpful.Source: §5 · p. 4, Figure 5 · p. 4
- Scaling RNN width in NSnet2-R and CRUSE4-C4-1xGRU1 shows a clear monotonic relationship between throughput or memory and quality; overly large models worsen the complexity-quality tradeoff.For CRUSE4, width is scaled by changing the last encoder layer’s filters C4; grouping its fully connected RNN into P disconnected RNNs while keeping feedforward flow fixed achieves significantly better tradeoffs at P = 2 and P = 4.Source: §5 · p. 4, Figure 6 · p. 4
- For CRUSE4-128-1GRU4, addition skips outperform no skips, while cheap 1×1 convolutions achieve the same MOS as concat skips.The comparison is the skip-connection ablation reported in Table 1.Source: §5 · p. 4, Table 1 · p. 4
- On an Intel Core i7 QuadCore at 3.5 GHz, CRUSE4-128-1xGRU4 processes a frame in 0.3 ms, using less than 3% of a 10-ms hop budget.The reported execution time is without further optimization and is specific to the tested hardware platform.Source: §5 · p. 4
6. CONCLUSIONS
The paper presents a scalable CRN for speech enhancement and reports quality–complexity tradeoff gains from modified skip connections and parallel RNNs, while noting remaining efficiency limits.
6. CONCLUSIONS
- The proposed CRN is trained on large data and tested on real recordings.The conclusion describes it as flexible and scalable.Source: §6 · p. 4
- Modified skip connections and disconnected parallel RNNs improve the speech-quality-versus-computational-complexity tradeoff.Source: §6 · p. 4
- The models use only a fraction of standard CPU computational budgets in real time, yet their quality gain per computational burden remains less efficient than traditional speech enhancement.Source: §6 · p. 4