Source-linked AI summary
Dual-Signal Transformation LSTM Network for Real-Time Noise Suppression
Nils L. Westhausen, Bernd T. Meyer
TL;DR
Real-time speech enhancement is challenging because many deep models use full sequences and future information. DTLN stacks STFT and learned signal transformations, and reports improved subjective scores over noisy conditions alongside slight improvements over baseline systems.
Problem
Many deep speech-enhancement methods process complete sequences and use future information, unlike causal frame-based processing needed for real-time operation.
Source: §1 · p. 1Method
DTLN stacks an STFT-based core with a learned-feature core to estimate magnitude and further enhance speech using phase information.
Source: §1 · p. 1, §2.2 · p. 2Results
Takeaways & Limitations
The authors report that DTLN works robustly in noisy reverberant environments and benefits from using two types of analysis and synthesis bases in a stacked network.
Source: §5 · p. 4The learned representation compresses 512 audio samples to 256 features; larger representations would require more parameters, while the reduction was found not to greatly affect speech quality.
Source: §4 · p. 4
Abstract
from arXiv · showhide
This paper introduces a dual-signal transformation LSTM network (DTLN) for real-time speech enhancement as part of the Deep Noise Suppression Challenge (DNS-Challenge). This approach combines a short-time Fourier transform (STFT) and a learned analysis and synthesis basis in a stacked-network approach with less than one million parameters. The model was trained on 500 h of noisy speech provided by the challenge organizers. The network is capable of real-time processing (one frame in, one frame out) and reaches competitive results. Combining these two types of signal transformations enables the DTLN to robustly extract information from magnitude spectra and incorporate phase information from the learned feature basis. The method shows state-of-the-art performance and outperforms the DNS-Challenge baseline by 0.24 points absolute in terms of the mean opinion score (MOS).
1. Introduction
Many deep speech-enhancement models use complete sequences and future information, which does not meet real-time causal processing needs. DTLN combines STFT and learned signal transformations in a stacked network intended to retain real-time capability while incorporating phase information.
1. Introduction
- Bidirectional RNNs require full-sequence input, while many deep enhancement methods exploit future signal information and are not designed for real-time causal processing.Frame-based algorithms can provide low input-output delay, and RNNs are a common neural-network choice for them.Source: §1 · p. 1
- The DNS-Challenge NSNet baseline predicts a gain or mask from the noisy STFT log power spectrum but reconstructs speech using noisy-mixture phase.The paper notes that this approach does not incorporate phase information that could be useful for enhanced speech quality.Source: §1 · p. 1
- DTLN stacks an STFT-based separation core with a learned-representation core to estimate magnitude first and further enhance the signal with phase information.The authors propose this combination as a way to exploit complementary transformations while keeping the model relatively small and real-time capable.Source: §1 · p. 1
2. Methods
The methods define speech extraction using STFT masking and a learned-basis representation, then stack both transformations in a real-time architecture. The paper describes its training and test data, model setup, baselines, and objective and subjective evaluation procedures.
2.1. Signal transformations
- STFT masking applies a mask to noisy-signal magnitude while retaining noisy phase, whereas learned-basis processing projects mixture frames into features for speech-mask estimation and synthesis.Noise suppression returns the speech signal and discards noise; the estimated signal is reconstructed in the time domain.Source: §2.1 · pp. 1–2
2.2. Network architecture
- The network has two separation cores: an STFT core followed by learned-feature processing through 1D-Conv layers, with instant layer normalization applied frame by frame.The second core masks the unnormalized feature representation, which is transformed back to the time domain before overlap-add reconstruction.Source: §2.2 · p. 2
2.3. Datasets
- Training used 500 h of DNS-Challenge-derived data split into 400 h for training and 100 h for validation, with SNRs from -5 to 25 dB.Testing included four DNS-Challenge categories and a 3,000-mixture WHAMR set with reverberant speech and real-life noise.Source: §2.3 · pp. 2–3
2.4. Model configuration and training setup
- The DTLN uses 128 units in each of four LSTM layers, 32 ms frames with an 8 ms shift, and scale-sensitive negative SNR as its time-domain training objective.The objective is chosen to avoid possible level offsets and to allow phase information to be considered implicitly.Source: §2.4 · p. 3
2.5. Baselines
- The comparisons include the challenge-provided NSNet and four similarly trained models that vary between STFT or learned features and stacked or non-stacked topologies.The comparison models were configured with the aim of matching DTLN's parameter count.Source: §2.5 · p. 3
2.6. Objective and subjective evaluation
- Evaluation used PESQ, SI-SDR, and STOI for objective measures, plus ITU-T P.808 subjective ratings on known and blind test sets.Each subjective-test file was rated by five or ten judges in the first and second runs, respectively.Source: §2.6 · p. 3
3. Results
DTLN achieves the strongest reported objective results on the non-reverberant DNS-Challenge and WHAMR tests, while subjective evaluation favors it over noisy speech and the baseline in reverberant conditions. Execution-time measurements show frame-wise processing is faster than a 32 ms frame duration after conversion to SavedModel format.
Objective evaluation
- DTLN reached the best PESQ, SI-SDR, and STOI results on the non-reverberant DNS-Challenge test and the best objective scores on WHAMR.On the reverberant DNS-Challenge test, objective measures were mixed: all models improved SI-SDR, but STOI favored noisy speech and only B4 slightly improved PESQ over noisy speech.Source: §3 · p. 3, Table 2 · p. 4
Subjective evaluation
- Subjective evaluation showed a clear DTLN benefit over noisy speech and the baseline in reverberant conditions, with consistent results on known and blind real recordings.The objective measures did not consistently reflect the reverberant subjective benefit, except for SI-SDR.Source: §3 · p. 4, Table 3 · p. 4
Execution time
- 0.65 ms per 32 ms frame was measured for frame-wise processing after conversion to TensorFlow SavedModel format.The original Keras frame-wise measurement was 2.08 ms; sequence processing took 0.23 ms.Source: §3 · p. 4
4. Discussion
The discussion finds that combining STFT and learned-feature transformations slightly improves baseline systems while using fewer LSTM units. It also considers possible reasons for STFT-feature systems’ stronger results and notes that training data and setup may account for part of the performance.
Baseline comparison
- STFT-feature systems B1 and B3 perform better than learned-feature systems B2 and B4 on non-reverberant, reverberant, and WHAMR tests.One proposed explanation is that fixed STFT representations leave more parameters available for LSTM layers.Source: §4 · p. 4
Possible explanations
- The authors hypothesize that discarding phase may make STFT features more robust to noisy input, while learned features must implicitly handle magnitude and phase.They also suggest that the learned representation compresses 512 samples to 256 features, and report that this reduction did not greatly affect speech quality.Source: §4 · p. 4
Overall comparison
- Stacking STFT and learned-feature transformations slightly improves overall baseline systems while using fewer LSTM units than pure STFT systems.The authors caution that the small objective-measure differences between DTLN and B1–B4 suggest that training data volume and setup account for part of the performance.Source: §4 · p. 4
5. Conclusions
The conclusion presents DTLN as a real-time noise-suppression approach combining two analysis and synthesis bases in a stacked network. It reports robust performance in noisy reverberant environments and a 0.22-point absolute MOS improvement over noisy conditions in subjective evaluations.
Conclusion
- 0.22 points absolute MOS improvement over noisy conditions was observed across subjective evaluations.The conclusion attributes this result to a stacked dual-signal-transformation LSTM approach trained on a large-scale dataset, and reports robust operation in noisy reverberant environments.Source: §5 · p. 4