Source-linked AI summary
FRCRN: Boosting Feature Representation using Frequency Recurrence for Monaural Speech Enhancement
Shengkui Zhao, Bin Ma, Karn N. Watcharasupat, Woon-Seng Gan
TL;DR
Monaural speech enhancement is limited by the restricted frequency context of conventional convolutional recurrent encoder-decoders. The paper introduces FRCRN, which uses FSMN-based frequency recurrence in a convolutional recurrent encoder-decoder, complex cIRM prediction, and joint losses. It reports state-of-the-art wideband benchmark performance and second place in the DNS-2022 real-time fullband track.
Problem
Conventional CRN convolutions have limited receptive fields, constraining feature representation across frequency context in monaural speech enhancement.
Method
FRCRN adds FSMN-based frequency recurrence to CRED feature maps, inserts stacked FSMN temporal layers, predicts complex cIRM, and uses joint time-frequency and time-domain losses.
Results
FRCRN achieves state-of-the-art performance on wideband benchmarks and second place in the DNS-2022 real-time fullband track by MOS and WAcc.
Takeaways & Limitations
Frequency recurrence enables the proposed model to capture long-range frequency correlations while retaining temporal dynamics modeling in the FRCRN framework.
Abstract
from arXiv · showhide
Convolutional recurrent networks (CRN) integrating a convolutional encoder-decoder (CED) structure and a recurrent structure have achieved promising performance for monaural speech enhancement. However, feature representation across frequency context is highly constrained due to limited receptive fields in the convolutions of CED. In this paper, we propose a convolutional recurrent encoder-decoder (CRED) structure to boost feature representation along the frequency axis. The CRED applies frequency recurrence on 3D convolutional feature maps along the frequency axis following each convolution, therefore, it is capable of catching long-range frequency correlations and enhancing feature representations of speech inputs. The proposed frequency recurrence is realized efficiently using a feedforward sequential memory network (FSMN). Besides the CRED, we insert two stacked FSMN layers between the encoder and the decoder to model further temporal dynamics. We name the proposed framework as Frequency Recurrent CRN (FRCRN). We design FRCRN to predict complex Ideal Ratio Mask (cIRM) in complex-valued domain and optimize FRCRN using both time-frequency-domain and time-domain losses. Our proposed approach achieved state-of-the-art performance on wideband benchmark datasets and achieved 2nd place for the real-time fullband track in terms of Mean Opinion Score (MOS) and Word Accuracy (WAcc) in the ICASSP 2022 Deep Noise Suppression (DNS) challenge (https://github.com/modelscope/ClearerVoice-Studio).
1. INTRODUCTION
Monaural speech enhancement seeks to recover cleaner, more intelligible speech and improve recognition from noisy, reverberant inputs. FRCRN addresses limited frequency-context representation in CRNs by adding frequency recurrence to convolutional encoder-decoder features, with strong reported benchmark and challenge results.
- Noisy and reverberant speech degrades perceptual quality, intelligibility, and automatic speech recognition performance.
- CRNs combine convolutional feature extraction with recurrent sequence modeling, but conventional convolutions mainly capture local temporal-spectral patterns.
- FRCRN adds frequency recurrence after each convolution in CRED to model long-range frequency dependencies alongside local structures.Frequency sequences are processed with FSMN, and stacked convolutional recurrent blocks form the encoder-decoder.
- FRCRN uses complex-valued cIRM prediction and joint time-frequency and time-domain optimization, achieving SOTA results on wideband datasets and second place in the DNS-2022 real-time fullband track.The challenge ranking is reported in terms of MOS and WAcc.
2. THE PROPOSED FRCRN MODEL
FRCRN combines complex-valued convolutional processing with frequency and temporal recurrence to estimate speech from noisy spectrograms. It jointly optimizes signal-level and cIRM estimation losses.
- Overall architecture: FRCRN transforms the corrupted waveform into an STFT and predicts a bounded complex Ideal Ratio Mask for speech enhancement.The input signal is processed by the FRCRN model after STFT, with tanh bounding the estimates to [−1, 1].
- CR block: Each CR block applies complex Conv2d, batch normalization, LeakyReLU, and a CFSMN layer to complex feature maps.The complex convolution operates on channel, frame, and frequency dimensions before CFSMN processing.
- Frequency recurrence: CFSMN forms a frequency sequence for each time frame, applying recurrence across frequency features separately through real and imaginary cells.The feature map is permuted to time × frequency × channels, and the real and imaginary outputs are combined into a complex result.
- Temporal modeling: The CFSMN recurrence uses causal temporal memory with look-back order 20 and no lookahead in the reported experiments.The same recurrence formulation is applied to each frequency sequence, with a single CFSMN layer in each CR block.
- Joint loss function: FRCRN optimizes a joint objective combining SI-SNR signal loss with mean squared errors for the real and imaginary cIRM components.The signal-estimation and cIRM-estimation terms receive equal weighting by setting λ to 1.
3. EXPERIMENTS
Experiments evaluate FRCRN on wideband and fullband speech enhancement settings, including ablations, benchmark comparisons, and DNS-2022 development and blind-test results. The model outperforms baselines broadly while maintaining a 30 ms wideband latency and a 10.27-million-parameter fullband configuration.
- Evaluation setup: The wideband FRCRN uses 20 ms windows, 10 ms frame shifts, 640-point STFTs, and a 30 ms total latency.The model uses one input channel and six CRED encoder and decoder blocks.
- Ablation study: FRCRN outperforms DCCRN and PHASEN with a large margin on all WSJ0 evaluation metrics.FRCRN-Lite has lower complexity than DCCRN while retaining competitive performance.
- Ablation study: Ablations show that all examined components contribute, with frequency-recurrent CFSMN layers in CRED contributing especially strongly.The ablation progressively removes CRED frequency recurrence, the attention block, and the recurrent module.
- Benchmark comparisons: FRCRN outperforms previous state-of-the-art methods on most DNS-2020 and VoiceBank+Demand metrics, except PERL-AE on CSIG and COVL.The comparisons use published results on both wideband benchmark datasets.
- Fullband evaluation: On DNS-2022 fullband evaluation, FRCRN performs better than the baseline across development metrics and ranks top 2 in the non-personalized blind-test track.The evaluation uses DNSMOS P.835 quality measures and Word Accuracy; the fullband model has 10.27 million parameters and 12.30 GMACS per second.
4. CONCLUSIONS
FRCRN uses frequency recurrence within a CRED structure to enhance frequency representation, while its recurrent module models temporal dynamics. In the complex-valued domain, joint-loss optimization supports SOTA wideband performance and a second-place fullband DNS challenge result.
- 4. CONCLUSIONS: CRED applies frequency recurrence to 3D convolutional feature maps along the frequency axis, efficiently implemented with FSMN.This design boosts feature representation and captures long-range frequency correlations.
- 4. CONCLUSIONS: FRCRN combines CRED for long-range frequency correlations with a time-recurrent module for temporal dynamics.The architecture jointly addresses frequency and temporal structure in speech signals.
- 4. CONCLUSIONS: FRCRN operates in the complex-valued domain and uses a joint loss function for optimization.The model predicts complex-domain speech enhancement targets while combining time-frequency and time-domain objectives.
- 4. CONCLUSIONS: FRCRN achieved SOTA performance on wideband benchmarks and second place in the fullband non-personalized ICASSP 2022 DNS challenge.These results cover both benchmark evaluation and challenge ranking.