Source-linked AI summary
DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement
Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, Lei Xie
TL;DR
Speech enhancement must improve noisy speech despite limitations in conventional TF-domain and real-valued approaches to complex targets. The paper proposes DCCRN, combining complex-valued convolution and recurrent modeling, and reports competitive objective and subjective performance, including top DNS challenge MOS rankings with 3.7M parameters.
Problem
Conventional TF-domain methods often predict masks or spectra with real-valued networks, while phase remains difficult to model effectively.
Method
DCCRN combines complex-valued CNN and LSTM structures for complex-spectrum modeling while reducing trainable parameters and computational cost.
Results
With 3.7M parameters, DCCRN achieved the best MOS in the DNS real-time track and the second-best MOS in the non-real-time track.
Takeaways & Limitations
DCCRN provides competitive speech-enhancement performance with substantially lower computation than DCUNET under similar model-parameter configurations.
Abstract
from arXiv · showhide
Speech enhancement has benefited from the success of deep learning in terms of intelligibility and perceptual quality. Conventional time-frequency (TF) domain methods focus on predicting TF-masks or speech spectrum, via a naive convolution neural network (CNN) or recurrent neural network (RNN). Some recent studies use complex-valued spectrogram as a training target but train in a real-valued network, predicting the magnitude and phase component or real and imaginary part, respectively. Particularly, convolution recurrent network (CRN) integrates a convolutional encoder-decoder (CED) structure and long short-term memory (LSTM), which has been proven to be helpful for complex targets. In order to train the complex target more effectively, in this paper, we design a new network structure simulating the complex-valued operation, called Deep Complex Convolution Recurrent Network (DCCRN), where both CNN and RNN structures can handle complex-valued operation. The proposed DCCRN models are very competitive over other previous networks, either on objective or subjective metric. With only 3.7M parameters, our DCCRN models submitted to the Interspeech 2020 Deep Noise Suppression (DNS) challenge ranked first for the real-time-track and second for the non-real-time track in terms of Mean Opinion Score (MOS).
1. Introduction
Noise interference degrades speech quality, intelligibility, and related tasks, motivating deep-learning speech enhancement. The paper focuses on complex-valued TF modeling to address limitations of magnitude-only or real-valued approaches.
- Motivation: Noise interference can severely reduce perceptual quality and intelligibility and heavily affect automatic speech recognition.Speech enhancement takes noisy speech as input and produces enhanced speech for improved quality, intelligibility, or downstream criteria.
- Motivation: Deep-learning methods have achieved promising speech-enhancement results, including under challenging non-stationary noise conditions.The paper notes applications in both single-channel and multi-channel enhancement.
- Related work: Time-frequency approaches use spectrograms because speech and noise may be more separable after short-time Fourier transformation.Convolution recurrent networks combine convolutional encoder-decoder structures with LSTM-based temporal modeling.
- Related work: Magnitude-focused targets ignore phase, limiting performance when estimated speech phase deviates substantially under serious interference.Later approaches model phase-sensitive or complex targets, including real and imaginary components, but some retain real-valued network operations.
- Contribution: DCCRN combines complex-valued CNN and RNN operations to train complex targets more effectively, using 3.7M parameters for DNS challenge submissions.The model combines DCUNET and CRN advantages while reducing trainable parameters and computational cost.
2. The DCCRN Model
DCCRN extends the convolution recurrent network with complex-valued convolution, normalization, and recurrent operations to model complex speech spectrograms. It estimates complex or magnitude-related targets using several masking patterns and trains with SI-SNR on reconstructed waveforms.
- Convolution recurrent network architecture: The CRN uses a causal convolutional encoder-decoder with LSTM layers between encoder and decoder to model temporal dependencies.Skip connections connect encoder and decoder layers and support gradient flow.
- Complex network: DCCRN replaces shared real-valued convolution on real and imaginary channels with complex CNN, complex batch normalization, and complex LSTM operations.The complex module simulates complex multiplication to model correlations between magnitude and phase.
- Complex network: Complex convolution computes real and imaginary output components from four real convolutions involving the input and filter components.For X = Xr + jXi and W = Wr + jWi, the output follows the stated complex multiplication structure.
- Training target: DCCRN estimates a complex ratio mask from noisy and clean complex STFT spectrograms and applies signal approximation losses to reconstructed speech representations.The method compares CRM-based and magnitude-mask signal approximation objectives.
- Training target: Three multiplicative patterns produce estimated clean speech: separate real-imaginary multiplication, complex multiplication, or polar-coordinate magnitude and phase processing.DCCRN-C uses complex multiplication, while DCCRN-E limits mask magnitude with tanh activation.
- Loss function: SI-SNR is the training loss, computed after STFT-initialized convolution and deconvolution modules analyze and synthesize the waveform.The clean and estimated time-domain waveforms are compared using the SI-SNR formulation.
3. Experiments
The experiments compare LSTM, CRN, DCCRN, and DCUNET variants on simulated WSJ0 and DNS challenge data using PESQ and MOS-related evaluations. DCCRN-CL performs best among DCCRNs on WSJ0, while DCCRN-E and its augmented version are selected for DNS tracks based on objective and subjective considerations.
- Datasets and setup: 24,500 WSJ0 utterances were split into 20,000 training, 3,000 validation, and 1,500 evaluation examples.The corpus contains 131 speakers, with noise drawn from MUSAN.
- Datasets and setup: All models used 25 ms windows, 6.25 ms hops, 512-point FFTs, Adam optimization, learning-rate decay, and early stopping.Waveforms were resampled to 16 kHz.
- Model configurations: DCCRN models represent complex spectrograms with complex convolutional and recurrent components, while CRN separates real and imaginary processing in its decoders.DCCRN variants include DCCRN-R, DCCRN-C, DCCRN-E, and DCCRN-CL.
- WSJ0 results: DCCRN-CL outperformed the baseline LSTM and CRN on simulated WSJ0, and exceeded the other DCCRN variants in PESQ.The results support the reported benefit of complex convolution and complex LSTM for complex-target training.
- WSJ0 results: DCCRN and DCUNET achieved similar PESQ on WSJ0, while DCUNET required almost six times the computational complexity of DCCRN-CL.The comparison is based on the authors’ runtime test.
- DNS challenge results: DCCRN-E was selected for the real-time DNS track, and DCCRN-E-Aug for the non-real-time track after subjective listening and reverb-focused augmentation.DCCRN-CL slightly improved PESQ generally but could over-suppress speech; augmentation improved reverb-set MOS by 0.02.
4. Conclusions
The paper proposes DCCRN, a complex network for complex-valued spectrum modeling in speech enhancement. Under a complex multiply constraint, it reports better PESQ and MOS than other models with similar parameter counts, while identifying edge deployment and reverberation suppression as future directions.
- Conclusion: DCCRN uses complex-valued network operations to model complex speech spectra for speech enhancement.The proposed architecture is intended to handle complex-valued spectrum modeling directly.
- Conclusion: DCCRN achieved better PESQ and MOS than other models with similar parameter counts under the complex multiply rule constraint.The conclusion states this comparison at the model-configuration level without reporting a specific value here.
- Conclusion: Future work targets deployment in low-computational scenarios and improved noise suppression under reverberation.The paper specifically mentions edge devices and reverberation conditions.