Source-linked AI summary
Complex spectrogram enhancement by convolutional neural network with multi-metrics learning
Szu-Wei Fu, Ting-yao Hu, Yu Tsao, Xugang Lu
TL;DR
Speech enhancement commonly leaves noisy phase unchanged because clean-phase estimation is difficult, while phase influences synthesized speech quality. This paper estimates clean RI spectrograms with a CNN and adds LPS reconstruction to the objective; experiments report simultaneous improvement of LSD and SSNR when β is properly specified.
Problem
Existing enhancement models often process magnitude while leaving phase noisy, although phase affects the quality of resynthesized speech.
Method
The proposed CNN estimates clean RI spectrograms directly from noisy ones and combines RI and LPS reconstruction objectives through MML.
Results
MML can simultaneously improve LSD and SSNR when β is properly specified.
Takeaways & Limitations
Representing enhancement through RI spectrograms enables more accurate phase information and a unified objective for multiple metrics.
Abstract
from arXiv · showhide
This paper aims to address two issues existing in the current speech enhancement methods: 1) the difficulty of phase estimations; 2) a single objective function cannot consider multiple metrics simultaneously. To solve the first problem, we propose a novel convolutional neural network (CNN) model for complex spectrogram enhancement, namely estimating clean real and imaginary (RI) spectrograms from noisy ones. The reconstructed RI spectrograms are directly used to synthesize enhanced speech waveforms. In addition, since log-power spectrogram (LPS) can be represented as a function of RI spectrograms, its reconstruction is also considered as another target. Thus a unified objective function, which combines these two targets (reconstruction of RI spectrograms and LPS), is equivalent to simultaneously optimizing two commonly used objective metrics: segmental signal-to-noise ratio (SSNR) and logspectral distortion (LSD). Therefore, the learning process is called multi-metrics learning (MML). Experimental results confirm the effectiveness of the proposed CNN with RI spectrograms and MML in terms of improved standardized evaluation metrics on a speech enhancement task.
1. INTRODUCTION
Existing speech-enhancement models often process only magnitude spectrograms because clean-phase estimation is difficult, despite phase affecting resynthesized waveform quality. The paper addresses this by estimating clean RI spectrograms with a CNN and using them to support multi-metric optimization.
- Motivation: Phase estimation is difficult because phase spectrograms lack clear structure, although phase strongly affects resynthesized speech quality.Prior work found phase especially important when magnitude and phase are inconsistent or when Fourier-transform windows and overlap increase.
- Proposed approach: The proposed CNN estimates clean real and imaginary spectrograms directly from noisy RI spectrograms rather than from complementary features.Real and imaginary spectrograms are treated as separate input channels to exploit their relationship.
- Multi-metrics learning: Because RI outputs can produce other signal representations, the objective combines RI reconstruction and LPS reconstruction to optimize multiple metrics.The paper connects these targets to waveform-domain SSNR and spectrogram-domain LSD.
2. NOISY PHASE
Noisy phase can resemble clean phase at high SNR but diverges more seriously at low SNR. This motivates enhancing phase through RI representations rather than directly estimating an unstructured phase spectrogram.
- Phase behavior: At high SNR and high speech energy, noisy phase is similar to clean phase because speech components dominate the corresponding real and imaginary noise components.The noisy phase is defined from the imaginary and real parts of the noisy complex spectrogram.
- Phase behavior: At low SNR, using noisy phase can substantially degrade synthesized speech, while enhanced phase can considerably improve synthesized-signal quality.The paper links this degradation to larger clean–noisy phase differences under low-SNR conditions.
3. ENHANCEMENT OF RI SPECTROGRAMS BY CNN
The proposed CNN enhances clean RI spectrograms using a channelized representation and can incorporate pseudo layers for additional objectives. This design supports reconstruction of enhanced speech representations beyond the primary RI output.
- RI enhancement: RI spectrograms are enhanced by a CNN that treats real and imaginary spectrograms as different input channels.The approach exploits the similar structure of RI and magnitude spectrograms while avoiding direct estimation of phase.
- Pseudo objectives: Pseudo hidden and output layers allow other objectives, such as log-power spectra or waveform reconstruction, to be connected to the RI-enhancement network.These representations can be depicted as functions of RI spectrograms because the RI outputs retain the same information amount as the raw waveform.
- RI reconstruction: The primary objective reconstructs clean real and imaginary spectrograms from their enhanced counterparts.The formulation uses vertically cascaded clean and enhanced RI vectors and distinguishes their real and imaginary components.
4. MULTI-METRICS LEARNING
The paper combines RI-spectrogram reconstruction with LPS reconstruction in a unified objective, linking waveform-oriented and spectrogram-oriented enhancement criteria. The resulting multi-metrics learning objective uses weighting factors to balance these targets.
- 4.1. Relation between RI spectrogram and waveform: RI-spectrogram enhancement differs from waveform enhancement only by a linear matrix multiplication, so their enhancement results follow similar trends.The matrix multiplication does not introduce nonlinear effects during back-propagation.
- 4.1. Relation between RI spectrogram and waveform: Optimizing RI spectrograms is related to maximizing segmental signal-to-noise ratio because RI spectrograms preserve the information needed for waveform reconstruction.
- 4.2. Incorporating LPS reconstruction term into the objective function: LPS is a nonlinear function of RI spectrograms, so reconstructing LPS through RI outputs affects enhancement results beyond the RI reconstruction term alone.
- 4.2. Incorporating LPS reconstruction term into the objective function: The LPS reconstruction term targets log-spectral distortion, while the RI reconstruction term is the objective associated with maximizing SSNR.The unified objective combines both reconstruction terms.
- 4.2. Incorporating LPS reconstruction term into the objective function: The unified objective uses α and β as weighting factors for the different target objectives.
5. EXPERIMENTS
Experiments used mismatched noise types and SNR levels for training and testing, evaluating RI-based models and multi-metrics learning with signal, spectral, intelligibility, and quality metrics. Results support benefits from RI spectrogram modeling and MML, including improved SSNR for small β.
- 5.1. Experimental setups: 600 TIMIT utterances were corrupted with five noise types at six SNR levels for training, while 100 separate utterances formed a mismatched test set.Training and test noise types and SNR levels were intentionally mismatched.
- 5.1. Experimental setups: The evaluation used SSNR and LSD for signal differences, plus PESQ and STOI for speech quality and intelligibility.PESQ and STOI were reported although they were not included in the MML objective.
- 5.2.1. Phase effects: Using noisy phase with clean magnitude degrades synthesized-waveform SSNR more severely at low input SNRs.The reported SSNR decreases from 13.43 dB at 12 dB input SNR to 2.149 dB at −12 dB input SNR.
- 5.2.2. Comparison of different models: RI-DNN improves SSNR, STOI, and PESQ over the magnitude-enhancing DNN baseline, while the baseline achieves the lowest LSD.RI-CNN further outperforms RI-DNN, indicating stronger feature extraction in the CNN model.
- 5.2.3. Results of MML: For small β, the RI and LPS objective terms do not conflict because both have clean speech as their optimal solution in different representations.The LPS term makes estimated real and imaginary spectrograms influence each other.
6. CONCLUSIONS
The paper concludes that its CNN estimates clean RI spectrograms and that MML jointly improves LSD and SSNR when β is properly specified.
- The proposed CNN estimates clean real and imaginary spectrograms from noisy ones for enhanced-waveform synthesis with more accurate phase information.
- MML incorporates multiple metrics into the objective function by using signal representations expressible as functions of RI spectrograms.
- MML can simultaneously improve LSD and SSNR when β is properly specified.
- The reported improvements are explained by treating MML as constraints, or pseudo layers, added to the original objective function to enhance generalization.
- Future work will investigate incorporating STOI and PESQ and developing MML objectives beyond simple weighted sums.