Source-linked AI summary
CMGAN: Conformer-based Metric GAN for Speech Enhancement
Ruizhe Cao, Sherif Abdulatif, Bin Yang
TL;DR
Speech enhancement must improve perceived quality and intelligibility, but time-domain methods can struggle to capture frequency-domain phonetics, while time-frequency methods face challenges in estimating phase and complex spectra. CMGAN addresses these issues with a conformer-based generator that jointly uses magnitude and complex spectrogram information, plus a metric discriminator, and outperforms prior methods on Voice Bank+DEMAND with relatively few parameters.
Problem
Speech enhancement quality affects ASR, telecommunications, and hearing aids, while existing time-domain and time-frequency approaches face limitations in frequency representation, phase handling, and bounded complex-spectrogram estimation.
Method
CMGAN uses two-stage time-frequency conformer blocks with decoupled magnitude-mask and complex decoders, jointly reconstructing enhanced speech under a metric discriminator.
Results
CMGAN outperforms current state-of-the-art methods on Voice Bank+DEMAND and achieves this with relatively few parameters, totaling 1.83 M.
Takeaways & Limitations
Combining conformers across time and frequency with metric-based optimization supports strong speech-enhancement performance across the evaluated metrics.
Takeaways & Limitations
The discussed time-domain paradigm lacks direct frequency representation and can require computationally complex frameworks because of its large raw-waveform input space.
Abstract
from arXiv · showhide
Recently, convolution-augmented transformer (Conformer) has achieved promising performance in automatic speech recognition (ASR) and time-domain speech enhancement (SE), as it can capture both local and global dependencies in the speech signal. In this paper, we propose a conformer-based metric generative adversarial network (CMGAN) for SE in the time-frequency (TF) domain. In the generator, we utilize two-stage conformer blocks to aggregate all magnitude and complex spectrogram information by modeling both time and frequency dependencies. The estimation of magnitude and complex spectrogram is decoupled in the decoder stage and then jointly incorporated to reconstruct the enhanced speech. In addition, a metric discriminator is employed to further improve the quality of the enhanced estimated speech by optimizing the generator with respect to a corresponding evaluation score. Quantitative analysis on Voice Bank+DEMAND dataset indicates the capability of CMGAN in outperforming various previous models with a margin, i.e., PESQ of 3.41 and SSNR of 11.10 dB.
1. Introduction
Speech enhancement is important for speech quality and intelligibility in ASR, telecommunications, and hearing aids, but existing time- and TF-domain paradigms have distinct limitations. CMGAN addresses these issues with two-stage conformer blocks and a metric discriminator, and outperforms prior approaches on Voice Bank+DEMAND.
- Speech enhancement directly affects perceived speech quality and intelligibility in ASR, telecommunications, and hearing-aid systems.
- Time-domain methods preserve waveform information but struggle to capture frequency-domain phonetics and often require computationally complex frameworks.
- TF-domain methods face challenges from phase modeling, magnitude–phase compensation, and unbounded complex-spectrogram estimates.
- CMGAN combines a TF-domain generator with two-stage conformer blocks that sequentially capture time and frequency dependencies.Its generator uses magnitude and complex inputs with separate mask and complex decoders.
- A metric discriminator improves the corresponding evaluation metric without adversely affecting other metrics.
- CMGAN outperforms former approaches on the Voice Bank+DEMAND dataset.
2. Methodology
CMGAN processes compressed time-frequency features with two-stage conformer blocks that separately model time and frequency dependencies. Its decoupled decoders estimate magnitude and complex components, while metric-aware adversarial and waveform losses guide enhancement.
- Generator architecture: STFT converts noisy waveforms into complex spectrograms, followed by power-law compression before generator processing.The compressed magnitude, phase, real, and imaginary components are used to form the generator input.
- Encoder: The encoder uses convolution blocks and a dilated DenseNet to extract multiscale features while reducing the frequency dimension.Dilated convolutions enlarge the receptive field, and the final convolution block halves the frequency dimension to reduce complexity.
- Two-stage conformer blocks: Sequential conformer blocks capture time dependency first and frequency dependency second through reshaping and residual connections.Conformers combine transformer-style long-distance modeling with convolutional local-feature extraction.
- Decoder: The decoder separately predicts a magnitude mask and complex real-imaginary components, then combines them to reconstruct the enhanced spectrogram and waveform.The magnitude path uses the noisy phase, the complex path predicts real and imaginary parts, and inverse compression plus ISTFT produces the time-domain signal.
- Metric-aware training: The metric discriminator learns normalized PESQ behavior because evaluation metrics may be nondifferentiable and poorly aligned with objective losses.The generator combines time-frequency reconstruction, adversarial, and waveform losses with weights chosen to reflect equal importance.
3. Experiments
The experiments use the Voice Bank+DEMAND corpus with speaker-disjoint training and test utterances mixed with multiple noise types at several signal-to-noise ratios. Training uses fixed two-second segments, whereas test utterances retain variable duration.
- Dataset: Voice Bank+DEMAND contains 11,572 training utterances from 28 speakers and 824 test utterances from 2 unseen speakers.Training mixtures include eight DEMAND noise types and two artificial noise types at SNRs of 0 dB, 5 dB, and 10 dB.
- Preprocessing: Training utterances are sliced into 2-second segments, while test utterances remain unsliced with variable duration.This creates different segment-length handling between training and evaluation.
- Spectral representation: The spectrogram uses a 25 ms Hamming window, 400-point FFT, and 6.25 ms hop, yielding 200 frequency bins.The time dimension varies with the duration of each test track.
4. Results and discussion
CMGAN outperforms the compared speech-enhancement baselines on Voice Bank+DEMAND, while ablations support combining magnitude and complex processing, metric discrimination, and sequential time-frequency conformers.
- Baseline comparison: CMGAN outperforms recent transformer-based methods across all evaluation scores with only 1.83 M parameters.The comparison includes DB-AIAT and DPT-FSNet.
- Baseline comparison: 0.26, 0.49, 0.78 and 0.48 improvements over MetricGAN+ are reported for PESQ, CSIG, CBAK and COVL, respectively.These comparisons are reported for frameworks involving metric discriminators.
- Input ablation: 0.18 PESQ points are lost without phase enhancement, while pure complex-spectrogram input reduces SSNR by 1.91 dB.The ablation compares magnitude-only and complex-only inputs.
- Discriminator ablation: Removing the discriminator negatively impacts all reported scores, whereas a patch discriminator provides only marginal improvement.The metric discriminator is therefore more beneficial for directly improving evaluation scores than the tested patch discriminator.
- Loss ablation: PESQ reaches 3.45 without time loss, but SSNR is slightly lower than in the original CMGAN.The authors interpret time loss as balancing PESQ and SSNR performance.
- Conformer ablation: 0.06 PESQ points and 0.47 dB SSNR are lost with parallel rather than sequential conformer blocks.The sequential configuration models time and frequency dimensions in the proposed order.
5. Conclusions
The paper introduces CMGAN for speech enhancement using both magnitude and complex spectrogram components, combining conformers with a metric discriminator. It reports state-of-the-art performance on Voice Bank+DEMAND with 1.83 M parameters and identifies subjective evaluation and broader task coverage as future work.
- Conclusion: CMGAN combines conformers for time-frequency dependencies with a metric discriminator that directly enhances non-differentiable evaluation scores.The approach operates on both magnitude and complex spectrogram components.
- Conclusion: CMGAN outperforms current state-of-the-art methods on Voice Bank+DEMAND with relatively few parameters, using 1.83 M parameters.An ablation study verifies the benefits of the framework's components and losses.
- Future work: Future work includes subjective listening tests and expansion to dereverberation and audio superresolution.