PaperlayerView exact revision on arXiv

Source-linked AI summary

CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement

Sherif Abdulatif, Ruizhe Cao, Bin Yang

arXiv:2209.11112v3cs.SDcs.AIcs.LGeess.AS

TL;DR

Speech enhancement in the TF domain must address denoising, dereverberation, and super-resolution while handling magnitude, phase, and perceptual-metric challenges. CMGAN combines shared magnitude-complex processing, conformer-based temporal-frequency modeling, and a metric discriminator, and achieves superior or competitive performance across the three tasks with 1.83 M parameters. The study also reports robust generalization to unseen noise types and distortions, while noting that real-time operation and combined-task evaluation remain open.

  • Problem

    TF-domain enhancement faces magnitude-phase compensation, while existing work has focused more narrowly on denoising and has limited coverage of complex TF-domain super-resolution.

  • Method

    CMGAN uses a shared encoder for magnitude and complex spectrogram components, two-stage conformer blocks, task-specific decoders, and a metric discriminator.

  • Results

    CMGAN achieves superior or competitive performance against state-of-the-art methods across denoising, dereverberation, and super-resolution, with robust generalization to unseen noise types and distortions.

  • Takeaways & Limitations

    The framework provides a unified TF-domain approach for three major speech enhancement tasks using relatively few parameters.

  • Takeaways & Limitations

    CMGAN has not been tested for real-time speech enhancement, accesses the whole track, and was mainly evaluated on individual rather than superimposed tasks.

Abstract

from arXiv · show

In this work, we further develop the conformer-based metric generative adversarial network (CMGAN) model for speech enhancement (SE) in the time-frequency (TF) domain. This paper builds on our previous work but takes a more in-depth look by conducting extensive ablation studies on model inputs and architectural design choices. We rigorously tested the generalization ability of the model to unseen noise types and distortions. We have fortified our claims through DNS-MOS measurements and listening tests. Rather than focusing exclusively on the speech denoising task, we extend this work to address the dereverberation and super-resolution tasks. This necessitated exploring various architectural changes, specifically metric discriminator scores and masking techniques. It is essential to highlight that this is among the earliest works that attempted complex TF-domain super-resolution. Our findings show that CMGAN outperforms existing state-of-the-art methods in the three major speech enhancement tasks: denoising, dereverberation, and super-resolution. For example, in the denoising task using the Voice Bank+DEMAND dataset, CMGAN notably exceeded the performance of prior models, attaining a PESQ score of 3.41 and an SSNR of 11.10 dB. Audio samples and CMGAN implementations are available online.

I. INTRODUCTION

Speech enhancement supports applications such as ASR, telecommunications, and hearing aids, but existing approaches face representation and computational challenges. CMGAN addresses these issues with a shared magnitude-complex generator, conformer-based temporal-frequency modeling, and a metric discriminator across denoising, dereverberation, and super-resolution.

  • I. INTRODUCTION: Speech enhancement is important for perceived quality and intelligibility in ASR, telecommunication systems, and hearing aids.
  • I. INTRODUCTION: TF-domain systems commonly prioritize magnitude or jointly enhance complex spectra, but magnitude-phase compensation can produce inaccurate magnitude estimates.This motivates combining magnitude masking with complex refinement rather than relying on complex estimation alone.
  • I. INTRODUCTION: CMGAN combines a generator with a metric discriminator that estimates and optimizes black-box, non-differentiable speech-quality metrics.The discriminator adds perceptual optimization beyond point-wise losses.
  • I. INTRODUCTION: The generator uses a shared encoder for concatenated magnitude and complex spectrogram components, then separates magnitude masking from complex refinement.This replaces separate dual-branch networks and reduces parameters through shared representation learning.
  • I. INTRODUCTION: Two-stage conformer blocks capture temporal and frequency dependencies while dual-path transformers reduce the conformer’s computational intensity.
  • I. INTRODUCTION: CMGAN extends the authors’ earlier denoising work to dereverberation and super-resolution while studying generalization, metric discrimination, and complex TF-domain super-resolution.The expanded evaluation includes unseen real and simulated distortions, multiple opinion-score evaluations, and architectural analysis for each task.

A. Denoising

Speech denoising targets suppression of background noise while preserving speech quality and intelligibility in a non-stationary source-separation problem. TF-domain methods commonly use mapping or masking, but phase handling and magnitude-phase compensation remain central challenges.

  • A. Denoising: Denoising suppresses background noise while predicting clean speech, with difficulty depending on the non-stationary speech and noise characteristics.Noise may be stationary, such as fan noise, or non-stationary, such as babble and street noise.
  • A. Denoising: TF representations are widely used because they expose time-varying frequency properties, while the unstructured phase remains a key limitation.
  • A. Denoising: TF-domain denoising methods are broadly divided into mapping-based and masking-based approaches.Mapping methods learn nonlinear transformations, whereas masking methods determine active sources in individual TF units.
  • A. Denoising: Magnitude-only masking methods retain noisy phase, motivating separate phase reconstruction or phase-denoising strategies.
  • A. Denoising: Ideal binary masking assigns 1 to TF units above an SNR threshold and 0 otherwise, improving intelligibility but potentially introducing musical-noise distortions.
  • A. Denoising: Magnitude-phase compensation allows complex-loss models to trade magnitude accuracy against difficult phase estimation, motivating bounded magnitude masks followed by complex refinement.This design combines masking-based magnitude estimation with mapping-based complex refinement.

C. Super-resolution

CMGAN treats super-resolution as reconstructing missing samples from a low-sampling-frequency input, unlike denoising and dereverberation, which suppress unwanted noise or reflections. Its generator processes compressed time-frequency representations with shared convolutional features and task-specific masking operations.

  • C. Super-resolution: Super-resolution reconstructs missing samples from a low sampling frequency input, whereas denoising and dereverberation suppress superimposed noise or reflections.
  • C. Super-resolution: For super-resolution, the masking operator uses element-wise addition rather than multiplication, reflecting reconstruction of missing frequency content.
  • C. Super-resolution: Deep-learning audio super-resolution methods have improved on traditional approaches, but limited frequency components remain a constraint for further performance gains.
  • C. Super-resolution: CMGAN’s generator converts the distorted waveform into a compressed complex spectrogram whose magnitude, real, and imaginary components form the generator input.Power-law compression uses c = 0.3 and equalizes quieter sounds relative to louder ones.
  • C. Super-resolution: The encoder combines convolutional blocks with a dilated DenseNet and halves the frequency dimension to F′ = F/2, reducing computational complexity.Dense connections aggregate feature levels, while dilation enlarges the receptive field without increasing kernel or layer counts.

1) Encoder:

CMGAN’s encoder and decoder use two-stage conformer processing and decoupled mask and complex paths. The architecture captures time and frequency dependencies while reconstructing magnitude and complex spectrogram components.

  • 1) Encoder:: Two sequential conformer blocks capture time dependency first and frequency dependency second through reshaping and residual connections.The first block operates across time, while the second operates across frequency before restoring the original feature-map shape.
  • 1) Encoder:: Each conformer combines half-step feed-forward networks, four-head self-attention, and a convolution module containing GLU, depthwise convolution, and swish activation.
  • 1) Encoder:: The decoder separates processing into mask and complex paths, with the mask decoder predicting a mask applied element-wise to the input magnitude.
  • 1) Encoder:: Both decoder paths use dilated DenseNets and sub-pixel convolution to upsample the frequency dimension back to F.
  • 1) Encoder:: The masked magnitude is combined with the noisy phase and then summed with the complex decoder output to form the final complex spectrogram.
  • 1) Encoder:: The predicted complex spectrogram is converted back to the time domain by inverse power-law compression followed by inverse short-time Fourier transform.

B. Metric discriminator

The metric discriminator addresses the mismatch between differentiable training losses and perceptual evaluation metrics by learning to mimic PESQ.

  • B. Metric discriminator: CMGAN’s discriminator estimates a metric score and incorporates it into training because PESQ and STOI are non-differentiable and objective losses may not correlate with evaluation metrics.PESQ is used as the discriminator label following MetricGAN.

C. Loss functions

CMGAN combines magnitude, complex, adversarial, and waveform losses to train the generator. The losses balance TF-domain reconstruction with perceptual metric optimization and time-domain fidelity.

  • C. Loss functions: The TF-domain loss linearly combines magnitude and complex losses, with grid search selecting α = 0.7 for best performance.
  • C. Loss functions: Adversarial training uses least-squares GAN optimization over discriminator and generator losses, with the discriminator targeting the normalized PESQ score.The normalized PESQ score lies in the range [0,1].
  • C. Loss functions: An additional waveform penalty is included because it has been shown to improve restored speech quality.
  • C. Loss functions: Figure 4 illustrates propagation of magnitude and complex losses using three-channel target and estimated spectrogram representations.
  • C. Loss functions: The final generator objective weights the corresponding losses through γ1, γ2, and γ3, chosen to reflect equal importance.

A. Datasets

CMGAN is evaluated across denoising, dereverberation, and super-resolution benchmarks using standard train/test splits, with extensive architectural and loss-function ablations. The reported studies show that phase-aware masking, sequential TS-Conformer blocks, and a metric discriminator materially influence performance.

  • Datasets and evaluation: The experiments use Voice Bank+DEMAND for denoising, REVERB for single-channel dereverberation, and VCTK-based low-resolution signals for super-resolution under standard benchmark splits.Denoising uses unseen noise types in testing, REVERB includes simulated and real recordings, and super-resolution varies the upscaling ratio.
  • Input and decoder ablations: PESQ falls from 3.35 with Single-Complex input to 3.23 with Single-Mask input, while SSNR rises from 9.19 dB to 9.82 dB with Single-Mask masking.The ablation indicates that phase information benefits PESQ, whereas explicit masking benefits SSNR.
  • Input and decoder ablations: With matched parameter counts, Dual-Mask improves SSNR by 0.38 dB over Single-Mask, whereas Dual-Complex remains below CMGAN, supporting the combination of mask and complex decoders.Single-Path decoding decreases SSNR by 0.91 dB, and Mask + cIRM reduces both PESQ and SSNR relative to CMGAN.
  • Loss-function ablations: Removing the discriminator harms all reported scores, while a patch discriminator provides only marginal improvement; the metric discriminator is beneficial for directly targeting evaluation scores.The metric-discriminator design addresses the mismatch between differentiable training losses and non-differentiable metrics such as PESQ and STOI.
  • Conformer ablations: Sequential time-then-frequency conformers outperform parallel conformers by 0.06 PESQ and 0.47 dB SSNR, while performance improves with block count until no further gain after four blocks.One conformer block improves PESQ by 0.4 compared with no conformer blocks.

B. Dereverberation

CMGAN is evaluated for dereverberation against recent TF-domain methods across simulated room conditions and real recordings, showing generally strong performance with metric-dependent exceptions.

  • B. Dereverberation: Larger rooms and farther microphones produce lower dereverberation scores because they introduce greater speech distortion.Tables III and IV cover simulated small, medium, and large rooms plus real-recording SRMR for near and far microphones.
  • B. Dereverberation: CMGAN outperforms competing methods in average CD and FWSegSNR for near microphones by 0.3 and 1.65 dB, respectively.Xiao et al. and SkipConvNet retain advantages on some SRMR conditions.
  • B. Dereverberation: CMGAN maintains gains in overall far-microphone scores, especially FWSegSNR, while Xiao et al. remains only 0.05 better in average simulated SRMR.SkipConvNet also reports slightly better real-recording SRMR.
  • B. Dereverberation: Replacing PESQ with normalized LLR in the metric discriminator improves SRMR, especially for near recordings, but standard CMGAN remains more balanced across metrics.CMGAN-LLR exceeds SkipConvNet in near and far real recordings by 0.44 and 0.75 SRMR, respectively.

C. Super-resolution

For super-resolution, CMGAN replaces masking multiplication with addition to reconstruct missing high-frequency bands, achieving strong results across upsampling scales and datasets.

  • C. Super-resolution: CMGAN modifies the mask decoder from element-wise multiplication to addition because super-resolution must complete missing high-frequency bands while preserving low-frequency input.The modified decoder therefore learns completion activations rather than conventional suppression masks bounded between 0 and 1.
  • C. Super-resolution: CMGAN outperforms all compared methods on every metric and upscaling ratio in VCTK-Multi.The reported SNR improvements across scales 2, 4, and 8 are 2.3 dB, 1.0 dB, and 2.1 dB.
  • C. Super-resolution: On VCTK-Single scale 2, CMGAN achieves the best score on all three metrics, including a 2.3 dB SNR improvement over the SOTA AE method.At scale 4, AE is marginally better in SNR and LSD10; at scale 8, CMGAN leads LSD while trailing in SNR.
  • C. Super-resolution: Removing the complex decoder and metric discriminator improves LSD but substantially reduces SNR, exposing a trade-off between magnitude fidelity and reconstructed time-domain quality.The CMGAN-Mag. variant is evaluated using magnitude loss only.
  • C. Super-resolution: In a scale-4 example, CMGAN reconstructs missing high-frequency content and accurately interpolates intermediate time-domain samples.Figure 7 displays corresponding TF-magnitude representations and 20 ms time-domain segments.

VI. OPINION SCORE EVALUATION

The paper supplements objective metrics with DNSMOS-based opinion-score evaluation because objective measures do not fully replace subjective quality judgments.

  • VI. OPINION SCORE EVALUATION: Objective scores are treated as indicators rather than complete substitutes for subjective quality measures, motivating the additional opinion-score analysis.Subjective listening tests are costly and time-consuming, creating interest in objective measures that correlate with listener judgments.
  • VI. OPINION SCORE EVALUATION: DNSMOS provides a non-intrusive subjective-quality alternative trained to mimic listener ratings for distortion, background noise, and overall quality.Its scores range from 1 to 5, and reported correlations with subjective assessments are 0.94 to 0.98.
  • VI. OPINION SCORE EVALUATION: The evaluation covers four denoising use cases to test generalization to unseen noise conditions, real noise, and additional distortions.The study includes datasets such as CHiME-3 and DNS-Challenge, with some cases lacking clean references.
  • VI. OPINION SCORE EVALUATION: Comparisons use representative models from metric-discriminator, time-domain, and TF-domain complex-denoising paradigms under standardized 16 kHz, 50% male-to-female speaker conditions.CMGAN is compared with MetricGAN+, DEMUCS, PHASEN, and PFPL; DEMUCS is retrained on Voice Bank+DEMAND for fairness.

A. DNN-Based MOS

DNSMOS evaluation shows CMGAN achieving the strongest overall speech-quality scores across four denoising use cases, with narrower score distributions than competing methods.

  • A. DNN-Based MOS: CMGAN outperforms all compared methods in overall DNSMOS across the four denoising use cases.In the first three use cases, its average improvement over PFPL is 0.15.
  • A. DNN-Based MOS: CMGAN has a much narrower DNSMOS interquartile range than the other evaluated methods.Figure 8 summarizes means, medians, interquartile ranges, and non-outlier extrema across four datasets.
  • A. DNN-Based MOS: Additional reverberation lowers DNSMOS by 0.5 on average relative to the DNS-Challenge condition without reverberation.The paper characterizes unseen reverberation as more challenging to generalize to than unseen noise types.

B. Listening test

The listening test found CMGAN had the highest subjective overall quality, while the study also identifies important evaluation boundaries for real-time and combined-task settings.

  • B. Listening test: CMGAN received the highest subjective overall-quality rating, outperforming PHASEN, the second-best method, by 0.81 on average.The listening test used 25 participants and 10 samples drawn from three speech datasets.
  • B. Listening test: CMGAN was evaluated with whole-track access rather than under real-time constraints, leaving its real-time operation and computational cost untested.Future work is described as restricting access to a few TF bins from older samples and studying floating-point operations.
  • B. Listening test: The experiments primarily treated denoising, dereverberation, and super-resolution separately, while combined denoising-and-dereverberation training and evaluation remain an extension.The superimposed effect was only briefly addressed in subjective evaluation.
  • B. Listening test: The current evaluation does not include automatic speech recognition performance, which the authors identify as a direction for subsequent research.This leaves ASR-related effectiveness outside the reported evaluation scope.
  • B. Listening test: The study concludes that CMGAN achieves superior or competitive performance against state-of-the-art methods across denoising, dereverberation, and super-resolution with 1.83 M parameters.The conclusion also reports robust generalization to unseen noise types and distortions.

APPENDIX

The appendix describes CMGAN’s architecture and visual comparisons across challenging noise conditions, highlighting differences in speech preservation, phase behavior, and generalization.

  • APPENDIX: CMGAN’s architecture is specified through convolutional, dilated dense, linear, and conformer blocks with their associated hyperparameters.Table VI provides kernel sizes, strides, channels, dilation factors, and linear-layer channel counts.
  • APPENDIX: The appendix uses baseband phase difference to visualize phase because the phase is unstructured.This helps compare phase behavior alongside time-frequency magnitude representations.
  • APPENDIX: Under challenging cafe and doorbell noise, MetricGAN+, DEMUCS, and PHASEN confuse speech with noise and produce distortions or missing speech segments.The errors are especially visible around 1.5 to 2 seconds, where speech and noise powers are similar.
  • APPENDIX: CMGAN perfectly attenuates the doorbell distortions shown in the visualization, whereas MetricGAN+ and PHASEN generalize poorly and DEMUCS and PFPL perform somewhat better.The doorbell artifacts appear at 3.5, 5, and 7 kHz for the poorer-generalizing methods.
Loading 2209.11112v3…