Source-linked AI summary
MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement
Szu-Wei Fu, Cheng Yu, Tsun-An Hsieh, Peter Plantinga, Mirco Ravanelli, Xugang Lu, Yu Tsao
TL;DR
Speech-enhancement training losses often poorly align with speech quality, intelligibility, or recognition goals. MetricGAN+ adds three speech-processing-informed training techniques to MetricGAN and achieves state-of-the-art VoiceBank-DEMAND results, with PESQ increasing by 0.3 over MetricGAN.
Problem
Losses based on signal-level differences are not highly correlated with speech quality, intelligibility, or word error rate, motivating task-specific objectives.
Source: §1 · p. 1Method
MetricGAN+ improves MetricGAN with three speech-processing-informed training techniques, including noisy-speech discriminator training, replay-buffer reuse, and learnable sigmoid scaling.
Source: §1 · p. 1, §3 · p. 2, §4.3 · p. 4Results
0.3 PESQ increase over MetricGAN accompanies state-of-the-art results on the VoiceBank-DEMAND dataset.
Source: §6 · p. 4Takeaways & Limitations
The framework can be further improved and applied to different tasks, and its code was made publicly available.
Source: §6 · p. 4MetricGAN has not yet been used for word error rate minimization under noisy conditions with a black-box automatic speech recognition model.
Source: §5 · p. 4
Abstract
from arXiv · showhide
The discrepancy between the cost function used for training a speech enhancement model and human auditory perception usually makes the quality of enhanced speech unsatisfactory. Objective evaluation metrics which consider human perception can hence serve as a bridge to reduce the gap. Our previously proposed MetricGAN was designed to optimize objective metrics by connecting the metric with a discriminator. Because only the scores of the target evaluation functions are needed during training, the metrics can even be non-differentiable. In this study, we propose a MetricGAN+ in which three training techniques incorporating domain-knowledge of speech processing are proposed. With these techniques, experimental results on the VoiceBank-DEMAND dataset show that MetricGAN+ can increase PESQ score by 0.3 compared to the previous MetricGAN and achieve state-of-the-art results (PESQ score = 3.15).
1. Introduction
Task-specific speech enhancement requires objectives aligned with quality, intelligibility, or recognition goals, because simple signal-distance losses correlate poorly with these outcomes. MetricGAN addresses black-box metric optimization, and MetricGAN+ adds three speech-processing techniques to improve it.
Motivation
- Speech enhancement targets differ across human communication and human-machine communication, motivating task-specific models.Quality or intelligibility may matter in human communication, whereas recognition performance such as WER matters in ASR.Source: §1 · p. 1
- L1 and L2 losses are not highly correlated with speech quality, intelligibility, or WER.Source: §1 · p. 1
Objective metrics
- PESQ and STOI serve as human-perception-related objective metrics for speech quality and intelligibility.They show higher correlation with subjective listening tests than simple L1 or L2 distances.Source: §1 · p. 1
Existing approaches
- White-box methods require metric details and target a specified metric, whereas black-box methods treat metrics as rewards but are usually inefficient.Source: §1 · p. 1
MetricGAN
- MetricGAN improves training efficiency and increases average PESQ by more than 0.1 compared with conventional L1 loss.It can optimize different evaluation metrics, including PESQ, STOI, and WER.Source: §1 · p. 1
MetricGAN+
- MetricGAN+ adds noisy-speech discriminator training, replay-buffer reuse, and a per-frequency learnable sigmoid for mask estimation.The discriminator receives noisy speech and historical generated speech, while the generator uses frequency-specific sigmoid parameters.Source: §1 · p. 1
2. Introduction to MetricGAN
MetricGAN learns a neural surrogate for a target evaluation metric and uses that surrogate to train speech enhancement. Its discriminator matches target scores, while the generator seeks a desired assigned score through adversarial loss.
MetricGAN framework
- MetricGAN trains a neural network to mimic a target evaluation function and uses the learned surrogate as the enhancement model’s loss.The target metric is treated as a black box and contributes raw scores during surrogate learning.Source: §2 · p. 2
Metric representation
- The target metric Q′ is normalized between 0 and 1 and evaluates enhanced or noisy speech together with corresponding clean speech.Source: §2 · p. 2
Discriminator
- The discriminator objective minimizes its squared-score difference from Q′ on clean and enhanced speech.For clean speech, Q′(y, y) = 1; enhanced-speech scores satisfy 0 ≤ Q′(G(x), y) ≤ 1.Source: §2 · p. 2
Generator
- The generator is trained solely with adversarial loss to make the discriminator output a desired assigned score.For clean-speech generation, the assigned score s can be set to 1.Source: §2 · p. 2
3. From MetricGAN to MetricGAN+
MetricGAN+ improves MetricGAN through noisy-speech discriminator training, replay-buffer reuse, and a learnable frequency-specific sigmoid for mask estimation. These modifications target discriminator stability and greater flexibility in speech enhancement.
Design overview
- MetricGAN+ introduces three modifications intended to improve performance and training efficiency.Source: §3 · p. 2
Noisy-speech discriminator training
- Noisy speech is added to discriminator training so its outputs also match target metric scores for noisy-clean pairs.The added loss term minimizes the difference between D and Q′ for noisy speech.Source: §3.1 · p. 2
Replay buffer
- Historical enhanced samples are reused from a replay buffer to reduce discriminator forgetting of previously generated speech.MetricGAN+ samples a portion of the buffer to update the discriminator alongside current data.Source: §3.2 · p. 2, §3.3 · p. 3
Learnable sigmoid
- The sigmoid scale β is set to 1.2 because the ideal magnitude mask is not guaranteed to be below 1.Source: §3.3 · p. 2
- The learnable sigmoid gives different frequency bands their own learned compression parameter α for mask estimation.This flexibility reflects distinct speech and noise patterns across frequency bands.Source: §3.3 · p. 3
- Large α produces a threshold-like, saturated mask response, while small α produces a more linear response.The response shapes are illustrated in Figure 2.Source: Figure 2 · p. 2, §3.3 · p. 3
4. Experiments
Experiments evaluate MetricGAN+ on VoiceBank-DEMAND using PESQ and additional speech-quality metrics, while ablations examine replay-buffer history, learnable sigmoid behavior, and training settings.
Experimental setup
- VoiceBank-DEMAND contains 11,572 training utterances from 28 speakers and 824 test utterances from 2 speakers across four SNR conditions.The experiment evaluates PESQ, CSIG, CBAK, and COVL.Source: §4.1 · p. 3
- The generator is a BLSTM with two bidirectional LSTM layers and learnable sigmoid mask-estimation outputs.The mask is multiplied with the noisy magnitude spectrogram to remove noise components.Source: §4.2 · p. 3
Ablation study
- Removing input spectrogram normalization and adding noisy speech to discriminator training improve performance more than further increasing replay-buffer sample size.The ablation study evaluates cumulative additions of MetricGAN+ training techniques.Source: §4.3 · p. 3, Table 1 · p. 3
Replay buffer
- 2.82 PESQ is obtained without replay-buffer history, while sampling 10% and 20% historical enhanced data raises the score by 0.2 and 0.23, respectively.No further improvement is observed at history_portion=0.3.Source: §4.3 · pp. 3–4, §4.2 · p. 3
Learnable sigmoid
- Learnable sigmoid further increases scores; most learned α values are below 1 and near 0.5, while high-frequency bins learn substantially larger values.Smaller α produces more linear behavior, whereas larger α produces more binary-mask-like behavior.Source: Figure 3 · p. 3, §4.3 · p. 4
Comparative results
- The experiments compare MetricGAN+ with other methods on the VoiceBank-DEMAND dataset.The comparison is reported in Table 3.Source: Table 3 · p. 4
5. Future work
The authors identify extensions for MetricGAN+ involving additional metrics, discriminator design, and more efficient replay-buffer training.
Additional metrics
- MetricGAN could be applied to WER minimization for black-box ASR models under noisy conditions, which remains unexplored to the authors’ knowledge.Speech intelligibility optimization has already been studied in related work.Source: §5 · p. 4
Discriminator design
- The discriminator could use attention instead of global pooling, and more complicated target functions may require a more complex discriminator.The current discriminator is described as a simple CNN with global average pooling.Source: §5 · p. 4
Training efficiency
- Replay-buffer training is time-consuming when the buffer contains substantial historical data; incremental learning is suggested as a possible solution.This is presented as a direction for improving training efficiency.Source: §5 · p. 4
6. Conclusion
MetricGAN+ improves MetricGAN through speech-processing-informed training techniques and achieves state-of-the-art results on VoiceBank-DEMAND.
Conclusion
- Including noisy speech for discriminator training and applying learnable sigmoid are identified as the most useful techniques.These techniques are among the proposed modifications to MetricGAN.Source: §6 · p. 4
- 0.3 PESQ is the increase achieved by MetricGAN+ over MetricGAN, while the increase over BLSTM (MSE) is 0.45.The reported results are on the VoiceBank-DEMAND dataset.Source: §6 · p. 4
- MetricGAN+ achieves state-of-the-art results on the VoiceBank-DEMAND dataset.The authors made the code publicly available for further improvement and application to different tasks.Source: §6 · p. 4