Source-linked AI summary
MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement
Szu-Wei Fu, Cheng Yu, Tsun-An Hsieh, Peter Plantinga, Mirco Ravanelli, Xugang Lu, Yu Tsao
TL;DR
Speech-enhancement training losses often poorly align with speech quality, intelligibility, or recognition goals. MetricGAN+ adds three speech-processing-informed training techniques to MetricGAN and achieves state-of-the-art VoiceBank-DEMAND results, with PESQ increasing by 0.3 over MetricGAN.
Problem
Losses based on signal-level differences are not highly correlated with speech quality, intelligibility, or word error rate, motivating task-specific objectives.
Method
MetricGAN+ improves MetricGAN with three speech-processing-informed training techniques, including noisy-speech discriminator training, replay-buffer reuse, and learnable sigmoid scaling.
Results
0.3 PESQ increase over MetricGAN accompanies state-of-the-art results on the VoiceBank-DEMAND dataset.
Takeaways & Limitations
The framework can be further improved and applied to different tasks, and its code was made publicly available.
Takeaways & Limitations
MetricGAN has not yet been used for word error rate minimization under noisy conditions with a black-box automatic speech recognition model.
Abstract
from arXiv · showhide
The discrepancy between the cost function used for training a speech enhancement model and human auditory perception usually makes the quality of enhanced speech unsatisfactory. Objective evaluation metrics which consider human perception can hence serve as a bridge to reduce the gap. Our previously proposed MetricGAN was designed to optimize objective metrics by connecting the metric with a discriminator. Because only the scores of the target evaluation functions are needed during training, the metrics can even be non-differentiable. In this study, we propose a MetricGAN+ in which three training techniques incorporating domain-knowledge of speech processing are proposed. With these techniques, experimental results on the VoiceBank-DEMAND dataset show that MetricGAN+ can increase PESQ score by 0.3 compared to the previous MetricGAN and achieve state-of-the-art results (PESQ score = 3.15).
1. Introduction
Task-specific speech enhancement requires objectives aligned with quality, intelligibility, or recognition goals, because simple signal-distance losses correlate poorly with these outcomes. MetricGAN addresses black-box metric optimization, and MetricGAN+ adds three speech-processing techniques to improve it.
- Motivation: Speech enhancement targets differ across human communication and human-machine communication, motivating task-specific models.Quality or intelligibility may matter in human communication, whereas recognition performance such as WER matters in ASR.
- Motivation: L1 and L2 losses are not highly correlated with speech quality, intelligibility, or WER.
- Objective metrics: PESQ and STOI serve as human-perception-related objective metrics for speech quality and intelligibility.They show higher correlation with subjective listening tests than simple L1 or L2 distances.
- Existing approaches: White-box methods require metric details and target a specified metric, whereas black-box methods treat metrics as rewards but are usually inefficient.
- MetricGAN: MetricGAN improves training efficiency and increases average PESQ by more than 0.1 compared with conventional L1 loss.It can optimize different evaluation metrics, including PESQ, STOI, and WER.
- MetricGAN+: MetricGAN+ adds noisy-speech discriminator training, replay-buffer reuse, and a per-frequency learnable sigmoid for mask estimation.The discriminator receives noisy speech and historical generated speech, while the generator uses frequency-specific sigmoid parameters.
2. Introduction to MetricGAN
MetricGAN learns a neural surrogate for a target evaluation metric and uses that surrogate to train speech enhancement. Its discriminator matches target scores, while the generator seeks a desired assigned score through adversarial loss.
- MetricGAN framework: MetricGAN trains a neural network to mimic a target evaluation function and uses the learned surrogate as the enhancement model’s loss.The target metric is treated as a black box and contributes raw scores during surrogate learning.
- Metric representation: The target metric Q′ is normalized between 0 and 1 and evaluates enhanced or noisy speech together with corresponding clean speech.
- Discriminator: The discriminator objective minimizes its squared-score difference from Q′ on clean and enhanced speech.For clean speech, Q′(y, y) = 1; enhanced-speech scores satisfy 0 ≤ Q′(G(x), y) ≤ 1.
- Generator: The generator is trained solely with adversarial loss to make the discriminator output a desired assigned score.For clean-speech generation, the assigned score s can be set to 1.
3. From MetricGAN to MetricGAN+
MetricGAN+ improves MetricGAN through noisy-speech discriminator training, replay-buffer reuse, and a learnable frequency-specific sigmoid for mask estimation. These modifications target discriminator stability and greater flexibility in speech enhancement.
- Design overview: MetricGAN+ introduces three modifications intended to improve performance and training efficiency.
- Noisy-speech discriminator training: Noisy speech is added to discriminator training so its outputs also match target metric scores for noisy-clean pairs.The added loss term minimizes the difference between D and Q′ for noisy speech.
- Replay buffer: Historical enhanced samples are reused from a replay buffer to reduce discriminator forgetting of previously generated speech.MetricGAN+ samples a portion of the buffer to update the discriminator alongside current data.
- Learnable sigmoid: The sigmoid scale β is set to 1.2 because the ideal magnitude mask is not guaranteed to be below 1.
- Learnable sigmoid: The learnable sigmoid gives different frequency bands their own learned compression parameter α for mask estimation.This flexibility reflects distinct speech and noise patterns across frequency bands.
- Learnable sigmoid: Large α produces a threshold-like, saturated mask response, while small α produces a more linear response.The response shapes are illustrated in Figure 2.
4. Experiments
Experiments evaluate MetricGAN+ on VoiceBank-DEMAND using PESQ and additional speech-quality metrics, while ablations examine replay-buffer history, learnable sigmoid behavior, and training settings.
- Experimental setup: VoiceBank-DEMAND contains 11,572 training utterances from 28 speakers and 824 test utterances from 2 speakers across four SNR conditions.The experiment evaluates PESQ, CSIG, CBAK, and COVL.
- Experimental setup: The generator is a BLSTM with two bidirectional LSTM layers and learnable sigmoid mask-estimation outputs.The mask is multiplied with the noisy magnitude spectrogram to remove noise components.
- Ablation study: Removing input spectrogram normalization and adding noisy speech to discriminator training improve performance more than further increasing replay-buffer sample size.The ablation study evaluates cumulative additions of MetricGAN+ training techniques.
- Replay buffer: 2.82 PESQ is obtained without replay-buffer history, while sampling 10% and 20% historical enhanced data raises the score by 0.2 and 0.23, respectively.No further improvement is observed at history_portion=0.3.
- Learnable sigmoid: Learnable sigmoid further increases scores; most learned α values are below 1 and near 0.5, while high-frequency bins learn substantially larger values.Smaller α produces more linear behavior, whereas larger α produces more binary-mask-like behavior.
- Comparative results: The experiments compare MetricGAN+ with other methods on the VoiceBank-DEMAND dataset.The comparison is reported in Table 3.
5. Future work
The authors identify extensions for MetricGAN+ involving additional metrics, discriminator design, and more efficient replay-buffer training.
- Additional metrics: MetricGAN could be applied to WER minimization for black-box ASR models under noisy conditions, which remains unexplored to the authors’ knowledge.Speech intelligibility optimization has already been studied in related work.
- Discriminator design: The discriminator could use attention instead of global pooling, and more complicated target functions may require a more complex discriminator.The current discriminator is described as a simple CNN with global average pooling.
- Training efficiency: Replay-buffer training is time-consuming when the buffer contains substantial historical data; incremental learning is suggested as a possible solution.This is presented as a direction for improving training efficiency.
6. Conclusion
MetricGAN+ improves MetricGAN through speech-processing-informed training techniques and achieves state-of-the-art results on VoiceBank-DEMAND.
- Conclusion: Including noisy speech for discriminator training and applying learnable sigmoid are identified as the most useful techniques.These techniques are among the proposed modifications to MetricGAN.
- Conclusion: 0.3 PESQ is the increase achieved by MetricGAN+ over MetricGAN, while the increase over BLSTM (MSE) is 0.45.The reported results are on the VoiceBank-DEMAND dataset.
- Conclusion: MetricGAN+ achieves state-of-the-art results on the VoiceBank-DEMAND dataset.The authors made the code publicly available for further improvement and application to different tasks.