Source-linked AI summary
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement
Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, Shou-De Lin
TL;DR
Conventional GAN losses may fail to optimize complex target metrics because discriminator judgments are not matched to evaluation scores. MetricGAN learns a metric-based discriminator surrogate and uses continuous score labels to guide generators, achieving higher metric-improvement training efficiency while supporting specified single or multiple scores.
Problem
Conventional adversarial and Lp losses may not directly optimize complex evaluation metrics, creating discriminator-evaluation mismatch in speech enhancement.
Method
MetricGAN associates the discriminator with evaluation metrics, learns a surrogate from data, and uses continuous metric-score labels to guide generator updates.
Results
MetricGAN increases metric scores more efficiently than conventional supervised learning with Lp loss and achieves higher scores than other loss functions and state-of-the-art speech-enhancement models.
Takeaways & Limitations
MetricGAN can generate speech with user-specified evaluation scores and, under some non-extreme conditions, assign multiple metrics simultaneously.
Abstract
from arXiv · showhide
Adversarial loss in a conditional generative adversarial network (GAN) is not designed to directly optimize evaluation metrics of a target task, and thus, may not always guide the generator in a GAN to generate data with improved metric scores. To overcome this issue, we propose a novel MetricGAN approach with an aim to optimize the generator with respect to one or multiple evaluation metrics. Moreover, based on MetricGAN, the metric scores of the generated data can also be arbitrarily specified by users. We tested the proposed MetricGAN on a speech enhancement task, which is particularly suitable to verify the proposed approach because there are multiple metrics measuring different aspects of speech signals. Moreover, these metrics are generally complex and could not be fully optimized by Lp or conventional adversarial losses.
1. Introduction
MetricGAN addresses discriminator-evaluation mismatch by learning a metric surrogate and using it to optimize speech-enhancement generators. It also supports user-specified single or multiple evaluation scores.
- Motivation: Conventional adversarial and Lp losses may not align generator training with complex speech-enhancement metrics.The paper identifies this mismatch as a reason conventional GAN-based enhancement can yield marginal or degraded objective scores.
- MetricGAN approach: MetricGAN learns a discriminator surrogate of the evaluation metrics directly from data without requiring computational details of the metric function.The discriminator’s labels are tied to evaluation scores rather than fixed binary real/fake labels.
- Results: MetricGAN training efficiency for increasing metric scores is higher than conventional supervised learning with Lp loss.This is reported as an experiment result in the introduction.
- Capabilities: Continuous discriminator labels let users assign desired evaluation scores to generated speech.The approach is designed to generate speech according to specified metric scores.
- Capabilities: Under some non-extreme conditions, multiple discriminators allow MetricGAN to assign multiple metrics simultaneously.The paper reports this capability while qualifying it for non-extreme conditions.
2. CGAN for SE
In conditional GAN speech enhancement, the generator maps noisy speech to clean speech while adversarial training encourages generated speech to resemble real data. The paper motivates associating discriminator training with the evaluation metric instead.
- CGAN formulation: GAN-based speech enhancement commonly uses a conditional generator to map noisy speech x to corresponding clean speech y.The generator is trained in conjunction with a discriminator under a least-squares GAN formulation.
- CGAN formulation: The conventional generator loss combines an adversarial term, weighted by λ, with an Lp reconstruction loss.The adversarial term trains the generator to cheat the discriminator, while the reconstruction term measures distance to clean speech.
- CGAN formulation: The discriminator’s conventional objective is to distinguish real data from generated data.This binary real/fake discrimination is the objective that MetricGAN later associates with evaluation metrics.
- Motivation: MetricGAN instead associates discriminator training with the evaluation metric to optimize metric scores.The paper explicitly proposes linking discriminator training to the metric of interest.
3. MetricGAN
MetricGAN trains the discriminator to approximate the evaluation metric and trains the generator against that learned evaluator. Continuous metric-based labels allow optimization toward user-specified scores.
- Associating the Discriminator with the Metrics: MetricGAN defines Q as the evaluation metric to be optimized and aligns the discriminator's behavior with Q.For PESQ and STOI, Q takes the generated-or-evaluated speech together with clean speech as input.
- Associating the Discriminator with the Metrics: Generated samples receive metric-based continuous targets rather than the constant false label used in conventional CGANs.This lets D estimate the degree of clean-speech realness instead of only distinguishing real from fake.
- Associating the Discriminator with the Metrics: MetricGAN uses the clean speech y as D's condition because D is intended to behave like the metric Q.The discriminator input is therefore chosen to match Q's input rather than the noisy-speech condition used in CGAN.
- Continuous Space of the Discriminator Label: The generator can be trained solely with MetricGAN's adversarial loss because the discriminator provides a more efficient gradient than Lp loss.The generator is trained to make D output the desired assigned score s.
- Continuous Space of the Discriminator Label: Users can specify the target score s: assigning s = 1 aims at clean speech, whereas smaller s can generate noisier speech.This flexibility follows from the discriminator labels being continuous and related to the evaluation metric.
- Learning Process: MetricGAN alternates local metric approximation by D with generator weight updates until convergence.D approximates Q in the observed region, while G adjusts its weights toward D's optimum.
4.1. Network Architecture
The speech-enhancement system uses a BLSTM generator for time-frequency mask estimation and a CNN discriminator designed for variable-length inputs and smooth score prediction.
- Generator: The generator receives normalized noisy magnitude-spectrogram utterances as input.Its estimated mask is multiplied with the noisy magnitude spectrogram to remove noise components.
- Generator: The generator is a BLSTM with two bidirectional LSTM layers followed by fully connected layers producing a 257-dimensional sigmoid mask.The LSTM layers have 200 nodes each, and the fully connected layers have 300 LeakyReLU nodes and 257 sigmoid nodes.
- Discriminator: The discriminator is a four-layer 2-D CNN followed by global average pooling and fully connected layers.Global pooling fixes the feature representation at 50 dimensions for variable-length utterances.
- Evaluation: Table 1 compares the loss functions using PESQ and STOI, with an asterisk marking models pretrained from another model.These metrics assess speech quality and intelligibility, respectively.
- Discriminator: Spectral normalization constrains the discriminator to be 1-Lipschitz continuous, promoting smooth score estimates.The design aims to prevent small spectrogram changes from causing large estimated-score differences.
4.2. Experiment on the TIMIT Dataset
On TIMIT speech enhancement, MetricGAN trains generators against PESQ or STOI through metric-associated discriminators, outperforming conventional objectives in score optimization, efficiency, and selected perceptual tests. It also supports assigning desired single or multiple metric scores, although extreme multi-metric assignments can be difficult to converge.
- PESQ ranges from -0.5 to 4.5, while STOI ranges from 0 to 1; higher scores indicate better speech quality or intelligibility.
- The TIMIT experiment used 300 training utterances corrupted by ten noise types and evaluated on 2,000 utterances containing four unseen noise types.
- Objective evaluation: MetricGAN(P) achieved the highest PESQ score and the second-highest STOI score among the evaluated models, while MetricGAN(S) achieved the best STOI score.MetricGAN(S) showed its clearest improvement under low-SNR conditions.
- Objective evaluation: MetricGAN learned metric scores more efficiently than the other objectives, but performance was sub-optimal when its training metric Q did not match the evaluation metric.The authors report that training without Q did not converge.
- Subjective evaluation: MetricGAN(S) produced spectrograms with clearer recovered speech structures and the highest STOI score, whereas increasing conventional adversarial-loss weight introduced unpleasant artifacts.
- Subjective evaluation: In AB preference tests, MetricGAN(P) significantly outperformed both baselines, while IRM(L1) and IRM(CGAN) had overlapping confidence intervals.The tests compared randomized sample pairs using 20 pairs and 15 listeners per comparison.
- Metric-score assignment: MetricGAN can assign desired single-metric scores, but larger scores are harder to obtain because the number of solutions decreases as the target increases.The generated speech components gradually disappear when assigning low STOI scores.
- Multi-metric score assignment: For multiple metrics, MetricGAN uses one discriminator per metric and trains the generator using the discriminator with the largest achieved-to-assigned score distance.All discriminators are updated during discriminator training.
4.3. Comparison with Other State-of-the-Art SE Models
MetricGAN is compared with five state-of-the-art speech-enhancement baselines using the same dataset split and multiple test-set metrics. It outperforms the previous methods on three reported metrics despite being trained only to optimize PESQ.
- The comparison uses a publicly available noisy-clean dataset and the exact same training and test split as previous speech-enhancement works.
- The test set reports PESQ, CSIG, CBAK, and COVL to evaluate speech quality, distortion, background-noise interference, and overall quality.
- Five baselines are compared with MetricGAN: SEGAN, MMSE-GAN, WGAN-GP, SERGAN, and Deep Feature Loss.The baselines use adversarial, L1, L2, Wasserstein, relativistic least-square, or deep-feature losses.
- MetricGAN outperforms all previous works with respect to three metrics.The cited passage states that the highest score per metric is highlighted in Table 2.
5. Conclusion
The paper proposes MetricGAN to directly optimize generators for one or multiple evaluation metrics. In speech enhancement, it outperforms other loss functions and state-of-the-art models while supporting specified metric scores.
- MetricGAN directly optimizes generators based on one or multiple evaluation metric scores.
- Associating the discriminator with target metrics lets MetricGAN learn a surrogate that captures metric behavior and provides gradients for generator updates.
- MetricGAN outperforms other loss functions and state-of-the-art speech-enhancement models.
- MetricGAN can be trained to generate data according to designated metric scores.
- The paper identifies this as the first GAN-based work to train a generator directly with respect to multiple evaluation metrics.