Source-linked AI summary
Improving GANs for Speech Enhancement
Huy Phan, Ian V. McLoughlin, Lam Pham, Oliver Y. Chén, Philipp Koch, Maarten De Vos, Alfred Mertins
TL;DR
Existing speech-enhancement GANs commonly use a single generator for one-stage mapping, while this paper investigates chained generators for gradual, multi-stage refinement. It compares shared and independent stage mappings; DSEGAN with independent generators achieves the best reported speech-quality results, and both multi-stage models outperform SEGAN on intelligibility.
Problem
Existing speech-enhancement GANs generally use one generator for a single enhancement stage, leaving multi-stage enhancement mappings to be examined.
Source: §I · p. 1Method
ISEGAN and DSEGAN chain generators to refine noisy speech in stages, with ISEGAN sharing parameters and DSEGAN using independent parameters.
Source: §III · p. 2Results
With N = 2, DSEGAN's relative improvements over SEGAN were 7.3% PESQ, 4.7% CSIG, 6.9% CBAK, 6.2% COVL, and 18.2% SSNR; both multi-stage models outperformed SEGAN on STOI.
Source: D. Objective evaluation · pp. 3–4Takeaways & Limitations
Independent mappings yield better performance than a common mapping, while parameter sharing gives ISEGAN a smaller expected model footprint.
Source: §I · p. 1, §VI · p. 4
Abstract
from arXiv · showhide
Generative adversarial networks (GAN) have recently been shown to be efficient for speech enhancement. However, most, if not all, existing speech enhancement GANs (SEGAN) make use of a single generator to perform one-stage enhancement mapping. In this work, we propose to use multiple generators that are chained to perform multi-stage enhancement mapping, which gradually refines the noisy input signals in a stage-wise fashion. Furthermore, we study two scenarios: (1) the generators share their parameters and (2) the generators' parameters are independent. The former constrains the generators to learn a common mapping that is iteratively applied at all enhancement stages and results in a small model footprint. On the contrary, the latter allows the generators to flexibly learn different enhancement mappings at different stages of the network at the cost of an increased model size. We demonstrate that the proposed multi-stage enhancement approach outperforms the one-stage SEGAN baseline, where the independent generators lead to more favorable results than the tied generators. The source code is available at http://github.com/pquochuy/idsegan.
I. INTRODUCTION
Speech enhancement seeks to improve the quality and intelligibility of noise-degraded speech and can support speech-driven applications. This section motivates replacing single-generator SEGAN with chained enhancement stages and reports that independent mappings perform better than shared mappings.
I. INTRODUCTION
- Speech enhancement aims to improve the quality and intelligibility of speech degraded by background noise and can serve as a front end for automatic speech recognition.The passage also identifies communication systems, hearing aids, and cochlear implants as applications.Source: §I · p. 1
- The proposed ISEGAN and DSEGAN chain generators to refine noisy speech stage by stage, using either a shared mapping or independent mappings.Parameter sharing gives ISEGAN a smaller expected footprint; DSEGAN's independent parameters allow different mappings at different stages.Source: §I · p. 1
- Independent mappings in DSEGAN lead to better performance than the common mapping in ISEGAN, while the proposed methods outperform the SEGAN baseline on objective and subjective metrics.The passage reports this comparison without specifying metric values.Source: §I · p. 1
II. SEGAN
SEGAN maps noisy speech to an enhanced signal with one generator, while a discriminator distinguishes clean/noisy pairs from generated/noisy pairs. Its least-squares variant replaces discriminator cross-entropy loss, and an ℓ1 term encourages finer, more realistic outputs.
II. SEGAN
- SEGAN's generator maps noisy speech, together with a latent representation, to an enhanced signal, while its discriminator distinguishes clean/noisy pairs from generated/noisy pairs.The generator is trained to fool the discriminator into classifying the generated pair as real.Source: §II · p. 2
- SEGAN uses least-squares GAN loss to replace the discriminator's cross-entropy loss for improved training stability.Source: §II · p. 2
- An ℓ1 distance term between clean speech and generated speech encourages finer-grained, more realistic results, with its influence regulated by λ.The passage reports λ = 100 in.Source: §II · p. 2
III. ITERATED SEGAN AND DEEP SEGAN
ISEGAN and DSEGAN chain generators so each stage refines its predecessor's output, with shared parameters in ISEGAN and independent parameters in DSEGAN. Training evaluates every stage output as fake and uses a weighted objective to balance the stages.
III. ITERATED SEGAN AND DEEP SEGAN
- The chain maps each generator's predecessor output to a successively enhanced signal; ISEGAN shares one mapping, whereas DSEGAN uses independent mappings.The final generator output is treated as the enhanced signal, and intermediate outputs are checkpoints encouraged to improve progressively.Source: §III · p. 2
- The discriminator classifies the clean/noisy pair as real and every stage's enhanced/noisy pair as fake, while the generators learn to fool it.For N = 2, Fig. 2 illustrates this adversarial training arrangement.Source: Figure 2 · p. 2, §III · p. 2
- The discriminator's fake-example loss and the first generator-loss term are divided by N to balance their magnitudes against real examples and the ℓ1 term.The objective also assigns stage-specific λ_n weights, with each weight double its predecessor and the final weight fixed at 100.Source: §III · p. 2
- The stage-wise curriculum expects each generator's output to be twice as good as its predecessor in ℓ1-norm, so a stage need not produce a perfect result alone.Its successor can refine that output.Source: §III · p. 2
A. Generators Gn
ISEGAN and DSEGAN use an encoder-decoder generator for raw waveform segments. The encoder has 11 strided convolutional layers, and the decoder mirrors it with deconvolutions and skip connections.
A. Generators Gn
- Each generator processes 16,384 raw-signal samples, approximately one second at 16 kHz, through an encoder-decoder architecture.Source: A. Generators Gn · p. 2
- The encoder comprises 11 one-dimensional strided convolutional layers, each using filter width 31 and stride 2, followed by PReLUs.Its filter counts increase with depth as convolutional output sizes shrink.Source: A. Generators Gn · pp. 2–3, Figure 3 · p. 3
- The decoder mirrors the encoder with deconvolutions, and skip connections link corresponding layers to carry waveform information into decoding.Source: A. Generators Gn · p. 3, Figure 3 · p. 3
B. Discriminator D
The discriminator uses an encoder-like architecture with two-channel input and a classification head. Virtual batch normalization and a 1 × 1 convolution precede softmax classification.
B. Discriminator D
- The discriminator’s architecture resembles the generators’ encoder, but accepts two input channels.It applies virtual batch-norm before LeakyReLU with α = 0.3.Source: B. Discriminator D · p. 3
- A 1 × 1 convolution reduces the final convolutional output from 8 × 1024 to 8 features.Source: B. Discriminator D · p. 3
- A softmax layer classifies the resulting eight features.Source: B. Discriminator D · p. 3
A. Dataset
The experiments use a Voice Bank-derived corpus split by speaker, with noisy conditions constructed from specified noise types and SNRs. SEGAN and two discriminative systems provide comparison baselines.
A. Dataset
- Data from 28 speakers was used for training and data from two remaining speakers for testing.The corpus was used previously to evaluate SEGAN.Source: A. Dataset · p. 3
- Training covered 40 noise conditions; testing covered 20, and all utterances were downsampled to 16 kHz.Training combined ten noises with SNRs of 15, 10, 5, and 0 dB; testing combined five noises with SNRs of 17.5, 12.5, 7.5, and 2.5 dB.Source: A. Dataset · p. 3
B. Baseline system
- SEGAN was retrained as a baseline, alongside the DNN in and TSN in.The DNN baseline used wideband operation, smaller frames and shifts, Adam, simplified training, and validation-based early stopping; TSN followed except for wideband speech.Source: B. Baseline system · p. 3
C. Network parameters
The evaluation compares chained enhancement systems and baselines using objective metrics, finding the strongest speech-quality results with independent-stage DSEGAN. Performance trends differ across DSEGAN’s depth and ISEGAN’s iterations.
C. Network parameters
- Networks were trained for 100 epochs with RMSprop at a learning rate of 0.0002; ISEGAN and DSEGAN used smaller minibatches than SEGAN.The minibatch sizes were 50 for ISEGAN and DSEGAN versus 100 for SEGAN; tested stage counts or depths were N = {2, 3, 4}.Source: C. Network parameters · p. 3
- Training used 16,384-sample speech segments with 50% overlap and 0.95 preemphasis; test segments were processed without overlap, deemphasized, and concatenated.Source: C. Network parameters · p. 3
D. Objective evaluation
- The evaluation averages six metrics over 824 test files and reports means and standard deviations across the five latest checkpoints.The metrics are PESQ, CSIG, CBAK, COVL, SSNR, and STOI.Source: D. Objective evaluation · p. 3
- ISEGAN performs comparably to SEGAN on speech-quality metrics, while ISEGAN and DSEGAN obtain similar STOI results and both outperform SEGAN.ISEGAN slightly surpasses SEGAN in PESQ, CBAK, and SSNR for N = 2 and N = 4, but marginally underperforms in CSIG and COVL.Source: D. Objective evaluation · pp. 3–4
- DSEGAN beats SEGAN and ISEGAN across speech-quality metrics; at N = 2, its relative gains over SEGAN are 7.3% PESQ, 4.7% CSIG, 6.9% CBAK, 6.2% COVL, and 18.2% SSNR.The results are shown in Table I.Source: D. Objective evaluation · pp. 3–4, Table I · p. 4
- DNN and TSN outperform ISEGAN and DSEGAN on PESQ, CSIG, and COVL, but underperform on CBAK and especially SSNR.Both discriminative systems perform poorly on intelligibility; DNN degrades STOI, while TSN brings modest improvement, and the authors suggest more aggressive input alteration may introduce artifacts.Source: D. Objective evaluation · p. 4
- With further stages, DSEGAN’s performance improves overall, while ISEGAN trends downward on most metrics except SSNR.The metrics are tracked along DSEGAN’s depth and ISEGAN’s iterations.Source: D. Objective evaluation · p. 4, Figure 4 · p. 4
E. Subjective evaluation
A small listening study compared noisy, SEGAN, ISEGAN, and DSEGAN signals using forced binary quality judgments. The multi-stage systems beat SEGAN, but the test did not distinguish ISEGAN from DSEGAN.
E. Subjective evaluation
- Twenty volunteers assessed 20 randomly presented sentence pairs, balanced across speakers and noise types.Each pair differed only in system; preference scores were based on accumulated rewards divided by system occurrences.Source: E. Subjective evaluation · p. 4
- The three enhanced conditions were preferred over noisy signals, with scores of 0.55 to 0.45 (p < 0.05).Source: E. Subjective evaluation · p. 4
- DSEGAN and ISEGAN together significantly outperformed SEGAN (0.67 to 0.33, p < 0.001), but their scores did not differ significantly (0.48 to 0.52).The authors characterize this as a small-scale evaluation.Source: E. Subjective evaluation · p. 4
VI. CONCLUSIONS
ISEGAN and DSEGAN use chained generators for multi-stage speech enhancement, with shared parameters in ISEGAN and independent mappings in DSEGAN. Both outperform SEGAN on speech-quality and intelligibility metrics and subjective perceptual testing; independent mappings perform better than a common mapping.
VI. CONCLUSIONS
- Both ISEGAN and DSEGAN outperform SEGAN on speech-quality metrics, speech-intelligibility metrics, and subjective perceptual testing.The two proposed systems perform comparably on speech-quality metrics.Source: §VI · p. 4
- Independent generators in DSEGAN achieve better performance than ISEGAN's shared-parameter generators learning a common mapping.ISEGAN's shared parameters constrain its generators to use a common mapping across enhancement stages.Source: §VI · p. 4
- Chained generators implement multi-stage enhancement, with each generator corresponding to an enhancement mapping in the chain.The method aims to enhance speech through multiple stages.Source: §VI · p. 4