Source-linked AI summary
SEGAN: Speech Enhancement Generative Adversarial Network
Santiago Pascual, Antonio Bonafonte, Joan Serrà
TL;DR
Speech enhancement has largely relied on spectral or higher-level representations and has often covered limited noise conditions. The paper proposes SEGAN, an end-to-end waveform-level GAN that shares parameters across speakers and noise types, and reports viable, effective enhancement in objective and subjective evaluations.
Problem
Existing speech enhancement methods mainly operate in the spectral domain or use higher-level features, often addressing limited speakers and noise conditions.
Method
SEGAN uses a fully convolutional generator to map noisy speech and latent noise to enhanced waveforms within an end-to-end adversarial framework.
Results
Objective and subjective evaluations report that SEGAN is a viable and effective alternative to current speech enhancement approaches.
Takeaways & Limitations
The work opens exploration of generative architectures for waveform-level speech enhancement with shared parameters across speakers and noise types.
Abstract
from arXiv · showhide
Current speech enhancement techniques operate on the spectral domain and/or exploit some higher-level feature. The majority of them tackle a limited number of noise conditions and rely on first-order statistics. To circumvent these issues, deep networks are being increasingly used, thanks to their ability to learn complex functions from large example sets. In this work, we propose the use of generative adversarial networks for speech enhancement. In contrast to current techniques, we operate at the waveform level, training the model end-to-end, and incorporate 28 speakers and 40 different noise conditions into the same model, such that model parameters are shared across them. We evaluate the proposed model using an independent, unseen test set with two speakers and 20 alternative noise conditions. The enhanced samples confirm the viability of the proposed model, and both objective and subjective evaluations confirm the effectiveness of it. With that, we open the exploration of generative architectures for speech enhancement, which may progressively incorporate further speech-centric design choices to improve their performance.
1. Introduction
The paper motivates waveform-level, adversarial speech enhancement as an alternative to predominantly spectral methods, targeting broader speaker and noise coverage. SEGAN is presented as an end-to-end shared model with fast, raw-audio processing and no hand-crafted features.
- Limitations of current approaches: Most speech enhancement systems use short-time Fourier analysis and modify spectrum magnitude, while phase can also affect speech quality.Earlier methods include spectral subtraction, Wiener filtering, statistical model-based methods, subspace algorithms, and neural architectures.
- Motivation for GANs: GANs had succeeded in generating realistic high-dimensional data, but had not yet been applied to speech generation or enhancement in this work’s account.The paper positions SEGAN as the first adversarial approach for generating speech signals.
- SEGAN contributions: SEGAN enhances raw audio end-to-end without extracting hand-crafted features or imposing explicit assumptions about the raw data.The model operates directly on waveform inputs rather than a spectral representation.
- SEGAN contributions: SEGAN shares one parameterization across different speakers and noise types, making the system simple and generalizable along those dimensions.The contribution is framed as incorporating multiple speakers and noise conditions into one model.
- SEGAN contributions: SEGAN provides quick enhancement without causality or recursive operations such as those used in recurrent neural networks.The stated speed advantage follows from avoiding recursive processing.
2. Generative Adversarial Networks
GANs learn a mapping from prior samples to the training-data distribution through adversarial interaction between a generator and discriminator. The discriminator distinguishes real from generated samples while the generator adapts to make its outputs appear real, with conditioned and least-squares variants extending the framework.
- GAN components: A GAN generator maps samples z from a prior distribution Z to novel samples x resembling the training distribution X.The generator learns distribution characteristics rather than memorizing input-output pairs.
- Adversarial training: The discriminator classifies dataset samples as real and generator outputs as fake, while the generator learns to fool the discriminator.This adversarial interaction moves generated outputs toward the real-data manifold described by the training data.
- Adversarial training: GAN training alternates discriminator back-propagation on real and fake batches with generator back-propagation while discriminator parameters are frozen.The generator update seeks discriminator misclassification of generated examples.
- GAN variants: Conditioned GANs add extra information to the generator and discriminator to perform the mapping and classification.The paper denotes this additional input as x_c.
- GAN variants: LSGAN replaces sigmoid cross-entropy with a binary-coded least-squares loss to address vanishing gradients and improve generated-sample quality.The binary coding assigns 1 to real samples and 0 to fake samples.
3. Speech Enhancement GAN
SEGAN enhances noisy speech directly from raw waveforms using a fully convolutional encoder-decoder generator with latent input and skip connections. Its adversarial objective is supplemented with an L1 distance term to encourage realistic, clean-signal generations.
- Architecture: SEGAN maps noisy speech and a latent vector to an enhanced waveform using a fully convolutional generator.The generator has no dense layers and outputs ˆx = G(˜x).
- Architecture: The generator compresses the input into a thought vector, concatenates it with latent noise, and reconstructs the signal through mirrored decoding layers.Strided convolutions and PReLUs perform the encoding and decoding stages.
- Architecture: Skip connections pass fine-grained waveform information from encoding layers directly to corresponding decoding layers.They help preserve details such as phase and alignment that could be lost through the compression bottleneck.
- End-to-end processing: The model processes raw speech sampled at 16 kHz without intermediate acoustic-feature transformations.This gives SEGAN an end-to-end waveform-level structure.
- Training objective: An L1 distance term supplements the adversarial loss to keep generated signals close to clean examples while allowing realistic detail.The L1 contribution is controlled by the hyper-parameter λ.
4. Experimental Setup
The experiments use separate speakers, noise types, and SNR conditions for training and testing, with waveform-based adversarial training and convolutional generator and discriminator architectures. Evaluation compares noisy, Wiener-enhanced, and SEGAN-enhanced speech using objective metrics and released implementation resources.
- Data Set: The dataset contains 28 training speakers and 2 test speakers drawn from a 30-speaker Voice Bank selection.The test speakers are held out from training.
- Data Set: Training uses 40 noise conditions, while testing uses 20 different conditions with alternative noise types and SNR values.Training includes 10 noise types at 15, 10, 5, and 0 dB; testing includes 5 noise types at 17.5, 12.5, 7.5, and 2.5 dB.
- SEGAN Setup: Training pairs noisy-clean signals for the real pair and noisy-enhanced signals for the fake pair in adversarial learning.The model is trained for 86 epochs with RMSprop at a learning rate of 0.0002 and effective batch size 400.
- SEGAN Setup: Waveform preprocessing downsamples utterances to 16 kHz and uses approximately one-second training chunks with 50% overlap.A high-frequency preemphasis filter with coefficient 0.95 is applied during training and testing.
- Evaluation: Objective evaluation compares enhanced speech with clean references using PESQ, CSIG, CBAK, COVL, and SSNR, alongside noisy and Wiener baselines.The measures are computed on 824 test files.
5. Results
Objective and subjective evaluations compare SEGAN with noisy signals and a Wiener baseline. SEGAN improves most speech/noise distortion metrics and is preferred in comparative listening tests, although its advantage over Wiener is not significant.
- 5.1. Objective Evaluation: The objective measures assess perceptual quality, speech distortion, background-noise intrusiveness, overall effect, and segmental SNR.All metrics are defined with higher values interpreted as better.
- 5.1. Objective Evaluation: SEGAN slightly worsens PESQ but outperforms Wiener on CSIG, CBAK, SSNR, and COVL.The metrics compare enhanced signals with clean references across 824 test files.
- 5.2. Subjective Evaluation: 16 listeners evaluated 20 test sentences containing noisy, Wiener-enhanced, and SEGAN-enhanced versions.Listeners rated overall quality on a 1–5 scale and could replay signals freely.
- 5.2. Subjective Evaluation: SEGAN was preferred over the noisy signal in 67% of cases, versus 8% preferring the noisy signal.No preference occurred in 25% of comparisons.
- 5.2. Subjective Evaluation: SEGAN was preferred over Wiener in 53% of cases, while Wiener was preferred in 23%.The remaining 24% of comparisons showed no preference, and the Wiener–SEGAN difference was not significant.
6. Conclusions
SEGAN implements end-to-end waveform enhancement with a fully convolutional encoder-decoder in a generative adversarial framework. The results support its viability and effectiveness, while future work targets improved convolutions, perceptual weighting, and broader comparisons.
- 6. Conclusions: SEGAN performs end-to-end speech enhancement within a generative adversarial framework.Its fully convolutional encoder-decoder operates quickly on waveform chunks.
- 6. Conclusions: The results show that SEGAN is viable and an effective alternative to current approaches.The authors identify reducing possible high-frequency artifacts and comparing additional competitive approaches as future work.