Source-linked AI summary
HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks
Jiaqi Su, Zeyu Jin, Adam Finkelstein
TL;DR
Real-world speech recordings contain combined noise, reverberation, and equalization distortions that existing methods do not sufficiently address. HiFi-GAN uses a feed-forward WaveNet with multi-scale adversarial discriminators and deep feature matching across waveform and time-frequency domains. It generalizes across speakers, content, and environments and outperforms state-of-the-art baselines in denoising and dereverberation evaluations.
Problem
Real-world recordings contain combined noise, reverberation, and equalization distortion, while existing methods do not sufficiently address these degradations together.
Method
HiFi-GAN combines a feed-forward WaveNet with multi-scale adversarial discriminators in waveform and time-frequency domains and deep feature matching losses.
Results
HiFi-GAN outperforms all other state-of-the-art baselines in denoising and dereverberation tasks.
Takeaways & Limitations
The method generalizes to new speakers, speech content, and environments while improving the perceptual quality of enhanced speech.
Takeaways & Limitations
Experiments use 16kHz audio, whereas high-fidelity speech should target 44kHz; retraining for that rate is computationally expensive.
Abstract
from arXiv · showhide
Real-world audio recordings are often degraded by factors such as noise, reverberation, and equalization distortion. This paper introduces HiFi-GAN, a deep learning method to transform recorded speech to sound as though it had been recorded in a studio. We use an end-to-end feed-forward WaveNet architecture, trained with multi-scale adversarial discriminators in both the time domain and the time-frequency domain. It relies on the deep feature matching losses of the discriminators to improve the perceptual quality of enhanced speech. The proposed model generalizes well to new speakers, new speech content, and new environments. It significantly outperforms state-of-the-art baseline methods in both objective and subjective experiments.
1. Introduction
Real-world recordings combine noise, reverberation, and equalization distortion, while existing methods do not sufficiently address these degradations together. HiFi-GAN combines waveform enhancement, multi-domain adversarial discrimination, and deep feature matching to improve studio-quality speech enhancement.
- Motivation: Real-world recordings often contain noise, reverberation, and equalization distortion, but their combined reduction remains insufficiently addressed.The paper targets inexpensive single-channel recordings and aims to make them sound studio-quality.
- Prior methods: Traditional signal-processing methods generalize across environments but provide only modest noise and reverberation reduction.
- Prior methods: Waveform-based enhancement reduces phase-induced artifacts but remains sensitive to training data and unfamiliar noises and reverberation.
- Proposed approach: HiFi-GAN combines an end-to-end feed-forward WaveNet with deep feature matching and discriminators operating in time and time-frequency domains.Its discriminators evaluate waveforms at multiple rates and mel-spectrograms.
- Contribution: The method is reported to generalize across speakers, speech content, and environments while significantly outperforming state-of-the-art baselines.
2. Method
HiFi-GAN uses a feed-forward WaveNet enhancement network with a convolutional Postnet, multi-scale waveform and mel-spectrogram discriminators, and dynamically learned feature matching. Training combines reconstruction, spectrogram, adversarial, and feature-matching losses using simulated noisy-reverberant data.
- Generator: The enhancement network is a feed-forward WaveNet with non-causal dilated convolutions and a Postnet that removes residual artifacts.The Postnet is a stack of 12 one-dimensional convolutional layers, and losses apply before and after it.
- Discriminators: Waveform discriminators operate at 16kHz, 8kHz, and 4kHz, while a mel-spectrogram discriminator evaluates the time-frequency representation.The two domains cover different resolutions and make long-span reverberation residuals easier to identify.
- Adversarial training: Adversarial training uses hinge losses, and discriminators from two domains stabilize training while balancing perceptual factors.
- Training behavior: Adversarial training works better with the Postnet than without it, although both pre- and post-Postnet outputs improve.
- Feature matching: Deep feature matching compares intermediate discriminator feature maps, forming a partially learned perceptual loss that evaluates multiple aspects of speech quality.
- Training data: Training data are simulated by convolving studio recordings with measured room impulse responses and adding noise, with augmentation across speakers, content, and environments.
3. Experiments
Experiments evaluate HiFi-GAN with objective and subjective measures on synthetic, DAPS, and VCTK noisy datasets. The method achieves strong perceptual and denoising performance, while objective metrics do not consistently agree.
- Experimental setup: The experiments compare HiFi-GAN components and six state-of-the-art denoising and dereverberation methods.The ablation includes Base, Postnet, SpecGAN, and WaveGAN variants, alongside W+W, Deep FL, Wave-U-Net, BLSTM, and MetricGAN.
- Objective evaluations: The evaluation uses PESQ, STOI, SRMR, and FW-SSNR on synthetic and DAPS data.These are commonly used objective measures for denoising and dereverberation.
- Objective evaluations: HiFi-GAN achieves the best PESQ in both evaluated cases.The waveform discriminator alone can degrade scores, indicating that discriminator configuration matters.
- Objective evaluations: Objective metrics show no consistent ranking, so no single measure adequately captures overall subjective quality.The study therefore adds human listening evaluations.
- Subjective evaluations: HiFi-GAN scores highest in MOS, and its variants are among the best in the DAPS listening study.The study collected about 1000 valid HITs per voice, totaling 14K ratings per voice.
- Subjective evaluations: HiFi-GAN is preferred over other baselines more than 90% of the time and shows an evident advantage over SpecGAN in MOS.Each method pair received 900 ratings per voice.
- Denoising task: HiFi-GAN outperforms all other methods on the VCTK noisy denoising benchmark.The result supports generalization to different speech enhancement tasks and datasets, while background distortion scores are lower because VCTK clean recordings retain some background noise.
4. Discussion and Future Work
HiFi-GAN is presented as an end-to-end method that outperforms state-of-the-art baselines in denoising and dereverberation. Its experiments use 16kHz audio, whereas real high-fidelity applications should target 44kHz.
- Discussion: HiFi-GAN combines a feed-forward WaveNet with multi-scale adversarial training in time and time-frequency domains.Dynamic discriminator feature-matching losses are used to improve perceptual quality.
- Discussion: Evaluations show HiFi-GAN outperforms state-of-the-art baselines in denoising and dereverberation tasks.
- Future work: The experiments use a 16kHz sample rate, while real high-fidelity audio should target 44kHz.Retraining at the desired rate is computationally expensive; band extension is proposed as an alternative future direction.