PaperlayerView exact revision on arXiv

Source-linked AI summary

Speech Denoising with Deep Feature Losses

Francois G. Germain, Qifeng Chen, Vladlen Koltun

arXiv:1806.10522v2eess.AScs.SD

TL;DR

Speech denoising must recover clean speech from waveforms corrupted by unknown additive background signals, particularly under difficult single-channel conditions. The paper trains a fully convolutional context aggregation network with a deep feature loss based on internal activations from an audio classification network, and reports better objective and listener-rated quality than state-of-the-art baselines and traditional regression losses, especially for the noisiest inputs.

  • Problem

    Single-channel speech denoising must remove unknown background content from speech despite complex signals and perceptually noticeable errors.

  • Method

    A fully convolutional context aggregation network is trained on raw waveforms using a weighted deep feature loss comparing clean and denoised signals through internal activations of an audio classification network.

  • Results

    The method outperforms state-of-the-art baselines in objective speech-quality metrics and large-scale human listening experiments, with the strongest advantage under the noisiest conditions.

  • Takeaways & Limitations

    Deep feature losses improve denoising quality over identical networks trained with L1 or L2 regression losses, particularly when background noise is most intrusive.

Abstract

from arXiv · show

We present an end-to-end deep learning approach to denoising speech signals by processing the raw waveform directly. Given input audio containing speech corrupted by an additive background signal, the system aims to produce a processed signal that contains only the speech content. Recent approaches have shown promising results using various deep network architectures. In this paper, we propose to train a fully-convolutional context aggregation network using a deep feature loss. That loss is based on comparing the internal feature activations in a different network, trained for acoustic environment detection and domestic audio tagging. Our approach outperforms the state-of-the-art in objective speech quality metrics and in large-scale perceptual experiments with human listeners. It also outperforms an identical network trained using traditional regression losses. The advantage of the new approach is particularly pronounced for the hardest data with the most intrusive background noise, for which denoising is most needed and most challenging.

I. INTRODUCTION

Speech denoising removes background content, but single-channel signals are under-determined and difficult to process without perceptual artifacts. This work addresses these challenges with end-to-end raw-waveform learning and a deep feature loss that outperforms prior approaches, especially for highly intrusive noise.

  • Motivation: Single-channel speech denoising is under-determined because speech is complex and the non-speech material is unknown.The task is further complicated by dense audio sampling and the perceptual noticeability of small errors.
  • Approach: The proposed system trains a fully-convolutional denoising network with a deep feature loss computed from internal activations of a pretrained audio network.The loss compares feature patterns across multiple scales in the input waveforms.
  • Results: The approach outperforms recent state-of-the-art end-to-end methods in objective speech-quality metrics and large-scale human perceptual experiments.The reported experiments indicate that it is more effective than the baselines.
  • Results: The advantage is particularly pronounced for the hardest, noisiest inputs, where denoising is most challenging.These are the conditions in which background degradation is most intrusive.
  • Prior work: Most recent denoising pipelines operate in the spectrogram domain, where inverse transforms can introduce time-aliasing artifacts.End-to-end raw-waveform methods aim to avoid time-frequency transformations and phase-information loss.

II. METHOD

The denoiser is a fully convolutional context aggregation network that maps a corrupted waveform to an enhanced waveform. Dilated convolutions provide broad temporal context, while adaptive normalization combines batch normalization with identity mapping.

  • Denoising network: The network learns a denoising operator that maps an additive-noise-corrupted speech waveform to an approximation of clean speech.The output is synthesized sample by sample as the network slides along the input.
  • Denoising network: The architecture is fully convolutional and uses context aggregation to process input signals whose sample count is not fixed in advance.Its output is synthesized sample by sample while sliding along the input.
  • Denoising network: The architecture is simpler than WaveNet because it omits skip connections, conditioning, and gated activations.Its more advanced loss function is described separately.
  • Context aggregation: Dilated convolutions aggregate long-range context without changing the sampling frequency across layers.The dilation factor increases exponentially with depth before the final output layers.
  • Context aggregation: The receptive field spans 2^14 + 1 samples, about 1 second at 16 kHz, targeting context on spoken-word time scales.The output layer uses a linear 1 × 1 convolution to synthesize each output sample.
  • Normalization: Adaptive normalization learns a weighted combination of the input identity mapping and batch normalization.The scalar weights are learned by backpropagation for each layer.

B. Feature loss

The feature loss compares clean and denoised signals through internal activations of a pretrained audio classification network. Its multi-depth representations capture signal differences at varied time resolutions and are used in a weighted L1 objective.

  • Motivation: Simple regression losses degraded output quality at low SNRs by improperly processing perceptually important low-energy speech.This motivates replacing waveform-level regression with a deep feature loss.
  • Denoising loss: Feature activations at different depths represent features with different time resolutions.Comparing multiple depths lets the loss assess signal structure across time scales.
  • Feature-loss network: Because standard pretrained classification networks were not established for audio, the authors design and train an audio feature-loss network.The network is inspired by the VGG architecture used in computer vision.
  • Feature-loss network: The feature-loss network uses 15 convolutional layers and is trained by feeding its output features to logistic classifiers with cross-entropy loss.Its layers use 3×1 kernels, batch normalization, LReLU units, zero padding, and progressive temporal decimation.
  • Denoising loss: The feature loss penalizes differences between activations produced by clean reference speech and the denoiser output in multiple network layers.The loss is a weighted L1 difference between feature activations.
  • Denoising loss: The layer weights are initialized equally and later set inversely to the relative feature discrepancies measured after 10 training epochs.The weights λ_m balance each layer’s contribution to the loss.

III. TRAINING

The feature-loss network is trained jointly on acoustic scene classification and domestic audio tagging data. Training alternates between tasks with randomized file order and uses randomly cropped continuous sections for augmentation.

  • Training tasks: The feature-loss network is trained jointly on acoustic scene classification and domestic audio tagging tasks.Only the logistic classifier parameters are task-dependent.
  • Training data: The scene-classification data contains 30-second audio files from 15 scene classes, resampled for the denoiser’s 16 kHz operating frequency.The training set contains 2,340 files after stereo channels are split into mono files.
  • Optimization: Training alternates between files from the two tasks, randomizes file order each epoch, and runs for 2,500 epochs.The network uses Xavier initialization and Adam with a learning rate of 10^-4.
  • Data augmentation: Random continuous sections rather than entire clips are presented at each iteration as data augmentation.The sampled sections have a minimum duration of 2^15 samples.

B. Speech Denoising

The study trains and evaluates speech-denoising systems using noisy speech data, baseline methods, and a mismatched test set partitioned by background conditions.

  • Data: The training dataset contains 11,572 files generated from 28 speakers, 10 background types, and four SNR levels.The data are resampled to 16 kHz before use.
  • Training: Training uses Xavier initialization, Adam with a 10^-4 learning rate, six feature-loss layers, and 320 epochs.Training takes approximately 80 hours on a Titan X GPU.
  • Baselines: The baselines include Wiener filtering, SEGAN, and a WaveNet-based raw-waveform denoiser.The WaveNet-based baseline uses stacked context aggregation modules with gated activation units.
  • Testing: The test set is partitioned into eight tranches according to composite background score.The tranches are demarcated by red dashed lines.
  • Testing: The test set contains 824 files from unseen speakers and background types, with mixtures at 2.5, 7.5, 12.5, and 17.5 dB SNR.The pipeline processes approximately 1 second of audio in 12 ms.

C. Quantitative measures

Performance is evaluated with objective quality measures and perceptual experiments across degradation levels. The reported results favor the proposed feature-loss system, especially under stronger background intrusiveness.

  • Objective measures: Objective evaluation reports SNR, SIG, BAK, and OVL scores against clean speech, with higher values indicating better performance.SIG, BAK, and OVL are composite scores on a 1.0–5.0 scale, while SNR is measured in dB.
  • Objective measures: The test set is divided into eight equal-size tranches because SNR correlates poorly with perceived degradation, whereas BAK better represents background intrusiveness.The analysis therefore evaluates performance across a continuum of degradation levels.
  • Objective results: The proposed method outperforms all baselines on every objective measure and achieves the best quality across all background-intrusiveness tranches.The margin is particularly significant for the perceptually motivated composite measures.
  • Loss comparison: Training the same denoising network with a feature loss outperforms training with L1 or L2 losses.Compared with L1, the feature loss especially improves BAK and OVL while also scoring well on SIG in noisier tranches.
  • Perceptual results: Human listeners rated the proposed output cleaner than state-of-the-art deep-network baselines in more than 83% of hardest-tranche comparisons.All reported results were statistically significant with p < 10^-3.

V. CONCLUSION

The paper presents an end-to-end speech-denoising pipeline combining a fully convolutional network with a pretrained deep feature-loss network. Experiments report better objective and human-perceptual performance, particularly in the noisiest conditions.

  • Conclusion: The pipeline uses a fully-convolutional network trained with a deep feature loss from a network pretrained on relevant audio classification tasks.The loss is intended to capture speech structure at multiple scales without added system complexity or expert loss design.
  • Conclusion: The approach significantly outperforms recent state-of-the-art baselines on objective speech-quality measures and large-scale perceptual experiments.The conclusion characterizes the combined context-aggregation and feature-loss approach as achieving state-of-the-art performance.
  • Conclusion: The performance advantage is particularly strong in the noisiest conditions, where speech denoising is most challenging.This conclusion is also reflected in the reported hardest-tranche listener comparisons.

APPENDIX

The appendix provides additional architectural details for the denoising and feature-loss networks described in the main paper.

  • APPENDIX: The appendix supplements the main text with architectural details for the denoising and feature-loss networks.

A. Denoising Network

The denoising network is a fully convolutional, 16-layer context aggregation model that transforms degraded raw audio into an enhanced output using deep feature loss training. Dilated convolutions, adaptive normalization, zero-padding, and a linear output layer provide broad temporal context while preserving signal length.

  • Context aggregation: Dilated convolutions compute intermediate representations, with dilation factor r_k = 2^(k−1) for the dilated layers.Dilation is removed between layers 13 and 14, where r_14 = 1.
  • Output synthesis: The output layer applies a linear 1×1 convolution without a nonlinearity to synthesize output samples, with a receptive field of 16385 samples.The output layer includes a learned bias term.
  • Adaptive normalization: Adaptive normalization combines batch normalization and identity mapping using learned scalar weights α_k and β_k.These layers omit bias terms because adaptive normalization is present.
  • Zero padding: Zero-padding keeps the effective length of every layer tensor equal to N.This preserves the signal length throughout the network despite convolutional processing.
  • Training loss: The network is trained by backpropagation using the deep feature loss described for the feature-loss classification network.The loss compares representations produced by a separate pretrained network.

B. Feature Loss Network

The feature-loss network is a VGG-inspired, multilayer audio classifier whose internal representations support the denoising objective. It progressively changes feature-map width and temporal resolution before producing task-specific classification outputs trained with cross-entropy.

  • Feature layer structure: The feature-loss network has 15 consecutive layers and is inspired by the VGG architecture from computer vision.Its first layer receives the input signal, while intermediate layers use progressively structured feature representations.
  • Feature layer structure: Intermediate feature layers have dimensionality N/2^m × W_m, with the number of features doubled every five layers.The network decimates between layers, reducing the length of subsequent representations.
  • Feature layer structure: The feature-loss network has a receptive field of 32767 samples and uses zero-padding so adjacent representations retain the same effective length.Temporal resolution is reduced by decimation even though effective-length alignment is maintained.
  • Classification layer: For each classification task, average pooling produces a 1×W_14 feature vector that a linear layer maps to a 1×C_p logit vector.C_p is the number of classes for task p.
  • Classification layer: The output classification vector applies a task-specific logistic nonlinearity, using softmax for unique-label tasks or sigmoid for multilabel tasks.The resulting vector has dimension 1×C_p with elements in the range [0, 1].
  • Training loss: Training uses cross-entropy between the task-specific output vector and its corresponding ground-truth classification vector.The ground-truth vector marks associated labels with 1 and other labels with 0.
Loading 1806.10522v2…