PaperlayerView exact revision on arXiv

Source-linked AI summary

Raw Waveform-based Speech Enhancement by Fully Convolutional Networks

Szu-Wei Fu, Yu Tsao, Xugang Lu, Hisashi Kawai

arXiv:1703.02205v3stat.MLcs.LGcs.SD

TL;DR

Speech enhancement systems commonly rely on magnitude-spectrum features, while raw waveform enhancement must preserve local temporal relationships needed to generate frequency components. The paper proposes a fully convolutional network that performs waveform-in, waveform-out enhancement without fully connected layers. FCN preserves local waveform structure, outperforms the LPS-based DNN baseline on STOI and PESQ, and uses approximately 0.2% as many weights as DNN and CNN.

  • Problem

    Magnitude-spectrum speech enhancement inadequately considers phase and requires time-frequency transformations, while fully connected layers may not preserve local waveform information needed for high-frequency components.

  • Method

    The paper applies a fully convolutional network to end-to-end raw waveform speech enhancement, removing fully connected layers so outputs depend locally on neighboring input regions.

  • Results

    FCN outperforms the LPS-based DNN baseline on STOI and PESQ, while using approximately 0.2% of the weights in DNN and CNN.

  • Takeaways & Limitations

    FCN can recover waveform speech components while effectively removing noise and preserving local temporal structure with relatively few weights.

Abstract

from arXiv · show

This study proposes a fully convolutional network (FCN) model for raw waveform-based speech enhancement. The proposed system performs speech enhancement in an end-to-end (i.e., waveform-in and waveform-out) manner, which dif-fers from most existing denoising methods that process the magnitude spectrum (e.g., log power spectrum (LPS)) only. Because the fully connected layers, which are involved in deep neural networks (DNN) and convolutional neural networks (CNN), may not accurately characterize the local information of speech signals, particularly with high frequency components, we employed fully convolutional layers to model the waveform. More specifically, FCN consists of only convolutional layers and thus the local temporal structures of speech signals can be efficiently and effectively preserved with relatively few weights. Experimental results show that DNN- and CNN-based models have limited capability to restore high frequency components of waveforms, thus leading to decreased intelligibility of enhanced speech. By contrast, the proposed FCN model can not only effectively recover the waveforms but also outperform the LPS-based DNN baseline in terms of short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ). In addition, the number of model parameters in FCN is approximately only 0.2% compared with that in both DNN and CNN.

I. INTRODUCTION

The study examines raw waveform inputs for speech enhancement and proposes an FCN because fully connected layers may not preserve local information needed for high-frequency waveform components.

  • I. INTRODUCTION: Conventional speech enhancement methods commonly use spectral representations, including spectral subtraction, MMSE spectral amplitude estimation, Wiener filtering, NMF, DDAE, and DNN-based models.These approaches form the prior context for investigating raw waveform-based enhancement.
  • I. INTRODUCTION: Phase is difficult to estimate precisely from noisy speech because no clear structure exists in a phase spectrogram.This motivates approaches that model phase information more directly than magnitude-spectrum-only systems.
  • I. INTRODUCTION: Raw waveform enhancement is less studied than raw waveform modeling for automatic speech recognition because enhancement must generate denoised output samples as well as distinguish speech from noise.Each estimated time-domain sample must cooperate with neighboring samples to represent frequency components.
  • I. INTRODUCTION: Fully connected layers may not preserve local information well enough to generate high-frequency components in raw speech waveforms.The study therefore investigates deep-learning-based speech enhancement methods using raw waveform features.
  • I. INTRODUCTION: The proposed FCN removes top fully connected layers so each output sample can depend locally on neighboring input regions.The paper notes that convolution in the time domain corresponds to multiplication by a filter response in the frequency domain, avoiding an explicit waveform-to-spectrogram mapping.

II. RAW WAVEFORM SPEECH ENHANCEMENT

The section motivates raw waveform speech enhancement by identifying limitations of log power spectrum features, especially their treatment of phase and their required domain transformations.

  • II. RAW WAVEFORM SPEECH ENHANCEMENT: Log power spectrum features omit well-modeled phase information, so synthesis commonly reuses noisy phase and may degrade enhanced-speech perceptual quality.The section also identifies the Fourier transforms required to map between time and frequency domains as a drawback.

A. Characteristics of Raw Waveform

Raw waveform samples carry limited information individually and must be combined with neighboring samples to represent frequency components, motivating local convolutional processing.

  • A. Characteristics of Raw Waveform: A time-domain sample point must combine information from neighbors to represent a frequency component.Samples that are very different from neighbors represent high-frequency components, while similar samples represent low-frequency components.
  • A. Characteristics of Raw Waveform: Convolution efficiently finds useful local acoustic information because time-domain patterns can occur at any position.This is why most cited raw-waveform analysis methods employ CNN models.

B. Problems in Fully Connected Layers for Modeling Raw Waveform

Fully connected output mappings use fixed weights across waveform sample positions, making it difficult to learn high- and low-frequency structures simultaneously despite DNN representational capacity.

  • B. Problems in Fully Connected Layers for Modeling Raw Waveform: 1980s ANN waveform-enhancement studies reported enhanced waveforms without higher formant structures, a phenomenon also observed in the paper’s DNN and CNN outputs.The earlier work analyzed the weight matrix between the last hidden and output layers to explain this behavior.
  • B. Problems in Fully Connected Layers for Modeling Raw Waveform: The DNN and CNN output layers connect fully to the last hidden layer, and the paper argues this connection complicates simultaneous modeling of high- and low-frequency waveform components.The output-to-hidden relation is represented as y = Wh, with bias omitted for simplicity.
  • B. Problems in Fully Connected Layers for Modeling Raw Waveform: For a fixed hidden representation, high-frequency outputs require neighboring weight vectors to differ, whereas low-frequency outputs require them to correlate.Because the fully connected weight matrix is fixed after training, both conditions cannot be satisfied simultaneously.
  • B. Problems in Fully Connected Layers for Modeling Raw Waveform: The paper qualifies that fully connected layers make learning the waveform mapping more difficult, rather than making the mapping unrepresentable by DNNs.It notes that universal approximation guarantees representational capacity with appropriate parameters but not that those parameters can be learned.

III. FCN

The FCN removes fully connected layers and models waveform outputs through local convolutional connections. Its design is illustrated through receptive-field analysis and comparisons of speech-enhancement outputs and performance.

  • III. FCN: FCN removes all fully connected layers from a conventional CNN to model raw waveform outputs with fewer parameters.Discarding fully connected layers is described as a way to dramatically reduce the network's parameter count.
  • III. FCN: The section compares FCN with DNN and CNN through high-frequency signal generation, weight correlations, spectrograms, and STOI/PESQ performance.The supplied figures and table captions identify these qualitative and quantitative comparison targets.
  • III. FCN: Each output sample is computed from a local receptive field using a shared learned filter, allowing neighboring input structure to influence waveform generation.The filter is shared across output locations, while the receptive field captures local dependencies.

A. Experimental Setup

The experiments use mismatched-noise TIMIT training and test conditions to evaluate waveform-based enhancement models. Models are trained on waveform or LPS inputs and assessed with STOI and PESQ.

  • A. Experimental Setup: 600 TIMIT training utterances were corrupted with five noise types at five SNR levels, while 100 separate test utterances used three mismatched noises and five other SNR levels.Test results were averaged across white Gaussian, engine, and baby-cry noises.
  • A. Experimental Setup: The proposed model uses 512-sample waveform frames, while the baseline system uses 257-dimensional LPS vectors.CNN, FCN, and DNN architectures differ in their convolutional and fully connected layers.
  • A. Experimental Setup: STOI and PESQ are used to evaluate speech intelligibility and speech quality, respectively.These metrics assess the two stated evaluation dimensions of speech enhancement.

B. Experimental Results

The experiments show that fully connected architectures struggle to preserve high-frequency waveform components, whereas FCN preserves speech content, removes noise, and achieves the strongest overall enhancement balance with far fewer weights.

  • Qualitative Comparison: DNN and CNN struggle to generate high-frequency waveform components because fully connected layers poorly model their local structure.The paper attributes this difficulty to high-frequency components being rare and therefore sacrificed during optimization.
  • Qualitative Comparison: Approximately 0.9 correlation between neighboring DNN weight vectors indicates strongly correlated generated samples, helping explain the difficulty of producing high-frequency waveforms.The correlation drops toward zero for weight vectors separated by a considerable distance.
  • Qualitative Comparison: FCN-enhanced spectrograms preserve speech components while effectively removing noise, unlike DNN enhancement, which misses high-frequency speech components.CNN also exhibits the missing-component phenomenon, although less severely than DNN.
  • Quantitative Comparison: The waveform-based DNN achieves the highest PESQ but worst STOI, while FCN achieves the highest STOI and a satisfactory PESQ.These results indicate that DNN does not balance intelligibility and quality as effectively as FCN.
  • Quantitative Comparison: FCN uses approximately 0.2% as many weights as both DNN and CNN because its fully connected layers are removed.The parameter reduction accompanies FCN's stronger intelligibility result.

V. DISCUSSION

The discussion links missing high-frequency components to fully connected layers and shows that data overlap can change DNN waveform generation. A shift increment of 1 makes DNN behave more like a filter, but FCN is more efficient.

  • Fully connected layers: Increasing the number of fully connected layers makes missing high-frequency components more critical because these layers destroy correlations between waveform features.CNN has fewer fully connected layers and therefore shows relatively minor high-frequency loss than DNN.
  • Window shift increment: When L = 1, DNN successfully generates high-frequency components, unlike its nonoverlapping-window setting shown in Fig. 7.Each output sample is generated through fixed weights analogous to learned convolutional filters.
  • Window shift increment: With L = 1, each estimated sample depends on fixed weights and changing inputs, resolving the high-frequency problem caused by correlated output-layer weight vectors.The filter-like DNN approach remains less efficient than FCN.

VI. CONCLUSIONS

The study finds that fully connected layers are unnecessary for raw-waveform speech enhancement and uses FCN to preserve feature correlations more effectively. FCN yields better results than DNN with LPS inputs.

  • Conclusions: Fully connected layers increase model parameters and have limited ability to preserve feature correlations needed for waveform generation.These findings motivate replacing them with a fully convolutional architecture.
  • Conclusions: FCN produces better results than DNN with LPS inputs for raw-waveform speech enhancement.The authors identify FCN as their solution to the limitations of fully connected layers.
  • Conclusions: Future work will apply FCN in an utterance-based rather than frame-wise manner to optimize each utterance as a whole.This is presented as a planned extension of the current study.
Loading 1703.02205v3…