PaperlayerView exact revision on arXiv

Source-linked AI summary

PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network

Dacheng Yin, Chong Luo, Zhiwei Xiong, Wenjun Zeng

arXiv:1911.04697v1cs.SDeess.AS

TL;DR

Single-channel speech enhancement needs better phase prediction and exploitation of long-range harmonic structure than straightforward complex-mask estimation provides. PHASEN uses communicating amplitude and phase streams plus frequency transformation blocks, and reports superior performance across two benchmark settings, including a 1.76dB SDR gain in ablation against its baseline.

  • Problem

    Straightforward complex ideal ratio mask estimation may fail to recover informative phase structure, while conventional convolutions miss non-local harmonic correlations in T-F spectrograms.

  • Method

    PHASEN uses a two-stream DNN with two-way amplitude-phase information exchange and frequency transformation blocks for global frequency correlations.

  • Results

    PHASEN shows superior performance over existing systems across AVSpeech+AudioSet and Voice Bank+DEMAND, while its two innovations yield 1.76dB SDR improvement over the PHASEN-baseline.

  • Takeaways & Limitations

    The results support jointly modeling phase and amplitude while explicitly exploiting harmonic frequency structure for monaural T-F speech enhancement.

  • Takeaways & Limitations

    PHASEN’s current design does not support low-latency applications such as voice over IP.

Abstract

from arXiv · show

Time-frequency (T-F) domain masking is a mainstream approach for single-channel speech enhancement. Recently, focuses have been put to phase prediction in addition to amplitude prediction. In this paper, we propose a phase-and-harmonics-aware deep neural network (DNN), named PHASEN, for this task. Unlike previous methods that directly use a complex ideal ratio mask to supervise the DNN learning, we design a two-stream network, where amplitude stream and phase stream are dedicated to amplitude and phase prediction. We discover that the two streams should communicate with each other, and this is crucial to phase prediction. In addition, we propose frequency transformation blocks to catch long-range correlations along the frequency axis. The visualization shows that the learned transformation matrix spontaneously captures the harmonic correlation, which has been proven to be helpful for T-F spectrogram reconstruction. With these two innovations, PHASEN acquires the ability to handle detailed phase patterns and to utilize harmonic patterns, getting 1.76dB SDR improvement on AVSpeech + AudioSet dataset. It also achieves significant gains over Google's network on this dataset. On Voice Bank + DEMAND dataset, PHASEN outperforms previous methods by a large margin on four metrics.

1 Introduction

PHASEN addresses weaknesses in direct complex-mask estimation by coupling amplitude and phase prediction and modeling long-range harmonic correlations. Its experiments evaluate these design choices and report gains over existing methods.

  • Motivation: Direct cIRM estimation can collapse toward an ideal ratio mask because the predicted imaginary component is nearly zero despite informative ideal phase structure.This motivates a dedicated architecture for phase prediction rather than merely changing the training target.
  • Proposed architecture: Two-way information exchange between amplitude and phase streams is identified as critical for successful phase prediction.The amplitude prediction guides phase prediction, whose structure is otherwise difficult to estimate.
  • Harmonic modeling: Frequency transformation blocks capture global frequency correlations, especially harmonic correlations, that small conventional convolution kernels miss.The design targets non-local relationships between a fundamental frequency and its overtones.
  • Contributions: PHASEN combines a two-stream DNN with two-way communication and frequency transformation blocks for T-F speech enhancement.The architecture is designed to recover target-speech phase information while exploiting harmonic structure.
  • Evaluation: Comprehensive experiments compare PHASEN with existing noise-reduction methods and justify its design choices.The paper reports this evaluation across its experimental study and concludes with limitations and future work.

2 Related Work

Related work spans T-F masking, time-domain enhancement, and harmonic-aware reconstruction. Prior approaches progressively incorporate phase information, while PHASEN builds on T-F representations and explicit amplitude-phase modeling.

  • T-F domain methods: T-F masking methods transform waveforms into spectrograms, predict multiplicative masks, and invert the modified spectrogram to recover speech.The central design questions are which mask to use and how to predict it.
  • T-F domain methods: Early methods estimate only spectrogram amplitudes, whereas PSM and cIRM extend masking toward phase recovery.cIRM is complex-valued and can jointly represent amplitude and phase enhancement.
  • Phase estimation: DNN-based cIRM estimation has not consistently delivered substantial gains over PSM, suggesting that complex-mask potential remains underexploited.Later methods also treat phase reconstruction asynchronously or as discretized classification.
  • Time-domain methods: Time-domain methods avoid explicit phase prediction by directly modeling clean waveforms, but they give up benefits of T-F feature representations.Examples include GAN-, WaveNet-, and TasNet-based systems.
  • Harmonics: Harmonic structure has been used to regenerate distorted frequency bands and to reconstruct speech phase through correlations between harmonics.These findings motivate explicit harmonic modeling in T-F enhancement.

3 PHASEN Architecture

PHASEN uses parallel amplitude and phase streams within stacked two-stream blocks, with bidirectional information exchange to support phase estimation. Frequency transformation blocks extend frequency-axis context for harmonic-aware amplitude prediction, and the streams produce amplitude and phase estimates that are recombined into the predicted spectrogram.

  • 3.1 Overview: PHASEN separates amplitude-mask prediction and phase prediction into parallel streams processing the input complex STFT spectrogram.Stream A predicts M, while Stream P forms a normalized complex feature map whose phase prediction is Ψ.
  • 3.1 Overview: Information exchange occurs at the end of each two-stream block because amplitude features support phase prediction and predicted phase can improve amplitude prediction.The communication uses attention-based functions in both directions between the streams.
  • 3.4 Output and Training: The predicted spectrogram combines the noisy-spectrogram magnitude, predicted amplitude mask, and predicted phase through element-wise multiplication.The network is trained with amplitude and phase-aware losses on a power-law compressed STFT spectrogram.
  • 3.2 Two-Stream Blocks: Stream A uses three convolutional layers and FTBs for local and global correlations, while Stream P uses two convolutional layers and a 25×1 kernel for long-range time-domain correlation.Global Layer Normalization precedes Stream P’s convolutions, and Stream P omits activation functions.
  • 3.2 Two-Stream Blocks: Frequency transformation blocks apply learned frequency-axis transformations so features can capture non-local correlations, including harmonic relationships, beyond local convolutional context.FTBs are placed before and after Stream A’s convolutional layers, giving subsequent features full-frequency receptive fields.
  • 3.3 Frequency Transformation Blocks (FTBs): Each FTB predicts a T-F attention map, applies a trainable frequency transformation to each time slice, then concatenates the transformed and input features through a 1×1 convolution.The attention module reduces channels to Cr = 5 and uses a 1D convolution kernel size of 9; the transformed slice retains the input slice’s dimensions.

4.1 Datasets

PHASEN is evaluated on two speech-enhancement datasets: the large AVSpeech+AudioSet mixture dataset and the open Voice Bank+DEMAND dataset. They differ in scale, speaker composition, and noise construction.

  • Dataset overview: Two datasets are used to evaluate PHASEN: AVSpeech+AudioSet and Voice Bank+DEMAND.The experiments therefore cover both a large-scale mixture dataset and an open benchmark dataset.
  • AVSpeech+AudioSet: AVSpeech+AudioSet combines clean speech from 4,700 hours of YouTube video segments with AudioSet noise spanning more than 1.7 million segments and 526 noise types.The clean speech includes approximately 150,000 distinct speakers and varied languages.
  • Voice Bank+DEMAND: Voice Bank+DEMAND uses speech from 30 Voice Bank speakers and mixtures with noise from the Diverse Environments Multichannel Acoustic Noise Database.Twenty-eight speakers are used for training, two for validation, and 40 noise conditions are considered.

4.2 Evaluation Metrics

PHASEN and competing speech-enhancement systems are evaluated with six metrics, all interpreted as better when higher. The metrics cover signal distortion, perceptual quality, and predicted speech or noise quality.

  • Evaluation metrics: The evaluation uses six metrics: SDR, PESQ, CSIG, CBAK, and two additional metrics listed in the paper’s metric set.SDR measures signal-to-distortion ratio, PESQ measures perceptual speech quality, and CSIG and CBAK predict speech distortion and noise intrusiveness.

4.3 Ablation Study

The ablations show that PHASEN’s two-way two-stream design and frequency transformation blocks each contribute to enhancement quality, while stream-specific choices also matter. Visualizations link information exchange to improved phase prediction and reduced amplitude artifacts, and learned frequency weights to harmonic correlation.

  • Core ablations: 1.76dB SDR and 0.53 PESQ improvements show that PHASEN’s two-stream architecture and FTBs outperform the single-stream cIRM baseline.The comparison uses PHASEN-baseline with a single stream and 5×5 convolutions replacing FTBs.
  • Core ablations: 0.85dB SDR and 0.42 PESQ gains show that the two-stream architecture improves both objective and perceptual performance over PHASEN-1strm.The large PESQ gain is reported as evidence of improved perceptual quality in denoised speech.
  • FTBs: 0.74dB SDR and 0.09 PESQ gains show that FTBs improve PHASEN relative to replacing all FTBs with 5×5 convolutions.Replacing FTBs at either location causes 0.31dB-0.39dB SDR and 0.03-0.05 PESQ drops, indicating both locations contribute.
  • FTBs: The learned frequency-transformation weights resemble harmonic correlation, especially when higher harmonics are included.The visualization is presented as evidence that FTBs capture global harmonic structure useful to the speech-enhancement network.
  • Information communication: 0.49dB SDR and 0.05 PESQ separate the partial-communication and no-communication settings, showing that amplitude information helps phase prediction.Adding phase information back into amplitude prediction provides a further 0.22dB SDR gain over PHASEN-w/o-P2A.
  • Information communication: The full model shows visible phase prediction in noisy T-F bins and fewer amplitude artifacts, unlike PHASEN-w/o-A2PP2A.The visualization supports the conclusion that bidirectional communication helps both phase prediction and amplitude reconstruction, with A2P identified as especially important.
  • Stream-specific choices: Removing activation functions from stream P is supported by ablations, whereas adding ReLU after its final convolution causes a 5.52dB SDR and 0.2 PESQ drop.Adding ReLU or Tanh after earlier convolutional layers causes smaller SDR drops of 0.02dB-0.16dB.
  • Stream-specific choices: Using gLN in stream A causes a 0.97dB SDR and 0.12 PESQ drop, while using BN in stream P causes a 0.09dB SDR and 0.02 PESQ drop.These differing effects support distinct normalization choices for amplitude and phase prediction.

4.4 System Comparison

PHASEN is compared with recent speech-enhancement systems on AVSpeech + AudioSet and Voice Bank + DEMAND. It outperforms competing methods broadly, including significant gains over a hybrid model on four metrics.

  • Experimental scope: The experiments compare systems on both AVSpeech + AudioSet and Voice Bank + DEMAND.The section introduces system comparisons on the two datasets and identifies their corresponding comparison tables.
  • AVSpeech + AudioSet: PHASEN outperforms both Conv-TasNet and “Google” on AVSpeech + AudioSet despite using fewer training steps and less data than “Google”.PHASEN uses 1M/5M of “Google”’s training steps and 100k/2.4M of its data.
  • Voice Bank + DEMAND: On Voice Bank + DEMAND, PHASEN achieves large gains over SEGAN, Wavenet, and DFL across all five reported metrics.The comparison is framed as evidence of an advantage in capturing phase-related information.
  • Voice Bank + DEMAND: PHASEN greatly improves over MMSE-GAN on all metrics in the Voice Bank + DEMAND comparison.MMSE-GAN is a time-frequency-domain method, whereas PHASEN uses the proposed network design.
  • Voice Bank + DEMAND: PHASEN significantly outperforms MDPhD on four metrics, with only about 0.04dB difference on SSNR.MDPhD is described as a hybrid time-domain and time-frequency-domain model.

5 Conclusion

The paper concludes that PHASEN combines two-way two-stream phase prediction with learnable frequency transformation that captures harmonic correlation. It reports superior performance on both evaluated datasets but identifies low latency as a current boundary.

  • Contributions: PHASEN uses two-way information exchange between streams for phase prediction and a learnable frequency transformation matrix that captures harmonic correlation.The learned pattern is described as consistent with harmonic correlation.
  • Validation: Comprehensive ablation studies support almost every design choice made in PHASEN.
  • Results: PHASEN demonstrates superior performance against state-of-the-art systems on AVSpeech+AudioSet and Voice Bank+DEMAND.
  • Limitation and future work: PHASEN cannot currently be used for low-latency applications such as voice over IP.The authors plan to explore low-latency and mobile settings in future work.
Loading 1911.04697v1…