PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense CNN with Self-Attention for Time-Domain Speech Enhancement

Ashutosh Pandey, DeLiang Wang

arXiv:2009.01941v2eess.AScs.SD

TL;DR

Low-SNR speech enhancement requires handling noise while improving both speech magnitude and phase, and spectral-magnitude training can introduce artifacts and inconsistent SNR improvement. The paper proposes a self-attentive dense convolutional time-domain network with a phase-constrained magnitude loss, reporting improvements across causal and non-causal enhancement settings. Its evaluation is limited by training on WSJ and testing on untrained WSJ speakers rather than untrained corpora.

  • Problem

    Time-domain speech enhancement jointly addresses magnitude and phase, but spectral-magnitude loss can introduce artifacts and inconsistent SNR improvement.

  • Method

    The paper combines an encoder-decoder DCN with dense blocks and self-attention, trained using a loss based on enhanced-speech and predicted-noise magnitudes.

  • Results

    16.2% STOI, 0.59 PESQ, and 9.9 dB SNR average improvement are obtained for causal enhancement without context, dilation, or attention.

  • Takeaways & Limitations

    Attention and context aggregation improve objective speech-enhancement scores, with relatively larger gains at −5 dB.

  • Takeaways & Limitations

    DCN is trained on WSJ and evaluated only on untrained WSJ speakers; evaluation on untrained corpora remains future work.

Abstract

from arXiv · show

Speech enhancement in the time domain is becoming increasingly popular in recent years, due to its capability to jointly enhance both the magnitude and the phase of speech. In this work, we propose a dense convolutional network (DCN) with self-attention for speech enhancement in the time domain. DCN is an encoder and decoder based architecture with skip connections. Each layer in the encoder and the decoder comprises a dense block and an attention module. Dense blocks and attention modules help in feature extraction using a combination of feature reuse, increased network depth, and maximum context aggregation. Furthermore, we reveal previously unknown problems with a loss based on the spectral magnitude of enhanced speech. To alleviate these problems, we propose a novel loss based on magnitudes of enhanced speech and a predicted noise. Even though the proposed loss is based on magnitudes only, a constraint imposed by noise prediction ensures that the loss enhances both magnitude and phase. Experimental results demonstrate that DCN trained with the proposed loss substantially outperforms other state-of-the-art approaches to causal and non-causal speech enhancement.

I. INTRODUCTION

Speech enhancement addresses noise-related degradation in intelligibility, quality, and speech applications, while time-domain methods jointly enhance magnitude and phase. The paper proposes a self-attentive dense convolutional network and a phase-constrained magnitude loss to address limitations of spectral-magnitude training.

  • Background noise reduces speech intelligibility and quality and can degrade automatic speech recognition, teleconferencing, and hearing-aid applications.
  • Time-frequency methods often enhance spectral magnitude while reusing noisy phase, motivating approaches that jointly enhance magnitude and phase.
  • Self-attention aggregates selective utterance-level context, potentially using high-SNR repetitions of phones to reconstruct low-SNR occurrences.
  • The proposed DCN combines encoder-decoder skip connections, dense blocks, and attention modules for feature reuse, deeper extraction, and context aggregation.
  • The proposed PCM loss uses enhanced-speech and predicted-noise magnitudes to remove spectral-magnitude artifacts and constrain phase toward the clean phase.

II. SPEECH ENHANCEMENT IN THE TIME DOMAIN

Time-domain enhancement estimates clean speech directly from noisy samples rather than first converting the signal to a time-frequency representation. The formulation supports frame-level prediction using current, past, and future frames according to the enhancement setting.

  • Given noisy speech y, the enhancement goal is to estimate clean speech s as bs.
  • A time-domain DNN computes bs directly from y through a parameterized function fθ.
  • The input signal is divided into overlapping frames defined by frame length L and frame shift J.
  • Frame-level enhancement can use the current frame together with K1 past frames and K2 future frames.

B. Causal Speech Enhancement

Causal enhancement restricts each prediction to current and past inputs, enabling real-time operation. The section describes convolution choices and supporting DCN components for preserving context, controlling boundaries, and upsampling.

  • Causal Speech Enhancement: A causal speech-enhancement prediction uses only the current and past frames, a requirement for real-time enhancement.
  • DCN Building Blocks: DCN uses 2D convolution, sub-pixel convolution, layer normalization, dense blocks, and self-attention as its building blocks.
  • Figures: Figures 1 and 2 respectively depict the DCN architecture and VALID, non-causal SAME, and causal SAME convolution arrangements.
  • 2-D Convolution: VALID convolution reduces output size, while SAME padding preserves input size and causal padding prevents dependence on future time inputs.
  • 2-D Convolution: Causal convolution uses inputs at time instances less than or equal to t and generally requires m−1 padding samples along time for kernel size m.

B. Sub-pixel Convolution

Sub-pixel convolution increases signal size by generating multiple same-sized feature maps and rearranging them into an upsampled signal. The section also describes layer normalization as a batch-size-insensitive training aid.

  • Sub-pixel Convolution: Sub-pixel convolution upsamples a signal at rate (r, s) using r·s convolutions that produce same-sized intermediate signals.
  • Sub-pixel Convolution: The intermediate signals are combined by an index-based rearrangement using remainder and floor operations to form the upsampled output.
  • Layer Normalization: Layer normalization improves generalization and facilitates DNN training as an alternative to batch normalization, which is sensitive to batch size.
  • Layer Normalization: For input shape [C, T, L], normalization is performed over the last dimension with trainable γ and β shared across channels and frames.

D. Dense Block

The dense block reuses features through concatenated connections, while self-attention aggregates context within DCN’s encoder–decoder processing flow. The architecture supports both causal and non-causal attention and convolutional processing.

  • Dense Block: Dense connections reuse outputs from several preceding layers, helping address vanishing gradients through direct connections.The input to a layer includes outputs from earlier layers rather than only the immediately preceding layer.
  • Dense Block: Each DCN dense block contains five convolutional layers followed by layer normalization and parametric ReLU nonlinearities.The convolutions use m × 3 kernels, with m set to 2 for causal and 3 for non-causal convolution.
  • Dense Block: Within a dense block, each layer receives a concatenation of its input and the previous layer’s output.This implements feature reuse across the block’s successive layers.
  • Self-Attention: Self-attention computes correlations between queries and keys, converts them to Softmax weights, and linearly combines value rows.In DCN, three 1 × 1 convolutions produce Q, K, and V before the attention operations.
  • Self-Attention: Causal attention masks entries above the main diagonal so future frames contribute zero to the attention computation.This restricts attention to the current and preceding frames.
  • Processing Flow: DCN processes chunked utterance frames through an encoder whose later layers downsample, apply attention, and feed concatenated features into dense blocks.The encoder’s first layer increases the channel count with a 1 × 1 convolution before dense-block processing.

IV. LOSS FUNCTIONS

This section defines time-domain loss functions for speech enhancement, including an utterance-level mean squared error objective. The loss compares the clean speech with the enhanced estimate over the utterance.

  • Time-Domain Loss: The time-domain objective is defined as an utterance-level mean squared error between clean speech and enhanced speech.The equation averages squared sample-wise differences across the utterance.
  • Time-Domain Loss: The objective aggregates sample-level squared errors into a single utterance-level training quantity.This provides one MSE value for the complete utterance.

B. STFT Magnitude Loss

The STFT magnitude loss (LSM) improves objective scores but can produce inconsistent SNR changes and an audible buzzing artifact. The proposed phase-constrained magnitude loss matches magnitudes of enhanced speech and predicted noise to address both problems while preserving objective-score gains.

  • LSM is a mean absolute error between the L1 norms of clean and estimated STFT coefficients.
  • LSM can improve objective intelligibility and quality scores but may worsen processed SNR and introduce an annoying steady buzzing artifact.
  • The artifact is revealed by higher-frequency-resolution spectrograms, where horizontal stripes appear for LSM-enhanced speech.
  • C. Time-frequency Loss: The time-frequency loss combines time-domain and STFT-magnitude losses, improving SNR consistency while retaining objective-score gains but not removing artifacts.
  • D. Phase Constrained Magnitude Loss: The proposed PCM loss matches STFT magnitudes for enhanced speech and predicted noise, adding a constraint beyond speech magnitude alone.
  • D. Phase Constrained Magnitude Loss: Equal speech and noise contributions yield consistent SNR improvement, remove LSM artifacts, and retain objective intelligibility and quality scores similar to LSM.
  • D. Phase Constrained Magnitude Loss: LSM permits arbitrarily many complex-valued estimates with the target speech magnitude, whereas PCM restricts the candidates by also matching noise magnitude.

V. EXPERIMENTAL SETTINGS

The experiments use speaker- and noise-independent evaluation on WSJ0 SI-84, with separate speakers for training and evaluation. Training mixtures use sound-effect noises across six SNR levels, while testing uses babble and cafeteria noises.

  • Models are evaluated in a speaker- and noise-independent setting on the WSJ0 SI-84 dataset.
  • WSJ0 SI-84 contains 7138 utterances from 83 speakers, with 77 speakers for training and six for evaluation.
  • Training uses 10000 non-speech sounds to generate 320000 noisy utterances at SNRs from −5 dB through 0 dB.
  • The test set uses babble and cafeteria noises from an Auditec CD.

B. System Setup

The study evaluates context aggregation and loss functions using STOI, PESQ, and SNR, comparing causal and non-causal DCN configurations. Results favor combining attention with convolutional context and using losses that avoid spectral-magnitude artifacts and inconsistent SNR improvement.

  • Context aggregation: Causal ablation begins with 16.2% STOI, 0.59 PESQ, and 9.9 dB SNR improvement without context.Adding m = 2, dilation, and attention produces further objective-score improvements, especially at −5 dB.
  • Context aggregation: Adding m = 2, dilation, and attention progressively improves objective scores, with most gains occurring at −5 dB SNR.The ablation attributes the gains to progressively broader context aggregation.
  • Context aggregation: Normal convolutions perform similarly or better than dilated convolutions, but reducing m from 2 to 1 degrades performance.The reported results support attention combined with some local convolutional context rather than attention alone.
  • Loss comparisons: LSM achieves similar STOI and PESQ to alternative losses but can produce substantially worse non-causal SNR and inconsistent improvement.Its SNR is sensitive to learning rate, initialization, and model architecture.
  • Loss comparisons: LTF and LPCM provide consistent SNR improvement while retaining STOI and PESQ performance comparable to LSM.LPCM removes the buzzing artifact observed with SM and TF losses, whereas LTF does not reliably remove artifacts across α values.

C. Comparison with Baselines

DCN is compared with masking, spectral-mapping, complex-spectral-mapping, and time-domain baselines under causal and non-causal settings. The proposed real-time and non-causal PCM systems outperform their strongest respective baselines on average.

  • Baseline comparisons: A frame-based DCN without context, dilation, or attention outperforms BLSTM time-frequency masking on average.BLSTM is slightly better at −5 dB SNR but uses the whole utterance and is non-causal.
  • Baseline comparisons: Causal DCN with m = 2 is significantly better than non-causal GRN and causal SEGAN, which uses a 1-second frame and is not real-time.It is also similar or better than causal GCRN except for babble noise at −5 dB.
  • Baseline comparisons: Non-causal DCN with m = 3 is comparable to NC-GCRN, the strongest non-causal baseline in the comparison.The reported comparison uses DCN trained with LT; the authors state that LPCM yields further improvement over baselines.
  • Loss variants: DCN-SM and DCN-PCM obtain similar STOI and PESQ scores, exceeding DCN-T except for babble noise at −5 dB.At that condition, the three losses produce similar scores.
  • Real-time comparison: Causal DCN-PCM exceeds TCNN by 1.5% STOI and 0.19 PESQ on average.TCNN is identified as the best real-time baseline.
  • Non-causal comparison: Non-causal NC-DCN-PCM exceeds NC-GCRN by 1.3% STOI and 0.21 PESQ on average, with p < 0.0001 at all SNRs for both metrics.The authors report these improvements as statistically significant.

D. Attention Maps

Attention maps show that DCN focuses mainly on voiced harmonic structure and selected high-frequency regions. Causal constraints shift attention toward earlier frames and produce sharper attention patterns than non-causal DCN.

  • Non-causal DCN: The attention module focuses mainly on voiced harmonic structure between frames 125 and 185, with additional attention to high-frequency sounds near the utterance end.These patterns are observed in the non-causal DCN attention map.
  • Causal DCN: Causal DCN shifts voiced-speech attention to earlier frames because future frames are unavailable.Attention to high-frequency sounds also shifts to earlier sounds between frames 155 and 185.
  • Causal versus non-causal attention: Attention is relatively sharper in causal DCN than in non-causal DCN, and the attended regions differ between the two configurations.The comparison comes from visualized attention maps for the same utterance.
Loading 2009.01941v2…