Source-linked AI summary
A Fully Convolutional Neural Network for Speech Enhancement
Se Rim Park, Jinwon Lee
TL;DR
Babble noise can severely reduce hearing intelligibility and cause speech distortion, motivating a compact enhancement method for hearing aids. The paper learns a supervised mapping from noisy to clean spectra with fully convolutional networks, including R-CED. CNNs achieved similar or better performance with fewer parameters, while R-CED's efficacy was associated with expanding encoder feature dimensions and reducing decoder dimensions.
Problem
Babble noise is difficult to model and can degrade speech intelligibility while causing distortion in hearing-aid speech enhancement.
Method
The paper learns a supervised mapping from noisy speech spectra to clean spectra using fully convolutional networks, including the proposed R-CED architecture.
Results
CNNs achieved similar or better performance than FNNs and RNNs with much fewer parameters, while R-CED yielded better results than CED.
Takeaways & Limitations
R-CED's efficacy was associated with increasing feature-space dimension along the encoder and decreasing dimension along the decoder.
Abstract
from arXiv · showhide
In hearing aids, the presence of babble noise degrades hearing intelligibility of human speech greatly. However, removing the babble without creating artifacts in human speech is a challenging task in a low SNR environment. Here, we sought to solve the problem by finding a `mapping' between noisy speech spectra and clean speech spectra via supervised learning. Specifically, we propose using fully Convolutional Neural Networks, which consist of lesser number of parameters than fully connected networks. The proposed network, Redundant Convolutional Encoder Decoder (R-CED), demonstrates that a convolutional network can be 12 times smaller than a recurrent network and yet achieves better performance, which shows its applicability for an embedded system: the hearing aids.
1. INTRODUCTION
Babble noise remains difficult to model and can severely reduce speech intelligibility while introducing speech distortion in hearing aids. The paper therefore explores CNN-based spectral mapping as a memory-efficient alternative for embedded speech enhancement.
- Babble noise is difficult to model because it arises from concurrent talkers and often dominates speech.
- Incorrect babble-noise modeling can distort speech and create discomfort for hearing-aid users.
- The proposed approach learns a mapping from noisy speech spectra to clean speech spectra instead of explicitly modeling babble noise.
- CNNs use weight sharing to reduce parameter counts relative to fully connected and recurrent networks, supporting embedded deployment.
- The paper targets a memory-efficient denoising algorithm that minimizes speech artifacts and can be implemented in hearing aids.
2. PROBLEM STATEMENT
The problem is to learn a mapping that converts noisy speech spectra into clean spectra. For convolutional networks, the mapping uses a temporal segment of past noisy spectra to denoise the current spectrum.
- The objective is to learn f so denoised spectra approximate clean spectra in the ℓ2 norm.
- For convolutional networks, the input includes the past nT noisy spectra when denoising the current spectrum.
- nT = 8 gives the network about 100 ms of input speech and 32 ms of output speech.
3. CONVOLUTIONAL NETWORK ARCHITECTURES
The paper presents convolutional encoder-decoder architectures for speech enhancement, including the proposed R-CED and its cascaded variant. These designs use convolutional processing, symmetric filter structures, and bypass connections to map noisy spectra to clean spectra.
- Convolutional Encoder-Decoder (CED): CED uses symmetric encoding and decoding layers, compressing features during encoding and reconstructing them during decoding.Its encoder repeats convolution, batch normalization, max-pooling, and ReLU; its decoder repeats convolution, batch normalization, and up-sampling.
- Redundant CED Network (R-CED): R-CED removes pooling and up-sampling, instead expanding feature dimensions in the encoder and compressing them in the decoder.Its filter counts increase toward the encoder and decrease toward the decoder, with symmetric numbers of filters across the architecture.
- Redundant CED Network (R-CED): R-CED ends with a convolution layer, making it a fully convolutional network.The architecture repeatedly applies convolution, batch normalization, and ReLU activation layers.
- Cascaded R-CED Network (CR-CED): CR-CED repeats R-CED networks and achieves better performance with less convergence time than an equally sized R-CED network.The comparison is made at the same network size, defined as the same number of parameters.
- Bypass Connections: Skip connections are added every other layer to facilitate training optimization and improve performance.The paper selects skip rather than residual connections because skip connections are considered more suitable for symmetric encoder-decoder designs.
- Convolutional Operations: Convolution is performed only along the frequency axis because this was found more efficient than two-dimensional convolution for 129 × 8 input spectra.The filter and input have the same dimension along the time axis.
4. EXPERIMENTAL METHODS
The experiments use noisy speech from TIMIT and evaluate reconstructed speech with objective and perceptual measures. Preprocessing includes STFT-based spectral features, phase-aware magnitude scaling, and standardized training procedures.
- Dataset: The dataset combines 4,620 training utterances and 200 testing utterances from TIMIT with one of 27 noise clips at 0dB SNR.The collected noises are mostly babble but also include instrumental sounds; 20% of training features were assigned to validation.
- Feature Transformation: Speech signals are downsampled to 8kHz and converted into 129-point magnitude spectra using a 256-point STFT with 32ms windows and 8ms shifts.The resulting frequency resolution is 31.25 Hz per bin after removing the symmetric half of the spectrum.
- Feature Transformation: CNN inputs contain 8 consecutive noisy STFT magnitude vectors, forming a 129 × 8 feature covering 100ms of speech.The features are standardized to zero mean and unit variance.
- Phase Aware Scaling: Phase-aware scaling encodes clean spectral magnitude so the noisy-clean phase difference remains below 45 degrees during reconstruction.Spectral phase is excluded from training, while noisy phase is used with the predicted magnitude for inverse STFT reconstruction.
- Training: Networks are trained by back propagation with Adam and mini-batch gradient descent, with batch normalization added after each convolution layer.Convolution layers are trained from scratch, while fully connected and recurrent layers use initialization and pretraining procedures.
- Evaluation Metric: SDR measures the ℓ2 error between clean and denoised speech, while STOI and PESQ measure subjective listening quality.SDR is inversely associated with the objective function, whereas STOI and PESQ measure nonlinear short-term perceptual error.
- Training: Adding batch-normalization layers to FNN and RNN did not improve convergence or performance in this experiment.This observation is stated as an experiment-specific implementation note.
5. EXPERIMENTAL SETUP
The experimental setup compares CNN, FNN, and RNN architectures, then evaluates CED and R-CED under matched parameter and depth constraints. A third experiment varies R-CED and CR-CED size and depth to assess performance scaling.
- Test 1: FNN vs. RNN vs. CNN: The first experiment compares CNN, FNN, and RNN using their best-performing network configurations.The best FNN and RNN models have four fully connected layers, while CNN has 16 convolutional layers.
- Test 2: CED vs. R-CED: 33,000 parameters and 10 convolutional layers are fixed when comparing R-CED with CED.Filter widths are selected to preserve symmetric encoder-decoder structure, gradual parameter changes, and equal frequency coverage.
- Test 2: CED vs. R-CED: The matched CED and R-CED comparison ensures both networks use the same number of input frequency bins to reconstruct each output frequency bin.This quantity is defined as the networks’ frequency coverage.
- Test 3: Network Size and Depth: The third experiment compares R-CED and CR-CED across 33K and 100K parameters and depths of 10, 16, and 20 convolutional layers.These settings correspond to approximately 132MB and 400MB of memory, respectively.
6. RESULTS
CNN-based denoising achieved similar or better speech-quality performance with substantially smaller models, while R-CED outperformed CED and benefited differently from skip connections. Network size dominated performance, with depth secondary.
- CNN achieved similar or better denoising performance than FNN and RNN while using about 68 times fewer parameters than FNN and 12 times fewer than RNN.The comparison used STOI, PESQ, and SDR measures; 33,000 CNN parameters corresponded to roughly 132MB of memory.
- R-CED with skip connections achieved the best performance among the initial CED and R-CED comparisons, while CED without skip connections performed worst.R-CED outperformed CED regardless of whether skip connections were present.
- CED performance rose from 5.96 to 7.92 with skip connections, indicating that its decoder could not reconstruct information lost during encoding without bypassed information.The resulting CED speech sounded artificial and mechanical without adequate reconstruction.
- R-CED performance changed only from 8.07 to 8.19 with skip connections because its encoder expands rather than compresses the input.Redundant feature representations were generated at the encoder and unwanted features removed at the decoder.
- For R-CED variants, network size was the dominant performance-associated factor, network depth was secondary, and CR-CED performed best under matched conditions.The matched comparison used 16 convolutional layers and 33K parameters.
7. CONCLUSION
The study targeted memory-efficient babble-noise denoising for embedded hearing-aid systems and found that CNNs could match or exceed FNN and RNN performance with fewer parameters. It also proposed R-CED and associated its success with expanding feature dimensions in the encoder and reducing them in the decoder.
- The study demonstrated that CNNs could achieve similar or better denoising performance than FNNs and RNNs with substantially fewer model parameters.This conclusion addresses the goal of memory-efficient denoising for embedded systems.
- The proposed R-CED architecture was associated with increasing feature-space dimension through the encoder and decreasing dimension through the decoder.The paper describes this design as expanding representations before compressing them.