Source-linked AI summary
TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids
Igor Fedorov, Marko Stamenovic, Carl Jensen, Li-Chia Yang, Ari Mandell, Yiming Gan, Matthew Mattina, Paul N. Whatmough
TL;DR
Hearing-aid speech enhancement needs models that fit tight latency, memory, and compute constraints. This paper applies structured pruning, integer quantization, and RNN state-update skipping. The compressed models reduce model size, retain statistically indistinguishable listening preference from the baseline, and meet the stated compute-latency requirement.
Problem
Method
The paper optimizes RNN speech enhancement with structured pruning, integer quantization of inputs, weights, and activations, and skipped recurrent state updates.
Source: §1 · pp. 1–2, §3.1 · p. 2Results
Model size is reduced by 11.9× versus baseline, operations per inference by 1.78× versus the smallest compressed model, and listening ratings show no statistical difference across SNRs.
Source: §5 · p. 4, §4.3 · p. 4Takeaways & Limitations
Pruned (INT8) model 2 and Pruned Skip RNN (INT8) can be deployed on a real hearing-aid MCU with compute latency of 2.39–6.71ms, meeting the 10ms requirement.
Source: §4.2 · p. 4Skipping LSTM updates produced audio artifacts when the mask was not updated, prompting the introduction of two exponential moving averages to smooth the model in time.
Source: §3.3 · p. 3
Abstract
from arXiv · showhide
Modern speech enhancement algorithms achieve remarkable noise suppression by means of large recurrent neural networks (RNNs). However, large RNNs limit practical deployment in hearing aid hardware (HW) form-factors, which are battery powered and run on resource-constrained microcontroller units (MCUs) with limited memory capacity and compute capability. In this work, we use model compression techniques to bridge this gap. We define the constraints imposed on the RNN by the HW and describe a method to satisfy them. Although model compression techniques are an active area of research, we are the first to demonstrate their efficacy for RNN speech enhancement, using pruning and integer quantization of weights/activations. We also demonstrate state update skipping, which reduces the computational load. Finally, we conduct a perceptual evaluation of the compressed models to verify audio quality on human raters. Results show a reduction in model size and operations of 11.9$\times$ and 2.9$\times$, respectively, over the baseline for compressed models, without a statistical difference in listening preference and only exhibiting a loss of 0.55dB SDR. Our model achieves a computational latency of 2.39ms, well within the 10ms target and 351$\times$ better than previous work.
1. Introduction
The paper targets RNN speech enhancement that can meet hearing-aid hardware constraints, introducing pruning, integer quantization, and state-update skipping. It evaluates optimized models with objective metrics, listening tests, and MCU profiling.
1. Introduction
- Speech enhancement could ease noisy-environment listening difficulty, a leading concern among hearing-aid users.Source: §1 · p. 1
- The frame-based causal model must meet a 10ms compute-latency limit per frame to satisfy a 30ms maximum audio-latency target.The paper defines audio latency as the delay from noise arriving at the hearing aid to clean sound being produced for the listener.Source: §1 · p. 1
- The target MCU's Flash and SRAM constrain model size and working memory, while efficient computation requires integer data types and few operations per second.The paper targets an STM32F746VE with 512KB Flash and 320KB SRAM.Source: §1 · p. 1
- Earlier approaches either reported models exceeding memory constraints or left open whether pruning and quantization could be jointly applied to RNN speech enhancement.Other cited approaches also rely on nonuniform quantization or floating-point execution, which the paper identifies as hardware drawbacks.Source: §1 · p. 1
- The method combines structured pruning, weight and activation quantization, and skipped RNN state updates to meet hearing-aid hardware requirements.The paper also reports learning pruning thresholds directly, avoiding a costly hyperparameter search.Source: §1 · pp. 1–2
- The evaluation combines objective metrics, human listening preferences, and MCU profiling to assess audio quality and hardware constraints.The paper notes that compression artifacts may not be reflected in objective metrics such as SNR.Source: §1 · p. 2
2. Background
The background frames speech enhancement as recovering clean speech from a noisy signal using a learned time-frequency mask. Its baseline uses causal recurrent layers and fully connected layers to produce that mask.
2. Background
- Lowercase symbols denote vectors and uppercase symbols denote matrices in the background notation.Source: §2 · p. 2
2.1. Speech Enhancement
- The speech-enhancement task transforms noisy speech into a time-frequency representation and learns a mask to estimate the clean-speech spectrogram.The mask is applied to the noisy spectrogram while retaining its phase.Source: §2.1 · p. 2
2.2. Baseline Model Architecture
- The causal baseline uses two 256-unit unidirectional LSTM layers and two 128-unit fully connected layers, with a 128-dimensional mel-space input and output.Batch normalization lies between the last LSTM and first fully connected layer; ReLU and sigmoid follow the first and second fully connected layers, respectively.Source: §2.2 · p. 2
3. Optimizing LSTMs for HA Hardware
The section combines structured pruning and integer quantization to reduce LSTM model size and computational cost, then adds state-update skipping to lower average operations. Its techniques target hearing-aid hardware constraints while addressing audio artifacts associated with skipped updates.
3. Optimizing LSTMs for HA Hardware
- Pruning reduces operations, while weight and activation quantization reduces model size and enables low-precision integer arithmetic.The operation count is independent of quantization, but integer arithmetic can increase real-hardware throughput.Source: §3 · p. 2
3.1. Structural Pruning
- Structured pruning groups weights by connected neuron or LSTM hidden-state element and learns layer thresholds to remove groups.Learning thresholds directly avoids a separate hyperparameter search that can grow costly with the number of layers.Source: §3.1 · pp. 2–3
3.2. Quantization
- Training-aware quantization exposes model outputs to quantization noise while backpropagating on real-valued weights; weights, activations, and inputs use 8 bits.The pruning mask uses 16-bit quantization.Source: §3.2 · p. 3
3.3. Skip RNN Cells
- Skip RNNs use a binary gate to control updates to candidate cell and hidden states, and update the state-update probability between skipped steps.Because that probability stays fixed while the LSTM is not updating, its update calculation is needed only when the LSTM updates.Source: §3.3 · p. 3
- Skipping updates can produce audio artifacts because the enhancement mask is not updated; two exponential moving averages were introduced to smooth the model over time.One EMA projects the input spectrogram frame into a context vector concatenated to the LSTM output; another smooths the masks.Source: §3.3 · p. 3
4. Experimental Results
The experiments evaluate CHiME2 enhancement against objective and human-preference measures, then test pruning, quantization, and state skipping under hearing-aid hardware constraints. The optimized models meet the stated 10 ms compute-latency requirement, and listeners show no statistical preference difference between baseline and compressed outputs.
4. Experimental Results
- Experiments use CHiME2 WSJ0 mono audio with SNRs from -6 to 9dB, and skip-RNN experiments use 25 ms frames with 6.25 ms hops.The dataset includes 7,138 training, 2,560 development, and 1,980 test utterances with living-room interference sources.Source: §4 · p. 3
4.1. Baseline Model
- 12.77dB SDR on CHiME2 development and 13.70dB SDR on test are comparable to.The baseline is evaluated with 32 ms frames and 16 ms hops; training uses SISDR, while final objective evaluation uses SDR.Source: §4 · p. 3, §4.1 · p. 4
4.2. Structured Pruning and Quantization
- 47% structural pruning preserves baseline performance; joint pruning and quantization yields 37% pruning at around 0.2 dB SISDR reduction and 66% pruning at around 0.5 dB decay.Figure 1 plots model size against SISDR for checkpoints and their Pareto front.Source: §4.2 · p. 4, Figure 1 · p. 4
- 12.07dB SDR on development and 12.96dB SDR on test accompany a 63% skip rate, which reduces average energy per inference versus Pruned (INT8) 2.The skip RNN requires more inferences per second.Source: §4.2 · p. 4
- 2.39–6.71 ms compute latency meets the 10 ms requirement, and Pruned (INT8) model 2 and Pruned Skip RNN (INT8) fit a real HA MCU despite violating constraints.The compared models also have slightly better SISDR/SDR performance, but violate model-size, working-memory, and operation constraints.Source: §4.2 · p. 4
4.3. Perceptual Evaluation
- Participants moderately preferred enhanced audio on average, with no statistical difference in ratings between baseline and optimized models across SNRs.Each survey had 50 participants; each rated 24 paired comparisons drawn from 12 utterances across six SNR levels.Source: §4.3 · p. 4, Figure 2 · p. 4
5. Conclusions
The conclusion reports that pruning and integer quantization reduce model size, while skip-RNN techniques further reduce operations per inference. The optimized implementation meets the 10 ms compute-latency target with no statistical difference in subjective evaluation.
5. Conclusions
- 11.9× smaller model size and 4.26ms optimized latency meet the 10ms target, with negligible SISDR degradation and no statistical difference in subjective evaluation.Skip RNN techniques reduce operations per inference by 1.78× compared with the smallest compressed model.Source: §5 · p. 4