Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement

1 Introduction

• Compact systems including RNNoise, PercepNet [2], NSNet2, DeepFilterNet, FSPEN, and LiSenNet pursue lightweight or structured processing for speech enhancement.Recent encoder–mixer–decoder models combine time–frequency mixing with sub-band processing and use fewer than 105 parameters.Source: §1 · p. 1
• Low parameter and MAC counts do not guarantee efficient embedded execution when models use recurrent layers, runtime normalization, dynamic operations, or unsupported nonlinearities.Unsupported operations can trigger host fallback or data-transfer and scheduling overheads that dominate arithmetic cost.Source: §1 · p. 1
• Prior hardware-aware speech-enhancement work targets processors supporting recurrent or sparse computation, whereas commercial MCU NPUs expose predominantly static, integer-quantized convolutional operators.The authors therefore frame deployment as architectural and execution-level codesign, not model compression alone.Source: §1 · p. 1
• LiSenNet’s NPU-compatible redesign replaces its dual-path GRU with convolutional frequency and temporal mixers, supported operators, bounded activations, and a stateful real-time implementation.The authors also provide a fully int8 model with no floating-point fallback to the host.Source: §1 · p. 1

2.1 Baseline and Deployment Gap

• LiSenNet retains approximately 37k parameters, but its bidirectional GRU, multidimensional LayerNorm, PReLU, Mish, and sub-pixel upsampling prevent direct STM32N6 deployment.Unsupported operations may run on the Cortex-M55, where host execution, transfers, and scheduling can outweigh supported arithmetic.Source: §2.1 · p. 2

2.2 NPU-Compatible Architecture

• The architecture preserves the sub-band encoder, progressive frequency reduction, U-Net skips, and mask decoder while replacing the recurrent bottleneck and unsupported operators.A depthwise-separable frequency mixer uses an 11-bin kernel and maps to supported convolutional primitives.Source: §2.2 · p. 2
• LayerNorm becomes foldable per-channel BatchNorm, PReLU and Mish become bounded ReLU6, and sub-pixel upsampling becomes a 1 × 3 transposed convolution with stride 3.The upsampling replacement avoids unsupported rank-5 intermediates and reduces decoder arithmetic by approximately a factor of three.Source: §2.2 · p. 2

2.3 Temporal Modeling and Streaming State

• The primary model uses a causal temporal convolutional network with increasing dilations and FIFO state, although the temporal GRU can also run efficiently with its hidden state exposed as graph input and output.The finite convolutional history flushes errors from the receptive field, addressing concerns about recurrent-state drift and accumulated quantization error.Source: §2.3 · pp. 2–3

3.1 Experimental Setup

• The models are evaluated on the complete 824-utterance VoiceBank-DEMAND test set using a 512-sample STFT window and a 256-sample hop.The hop corresponds to a 16 ms frame shift; real-time int8 evaluation uses static-int8 mask estimation followed by reconstruction with the noisy input phase.Source: §3.1 · p. 3

3.2 Enhancement Quality; 3.3 On-Device Results

• LiSenNet-NPU reaches PESQ 3.08 in float32 and 3.01 in int8, versus recurrent LiSenNet’s 3.01 and 2.93, and runs at 4.83 ms per frame.The quality comparison is on VoiceBank-DEMAND; on-device latency is measured on the STM32N6.Source: Table 1 · p. 3, §3.2 · p. 3, Table 3 · p. 4

3.2 Enhancement Quality

• At the same 68-frame receptive field, C = 24 reaches PESQ 3.01 in float32 and 3.00 in int8, while C = 20 and C = 28 perform worse.C = 24’s int8 quantization loss is 0.01 PESQ, versus 0.08 for the original LiSenNet.Source: Table 1 · p. 3, §3.2 · p. 3
• ReLU6 raises the deep model’s PESQ by 0.015 in float32 and 0.029 in int8, to 3.08 and 3.01, respectively.These gains hold at unchanged parameter count and receptive field.Source: Table 1 · p. 3, §3.2 · p. 3
• LiSenNet-NPU matches int8 STOI at 0.934 and raises DNSMOS SIG, BAK, and OVRL from 3.02, 3.63, and 2.64 to 3.05, 3.67, and 2.69, while SI-SDR falls from 18.8 to 17.3 dB.The comparison is against the original int8 LiSenNet.Source: Table 2 · p. 3, §3.2 · p. 3

3.3 On-Device Results

• LiSenNet-NPU reaches PESQ 3.01 at 4.83 ms per frame, while ConvFSENet reaches 2.91 at 4.40 ms and dense NSNet2 reaches 2.83 at 22.94 ms.The baselines have 1.45 M and 2.78 M parameters, respectively, compared with 46.2 k for LiSenNet-NPU.Source: Table 3 · p. 4, §3.3 · p. 4
• For the C = 20 model, storing history uses 0.92 M MACs and 2.59 ms per frame, versus 66.55 M MACs and 29.93 ms when recomputing it.None of the stateless variants meets the 16 ms deadline; LiSenNet-NPU stores 154 KiB across 25 FIFO tensors, compared with 45 KiB of weights.Source: Table 3 · p. 4, §3.3 · p. 4

• PESQ 3.01 with 4.83 ms per 16 ms frame, improving deployable PESQ by approximately 0.08 over the quantized recurrent baseline.Source: §4 · p. 4
• Replacing unsupported recurrent, normalization, activation, and upsampling operations with static convolutional primitives enables fully integer execution, while bounded decoder activations preserve post-training quantization performance.Source: §4 · p. 4
• Real-time performance requires supported operators, controlled activation ranges, explicit streaming state, and complete compiled-graph measurement beyond parameter or MAC counts.Source: §4 · p. 4
• Stateful inference is substantially faster than receptive-field recomputation, while temporal recurrence remains efficient when state memory and latency outweigh drift and quantization risks.Source: §4 · p. 4