Transformer-based Neural Beamforming for Real-Time Speech Enhancement on Smart Low-Power Hearable Devices

I. INTRODUCTION

• Multichannel enhancement can suppress noise while preserving spatial cues, but its computational and memory demands have led most existing approaches to use single-channel inputs.Hearables also face small battery capacity and MCU limits of a few MB of on-chip memory and a few billion MAC operations per second.Source: §I · p. 1
• The hybrid architecture separates slowly varying scene estimation by a CNN from fast per-frame adaptation by a lightweight Transformer correction to MVDR weights.The CNN estimates spectral masks used to compute MVDR weights, while the Transformer produces an additive correction.Source: §I · pp. 1–2
• The pipeline uses float32 MVDR, an int8 CNN on the NE16 accelerator, and a float16 Transformer, with per-frame enhancement interleaved with slower CNN weight estimation.Speech activity detection suppresses computation during silence; the work reports real-time operation and all-day autonomy on a 100 mAh battery.Source: §I · p. 2
• 98.01% STOI and 3.768 PESQ-WB on VoiceBank at 24.2 mJ per inference; the battery estimate exceeds 20 h under stated deployment assumptions.The estimate assumes a 100 mAh, 3.7 V battery, 30% speech activity, and includes six 0.9 mW microphones.Source: §I · p. 2

II. RELATED WORK

• Prior MCU speech-enhancement methods were predominantly single-channel, while proposed multichannel models had not yet been deployed on real hardware.The related work spans compact recurrent approaches and newer multichannel architectures, including UNet-like and Transformer-based models.Source: §II · p. 2

III. METHODS

• The pipeline follows G1 mask estimation and MVDR weight computation, then G2 applies a Transformer-predicted correction to produce the enhanced output.The methods overview identifies this dataflow as the two main processing stages.Source: §III · pp. 2–3

A. Signal model and CNN neural beamforming

• For a single target amid noise, the compact MUDV4 C4 512 CNN estimates masks from one reference microphone using real and imaginary STFT channels.The CNN uses dilated 2-D convolutions with residual connections; it has approximately 20 k parameters and requires about 342 MMACs for one second of audio.Source: A. Signal model and CNN neural… · p. 3
• The estimated masks and multichannel STFT observations form speech and noise spatial covariance matrices, from which MVDR weights minimize interference while preserving the target signal.The stated MVDR formulation uses the noise and speech covariance matrices and the trace operation.Source: A. Signal model and CNN neural… · p. 3

• The correction network stacks the current and previous frames across six microphones, using real and imaginary components to form an F × 24 input tensor.Source: B. Transformer-based beamforming… · pp. 3–4
• A frequency-axis 1D convolution feeds one 12-head attention encoder, whose projected output has the same F × M shape as the MVDR weights.Source: B. Transformer-based beamforming… · p. 4
• The Transformer’s causal correction is added to CNN-MVDR weights, then the corrected weights are normalized and applied to the multichannel STFT.Source: B. Transformer-based beamforming… · pp. 3–4

• The SAD network skips speech-mask estimation and beamforming when it detects no speech, when estimating speech–noise masks is ill-defined.Source: C. Speech Activity Detection (SAD)… · p. 4
• A 1-D ResNet-8 processes the raw waveform before the STFT and uses a fully connected head for its binary speech/non-speech decision.Source: C. Speech Activity Detection (SAD)… · p. 4
• The 27.6k-parameter SAD network requires 6× fewer operations than the full speech-enhancement pipeline, which requires 76 MMACs.Source: C. Speech Activity Detection (SAD)… · p. 4

• If run sequentially on every pass, the full pipeline requires 443.6 million MACs, including 76 MMACs for SAD and 367.6 MMACs for speech enhancement.Source: D. Interleaved execution strategy · p. 4
• The six-channel STFT computation requires 7.7 MMACs.Source: D. Interleaved execution strategy · p. 4
• G1 requires 354 MMACs, dominated by the 342-MMAC CNN mask estimator, while G2 requires 5.9 MMACs per frame.Source: D. Interleaved execution strategy · p. 4
• While G1 incrementally computes the next weights, G2 applies the current weights and Transformer correction at each hop, keeping enhanced output uninterrupted.Source: D. Interleaved execution strategy · p. 4
• The execution schedule begins by buffering the audio samples needed for initial beamforming weights, then uses the remaining hop budget to advance G1 alongside G2.Source: D. Interleaved execution strategy · p. 4, Figure 2 · p. 4

1) Hardware platform & deployment:

• GAP9 combines a RISC-V host with 1.5 MB of L2 scratchpad, a nine-core RISC-V cluster, the NE16 neural engine, and four FPUs.Source: 1) Hardware platform & deployment · p. 4
• A custom attention kernel uses the cluster architecture and native float16 SIMD instructions because GapFlow lacks optimized Transformer attention kernels.Source: 1) Hardware platform & deployment · p. 5
• The six microphone spectrograms remain in external RAM, while faster L2 holds per-microphone buffers for G2.Source: 1) Hardware platform & deployment · p. 5

IV. EXPERIMENTAL RESULTS

• The evaluation on GAP9 compares the pipeline with MCU-deployed speech-enhancement systems across memory, quantization, performance, and energy efficiency.Source: §IV · p. 5

A. Setup & model training

• Training mixes 28 speakers and 10 DEMAND noise types at 0, 5, 10, and 15 dB SNR; testing uses two held-out speakers, five unseen noise types, and intermediate SNRs.Room dimensions, reverberation, microphone and source positions are simulated; audio is sampled at 8 kHz.Source: A. Setup & model training · p. 5
• The SAD dataset combines Voice Bank speech with 121 noise types and composite noises; test evaluation reports accuracy, precision, and recall.Recall matters because a false negative would cause current speech to be ignored by the pipeline.Source: A. Setup & model training · p. 5

B. Baselines

• SI-SNR rises from 10.15 to 22.02 dB (+11.87 dB), PESQ-WB from 2.61 to 3.75 (+1.14), STOI from 92.38 to 98.04, and ESTOI from 86.97 to 95.52.These full-pipeline results are compared with unprocessed noisy input.Source: B. Baselines · p. 6
• The combined pipeline improves all four metrics over standalone Transformer and CNN, without adding fusion or projection layers or increasing model capacity.Gains over the standalone Transformer are +2.23 dB SI-SNR, +3.56 ESTOI, +2.06 STOI, and +0.49 PESQ-WB; over the standalone CNN they are +4.32 dB, +5.57, +3.85, and +0.80, respectively.Source: B. Baselines · p. 6
• The CNN integrates one second of scene context, while the Transformer processes an approximately 30 ms window for faster spectro-temporal detail.Both stages operate on the same STFT representation.Source: B. Baselines · p. 6

C. Mixed-precision quantization

• Quantization ranges are calibrated on a random training-data subset, and on-device configurations are compared using identical fixed-noise input.The noisy reference under this setup scores STOI 92.02%, ESTOI 86.21%, and PESQ-WB 2.609; its SI-SNR is reported in the following passage.Source: C. Mixed-precision quantization · p. 6
• Int8 CNN quantization yields STOI 97.98%, ESTOI 95.42%, PESQ-WB 3.760, and SI-SNR 21.54 dB; float16 Transformer quantization produces no measurable metric change.The float32 model scores STOI 98.01%, ESTOI 95.49%, PESQ-WB 3.768, and SI-SNR 21.90 dB.Source: C. Mixed-precision quantization · p. 6
• Int8 quantization collapses MVDR output, while float16 lowers intelligibility below noisy input; the beamforming stage therefore remains float32.The passage attributes sensitivity to covariance estimation and matrix inversion being numerically ill-conditioned.Source: C. Mixed-precision quantization · p. 6

D. Latency Constraint on Weight Estimation

• Enhancement quality improves monotonically as CNN-derived weights are held longer within an utterance, with a full-utterance hold optimal; weights still need refreshing across changing utterances.The effective weight-production interval must not exceed the audio span consumed per update, preventing accumulating lag in a continuous stream.Source: D. Latency Constraint on Weight… · p. 6

E. Deployment on GAP9 SoC

• Int8 CNN execution on NE16 reduces mask-estimation runtime from 476 ms on general-purpose cores to 203 ms, a 2.38× reduction.Float16 CNN execution takes approximately 1.1 s per second of audio, exceeding the real-time requirement.Source: E. Deployment on GAP9 SoC · p. 7
• G2 applies Transformer corrections and weighted summation on every output frame, so its processing must meet the 15 ms frame deadline.Source: E. Deployment on GAP9 SoC · p. 7

1) Transformer Deployment:

• The baseline Transformer implementation exceeds execution-time and memory budgets, partly because it materializes an F × F attention-score matrix for F = 257 frequency bins.Serializing computation across attention heads lowers execution to approximately 3.70 million cycles but still requires an additional 132 kB L2 buffer.Source: 1) Transformer Deployment · p. 7
• Row-wise attention avoids materializing the full score matrix, reducing Transformer execution to approximately 2.22 million cycles without an extra L2 buffer.Fusing query-key, softmax, and score-value operations further reduces Transformer execution to approximately 1.65 million cycles; the full G2 takes 1.85 million cycles.Source: 1) Transformer Deployment · p. 7

2) Full application Deployment:

• G1 occupies about 3.14 MB of L3 and 920 kB of L2, while peak L2 use reaches 1447 kB, within GAP9’s 1.5 MB L2.The peak includes program code, buffers, weights, and Transformer working memory.Source: 2) Full application Deployment · p. 7, EXECUTION. WE REPORT FOR G2 THE TOTAL… · p. 8

EXECUTION. WE REPORT FOR G2 THE TOTAL ENERGY CONSUMED

• The final pipeline achieves 97.65 STOI, 94.84 ESTOI, 3.676 PESQ-WB, and 20.26 dB SI-SNR at 45.9 mW average power.The slower operating point stretches the full-pipeline inference interval to 564 ms while reducing average draw by 31%.Source: Table III · p. 8, EXECUTION. WE REPORT FOR G2 THE TOTAL… · p. 8
• The final schedule allocates each 15 ms frame as 10 ms for G1 and 5 ms for G2; G1 updates weights every 366 ms in the initial configuration.The whole SoC operates at 0.8 V and 370 MHz; each inference costs 24.2 mJ and average power is 66.1 mW.Source: EXECUTION. WE REPORT FOR G2 THE TOTAL… · p. 8

F. System-Level Evaluation

• With continuous pipeline operation and six microphones, estimated battery life is about 7.2 h; re-estimating weights once per second raises it to about 8.3 h.Both estimates fall short of the approximately 16 h full-day target.Source: F. System-Level Evaluation · p. 8
• The battery estimate assumes a 100 mAh, 3.7 V Li-Po battery with approximately 1330 J of energy; speech is assumed to occupy no more than about 30% of the day.Source: F. System-Level Evaluation · p. 8
• Speech-gated operation yields about 18.2 h with continuous weight re-estimation or about 20.3 h when weights are re-estimated once per second, both above the approximately 16 h target.The SAD retains 98.5% accuracy, 97.8% precision, and 99.9% recall in int8.Source: F. System-Level Evaluation · pp. 8–9

V. COMPARISON WITH STATE-OF-THE-ART

• +4.29 STOI, +6.97 ESTOI, +0.64 PESQ-WB, and +1.43 dB SI-SNR over the reimplemented single-channel streaming TCN, placing the designs at complementary quality-efficiency points.The TCN exceeds the pipeline in utilization and energy efficiency, but cannot exploit spatial information.Source: §V · p. 9
• Up to 7.0×, 8.7×, 1.30×, and 1.49× higher MAC/cycle throughput than RNNoise, TinyLSTM, GRU256, and LSTM256, respectively.The corresponding energy-efficiency gains are 13.0×, over two orders of magnitude, 1.37×, and 1.35×, respectively.Source: §V · p. 9
• The CNN–Transformer pipeline improves on the preliminary CNN-branch results by +2.56 dB SI-SNR, +4.49 ESTOI, +3.46 STOI, and +0.74 PESQ-WB.It matches that branch’s MAC/cycle throughput at 1.01× while improving energy efficiency by 1.19×.Source: §V · p. 9
• Unlike SlowFast’s reported monotonic quality degradation as reuse grows, this system tolerates stale weights because its CNN estimates a near-stationary acoustic scene over roughly 1 s.Figure 5 reports the effect of recomputation interval on STOI and SI-SNR; the text says the system shows the opposite behavior.Source: §V · p. 9

VI. CONCLUSIONS

• Under a realistic duty cycle with ~70% non-speech, the pipeline sustains 20 hours of continuous operation on a 100 mAh battery, including microphone power.The conclusion reports 24.2 mJ per inference and identifies the system as the first real-time, multi-channel, Transformer-based neural beamforming pipeline running directly on an MCU.Source: §VI · p. 9