Source-linked AI summary
Does per-frame early exit pay? A compute-matched study of dynamic depth for on-device speech enhancement
Clément Laroche, Riccardo Miccini
TL;DR
On-device speech enhancement must balance quality and compute, but devices that accelerate only static int8 graphs complicate runtime depth selection. The paper trains monotonic exits and compares dynamic routing with extracted static models; its results show the static models can match the routed quality–latency frontier on the evaluated speech-dense dataset.
Problem
On-device speech enhancement must balance denoising performance with real-time compute and battery constraints, but evidence comparing dynamic models with compute-matched statics and measured accelerator latency is limited.
Source: §1 · p. 1Method
The authors regularize per-frame exit quality and fine-tune only exit heads, then compare dynamic routing and extracted fixed-depth models with recipe-matched statics after int8 quantization.
Source: §1 · p. 1Results
0.04–0.11 PESQ improvements under the same training budget; the repaired static frontier nearly overlaps the routed frontier, with matched-compute PESQ differences below 0.01.
Source: §3.2 · p. 3, §5 · p. 4Takeaways & Limitations
On speech-dense workloads, per-frame depth selection brings little additional benefit over the static quality–latency frontier, while routed models retain workload-dependent operating points.
Source: §3.2 · p. 3, §5 · p. 4The claims are limited to speech-dense VoiceBank–DEMAND data; workloads with long pauses are outside the study.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Deep learning-based speech enhancement is increasingly deployed on-device in hearing aids, headsets, and earbuds. Most of these devices, however, can only accelerate static int8 graphs, so a depth-varying network must be implemented as several graphs, orchestrated by a policy. In this paper, we supervise every intermediate depth of one causal model, then we fine-tune its output heads to guarantee that deeper outputs are never worse than shallower ones. Using this training protocol, we can derive a family of static models that are more Pareto-efficient than their equivalently-sized counterparts trained from scratch on the same budget. Specifically, we achieve up to 0.11 higher PESQ for equivalent compute, and match the best PESQ at 30% less compute. We then quantize the models to int8 and measure the latency-quality frontier on an STM32N6 microcontroller. On VoiceBank-DEMAND, the dynamic enhancer lies on the same frontier as the static models, rather than trading quality for dynamic execution. Running the policy on the companion Cortex-M55 takes only 26 $\mu$s per frame, while splitting the enhancer into separate NPU graphs adds 2.2% latency overhead. The cost of dynamic execution is therefore small.
1. INTRODUCTION
On-device speech enhancement must balance denoising quality with real-time compute and battery constraints, while supporting different operating points on devices limited to static int8 graphs. This paper addresses whether early-exit training improves the static models it yields, dynamic execution, or both.
1. INTRODUCTION
- Speech enhancement must meet real-time and shared-battery constraints across devices and workloads, while microcontroller-class accelerators support only static int8 graphs.Runtime depth selection therefore requires a policy and dispatch outside the compiled graph.Source: §1 · p. 1
- The study compares dynamic speech enhancement with compute-matched static baselines and measures quantized models on-device, addressing gaps in prior comparisons based on exits or nominal MACs.It also examines frame-level quality and whether routing benefits arise from runtime adaptation, improved extracted static models, or both.Source: §1 · p. 1
- Per-frame monotonicity regularization and heads-only fine-tuning encourage deeper exits to improve over shallower ones, addressing regressions seen on nearly half of frames with task-loss-only training.The approach targets the overthinking phenomenon previously observed in early-exiting networks.Source: §1 · p. 1
- The paper evaluates both runtime routing and fixed-depth models extracted from one weight-shared model against recipe-matched static models, using int8 quantization and measured STM32N6 latency.The resulting fixed-depth models form a competitive quality–compute frontier that also holds when measured latency replaces theoretical MACs.Source: §1 · p. 1
2. EARLY-EXIT CONV-FSENET
The model attaches mask exits along a causal Conv-FSENet stack and routes each frame to a selected depth. Training supervises the exits and policy, then repairs the exit ladder so deeper outputs do not worsen performance.
2.1. Architecture and exits
- A one-shot router selects an exit before stack execution, so each frame runs only the frontend and stacks up to its chosen depth.Mask heads follow the frontend and each stack; earlier exits avoid the cost of subsequent stacks.Source: §2.1 · p. 2
2.2. Training and exit policy
- Deep supervision trains the routed output and each exit, while a clean-signal knee target identifies the cheapest exit within ε of the best exit’s error.The policy is trained by cross-entropy against this target, jointly with the backbone or on a frozen enhancer.Source: §2.2 · p. 2
2.3. Repairing the exits
- Because deeper exits can perform worse, the repair freezes the backbone and policy and fine-tunes only exit heads with a regularizer penalizing frames whose error increases with depth.The regularizer encourages monotonicity as a soft training constraint rather than enforcing it post hoc.Source: §2.3 · p. 2
3. RESULTS
The routed models beat dedicated static baselines, but after exit repair the routed and repaired-static quality–compute frontiers nearly overlap. Routing’s value is therefore adapting compute across frames without noticeably lowering the attainable quality frontier.
3.1. Experimental setup
- Experiments use 16-kHz VoiceBank–DEMAND, compute-matched Conv-FSENet static baselines, and whole-clip SI-SDR and wideband PESQ on 824 test clips.The static grid spans 5.7–41.8 MMAC/s and brackets routed points; static models follow the same training protocol with routing-specific losses removed.Source: §3.1 · p. 3
3.2. Quality–compute frontier
- Against dedicated statics, routed H=64 reaches 2.719 PESQ at 8.48 MMAC/s, while the static frontier is around 2.60 at the same cost.Routed models remain above the dedicated-static frontier, although the gap shrinks as compute increases.Source: §3.2 · p. 3
- Routing does not raise the best quality at a given compute budget, but can adapt compute across frames without noticeably lowering that frontier.Bypass selection is 2.4–3.4× more frequent in silence than in speech; the benefit is workload-dependent.Source: §3.2 · p. 3
- At matched compute, routed and mono-repaired exits differ by less than 0.01 PESQ over their common range.The routed points at 8.48, 12.18, and 20.22 MMAC/s differ from the interpolated repaired frontier by +0.009, −0.001, and +0.007 PESQ, respectively.Source: Figure 3 · p. 3, §3.2 · p. 3
4. MEASURED LATENCY ON A MICROCONTROLLER
On the STM32N6, analytic MACs alone do not predict latency, and routing’s fixed-backbone savings do not consistently beat alternative static topologies. The measured results place routed models on a comparable latency–quality frontier, with implementation and workload details shaping the cost.
4.1. deployment platform
- The STM32N6570-DK combines a Cortex-M55 host with a Neural-ART NPU; unsupported operations and graph orchestration remain on the host.The models use a 16 ms hop, with STFT, iSTFT, and masking on the host and exported ring buffers as explicit state tensors.Source: §4.1 · p. 3
4.2. MACs versus measured latency
- Across ten STM32N6 topologies, execution time per analytic MMAC varies 1.7×, from 82 to 137 µs/MMAC, so nominal MACs can misorder runtime.For example, (H=96, R=1) uses 21% more MACs than (H=32, R=2) but runs 15% faster.Source: §4.2 · p. 3, Table 1 · p. 4
- On a difficult utterance, the router bypasses 74% of speech-inactive frames versus 12% of speech frames, reducing compute by 58.7% over warmed repeats versus the deepest exit.The first of four repeats warms the GRU and is discarded.Source: Figure 4 · p. 4
4.3. Dynamic execution
- The routed enhancer is deployed as nine subgraphs, with routing and dispatch on the host; shared frontend and stack graphs let the family use one ladder’s storage.Quantization shifts routed compute by at most 2%; the repaired family changes by −0.03 to +0.01 PESQ versus float.Source: §4.3 · p. 4
- Skipped stacks retain their own most-recently-processed ring-buffer state; deployed-path graph execution costs −0.005±0.011 PESQ over 824 clips and 0.14 dB SI-SDR.Zero-order hold and propagating the last computed state to deeper stacks do not recover the lost quality.Source: §4.3 · p. 4
4.4. On-Device results
- With the backbone fixed, repaired routed H=64 at ε=2·10−3 runs at 0.930 ms/frame versus 1.215 ms for static H=64, R=2, with PESQ changing from 2.716 to 2.694.At H=256, routing reduces latency from 2.472 to 1.668 ms/frame while retaining 2.773 versus 2.781 PESQ.Source: §4.4 · p. 4, Table 2 · p. 4
- Static alternatives can match or dominate routed operating points: routed H=32 gives 2.639 PESQ at 0.769 ms/frame, versus static H=64, R=1 at 2.684 PESQ and 0.751 ms.At the more quality-oriented end, routed H=64 with asym4 reaches 2.724 PESQ at 1.473 ms, while static H=256, R=1 reaches 2.762 PESQ at 1.364 ms.Source: §4.4 · p. 4, Table 2 · p. 4
5. CONCLUSION
The conclusion presents monotonicity-regularized early-exit training as a way to produce deployable int8 models and a family of fixed-depth models. It reports quality gains at matched training budget, while finding little additional routing benefit on speech-dense data and leaving sparser workloads as an open question.
5. CONCLUSION
- The training recipe yields 0.04–0.11 PESQ improvements under the same training budget and can match the best PESQ with 30% less compute, at a 1.0 dB SI-SDR drop.The models retain the option of dynamic routing and are quantized to int8 for STM32N6 deployment.Source: §5 · p. 4
- On the STM32N6, the extracted static models and routed models lie on a latency–quality frontier that routed models can also match.On speech-dense datasets, per-frame depth selection brings little additional benefit.Source: §5 · p. 4
- The apparent routing opportunity largely comes from frames where deeper exits degrade results, and monotonicity repair removes much of that signal.Whether automatic routing is more valuable on sparser workloads with pauses and cheap bypass opportunities remains an open next step.Source: §5 · p. 4