Source-linked AI summary
Sub-Model Short-Term Memory Convolutions for Keyword Spotting Systems on Device
Paweł Warlewski, Artur Czeczko, Artur Szumaczuk, Grzegorz Stefański, Szymon Klimaszewski
TL;DR
Resource-constrained devices make accurate, low-latency keyword spotting challenging, while sliding-window CNN inference can repeat computations. This paper adapts STMC through sub-model decomposition and scheduling; on Google Speech Commands, SM-STMC reduces MCPS by up to 82% versus the standard sliding-window baseline while retaining high recognition performance.
Problem
On resource-constrained edge devices, KWS must meet high accuracy with limited compute and memory, while online sliding-window CNN inference can introduce redundant calculations.
Source: §1 · p. 1, §2.1 · p. 1Method
SM-STMC extends STMC by decomposing a convolutional model into sub-models and scheduling their execution to remove redundant temporal states in online inference.
Source: §2.2 · p. 2, §4 · p. 4Results
Up to 82% MCPS reduction versus the standard sliding-window baseline, alongside a 65% memory footprint reduction versus original STMC, with high recognition performance on Google Speech Commands.
Source: §4 · p. 4Takeaways & Limitations
SM-STMC requires no retraining or additional parameters and can be deployed through standard frameworks such as TensorFlow Lite on resource-constrained edge devices.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Keyword Spotting (KWS) is becoming increasingly important as voice-controlled devices grow more widespread. While voice interaction with smartphones and smart TVs is already common, deploying KWS on heavily resource-constrained edge devices such as wearables remains challenging. These systems must meet high accuracy requirements while operating under strict constraints on computational power, memory footprint, and real-time latency. In this work, we present an application of the STMC (Short-Term Memory Convolutions) framework to adapt a modular CNN model for online, LSTM-like inference. Our approach reduces power consumption and redundant computations while maintaining the stability and simplicity of training CNNs. We achieve up to 82% and 46% MCPS reduction compared to equivalently frequent standard CNN execution and vanilla STMC, respectively. The best configuration achieves 93.8% accuracy on the 11-class Google Speech Commands task and 97.1% on the same task with zero-padded data.
1. Introduction
Keyword spotting on resource-constrained devices must balance accuracy with limited compute and memory. This paper adapts a modular CNN for online KWS while targeting redundant computation and preserving CNN training simplicity.
1. Introduction
- KWS detects phrases in incoming audio, and resource scarcity makes the task harder on wearable devices.CNNs are widely adopted for their small memory footprint and demonstrated performance; transformer complexity and memory use make them inadequate for resource-constrained devices.Source: §1 · p. 1
- Online KWS needs fine-grained inference on continuous audio, but sliding-window CNN execution with small hops creates redundant calculations that can exceed compute limits.Short target phrases may occur at any time in the stream.Source: §1 · p. 1
- SM-STMC extends STMC to reduce model footprint for online keyword spotting while preserving accuracy.Source: §1 · p. 1
2. Method
SM-STMC reduces redundant temporal states in online convolutional networks by decomposing the model into sub-models and scheduling their execution. It retains STMC-style memory buffers while avoiding retraining and enabling deployment through standard conversion pipelines.
2.1. State Redundancy in the Classical Approach
- For networks with pooling layers, STMC can maintain extra temporal states that are valid computations but unnecessary when classification outputs are not needed every frame.A stride-greater-than-one max-pooling layer multiplies internal temporal states by its stride.Source: §2.1 · p. 1
2.2. Sub-model Decomposition and State Scheduling
- SM-STMC decomposes the convolutional backbone into independent sub-models ending at pooling or output layers, then schedules the deepest required sub-model at each time step.The sub-models do not share layers or weights; memory buffers preserve the original data flow.Source: §2.2 · p. 2
2.3. Memory Management and Computational Complexity
- The network runs in full only once every 2N steps; intermediate steps execute fractions of the computation, reducing average operations while latency and alignment remain unchanged.Source: §2.3 · p. 2
2.4. Training and Deployment
- SM-STMC requires no retraining or added parameters, and independent sub-models avoid runtime control flow for standard deployment pipelines.Deployment still requires memory management and scheduling on the target device.Source: §2.4 · p. 2
3. Experiments
Experiments compare VGG, STMC, and SM-STMC on standard and silence-padded Google Speech Commands data, measuring recognition performance, computational cost, and latency. Results highlight accuracy and efficiency trade-offs across offline and online inference settings.
3.1. Dataset and Experimental Setup
- The experiments compare standard VGG, sliding-window VGG, STMC, and SM-STMC, using two models and a conventional LSTM comparison.Evaluation uses standard 1-second GSC samples and a 2-second version with 0.5 seconds of silence added at each end.Source: §3 · p. 2, §3.1 · pp. 2–3
3.2. Parameters
- Table 1 reports embedding parameters by convolutional block and classifier parameters; Table 2 reports int8 temporal-buffer sizes for STMC configurations.STMC buffer sizes depend on kernel, filters, and mel features; standard STMC buffer sizes are also multiplied by the number of states.Source: §3.2 · p. 3, Table 1 · p. 3, Table 2 · p. 3
3.3. Evaluation Metrics
- Weighted average recall equals classification accuracy in this single-label multiclass evaluation, with classifiers evaluated 7.8125 times per second.The reported measures are weighted average precision, recall, and F1-score.Source: §3.3 · p. 3
3.4. Results on Standard GSC
- The standard VGG model has 1.04% lower accuracy than STMC on the standard GSC dataset.VGG would need evaluation at every time bin to obtain performance comparable to STMC-based approaches.Source: §3.4 · p. 3, §3.3 · p. 3, Table 3 · p. 3
3.5. Results on Extended Inputs
- On the 2-second silence-padded dataset, standard VGG misclassifies more than half the samples when keywords span input-window boundaries.Figure 3 shows VGG accuracy drops at temporal shifts of -0.5 s and 0.5 s, while STMC remains stable.Source: §3.5 · p. 3, Figure 3 · p. 4
- 3.28% higher accuracy than the 1-second setting is reported for VGG evaluated eight times per second and the SM-STMC models.The keyword is captured across overlapping windows, increasing the chance that one contains a favorable representation for classification.Source: §3.5 · p. 4
3.6. Computational Cost and Latency
- 2.271 and 2.782 are the STMC computational-cost factors over standard VGG for the first and second models; SM-STMC lowers them to 1.531 and 1.472.The factors compare total computational cost against the standard VGG baseline.Source: §3.6 · p. 4
- SM-STMC2 processes an equivalent audio duration 1.78 times more efficiently than STMC and 4.44 times more efficiently than LSTM.SM-STMC1 classifies every fourth frame and SM-STMC2 every eighth; their average latency increases over frame-by-frame STMC but remains within classification time.Source: §3.6 · p. 4
4. Conclusion
The paper proposes SM-STMC, which uses sub-model decomposition and static scheduling to reduce redundant temporal states in online convolutional networks. On Google Speech Commands, it preserves high recognition performance while reducing computation and memory, and supports deployment without retraining or added parameters.
4. Conclusion
- SM-STMC removes redundant temporal states through sub-model decomposition and static scheduling, executing only the required network depth.Source: §4 · p. 4
- SM-STMC reduces MCPS by up to 82% versus the standard sliding-window baseline and memory footprint by 65% versus original STMC, with latency remaining within real-time requirements.The reduced classification frequency introduces a marginal latency increase.Source: §4 · p. 4
- SM-STMC requires no retraining or additional parameters and can be deployed via standard frameworks such as TensorFlow Lite on resource-constrained wearables and hearables.Source: §4 · p. 4
5. Generative AI Use Disclosure
The authors used an LLM to polish the manuscript and find grammar errors, but not to write it directly or generate code for the work.
5. Generative AI Use Disclosure
- The authors used an LLM for manuscript polishing and grammar checking, not direct writing or code generation.Source: §5 · p. 5