Source-linked AI summary
Temporal Convolution for Real-time Keyword Spotting on Mobile Devices
Seungwoo Choi, Seokjun Seo, Beomjun Shin, Hyeongmin Byun, Martin Kersner, Beomsu Kim, Dongyoung Kim, Sungjoo Ha
TL;DR
Mobile keyword spotting requires both low latency and high accuracy, yet prior work provided limited quantitative analysis of actual mobile inference time. The paper introduces TC-ResNet, which uses temporal convolution with a compact ResNet architecture, and reports 385x speedup with a 0.3%p accuracy increase versus a state-of-the-art CNN-based model on Google Pixel 1.
Problem
KWS must be immediate and accurate on mobile devices with restricted hardware, while prior studies provided limited direct quantitative analysis of actual mobile inference latency.
Method
TC-ResNet applies one-dimensional temporal convolution along time, treats MFCC features as input channels, and integrates it into a compact ResNet architecture.
Results
385x speedup and a 0.3%p accuracy increase were achieved on Google Pixel 1 versus the state-of-the-art CNN-based KWS model on the Google Speech Commands Dataset.
Takeaways & Limitations
Temporal convolution is empirically associated with reduced computation and increased accuracy compared with 2D convolution for mobile KWS.
Abstract
from arXiv · showhide
Keyword spotting (KWS) plays a critical role in enabling speech-based user interactions on smart devices. Recent developments in the field of deep learning have led to wide adoption of convolutional neural networks (CNNs) in KWS systems due to their exceptional accuracy and robustness. The main challenge faced by KWS systems is the trade-off between high accuracy and low latency. Unfortunately, there has been little quantitative analysis of the actual latency of KWS models on mobile devices. This is especially concerning since conventional convolution-based KWS approaches are known to require a large number of operations to attain an adequate level of performance. In this paper, we propose a temporal convolution for real-time KWS on mobile devices. Unlike most of the 2D convolution-based KWS approaches that require a deep architecture to fully capture both low- and high-frequency domains, we exploit temporal convolutions with a compact ResNet architecture. In Google Speech Command Dataset, we achieve more than \textbf{385x} speedup on Google Pixel 1 and surpass the accuracy compared to the state-of-the-art model. In addition, we release the implementation of the proposed and the baseline models including an end-to-end pipeline for training models and evaluating them on mobile devices.
1. Introduction
KWS must respond immediately and accurately on resource-constrained mobile devices, but existing CNN approaches require substantial computation and lack quantitative mobile-latency analysis. The paper proposes TC-ResNet with temporal convolution and reports substantially faster, more accurate performance while releasing implementations and benchmarking tools.
- KWS supports hands-free mobile interactions, requiring immediate and accurate keyword detection under restricted hardware resources.
- CNN-based KWS models provide reliable accuracy but demand considerable computation, while prior studies often used operation counts instead of directly measuring mobile inference time.Operation counts can diverge from latency because they omit memory-access costs and platform-specific optimization.
- TC-ResNet applies one-dimensional temporal convolution along time while treating MFCC features as input channels to improve accuracy and reduce mobile latency.
- 385x speedup and a 0.3%p accuracy increase were achieved on Google Pixel 1 versus the state-of-the-art CNN-based KWS model on the Google Speech Commands Dataset.
- The authors release proposed and baseline KWS models together with a complete benchmark tool for evaluating them on mobile devices.
- Ablation experiments identify temporal convolution as responsible for reduced computation and increased accuracy relative to 2D convolutions in mobile KWS.
2. Network Architecture
The architecture reformulates MFCC inputs as temporal sequences and uses temporal convolutions within compact ResNet models. This design broadens frequency access while reducing feature-map size, computation, and footprint, with width multipliers providing capacity flexibility.
- Temporal Convolution for KWS: MFCC transforms audio into a time-frequency representation, which conventional KWS models commonly process as a 2D tensor with time and frequency axes.
- Temporal Convolution for KWS: The method reshapes MFCC input to one-dimensional sequential data, setting the frequency dimension as input channels for temporal convolution along time.
- Temporal Convolution for KWS: Temporal convolution lets each layer use lower-level features across the full frequency range, avoiding many stacked layers while retaining informative audio features.
- Temporal Convolution for KWS: Smaller temporal-convolution feature maps reduce computation and footprint in subsequent layers, which supports fast mobile KWS.
- TC-ResNet Architecture: TC-ResNet8 is the base model, while TC-ResNet14 doubles its residual blocks; width multipliers adjust channels to fit capacity constraints.
3. Experimental Framework
The experiments evaluate TC-ResNet and baselines on Google Speech Commands using accuracy, ROC analysis, computational measures, and direct mobile-device latency. Models are trained and benchmarked with a reproducible TensorFlow-to-TensorFlow-Lite pipeline.
- Dataset: The Google Speech Commands Dataset contains 64,727 one-second utterances labeled across 30 target categories, with evaluation focused on 12 classes.
- Dataset: Models use 40 MFCC features extracted with 30 ms windows and 10 ms stride, following the prior study’s feature-extraction settings.
- Training: Training uses TensorFlow, stochastic gradient descent with momentum, weight decay, dropout, 30k iterations, learning-rate decay, and validation-based early stopping.
- Evaluation: Accuracy is the main metric, with models trained 15 times; ROC curves use false alarm and false reject rates with multiclass micro- and experiment-level averaging.
- Mobile benchmarking: The study reports FLOPs and all parameters, then directly measures average inference time on Google Pixel 1 using TensorFlow Lite with one little core.
- Baselines and tooling: Baselines include CNN, depthwise-convolution DS-CNN, and residual Res variants, while the released codebase supports training, conversion, evaluation, and benchmarking.
4. Experimental Results
TC-ResNet improves the accuracy–latency trade-off over baseline KWS models, while temporal convolution remains faster than parameter-matched 2D-convolution and pooling alternatives. Wider and deeper variants improve accuracy at the expense of inference speed.
- Baseline comparison: 385x speedup and 0.3%p higher accuracy are achieved by TC-ResNet8 versus Res15 on Google Pixel 1.
- Baseline comparison: 29x speedup and 5.4%p higher accuracy are achieved by TC-ResNet8 versus CNN-1, while accuracy improves 11.5%p over CNN-2 at comparable latency.
- Baseline comparison: 43x speedup and 6%p higher accuracy are achieved by TC-ResNet8 versus the slimmer Res8-Narrow baseline.
- Model scaling: Wider and deeper TC-ResNet variants achieve better accuracy at the expense of inference speed.
- ROC analysis: TC-ResNet14-1.5 has the smallest AUC among selected models, indicating fewer missed target keywords across false-alarm rates.
- Impact of temporal convolution: TC-ResNet8 is 9.2x faster than parameter-matched 2D-ResNet8 with comparable accuracy, and TC-ResNet8-1.5 surpasses 2D-ResNet8 using fewer computational resources.
- Impact of temporal convolution: 2D-ResNet8-Pool loses 1.2%p accuracy and remains 3.2x slower than TC-ResNet8 despite improving inference time at the same parameter count.
5. Related Works
Prior CNN-based KWS work commonly uses 2D convolutions over time–frequency representations, with successive studies targeting accuracy and resource constraints. The related models include small-footprint, architecture-searched, depthwise, residual, and dilated-convolution approaches.
- CNN-based KWS: Earlier KWS studies commonly apply 2D convolutions to time–frequency inputs such as MFCCs.
- CNN-based KWS: Prior work includes small-footprint CNNs, architecture searches under memory and computation constraints, and residual networks with dilated convolutions.
6. Conclusion
The investigation quantitatively evaluated mobile KWS inference speed and found that temporal convolution enabled substantial acceleration while improving accuracy. The authors identify broader architecture studies as a worthwhile next step.
- 385x speedup and 0.3%p higher accuracy were achieved against the state-of-the-art model on Google Pixel 1.The evaluation measured inference speed directly on the mobile device.
- Ablation results attributed the dramatic speedup and accuracy improvement to temporal convolution.
- Further studies should analyze temporal convolutions across diverse network architectures.