Source-linked AI summary
Deep Residual Learning for Small-Footprint Keyword Spotting
Raphael Tang, Jimmy Lin
TL;DR
Keyword spotting needs accurate, compact on-device models, but prior compact approaches traded accuracy for footprint and comparisons were often difficult. This paper applies residual learning and dilated convolutions to CNN keyword spotting on the Google Speech Commands Dataset, finding that full and compact variants outperform prior CNN baselines and compact models.
Problem
Keyword spotting requires balancing detection accuracy with a small footprint, while prior recurrent comparisons lacked public implementations and earlier work often used private datasets.
Method
The paper applies deep residual learning and dilated convolutions to CNN keyword spotting and varies model depth, width, and pooling on the Google Speech Commands Dataset.
Results
95.8% accuracy versus Google’s 91.7% marks the full residual network’s headline result, while compact variants outperform previous compact CNNs and can reduce parameters 50× and multiplies 18×.
Takeaways & Limitations
The results establish open-source state-of-the-art reference models on a common benchmark for future keyword-spotting research.
Takeaways & Limitations
The paper does not compare its CNN approaches with recurrent architectures because publicly available reference implementations were unavailable.
Abstract
from arXiv · showhide
We explore the application of deep residual learning and dilated convolutions to the keyword spotting task, using the recently-released Google Speech Commands Dataset as our benchmark. Our best residual network (ResNet) implementation significantly outperforms Google's previous convolutional neural networks in terms of accuracy. By varying model depth and width, we can achieve compact models that also outperform previous small-footprint variants. To our knowledge, we are the first to examine these approaches for keyword spotting, and our results establish an open-source state-of-the-art reference to support the development of future speech-based interfaces.
1. INTRODUCTION
Keyword spotting detects predefined words on devices, supporting privacy-preserving command recognition while balancing accuracy against model footprint.
- On-device keyword spotting detects predefined keywords in user utterances for mobile phones and smart-home devices.
- Local recognition of commands and frequent words can avoid transferring audio recordings to the cloud, sidestepping privacy concerns.
- Neural-network research focuses on balancing high detection accuracy with a small model footprint.
- Compact models are often derived from full models by sacrificing accuracy through sparsification.
- 95.8% accuracy lets the full residual network outperform Google’s previous CNN at 91.7% on the Google Speech Commands Dataset.
- A compact variant reaches accuracy only slightly below Google’s best CNN while reducing parameters 50× and feedforward multiplies 18×.
2. RELATED WORK
The paper places residual learning and dilated convolutions within CNN-based keyword spotting, where public benchmarks and implementations enable more comparable research.
- Residual networks had advanced deep learning and had already been applied to speaker identification and automatic speech recognition.
- Earlier keyword-spotting work progressed from multilayer perceptrons and HMM comparisons to CNNs with smaller footprints for low-power applications.
- The authors focus on CNNs because they remain the standard baseline, are straightforward to tune, and have implementations in multiple frameworks.
- Recurrent-model comparisons were constrained by the lack of publicly available implementations and, previously, a common benchmark.
- The paper identifies residual learning as an unexplored keyword-spotting direction and uses dilated convolutions to capture longer-range dependencies.
3. MODEL IMPLEMENTATION
The implementation combines residual blocks and dilation for keyword spotting, then varies depth, width, and pooling to create models with different computational footprints.
- The experiments use open-source code, band-pass filtering, MFCC features, and one-second stacked inputs.
- Residual connections add an earlier layer’s input to a downstream output, implementing H(x) = F(x) + x.
- Residual blocks use bias-free convolutions, ReLU units, batch normalization, and dilation to enlarge the receptive field with fewer layers.
- Base model: The res15 base model has six residual blocks and 45 feature maps, with exponentially increasing dilation and a 125×125 receptive field.
- Base model: Res15 uses roughly 238K parameters and 894M multiplies, the two quantities used to measure model footprint.
- Compact and deep variants: Res8 halves the residual-block count and adds 4 × 3 average pooling, making dilation unnecessary after reducing time and frequency dimensions.
- Compact and deep variants: Res26 doubles the residual-block count to 12, uses 2 × 2 average pooling, and omits dilation because its convolutional receptive field covers the input.
4. EVALUATION
The evaluation compares ResNet variants with Google CNN baselines on the Speech Commands Dataset, measuring accuracy and model footprint while also examining ROC behavior. Compact and wide residual models improve the accuracy–footprint tradeoff, whereas excessive depth and reduced width can degrade performance.
- Dataset and metrics: The models were evaluated on Google’s Speech Commands Dataset using the same procedure as Google’s reference implementation.The task uses 12 classes and an 80%/10%/10% training, validation, and test split.
- Dataset and metrics: Accuracy is the primary quality metric, while ROC curves report false alarm rate against false reject rate at sensitivity thresholds.The evaluation also records parameter count and multiply count as footprint measures.
- Results: The narrow res15 remained significantly better than Google CNNs despite sacrificing accuracy and using approximately 30% more multiplies.Compared with the base residual model, narrowing feature maps trades accuracy for a smaller model footprint.
- Results: The wide res8 strictly dominated all Google models, achieving significantly better accuracy with a smaller footprint.The narrow res8 reduced the footprint further, with a small performance degradation relative to tpool2, while using 50× fewer parameters and 18× fewer multiplies.
- Results: res26 had lower accuracy than res15, while width appeared to affect accuracy more strongly than depth across narrow and wide variants.The authors suggest that res26 exceeded the depth at which its parameters could be properly optimized.
- ROC analysis: The res15 ROC curve dominated the other selected models at all operating points, consistent with the accuracy results.Figure 3 compares two competitive baselines with res8, res8-narrow, and res15.
5. CONCLUSIONS AND FUTURE WORK
The paper presents residual learning and dilated convolutions as open-source reference approaches for keyword spotting on a common benchmark. Future work would compare these CNN approaches with recurrent architectures once comparable implementations and benchmarks are available.
- Conclusions: The work establishes new state-of-the-art open-source reference models on Google’s Speech Commands Dataset.The authors encourage others to build on these reference models.
- Future work: A comparison with recurrent architectures remains future work because publicly available reference implementations and a common benchmark were previously lacking.The authors note that the common-benchmark problem has since been addressed and that such comparisons would be interesting.