Source-linked AI summary
Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting
Sercan O. Arik, Markus Kliegl, Rewon Child, Joel Hestness, Andrew Gibiansky, Chris Fougner, Ryan Prenger, Adam Coates
TL;DR
KWS requires accurate keyword detection at low false-alarm rates while keeping latency, complexity, and model size small. This paper studies CRNNs that combine convolutional and recurrent layers and evaluates architecture and training strategies. With 229k parameters, the model achieves 97.71% accuracy at 0.5 FA/hour on 5 dB SNR test data.
Problem
KWS must achieve high detection accuracy at low false-alarm rates while minimizing footprint size, latency, and complexity across noisy and far-field conditions.
Method
The paper develops a small-footprint CRNN using convolutional layers for local structure, bidirectional recurrent layers for temporal context, and training strategies including hard negative mining.
Results
97.71% accuracy is achieved at 0.5 FA/hour for the 5 dB SNR test set with a 229k-parameter CRNN.
Takeaways & Limitations
The study identifies a CRNN configuration near the 250k-parameter limit as a performance-size trade-off for small-footprint KWS.
Abstract
from arXiv · showhide
Keyword spotting (KWS) constitutes a major component of human-technology interfaces. Maximizing the detection accuracy at a low false alarm (FA) rate, while minimizing the footprint size, latency and complexity are the goals for KWS. Towards achieving them, we study Convolutional Recurrent Neural Networks (CRNNs). Inspired by large-scale state-of-the-art speech recognition systems, we combine the strengths of convolutional layers and recurrent layers to exploit local structure and long-range context. We analyze the effect of architecture parameters, and propose training strategies to improve performance. With only ~230k parameters, our CRNN model yields acceptably low latency, and achieves 97.71% accuracy at 0.5 FA/hour for 5 dB signal-to-noise ratio.
1. Introduction
KWS must detect keywords accurately from continuous, noisy audio while maintaining low false alarms and a small computational footprint. The paper develops a small-footprint CRNN system that combines convolutional and recurrent modeling with training strategies for single-keyword detection.
- Motivation: KWS detects a particular keyword from continuous audio to control device states, making high accuracy and low false-alarm rates critical.Applications must also handle background audio, reverberation, and embedded-device speaker sounds.
- Related work: Deep learning-based KWS has replaced traditional Hidden Markov Model sequence-search approaches because of superior performance.The paper places its CRNN approach within this shift toward deep learning-based systems.
- Motivation: DNN-based KWS systems can ignore input structure and context, while CNNs exploit local connectivity but cannot model context.These limitations motivate combining complementary convolutional and recurrent layers.
- Contribution: The paper develops a production-quality single-keyword CRNN with cross-entropy loss for a small-footprint model.Training strategies are added to improve performance while retaining a small model size.
2. Small-footprint keyword spotting
The proposed small-footprint CRNN converts audio into PCEN mel spectrograms, models local time-frequency structure with convolutions, and captures temporal context with bidirectional recurrent layers. Cross-entropy training uses externally estimated character alignments and sequential decoding to obtain keyword boundaries.
- 2.1. End-to-end architecture: The CRNN converts raw time-domain audio into per-channel energy normalized mel spectrograms for compact representation and efficient training.The resulting two-dimensional features are provided to the convolutional layer.
- 2.1. End-to-end architecture: Two-dimensional convolutional filtering models time-frequency structure before bidirectional GRU or LSTM layers process the entire frame.A fully connected layer and two-neuron softmax decoder then produce a scalar score.
- 2.2. End-to-end training: The model uses cross-entropy rather than CTC because small-footprint architectures require a lower-complexity training objective.Unlike CTC, cross-entropy training requires precise alignment of training samples.
- 2.2. End-to-end training: Deep Speech 2 estimates per-time keyword-character probabilities, which are smoothed and aligned with a heuristic sequential algorithm.Short padding is added when chopping keyword segments to cover edge cases.
3. Experiments and Results
Experiments evaluate CRNN architecture, training-data strategies, noise robustness, and far-field robustness under constrained model size. The results identify effective architecture choices and training trade-offs while demonstrating degradation and mitigation across adverse conditions.
- Experimental setup: ~16k samples from more than 5k speakers are evaluated in streaming inference using overlapping 1.5-second frames shifted by 100 ms.The metrics are false rejection rate (FRR) and false alarms per hour, with noise added according to SNR.
- 3.2. Impact of the model architecture: Larger CRNNs typically perform better; increasing convolution filters or recurrent hidden units is effective, while additional recurrent layers have limited impact.GRUs are preferred over LSTMs because they achieve better performance at lower complexity.
- 3.2. Impact of the model architecture: At 250k parameters, the CNN baseline reaches 4.31% FRR at 1 FA/hour and 5.73% FRR at 0.5 FA/hour, about 51% higher than the chosen CRNN.The performance gap decreases at higher SNR values.
- 3.3. Impact of the amount of training data: Increasing positive samples saturates quickly, whereas hard negative mining decreases FRR on the 5 dB SNR test set.The capacity-limited model benefits from greater diversity of relevant negative samples.
- 3.4. Noise robustness: Higher SNR produces lower FRR and stable performance at lower FA rates, while training centered at 5 dB degrades beyond that SNR.Adding lower-SNR augmentation can improve low-SNR performance but reduces high-SNR performance because of limited learning capacity.
- 3.5. Far-field robustness: Far-field distance causes performance degradation, especially with higher noise; far-field augmentation reduces degradation at farther distances but worsens performance on original data because of mismatch.Far-field test sets use impulse responses representing varied configurations and distances.
4. Conclusions
The study finds that CRNN performance depends on balancing model capacity, training data, and application conditions. At 0.5 FA/hour, the model achieves high accuracy across tested noise levels, though direct comparisons with other KWS systems are not meaningful.
- 97.71%, 98.71% and 99.3% accuracy are achieved at 0.5 FA/hour for 5 dB, 10 dB and 20 dB SNR, respectively.
- Hard negative mining improves performance, whereas simply increasing the number of positive samples provides limited gains under the model's capacity constraint.
- Training sets should reflect deployment conditions, including noise level and far-field conditions.
- The model-size and performance trade-off determines the optimal architecture parameters for small-footprint KWS.
- Comparisons with other KWS models are not meaningful because datasets, keywords, and inference tasks differ.
- Further performance improvement remains possible because human performance on KWS is excellent.