Source-linked AI summary
A neural attention model for speech command recognition
Douglas Coimbra de Andrade, Sabato Leo, Martin Loesener Da Silva Viana, Christoph Bernkopf
TL;DR
The paper addresses speech command recognition models that are accurate yet suitable for local execution. It proposes an attention-based recurrent architecture and reports state-of-the-art performance across multiple keyword-spotting tasks while retaining a small model size. The attention mechanism also supports inspection of which audio regions inform predictions.
Problem
Speech command recognition needs models that can operate locally rather than relying on computation-intensive cloud speech pipelines.
Method
The paper introduces a convolutional recurrent network with attention for speech command recognition.
Results
The attention RNN achieves state-of-the-art performance on 20-command, 12-command, 35-word, and left-right tasks while maintaining a small model footprint.
Takeaways & Limitations
Attention improves recognition while making the model's considered audio regions inspectable and the architecture suitable for local deployment.
Takeaways & Limitations
The paper identifies the effectiveness of augmenting audio samples with noise as an unresolved limitation.
Abstract
from arXiv · showhide
This paper introduces a convolutional recurrent network with attention for speech command recognition. Attention models are powerful tools to improve performance on natural language, image captioning and speech tasks. The proposed model establishes a new state-of-the-art accuracy of 94.1% on Google Speech Commands dataset V1 and 94.5% on V2 (for the 20-commands recognition task), while still keeping a small footprint of only 202K trainable parameters. Results are compared with previous convolutional implementations on 5 different tasks (20 commands recognition (V1 and V2), 12 commands recognition (V1), 35 word recognition (V1) and left-right (V1)). We show detailed performance results and demonstrate that the proposed attention mechanism not only improves performance but also allows inspecting what regions of the audio were taken into consideration by the network when outputting a given category.
1. Introduction
Existing speech interfaces often depend on cloud-based computation, motivating lightweight local command-recognition models. This work introduces an attention-based recurrent architecture intended to improve recognition while remaining suitable for local deployment and interpretability.
- Motivation: Cloud-based speech pipelines require substantial computation, limiting speech recognition in settings without uninterrupted broadband connectivity.
- Motivation: Lightweight speech command models could support voice-controlled robots and assistive devices operating without internet coverage.
- Contribution: The architecture targets simple speech commands, supports local execution on mobile devices, and uses attention to improve explainability.
- Related work: Attention for single-word recognition had not been investigated, despite attention models' success in understanding which input regions support predictions.
- Related work: The proposed recurrent neural network with attention outperformed most larger models, although some spectrogram methods exceeded 95% accuracy on specific tasks.
2. Neural Network Implementation
The model converts raw WAV audio into a mel-scale spectrogram, applies time-only convolutions and bidirectional LSTMs, then uses attention-weighted outputs for classification. Its design assumes the command can occur anywhere in a one-second file while selecting the middle LSTM output as a query because commands are expected to be centered.
- Design assumptions: The implementation assumes a single-word command may occur anywhere in the one-second audio, while treating the middle-vector selection as arbitrary.The stacked LSTMs are expected to retain sufficient memory for other vector choices to work.
- Input representation: The model accepts raw WAV files and computes mel-scale spectrograms through non-trainable Kapre layers.The audio uses an approximately 16 kHz sampling rate, with an 80-band mel scale, 1024-point discrete Fourier transform, and hop size of 128 points.
- Feature extraction: Time-only convolutions extract local relations from the two-dimensional mel-scale spectrogram.The convolutions operate along the time dimension rather than across both spectrogram dimensions.
- Recurrent encoding: Two bidirectional LSTM layers capture forward and backward long-term dependencies in the audio.The architecture uses stacked recurrent units before applying attention.
- Attention and classification: A projected LSTM output serves as a query vector for identifying the most relevant audio region, and the resulting weighted average supports classification.The weighted average is passed through three fully connected layers; the middle LSTM vector is selected because commands are expected to be centered.
3. Results
Across five speech-command tasks, the attention-RNN model is evaluated against prior models, with attention improving reported accuracy and exposing which audio regions influence classification. The results also show consistent training and task-specific confusion patterns, including errors requiring contextual information.
- Five tasks evaluate recognition of 20 commands on V1 and V2, 12 commands on V1, all 35 words, and left-right words.
- Attention provides a substantial improvement over other models across the reported recognition tasks.
- 94.5% accuracy on V2 for 20 commands exceeds the 88.2% baseline from Warden (2018).
- 3.2. Attention Plots: Attention weights visualize which audio regions are most relevant for classification and emphasize vowel transitions in the illustrated speech examples.
- 3.3. Confusion Matrices: The confusion matrices show frequent confusion between “three” and “tree” and between “no” and “down,” where contextual information would help distinguish words.
- 93.9 ± 0.2% accuracy on V2 for 35-word recognition is slightly below V1 at 94.3 ± 0.2%, partly because V2 has 35 test words rather than 30.
4. Conclusion
The proposed attention RNN combines compact speech processing with attention-based localization of informative audio regions, achieving state-of-the-art results across multiple keyword-spotting tasks. The authors also identify unexplored audio augmentation and pretrained-model directions for future work.
- Efficiency: The model maintains a small trainable-parameter footprint while targeting speech models that can run locally.Source code is made available to enable further work.
- Architecture: The architecture processes raw WAV inputs into mel-scale spectrograms, extracts short- and long-term dependencies, and uses attention to pinpoint useful regions.The attended representation is then fed to dense layers.
- Performance: The attention RNN achieves state-of-the-art performance across 20-command, 12-command, 35-word, and left-right recognition tasks.The evaluation uses Google Speech Commands datasets V1 and V2.
- Interpretability: Attention indicates which audio regions are important for classification and aligns with the observation that vowel transitions help recognize words.The reported analysis also includes confusion matrices identifying tree-three and no-down as difficult word pairs.
- Limitations and future work: The study does not explore augmenting audio with noise from other datasets, and future work includes multiple datasets, pretrained models, and more complex command sequences.The authors also mention sequence-to-sequence models, multiple attention layers, language identification, and speech-pathology detection as future directions.