Source-linked AI summary
Keyword Transformer: A Self-Attention Model for Keyword Spotting
Axel Berg, Mark O'Connor, Miguel Tairum Cruz
TL;DR
Keyword spotting lacked evidence for fully self-attentional Transformer models, despite attention usually being combined with convolutional or recurrent encoders. The paper introduces KWT, a ViT-inspired fully self-attentional architecture for this task. KWT matches or exceeds state-of-the-art performance across tasks while retaining competitive real-world latency, without additional data or pre-training.
Problem
Fully-attentional Transformer models had not been investigated for keyword spotting, where attention had primarily been used with convolutional or recurrent architectures.
Method
The paper adapts ViT to keyword spotting by using audio-spectrogram patches in a fully self-attentional Keyword Transformer architecture.
Results
KWT matches or exceeds state-of-the-art performance across a range of keyword-spotting tasks and remains competitive in real-world latency.
Takeaways & Limitations
KWT provides a fully self-attentional drop-in replacement for existing keyword-spotting models without additional data or pre-training.
Takeaways & Limitations
The study is restricted to a non-streaming setting, while future improvements from pre-training, model compression, sparsity, and hardware codesign remain unexplored opportunities.
Abstract
from arXiv · showhide
The Transformer architecture has been successful across many domains, including natural language processing, computer vision and speech recognition. In keyword spotting, self-attention has primarily been used on top of convolutional or recurrent encoders. We investigate a range of ways to adapt the Transformer architecture to keyword spotting and introduce the Keyword Transformer (KWT), a fully self-attentional architecture that exceeds state-of-the-art performance across multiple tasks without any pre-training or additional data. Surprisingly, this simple architecture outperforms more complex models that mix convolutional, recurrent and attentive layers. KWT can be used as a drop-in replacement for these models, setting two new benchmark records on the Google Speech Commands dataset with 98.6% and 97.7% accuracy on the 12 and 35-command tasks respectively.
1. Introduction
The paper adapts Transformers directly to keyword spotting, introducing a fully self-attentional model inspired by ViT. KWT matches or outperforms existing models, supports drop-in replacement, and remains competitive for mobile use.
- Transformers had shown competitiveness across language, vision, speech, and other domains, motivating their direct application to keyword spotting.
- Attention for keyword spotting had primarily extended convolutional or recurrent architectures rather than forming a fully self-attentional model.
- Self-attention is more effective in the time domain than in the frequency domain for keyword spotting.
- KWT adapts ViT to keyword spotting as a fully self-attentional architecture and can serve as a drop-in replacement for existing models.
- KWT was evaluated across several Google Speech Commands tasks against convolutional, recurrent, and attention-based state-of-the-art models.
- KWT has competitive latency on a mobile phone, supporting its relevance to edge use cases.
2. Related Work
Keyword spotting operates in low-power, always-on settings and has traditionally relied on conventional neural architectures with audio preprocessing. This work addresses the limited investigation of fully self-attentional Transformer models for keyword spotting by adapting ViT-style spectrogram patching.
- Keyword spotting detects specific words from streaming audio, typically locally on low-power devices such as smart speakers and mobile phones.
- Existing keyword-spotting systems have used DNN, CNN, RNN, and Hybrid-Tree networks, typically with MFCC preprocessing.
- DS-CNN provided a strong accuracy, memory-footprint, and computational-resource tradeoff among small-scale architectures, while later work added synthesized data, temporal convolutions, or self-attention.
- ViT demonstrated that self-attention between image patches can learn high-level features, but its original approach required large-scale pre-training; later work improved data efficiency.
- Before this work, fully attentional Transformer models had not been investigated for keyword spotting; the proposed approach uses audio-spectrogram patches and a non-streaming setting.
3. The Keyword Transformer
The Keyword Transformer maps MFCC spectrogram features into a Transformer encoder using a class token, positional embeddings, and self-attention over time windows. Its configuration includes multi-head attention, PostNorm blocks, adjustable model size, and optional knowledge distillation.
- Model Architecture: MFCC spectrogram features are linearly projected into a higher-dimensional representation, concatenated with a learnable class embedding, and augmented with positional embeddings.The resulting sequence is fed into the Transformer encoder.
- Model Architecture: The encoder uses sequential multi-head attention and MLP blocks with PostNorm layer normalization, where normalization follows both sublayers.The multi-head output is formed by projecting the concatenated attention-head outputs.
- Model Architecture: Self-attention operates across time windows so the class embedding forms a global representation of the input spectrogram.This adapts the image-patch treatment of Vision Transformers to keyword spotting.
- Model Architecture: Model size is varied through the number of attention heads while fixing 12 Transformer encoder blocks and d/k = 64.The evaluated configurations use k = 1, 2, or 3 attention heads.
- Knowledge Distillation: Knowledge distillation adds a learned distillation token trained with hard teacher predictions, while inference averages class-token and distillation-token predictions.The teacher receives the same input augmentation as the student, and MHAtt-RNN serves as the teacher.
4. Experiments
Experiments evaluate KWT on Google Speech Commands, probe attention and normalization choices, and compare accuracy and mobile latency with existing models. Time-domain attention and PostNorm perform best in the reported ablations, while Transformer models remain competitive in latency.
- Keyword spotting evaluation: KWT is evaluated on Google Speech Commands V1 and V2, using one-second, 16-kHz keyword utterances across 12-label and 35-label tasks.V1 contains 65,000 snippets across 30 words, while V2 contains 105,000 snippets across 35 words.
- Keyword spotting evaluation: The best KWT models match or surpass prior state-of-the-art accuracy on both V2 tasks, with significant improvements reported for the 12-label and 35-label datasets.KWT does not outperform MHAtt-RNN on the smaller V1 dataset, while knowledge distillation improves accuracy in most scenarios.
- Ablation studies: Time-domain attention performs best among the tested spectrogram patch shapes.The default spectrogram has 98 time windows and 40 mel-scale frequencies; the study also tests frequency-domain and rectangular patches.
- Ablation studies: PostNorm improves keyword-spotting performance in these experiments, contrary to findings favoring PreNorm on other tasks.The authors encourage further study of normalization across Transformer application domains.
- Attention visualization: The learned attention mask focuses on important audio regions while suppressing background noise.Attention weights are propagated from each Transformer layer to the class token and averaged across heads before visualization.
- Attention visualization: Nearby learned position embeddings are highly similar, whereas distant embeddings are nearly orthogonal, with weaker structure near snippet boundaries.The paper hypothesizes that words typically occur near the middle or that boundary audio is less distinguishable.
- Latency measurements: Transformer-based models are competitive with existing state-of-the-art models in one-second mobile inference despite not being designed for latency.Latency is measured on a single thread of a OnePlus 6 using TensorFlow Lite.
5. Conclusion
The paper applies a standard Transformer directly to tokenized audio for keyword spotting and introduces KWT as a fully self-attentional model. KWT matches or exceeds state-of-the-art performance across tasks while retaining competitive real-world latency, motivating further exploration of Transformer techniques for keyword spotting.
- Conclusion: The paper directly applies Transformers to keyword spotting using a principled conversion of audio inputs into tokens.The approach uses a standard architecture rather than combining convolutional, recurrent, and attention components.
- Conclusion: KWT is a fully attentional model that matches or exceeds state-of-the-art performance across keyword-spotting tasks with competitive real-world latency.The model is presented as a practical alternative to existing keyword-spotting architectures.
- Conclusion: The results motivate future work applying Transformer advances such as pre-training, compression, and sparsity to keyword spotting.The paper identifies these techniques as potential ways to improve practical deployment impact.