Source-linked AI summary
Streaming keyword spotting on mobile devices
Oleg Rybakov, Natasha Kononenko, Niranjan Subrahmanya, Mirko Visontai, Stella Laurenzo
TL;DR
Converting non-streaming KWS models for incremental inference can require manual rewriting, and using the non-streaming representation for streaming can recompute convolutions. The paper provides a Keras-based conversion library and benchmarks streaming and non-streaming models on mobile phones; it reports a 10% relative classification-error reduction over state-of-the-art models on datasets V2.
Problem
Converting non-streaming KWS models to streaming mode may require manual rewriting, while using non-streaming models for incremental inference recomputes convolutions on overlapping input windows.
Source: §1 · p. 1, §1.1 · p. 1Method
Results
10% relative classification-error reduction compared with state-of-the-art models on datasets V2; Pixel 4 benchmarks also show different latency–accuracy tradeoffs between streaming and non-streaming models.
Source: §5 · p. 4Takeaways & Limitations
The library supports automatic streaming conversion and mobile benchmarking of KWS models, with code and experiments open-sourced.
Source: §2 · p. 2, §5 · p. 4The current Stream wrapper does not support striding or pooling greater than 1 in the time dimension.
Source: §2 · p. 2
Abstract
from arXiv · showhide
In this work we explore the latency and accuracy of keyword spotting (KWS) models in streaming and non-streaming modes on mobile phones. NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting. We address this by designing a Tensorflow/Keras based library which allows automatic conversion of non-streaming models to streaming ones with minimum effort. With this library we benchmark multiple KWS models in both streaming and non-streaming modes on mobile phones and demonstrate different tradeoffs between latency and accuracy. We also explore novel KWS models with multi-head attention which reduce the classification error over the state-of-art by 10% on Google speech commands data sets V2. The streaming library with all experiments is open-sourced.
1. Introduction
The paper focuses on model optimization, inference conversion, and mobile execution, with streaming KWS as a case where training and inference representations may differ. It proposes an automatic conversion workflow and benchmarks KWS models on mobile devices.
1. Introduction
- The work focuses on model optimization, conversion to inference mode, and execution on mobile devices, where sequence-classification models such as KWS may use different training and inference representations.It identifies quantization and transformation to a streaming model as optimization approaches.Source: §1 · p. 1
1.1. Model streaming example
- Streaming KWS processes audio incrementally; buffering prior convolution results avoids recomputing them, and the paper aims to let developers train a model once and convert it automatically.The illustrated implementations use either internal state or state passed in and returned from the model.Source: §1.1 · pp. 1–2
1.2. About speech frontend and modeling pipeline
- The pipeline embeds an MFCC-based speech feature extractor in Keras, converts the trained model to streaming Keras and then TFLite, and benchmarks accuracy and latency on Pixel 4.The paper reports implementing popular KWS models, improving classification error by 10% on datasets V2, and open-sourcing code, pretrained models, and results.Source: §1.2 · p. 2
2. Model streaming
The Stream wrapper automatically converts Keras models to streaming inference without requiring developers to rewrite them manually. Conversion inserts buffering for streamed layers or calls streaming-aware RNN functions, with support for internal or external state.
2. Model streaming
- The Stream wrapper automatically converts Keras models to streaming inference with internal or external state, avoiding manual model rewriting.It is intended for streaming inference without affecting training or default non-streaming inference.Source: §2 · p. 2
- Conversion sets the input feature size to one frame, then inserts ring buffers for streamed layers or calls streaming functions in streaming-aware layers such as RNNs.Streaming-aware RNN layers behave as standard RNNs during training and use the RNN cell during streaming inference.Source: §2 · p. 2
- The current Stream wrapper does not support striding or pooling greater than 1 in the time dimension.The authors state that these capabilities could be implemented in the future.Source: §2 · p. 2
3. Model architectures
The paper evaluates popular KWS neural-network architectures for streaming and non-streaming inference on mobile phones. Its DNN variant stacks per-feature outputs over 49 frames and uses pooling before further fully connected layers.
3. Model architectures
- The evaluated KWS pipeline frames one second of audio into overlapping 40 ms frames with 20 ms overlap, extracts MFCC features, and classifies them with neural networks.The paper implements popular architectures for streaming and non-streaming inference benchmarking on mobile phones.Source: §3 · p. 2
3.1. Deep Neural Network (DNN)
- The DNN applies ReLU fully connected layers to each speech feature, stacks outputs over 49 frames, and uses pooling before further fully connected layers.The authors report higher accuracy than the standard DNN in [5] with a similar number of parameters.Source: §3.1 · p. 2
3.2. Convolutional Neural Network (CNN)
The paper benchmarks convolutional KWS models, including a strided variant and a non-strided variant that can be automatically converted to streaming mode.
3.2. Convolutional Neural Network (CNN)
- The CNN architecture consists of 2D convolutions with ReLU non-linearities followed by fully connected layers.The paper benchmarks variants with and without striding.Source: §3.2 · p. 2
- The CNN+strd variant uses striding equal to 2.It is one of the CNN variations benchmarked in the paper.Source: §3.2 · p. 2
- The non-strided CNN can be automatically converted to streaming mode.The paper labels this variant CNN in Table 2.Source: §3.2 · p. 2
3.3. Recurrent Neural Networks: LSTM, GRU
The section compares sequential LSTM and GRU models for streaming keyword spotting, then describes CRNN as combining convolutional short-term features with recurrent longer-range context.
3.3. Recurrent Neural Networks: LSTM, GRU
- LSTM and GRU models process speech features sequentially while updating internal states, making this approach well suited to streaming inference.Both are compared with baseline [5] on Table 1.Source: §3.3 · p. 3
3.4. Convolutional Recurrent Neural Network (CRNN)
- CRNN combines 2D convolutions that capture short-term dependencies with a GRU that uses longer-range context.The CRNN library model is compared with baseline [5] on Table 1.Source: §3.4 · p. 3
3.5. Depthwise Separable Convolutional Neural Network (DSCNN)
The section describes DSCNN variants for keyword spotting, including a no-striding version that can be automatically converted to streaming mode.
3.5. Depthwise Separable Convolutional Neural Network (DSCNN)
- DSCNN processes speech features through 2D convolutional and depthwise layers, batch normalization with average pooling, and fully connected layers.The architecture follows [5].Source: §3.5 · p. 3
- The benchmark includes a striding-2 DSCNN, similar to the model in [5] and labeled DSCNN+strd in Tables 1 and 2.Source: §3.5 · p. 3
- The no-striding DSCNN is labeled DSCNN in Table 2 and can be automatically converted to streaming mode.Source: §3.5 · p. 3
3.6. Multihead attention RNN (MHAtt-RNN)
The section extends an attention-based bidirectional RNN for keyword spotting with four attention heads and GRU units, reporting a 10% classification-error reduction over the state of the art. Its bidirectional RNN architecture requires the full sequence and cannot be converted to streaming mode.
3.6. Multihead attention RNN (MHAtt-RNN)
- 10% lower classification error than the state of the art is reported for MHAtt-RNN on Table 1.The model uses four attention heads and replaces Att-RNN’s LSTM with a GRU.Source: §3.6 · p. 3
- MHAtt-RNN extends Att-RNN’s attention approach with four heads and replaces its LSTM with a GRU.Att-RNN derives a query from the center of the bidirectional LSTM output and uses attention-weighted output for classification [27].Source: §3.6 · p. 3
- Because both Att-RNN and MHAtt-RNN use bidirectional RNNs, they require the full sequence before classification and cannot be converted to streaming mode.Source: §3.6 · p. 3
3.7. Singular value decomposition filter (SVDF)
The section presents a simplified SVDF-based keyword-spotting model that does not require aligned audio annotations for training.
3.7. Singular value decomposition filter (SVDF)
- The simplified implementation of does not require aligned audio annotations for training.Source: §3.7 · p. 3
- The model comprises several SVDF and bottleneck layers, followed by a softmax layer.Source: §3.7 · p. 3
- Each SVDF block is a sequence of one-dimensional convolution and one-dimensional depthwise convolution layers.Source: §3.7 · p. 3
3.8. Temporal Convolution ResNet (TC-ResNet)
The section describes TC-ResNet14, a one-dimensional-convolution residual network, and an accuracy-improved variant with more parameters and SpecAugment. Table 1 provides the V1 and V2 accuracy comparison context.
3.8. Temporal Convolution ResNet (TC-ResNet)
- TC-ResNet14 is composed of residual blocks using one-dimensional convolution.Source: §3.8 · p. 3
- Table 1 compares baseline and paper-implemented model accuracy on datasets V1 and V2.Source: Table 1 · p. 3
- The improved TC-ResNet increases the parameter count from 305K to 365K and uses SpecAugment.Source: §3.8 · p. 3
4. Experimental results
The experiments compare KWS accuracy and mobile inference latency across datasets V1 and V2 and streaming and non-streaming models. SVDF, CRNN, and GRU are the most effective and accurate streaming models, while MHAtt-RNN is the most accurate non-streaming model.
4.1. Datasets and accuracy metrics
- The experiments use Google Speech Commands V1 and V2, twelve labels, one-second samples at 16 kHz, and an 80:10:10 train-validation-test split.Accuracy is measured by comparing test-set predictions with expected labels.Source: §4.1 · p. 3
4.2. Comparison with baseline
- The benchmark implements popular KWS models, converts trained models to TFLite, and reports their accuracy alongside paper-reported baselines.SVDF is also benchmarked on both datasets and described as having good streaming properties.Source: §4.2 · pp. 3–4
- MHAtt-RNN reduces classification error by 10% on V2 compared with Embed+head, but has twice as many parameters.Source: §4.2 · p. 4
4.3. Streaming and non-streaming latency with accuracy
- The streaming evaluation does not reset RNN states between test sequences; stateful retraining addresses this, while strided CNN, DSCNN, and MHAtt-RNN are not streamable with the library.Source: §4.3 · p. 4
- Non-streaming-to-streaming latency ratios are around 10x for convolutional models and 20x for RNN models.The authors attribute the difference to sequential frame-by-frame processing in non-streaming RNNs versus batched processing for convolutions.Source: §4.3 · p. 4
- SVDF, CRNN, and GRU are the most effective and accurate streaming models, while MHAtt-RNN is the most accurate non-streaming model.Source: Table 2 · p. 4, §4.3 · p. 4
5. Conclusion
The paper presents a Keras/TFLite library for streaming conversion and mobile KWS benchmarking. Its multi-head attention model reduces V2 classification error by 10% relative to state-of-the-art models.
5. Conclusion
- The library converts Keras models to streaming inference on mobile phones using Keras and TFLite.Source: §5 · p. 4
- The multi-head attention model reduces classification error by 10% relative on dataset V2 compared with state-of-the-art models.The improvement extends Att-RNN [27] with multi-head attention and SpecAugment [26].Source: §5 · p. 4
- The work benchmarks streaming and non-streaming latency and accuracy on a Pixel 4 and releases the code and experimental results.Source: §5 · p. 4