Source-linked AI summary
Broadcasted Residual Learning for Efficient Keyword Spotting
Byeonggeun Kim, Simyung Chang, Jinkyu Lee, Dooyong Sung
TL;DR
Keyword spotting requires low error and efficient operation on resource-limited devices, but 1D and 2D convolution each impose trade-offs. The paper introduces broadcasted residual learning and BC-ResNet to combine their advantages with lower computational cost. BC-ResNets achieve state-of-the-art accuracy while using fewer parameters and computations than prior approaches.
Problem
Keyword spotting must operate accurately and efficiently on resource-limited devices, while existing 1D and 2D convolutions have complementary efficiency and frequency-representation limitations.
Method
Broadcasted residual learning averages 2D features into temporal features and broadcasts temporal residual information back to the frequency-temporal dimension in BC-ResNet.
Results
BC-ResNets achieve state-of-the-art keyword-spotting accuracy while reducing model parameters and computation relative to previous approaches.
Takeaways & Limitations
BC-ResNet provides a family of width-scalable networks for applying broadcasted residual learning across target devices with different resource budgets.
Abstract
from arXiv · showhide
Keyword spotting is an important research field because it plays a key role in device wake-up and user interaction on smart devices. However, it is challenging to minimize errors while operating efficiently in devices with limited resources such as mobile phones. We present a broadcasted residual learning method to achieve high accuracy with small model size and computational load. Our method configures most of the residual functions as 1D temporal convolution while still allows 2D convolution together using a broadcasted-residual connection that expands temporal output to frequency-temporal dimension. This residual mapping enables the network to effectively represent useful audio features with much less computation than conventional convolutional neural networks. We also propose a novel network architecture, Broadcasting-residual network (BC-ResNet), based on broadcasted residual learning and describe how to scale up the model according to the target device's resources. BC-ResNets achieve state-of-the-art 98.0% and 98.7% top-1 accuracy on Google speech command datasets v1 and v2, respectively, and consistently outperform previous approaches, using fewer computations and parameters. Code is available at https://github.com/Qualcomm-AI-research/bcresnet.
1. Introduction
The paper addresses the efficiency–representation trade-off in keyword spotting by introducing broadcasted residual learning and the BC-ResNet family. The approach combines 1D temporal and 2D frequency processing, achieving high accuracy with reduced computation and model size.
- Motivation: Existing KWS systems use either efficient 1D temporal convolutions or more computationally demanding 2D frequency×temporal convolutions, each with distinct limitations.1D methods lack frequency-direction translation equivariance, while 2D methods require more computation.
- Method: Broadcasted residual learning repeatedly averages 2D features into temporal features and broadcasts them back to the frequency-temporal dimension.This allows residual information from 1D temporal processing to be added to 2D feature maps.
- Results: 98.0% test accuracy is achieved by BC-ResNet-8 on Google speech command dataset v1 after scaling BC-ResNet-1 by a channel-width factor of 8.The figure describes this configuration as state of the art and reports fewer than 10k parameters for BC-ResNet-1.
- Results: 96.6% and 96.9% top-1 accuracy are achieved on Google speech command datasets v1 and v2, respectively, with less than 10k parameters.These results are reported for the smallest BC-ResNet configuration.
- Method: BC-ResNet uses broadcasted residual learning to combine advantages of 1D and 2D convolution while minimizing additional computation.The method enables frequency-direction convolutional processing without adopting fully 2D computation throughout the network.
- Scaling: The BC-ResNet family is formed by increasing model width, supporting scaling according to target-device resources.The paper reports state-of-the-art accuracy while reducing model parameters and computation.
2. Proposed Method
Broadcasted residual learning combines frequency-wise 2D processing with temporal operations by averaging and then broadcasting residual features. BC-ResNet applies this design in efficient blocks and scales model width for different resource targets.
- Model Scaling: BC-ResNets form a model family by increasing channel width, supporting scaling according to computational or device-resource constraints.The paper contrasts this with scaling depth and width together, which can make fitting a target computational budget difficult.
- Broadcasted Residual Learning: Broadcasted residual learning decomposes residual processing into temporal and 2D operations, averaging 2D features by frequency before broadcasting temporal residuals back to 2D.This combines 1D temporal and 2D frequency-temporal representations while minimizing additional computation.
- BC-ResBlock: BC-ResBlock uses frequency-depthwise convolution with SubSpectralNorm, frequency averaging, temporal depthwise-separable convolution, and a broadcasted residual connection.The 2D branch uses a 3x1 convolution and the temporal branch uses a 1x3 convolution followed by normalization, activation, pointwise convolution, and dropout.
- Network Architecture: Transition blocks change channel width with an additional pointwise convolution and omit the identity shortcut.The architecture keeps frequency and temporal dimensions through zero-padding in depthwise convolutions, while dilation is used in temporal convolutions and stride in frequency.
- Computational Efficiency: Temporal depthwise and pointwise convolutions operate on frequency-averaged features, reducing their computing load by a factor of h versus 2D depthwise-separable convolutions.This design retains 2D features while reducing the cost of the most computationally intensive pointwise operations in tiny networks.
- Network Architecture: BC-ResNet-1 has fewer than 10k parameters and contains a front 5x5 convolution followed by 12 BC-ResBlocks arranged across four stages.The stages use 2, 2, 4, and 4 blocks, while later convolutions reduce the computations of the pointwise convolution behind them.
3. Related Works
CNN-based keyword-spotting research has combined residual learning and efficient convolutional designs, including depthwise separable convolutions and temporal convolutional approaches.
- Efficient CNN-based KWS: Handcrafted efficient CNNs such as MobileNets and ShuffleNet use depthwise separable convolution, inverted bottleneck blocks, and channel shuffle.CNN-based keyword-spotting methods build on these designs alongside residual learning and depthwise separable convolutions.
- Efficient CNN-based KWS: Prior keyword-spotting models include residual CNNs, DS-ResNet with depthwise separable convolutions, and TC-ResNet with temporal convolutions.
4. Experiments
Experiments evaluate BC-ResNets on Google Speech Commands v1 and v2, comparing accuracy, parameters, computation, and component ablations against 1D and 2D baselines. BC-ResNets achieve strong accuracy-efficiency trade-offs, with broadcasted residual learning and SSN contributing to performance.
- Experimental Setup: BC-ResNets are evaluated on Google Speech Commands v1 and v2 using Top-1 test accuracy, with results averaged across random seeds.Version 1 contains 64,727 utterances from 1,881 speakers, while version 2 contains 105,829 utterances from 2,618 speakers.
- Impact of Broadcasted Residual Learning: ResNet-1D has about three times more parameters yet is more than 1% less accurate, while ResNet-2D is about 2% less accurate and requires more computation.ResNet-2D has about 16% fewer parameters than BC-ResNet but performs more 2D operations.
- Ablation Study of BC-ResBlock: Removing SSN causes a more than 0.4% accuracy drop, while removing both SSN and the 2D residual increases error more than removing either component alone.The ablation identifies SSN and the auxiliary 2D residual as essential components of broadcasted residual learning.
- Comparison with Baseline: BC-ResNets are consistently efficient in accuracy per parameter and achieve higher accuracy than MatchboxNets and MHAtt-RNN with smaller computation on dataset v2.Compared with MatchboxNet, BC-ResNets require x2.6 fewer parameters while achieving higher accuracy.
- Comparison with Baseline: BC-ResNet-1 matches TC-ResNet14-1.5 and TeNet12 performance with x10.9 fewer parameters while using a similar number of multiplies.The comparison concerns 1D convolution-based approaches.
5. Conclusions
The conclusion presents broadcasted residual learning as a way to combine 1D efficiency with 2D feature processing. BC-ResNets use this approach to surpass state-of-the-art results on Google Speech Commands v1 and v2.
- 5. Conclusions: Broadcasted residual learning repeatedly averages 2D features to 1D and expands 1D features back to 2D.This combines temporal and frequency-temporal representations within the residual architecture.
- 5. Conclusions: BC-ResNets are designed by applying broadcasted residual learning and scaling network width.The resulting family of models is intended to support different resource targets.
- 5. Conclusions: BC-ResNets surpass state-of-the-art performance on Google Speech Commands datasets v1 and v2.