Source-linked AI summary
Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware
Peter Blouw, Xuan Choo, Eric Hunsberger, Chris Eliasmith
TL;DR
The paper asks whether neuromorphic hardware can improve the efficiency of real-time keyword spotting compared with conventional and low-power devices. It benchmarks equivalent keyword spotters across CPU, GPU, Jetson TX1, Movidius NCS, and Loihi, finding that Loihi has lower energy cost per inference and a stronger advantage as networks grow.
Problem
The paper evaluates whether neuromorphic hardware can provide energy-efficient, low-latency keyword spotting for mobile and IoT applications.
Method
The study benchmarks an inference-only keyword spotter across a CPU, GPU, Jetson TX1, Movidius NCS, and Loihi using non-spiking and architecturally identical spiking models.
Results
Loihi is more power efficient per inference than the alternative devices while achieving 93.8% test accuracy, compared with 92.7% for the non-spiking benchmarks.
Takeaways & Limitations
Loihi’s efficiency advantage over the Movidius NCS increases for larger networks, while the reported gains come with no cost to model accuracy.
Takeaways & Limitations
Power estimates for Jetson and Movidius are less precise because automated logging is unavailable, although their inference counts remain exact.
Abstract
from arXiv · showhide
Using Intel's Loihi neuromorphic research chip and ABR's Nengo Deep Learning toolkit, we analyze the inference speed, dynamic power consumption, and energy cost per inference of a two-layer neural network keyword spotter trained to recognize a single phrase. We perform comparative analyses of this keyword spotter running on more conventional hardware devices including a CPU, a GPU, Nvidia's Jetson TX1, and the Movidius Neural Compute Stick. Our results indicate that for this inference application, Loihi outperforms all of these alternatives on an energy cost per inference basis while maintaining equivalent inference accuracy. Furthermore, an analysis of tradeoffs between network size, inference speed, and energy cost indicates that Loihi's comparative advantage over other low-power computing devices improves for larger networks.
1. Introduction
The paper benchmarks Loihi for low-latency, energy-sensitive keyword spotting against conventional and low-power hardware. Loihi is reported as more energy efficient per inference while maintaining comparable accuracy.
- Keyword spotting is a real-time audio task suited to neuromorphic benchmarking because it requires low latency and energy efficiency for mobile and IoT applications.
- The study compares a keyword spotter on a Xeon CPU, Quadro GPU, Jetson TX1, Movidius NCS, and Loihi.The experiments use an inference-only TensorFlow model for the non-spiking benchmarks.
- 92.7% test classification accuracy is achieved across all non-spiking hardware devices on a small spoken-utterance dataset.
- 93.8% test classification accuracy is achieved both in Nengo simulation and on Loihi by an architecturally identical spiking model.
- Loihi is more power efficient on a cost-per-inference basis than the alternative devices for this keyword spotting application.The comparison uses power consumption and inference speed to compute energy cost per inference.
2. Methodology
The study evaluates a two-layer keyword spotter across conventional and neuromorphic hardware using matched computation, controlled power measurements, and network-scaling experiments. The methodology also documents dataset construction, batching, and measurement limitations.
- Network and data: The keyword spotter converts audio into MFCC frames with a 10ms stride, then predicts character probabilities through two hidden layers.Each frame has 390 dimensions, the hidden layers have 256 dimensions each, and the output has 29 character probabilities.
- Cross-device comparison: All devices use the same computational graph and parameters, while Loihi uses the same structure in spiking mode with Nengo.Audio features enter the input layer and character-probability distributions are read from the output layer, keeping computation matched across comparisons.
- Scaling experiments: Batch size and network size are varied to examine efficiency under offline batching and increasing computational load with fixed I/O.The default batch size is one, and network-scaling experiments use additional hidden layers or device-specific scaling architectures rather than the trained keyword-spotter parameters.
- Speech data and training: Approximately 2000 utterances from 96 speakers were collected and filtered from target and distractor phrases for model training.The dataset contains roughly three positive examples for every negative example.
- Power measurements: Dynamic joules per inference are estimated by subtracting idle power from runtime power and dividing by the inference rate.Power is logged during 15-minute runs; Loihi uses a separate input-spike script, while Jetson and Movidius lack automated power logging.
- Measurement limitations: Comparisons are constrained by non-identical software stacks and incomplete direct characterization of I/O energy on non-Loihi devices.The authors use conservative idle-power baselines and constant-I/O network scaling to assess whether software or I/O differences could affect efficiency comparisons.
3. Results
The benchmarks compare inference power efficiency across conventional, low-power, and neuromorphic hardware, then examine how speed and energy cost change with network size. Loihi maintains real-time speed as networks grow and gains a larger efficiency advantage over Movidius.
- Cross-device benchmarking: Table 1 combines mean power measurements and inference speed to compute each device’s energy cost per inference.Dynamic power is obtained by subtracting idle power from runtime power, then dividing by inferences per second.
- Cross-device benchmarking: Increasing batchsize improves CPU and GPU energy efficiency, whereas Movidius and Loihi remain represented at batch size 1.Loihi supports only batch size 1 because its deployment targets real-time, interactive applications.
- Scaling analyses: As network computation increases, inference speed decreases and energy cost per inference increases monotonically across devices.Dynamic power generally also increases, except for variability in the CPU results.
- Scaling analyses: Loihi’s dynamic power and energy cost rise more slowly with network size than Movidius’s, indicating a sizeable scaling advantage.The authors associate this pattern with Loihi’s architectural parallelism, temporal sparsity, and data locality.
- Practical implications: 100 inferences per second is required for real-time audio with a 10ms frame stride, and Loihi maintains this speed as network size increases while Movidius does not.The comparison is relevant because more sophisticated speech recognition beyond keyword spotting is expected to require more complicated networks.
- Scaling analyses: The scaling topology differs from the original keyword spotter in core allocation, making its N=0 Loihi inference slower and more energy-intensive despite identical neuron and weight counts.The scaling network distributes input neurons across eight Loihi cores instead of two to accommodate connectivity demands at larger N.
4. Conclusion
The study finds that Loihi improves energy efficiency per inference over tested low-power alternatives, with especially strong benefits for larger networks. These gains do not reduce model accuracy.
- Loihi improves energy efficiency on a cost-per-inference basis over the existing low-power computing devices tested.
- For N=10, Loihi offers a roughly 10x power-efficiency improvement over Movidius even under a conservative USB-I/O assumption.The comparison is attributed to Loihi’s modest dynamic-power increase and nearly constant inference speed.
- Loihi’s large-network energy-efficiency gains over Movidius come with no cost to model accuracy.