Source-linked AI summary
CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs
Liangzhen Lai, Naveen Suda, Vikas Chandra
TL;DR
IoT neural-network inference is difficult on resource-constrained edge devices because conventional models require substantial computation and memory. CMSIS-NN provides optimized kernels for Arm Cortex-M CPUs, combining specialized layer implementations with memory-conscious techniques. For a CIFAR-10 convolutional neural network, the kernels achieved 4.6X better runtime/throughput and 4.9X better energy efficiency.
Problem
Neural-network execution is computationally and resource intensive for IoT edge devices, which motivates reducing reliance on cloud-based processing.
Method
CMSIS-NN develops optimized Arm Cortex-M kernels for neural-network layers, including partial im2col, depthwise separable convolution, split x-y pooling, and SWAR-based ReLU.
Results
4.6X improvement in runtime/throughput and 4.9X improvement in energy efficiency was achieved for a CIFAR-10 convolutional neural network.
Takeaways & Limitations
CMSIS-NN kernels can be used directly by applications or as primitives by machine-learning frameworks to deploy trained models on Arm Cortex-M CPUs.
Takeaways & Limitations
The fully-connected layer is evaluated with batch size one, and larger kernels may be limited by the Cortex-M register count.
Abstract
from arXiv · showhide
Deep Neural Networks are becoming increasingly popular in always-on IoT edge devices performing data analytics right at the source, reducing latency as well as energy consumption for data communication. This paper presents CMSIS-NN, efficient kernels developed to maximize the performance and minimize the memory footprint of neural network (NN) applications on Arm Cortex-M processors targeted for intelligent IoT edge devices. Neural network inference based on CMSIS-NN kernels achieves 4.6X improvement in runtime/throughput and 4.9X improvement in energy efficiency.
1 Introduction
IoT edge computing moves data processing toward resource-constrained devices to reduce communication latency and energy use. CMSIS-NN addresses this need with optimized Arm Cortex-M kernels, demonstrated on CIFAR-10.
- IoT edge computing processes sensor data at the source, reducing latency and energy consumption for communication with the cloud.
- Deep neural networks offer near-human accuracy but have largely remained in cloud environments because of their computational and resource requirements.
- CMSIS-NN develops optimized software kernels for neural-network deployment on resource-constrained Arm Cortex-M CPUs.
2 Overview
CMSIS-NN organizes neural-network support into application-facing layer kernels and supporting utilities. Multiple kernel variants provide either universal operation or additional optimization under parameter constraints.
- CMSIS-NN separates its kernel code into NNFunctions for neural-network layers and NNSupportFunctions for utility functions.
- NNFunctions cover convolution, depthwise separable convolution, fully-connected, pooling, and activation layers for neural-network inference.
- Kernel APIs are kept simple so application code and machine-learning frameworks can use or retarget the functions.
- Basic kernel versions work universally, while optimized variants use transformed inputs or impose limitations on layer parameters.
3 Fixed-Point Quantization
CMSIS-NN uses low-precision fixed-point representations to reduce inference costs on resource-constrained processors. Its power-of-two scaling supports integer-oriented operations and avoids floating-point de-quantization between layers.
- Low-precision fixed-point representation avoids costly floating-point computation and reduces memory for weights and activations.
- CMSIS-NN represents q7_t, q15_t, and q31_t as int8, int16, and int32 data types, respectively.
- Power-of-two quantization represents values as A × 2^n, with scaling factors passed to kernels and implemented through bitwise shifts.
- This quantization avoids floating-point de-quantization between layers and enables simpler table-based activation functions.
4 Software Kernels
CMSIS-NN implements optimized Cortex-M neural-network kernels using SIMD-friendly transformations, matrix multiplication, convolution, pooling, and activation strategies that target constrained resources.
- CMSIS-NN targets Cortex-M systems with SIMD support, especially 16-bit MAC instructions useful for neural-network computation.
- Support Functions: q7_t inputs are expanded to q15_t using sign extension, while optional reordering is omitted when operand order already matches.This avoids unnecessary work inside computation-kernel inner loops.
- Matrix Multiplication: 2 × 2 matrix-multiplication kernels compute four dot-product outputs per loop while reusing data and reducing load instructions.Accumulation uses q31_t values with the __SMLAD instruction.
- Matrix Multiplication: 1 × 4 matrix-vector kernels reorder constant weights and match activation ordering to fit efficient computation within available registers.The approach interleaves weight rows and uses q7-to-q15 expansion without reordering.
- Convolution: Partial im2col expands only a limited number of columns, retaining matrix-multiplication benefits while minimizing the memory overhead caused by repeated pixels.Convolution combines im2col transformation with matrix multiplication.
- Pooling: In situ split x-y pooling is significantly faster than window-based pooling, achieving 4.5X speed-up without additional memory overhead.The operation overwrites the input while reusing intermediate x-direction results for y-direction pooling.
5 Experimental Results
The CMSIS-NN kernels were evaluated on a quantized CIFAR-10 CNN and reduced runtime while fitting the network within the Cortex-M7 board’s memory constraints.
- The evaluated CNN used 60,000 32x32 color images across 10 classes, with three convolution layers and one fully-connected layer.
- 99.1 ms per image corresponds to 10.1 images per second for the quantized CNN on an Arm Cortex-M7 core.The pre-quantized network reached 80.3% accuracy, while the 8-bit quantized network reached 79.9%.
- ∼133 KB maximum memory footprint enabled the network to fit, whereas ∼332 KB without partial im2col would not fit on the board.Convolutions use partial im2col followed by matrix multiplication to reduce memory use.
- 2.6X to 5.4X runtime/throughput improvement was achieved over baseline functions across the CNN application’s layer types.The baseline used CMSIS-DSP arm_conv, Caffe-like pooling, and ReLU implementations.
- The energy-efficiency improvement was in line with the throughput improvement when comparing CMSIS-NN kernels with the baseline functions.
6 Conclusion
CMSIS-NN targets faster, smaller neural-network inference on Arm Cortex-M CPUs. For a CIFAR-10 CNN, it achieved 4.6X runtime/throughput and 4.9X energy-efficiency improvements.
- CMSIS-NN was developed to maximize neural-network performance and minimize memory footprint on Arm Cortex-M CPUs.
- 4.6X runtime/throughput and 4.9X energy-efficiency improvements were achieved for a CIFAR-10 convolutional neural network.
- The kernels can be used directly by application code or as primitives within machine-learning frameworks deploying trained models.