CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs

Liangzhen Lai; Naveen Suda; Vikas Chandra

Source-linked AI summary

CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs

Liangzhen Lai, Naveen Suda, Vikas Chandra

arXiv:1801.06601v1Jan 19, 2018cs.NEcs.LGcs.MS

TL;DR

IoT neural-network inference is difficult on resource-constrained edge devices because conventional models require substantial computation and memory. CMSIS-NN provides optimized kernels for Arm Cortex-M CPUs, combining specialized layer implementations with memory-conscious techniques. For a CIFAR-10 convolutional neural network, the kernels achieved 4.6X better runtime/throughput and 4.9X better energy efficiency.

Problem
Neural-network execution is computationally and resource intensive for IoT edge devices, which motivates reducing reliance on cloud-based processing.
Source: §1 · p. 1
Method
CMSIS-NN develops optimized Arm Cortex-M kernels for neural-network layers, including partial im2col, depthwise separable convolution, split x-y pooling, and SWAR-based ReLU.
Source: §1 · p. 1, §4.3 · pp. 6–7, §4.4 · p. 7
Results
4.6X improvement in runtime/throughput and 4.9X improvement in energy efficiency was achieved for a CIFAR-10 convolutional neural network.
Source: §6 · p. 10
Takeaways & Limitations
CMSIS-NN kernels can be used directly by applications or as primitives by machine-learning frameworks to deploy trained models on Arm Cortex-M CPUs.
Source: §6 · p. 10
The fully-connected layer is evaluated with batch size one, and larger kernels may be limited by the Cortex-M register count.
Source: §4.2 · p. 4

Abstract

from arXiv · show

Deep Neural Networks are becoming increasingly popular in always-on IoT edge devices performing data analytics right at the source, reducing latency as well as energy consumption for data communication. This paper presents CMSIS-NN, efficient kernels developed to maximize the performance and minimize the memory footprint of neural network (NN) applications on Arm Cortex-M processors targeted for intelligent IoT edge devices. Neural network inference based on CMSIS-NN kernels achieves 4.6X improvement in runtime/throughput and 4.9X improvement in energy efficiency.

1 Introduction

IoT edge computing moves data processing toward resource-constrained devices to reduce communication latency and energy use. CMSIS-NN addresses this need with optimized Arm Cortex-M kernels, demonstrated on CIFAR-10.

IoT edge computing processes sensor data at the source, reducing latency and energy consumption for communication with the cloud.Source: §1 · p. 1
Deep neural networks offer near-human accuracy but have largely remained in cloud environments because of their computational and resource requirements.Source: §1 · p. 1
CMSIS-NN develops optimized software kernels for neural-network deployment on resource-constrained Arm Cortex-M CPUs.Source: §1 · p. 1

2 Overview

CMSIS-NN organizes neural-network support into application-facing layer kernels and supporting utilities. Multiple kernel variants provide either universal operation or additional optimization under parameter constraints.

CMSIS-NN separates its kernel code into NNFunctions for neural-network layers and NNSupportFunctions for utility functions.Source: §2 · p. 2
NNFunctions cover convolution, depthwise separable convolution, fully-connected, pooling, and activation layers for neural-network inference.Source: §2 · p. 2
Kernel APIs are kept simple so application code and machine-learning frameworks can use or retarget the functions.Source: §2 · p. 2
Basic kernel versions work universally, while optimized variants use transformed inputs or impose limitations on layer parameters.Source: §2 · p. 2

3 Fixed-Point Quantization

CMSIS-NN uses low-precision fixed-point representations to reduce inference costs on resource-constrained processors. Its power-of-two scaling supports integer-oriented operations and avoids floating-point de-quantization between layers.

Low-precision fixed-point representation avoids costly floating-point computation and reduces memory for weights and activations.Source: §3 · p. 2
CMSIS-NN represents q7_t, q15_t, and q31_t as int8, int16, and int32 data types, respectively.Source: §3 · p. 2
Power-of-two quantization represents values as A × 2^n, with scaling factors passed to kernels and implemented through bitwise shifts.Source: §3 · p. 2
This quantization avoids floating-point de-quantization between layers and enables simpler table-based activation functions.Source: §3 · p. 3

4 Software Kernels

CMSIS-NN implements optimized Cortex-M neural-network kernels using SIMD-friendly transformations, matrix multiplication, convolution, pooling, and activation strategies that target constrained resources.

CMSIS-NN targets Cortex-M systems with SIMD support, especially 16-bit MAC instructions useful for neural-network computation.Source: §4 · p. 3

Support Functions

q7_t inputs are expanded to q15_t using sign extension, while optional reordering is omitted when operand order already matches.This avoids unnecessary work inside computation-kernel inner loops.Source: §4.1 · p. 3

Matrix Multiplication

2 × 2 matrix-multiplication kernels compute four dot-product outputs per loop while reusing data and reducing load instructions.Accumulation uses q31_t values with the __SMLAD instruction.Source: §4.2 · p. 4, Figure 5 · p. 4
1 × 4 matrix-vector kernels reorder constant weights and match activation ordering to fit efficient computation within available registers.The approach interleaves weight rows and uses q7-to-q15 expansion without reordering.Source: §4.2 · p. 4, Figure 6 · p. 5, Figure 7 · p. 5

Convolution

Partial im2col expands only a limited number of columns, retaining matrix-multiplication benefits while minimizing the memory overhead caused by repeated pixels.Convolution combines im2col transformation with matrix multiplication.Source: §4.3 · p. 6

Pooling

In situ split x-y pooling is significantly faster than window-based pooling, achieving 4.5X speed-up without additional memory overhead.The operation overwrites the input while reusing intermediate x-direction results for y-direction pooling.Source: §4.4 · p. 7

5 Experimental Results

The CMSIS-NN kernels were evaluated on a quantized CIFAR-10 CNN and reduced runtime while fitting the network within the Cortex-M7 board’s memory constraints.

The evaluated CNN used 60,000 32x32 color images across 10 classes, with three convolution layers and one fully-connected layer.Source: §5 · p. 9
99.1 ms per image corresponds to 10.1 images per second for the quantized CNN on an Arm Cortex-M7 core.The pre-quantized network reached 80.3% accuracy, while the 8-bit quantized network reached 79.9%.Source: §5 · p. 9
∼133 KB maximum memory footprint enabled the network to fit, whereas ∼332 KB without partial im2col would not fit on the board.Convolutions use partial im2col followed by matrix multiplication to reduce memory use.Source: §5 · p. 9
2.6X to 5.4X runtime/throughput improvement was achieved over baseline functions across the CNN application’s layer types.The baseline used CMSIS-DSP arm_conv, Caffe-like pooling, and ReLU implementations.Source: §5 · p. 9
The energy-efficiency improvement was in line with the throughput improvement when comparing CMSIS-NN kernels with the baseline functions.Source: §5 · p. 9

6 Conclusion

CMSIS-NN targets faster, smaller neural-network inference on Arm Cortex-M CPUs. For a CIFAR-10 CNN, it achieved 4.6X runtime/throughput and 4.9X energy-efficiency improvements.

CMSIS-NN was developed to maximize neural-network performance and minimize memory footprint on Arm Cortex-M CPUs.Source: §6 · p. 10
4.6X runtime/throughput and 4.9X energy-efficiency improvements were achieved for a CIFAR-10 convolutional neural network.Source: §6 · p. 10
The kernels can be used directly by application code or as primitives within machine-learning frameworks deploying trained models.Source: §6 · p. 10