PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Pierre-Emmanuel Novac, Ghouthi Boukli Hacene, Alain Pegatoquet, Benoît Miramond, Vincent Gripon

arXiv:2105.13331v2cs.LGeess.SP

TL;DR

Deploying DNNs on microcontrollers is limited by memory, computation, power, and connectivity constraints. The paper presents MicroAI, an adaptable framework for training, quantization, and deployment, and evaluates it against existing inference engines. The results identify 16-bit fixed-point inference as a favorable alternative to 32-bit floating-point inference, while 8-bit quantization is less stable and requires quantization-aware training.

  • Problem

    Microcontroller DNN deployment requires improved memory, power, real-time, and edge-computing efficiency under severe hardware constraints.

  • Method

    The paper develops MicroAI, an end-to-end framework supporting training, quantization, and deployment, and compares it with TensorFlow Lite for Microcontrollers and STM32Cube.AI.

  • Results

    16-bit fixed-point inference reduces power consumption and memory footprint without an accuracy drop across the tested cases, while 8-bit quantization requires quantization-aware training and offers no substantial improvement over 16-bit quantization.

  • Takeaways & Limitations

    Fixed-point 16-bit post-training quantization is presented as a viable, preferable alternative to 32-bit floating-point inference for the evaluated microcontroller deployments.

  • Takeaways & Limitations

    TensorFlow Lite for Microcontrollers has a complex architecture and runtime-interpreted microcode, causing additional overhead and larger memory usage.

Abstract

from arXiv · show

Embedding Artificial Intelligence onto low-power devices is a challenging task that has been partly overcome with recent advances in machine learning and hardware design. Presently, deep neural networks can be deployed on embedded targets to perform different tasks such as speech recognition,object detection or Human Activity Recognition. However, there is still room for optimization of deep neural networks onto embedded devices. These optimizations mainly address power consumption,memory and real-time constraints, but also an easier deployment at the edge. Moreover, there is still a need for a better understanding of what can be achieved for different use cases. This work focuses on quantization and deployment of deep neural networks onto low-power 32-bit microcontrollers. The quantization methods, relevant in the context of an embedded execution onto a microcontroller, are first outlined. Then, a new framework for end-to-end deep neural networks training, quantization and deployment is presented. This framework, called MicroAI, is designed as an alternative to existing inference engines (TensorFlow Lite for Microcontrollers and STM32CubeAI). Our framework can indeed be easily adjusted and/or extended for specific use cases. Execution using single precision 32-bit floating-point as well as fixed-point on 8- and 16-bit integers are supported. The proposed quantization method is evaluated with three different datasets (UCI-HAR, Spoken MNIST and GTSRB). Finally, a comparison study between MicroAI and both existing embedded inference engines is provided in terms of memory and power efficiency. On-device evaluation is done using ARM Cortex-M4F-based microcontrollers (Ambiq Apollo3 and STM32L452RE).

1. Introduction

Deploying DNNs on microcontrollers is constrained by memory, computation, power, and connectivity requirements. The paper introduces MicroAI to support adaptable end-to-end training, quantization, and deployment while evaluating embedded efficiency.

  • Motivation: Cloud inference requires connectivity and data transmission, whereas edge execution can avoid transmission, privacy concerns, and non-deterministic latency.Running DNNs directly on resource-constrained devices remains difficult.
  • Motivation: Microcontrollers have limited memory, low processing speed, and constrained power budgets, making DNN deployment challenging.They often have less than 1 MiB of memory and operate at substantially lower frequencies than GPUs.
  • Motivation: DNN deployment requires trading off memory footprint, response time, power consumption, and model accuracy.Previous HAR results showed real-time feasibility on a Cortex-M4F microcontroller but left room to improve memory and power usage.
  • Quantization: Quantization reduces model memory by using fewer bits and enables integer operations that generally require fewer computations on microcontrollers.It also supports fixed-point rather than floating-point encoding.
  • Contribution: MicroAI provides an adaptable end-to-end framework for training, quantization, and deployment on microcontrollers.Its objective is to balance accuracy, energy efficiency, and memory footprint.
  • Contribution: The study compares MicroAI with TensorFlow Lite for Microcontrollers and STM32Cube.AI across two microcontrollers and three datasets.The comparisons cover memory footprint, inference time, and power efficiency using 8-bit and 16-bit quantization.

2. The State of the Art in Embedded Execution of Quantized Neural Networks

Embedded DNN optimization uses quantization, efficient architectures, pruning, and factorization, each reducing resource requirements with different implementation and performance trade-offs. Prior work increasingly targets microcontrollers, but hardware constraints remain important.

  • Quantization: Quantization reduces numerical precision, including through low-bit methods such as PACT, SAWB, LSQ, and Bit-Pruning.These approaches target weights, activations, or both.
  • Low-cost architectures: Efficient architectures such as MobileNet and SqueezeNet reduce computational cost, but selecting a suitable architecture remains difficult because of extensive hyperparameter combinations.These networks are often tailored to computer vision tasks.
  • Pruning: Pruning can remove up to 90% of parameters, but unstructured removal can make the resulting network difficult to exploit efficiently.The passage identifies parameter reduction but also notes implementation consequences.
  • Factorization: Factorization compresses networks by identifying similar components and replacing them with shared representations or memory pointers.Additional learning steps can improve the representativeness of the factorization.
  • Trade-offs: Compression techniques usually reduce performance, although parameter memory can commonly be reduced by half or two-thirds while maintaining similar performance.The trade-off varies across methods and implementations.
  • Hardware constraints: Microcontroller deployment is constrained by limited SIMD support, 32-bit registers, and non-open ARM instruction sets.Lower-precision values may still be computed one by one using 32-bit registers.
  • Hardware constraints: Custom RISC-V extensions for sub-byte quantization are difficult to use commercially because suitable microcontrollers are scarce and custom manufacturing is costly.This limits the practicality of hardware-specific approaches for IoT products.
  • Embedded deployment: Prior microcontroller implementations include 8-bit deployment and 32-bit microcontrollers with hardware floating-point or fixed-point computation.Some earlier methods required substantial assembly-level implementation work.

3. Representation of Real Numbers

Floating-point representation provides broad dynamic range and variable precision, while fixed-point representation uses fixed allocations for integer and fractional parts. On microcontrollers, hardware support and resource constraints make this choice especially consequential.

  • Floating-point: Floating-point numbers encode sign, significand, and exponent to provide broad dynamic range and increasing precision near zero.IEEE754 commonly defines these fields.
  • Floating-point: Microcontrollers commonly support single-precision floating-point in hardware, while double precision is slower and half precision may be converted before computation.The passage also notes emerging but unavailable half-precision support for the discussed Cortex-M55 context.
  • Floating-point: Binary32 uses 1 sign bit, 8 exponent bits, and 23 significand bits, providing a dynamic range of roughly [−10^38, 10^38].Its smallest values near zero are approximately ±1.4 × 10^-45.
  • Fixed-point: Fixed-point representation allocates fixed lengths to integer and fractional parts, directly limiting dynamic range and resolution.Its resolution remains constant across the represented range.
  • Fixed-point: For signed Qm.n numbers, the dynamic range is [−2^(m−1), 2^(m−1) − 2^−n] and the resolution is 2^−n.Increasing integer bits expands range but reduces fractional precision.
  • Fixed-point: A signed Q16.16 value in a 32-bit register has dynamic range [−32,768, 32,767.9999847] and resolution 1.5259 × 10^-5.This range is smaller and less precise near zero than the equivalent floating-point representation.

4. Training and Quantization of Deep Neural Networks

The paper converts floating-point-trained networks to fixed-point representations using power-of-two scaling and uniform quantization, then considers post-training and quantization-aware training. The design balances precision, dynamic range, computational cost, and deployment efficiency.

  • Conversion: Training uses single-precision floating-point offline, so weights must be converted and quantized before deployment on the target microcontroller.The conversion maps floating-point values to integer representations using scale factors.
  • Conversion: Power-of-two scale factors allow conversion using shifts, which are preferable to slower division instructions on Cortex-M4 architectures.The scale factor must cover the value range without overflow, reducing precision for smaller values.
  • Uniform quantization: Although convolutional weights are often close to Gaussian and could benefit from non-uniform quantization, the framework uses uniform quantization to avoid transformation overhead.Non-uniform schemes may require nonlinear functions or lookup tables.
  • Scale factors: Layer-specific scale factors balance quantization error against the memory overhead of storing many scale factors.A network-wide factor is too coarse, while one factor per weight is too costly.
  • Conversion method: The fixed-point conversion determines integer-bit requirements, allocates remaining fractional bits, and computes scaled integer values and their scale factor.Signed-number handling and the sign bit affect the available precision.
  • Post-training quantization: Post-training quantization freezes a fully floating-point-trained network before quantizing parameters for inference without further adjustment.Reducing bit width increases quantization error and can lower accuracy.
  • Quantization-aware training: Quantization-aware training performs the forward pass with quantized values while retaining non-quantized values for backpropagation.Floating-point pretraining can initialize parameters before this training stage.
  • Quantization-aware training: MicroAI quantizes inputs, weights, biases, and layer outputs during training, reassessing scale factors during training and freezing them for inference.Convolutional and fully connected layers require quantization-aware training for weights, whereas max-pooling does not.

5. Deployment of the Quantized Neural Network

MicroAI provides an end-to-end, adaptable pipeline for training, quantizing, converting, deploying, and evaluating neural networks on microcontrollers. It is positioned against existing engines by combining portability, extensibility, 8- and 16-bit support, and lightweight generated inference code.

  • Deployment process: The deployment process exports weights, generates inference code, compiles it, and uploads the program and weights to microcontroller ROM.These stages follow network training and optional quantization.
  • Existing inference engines: TensorFlow Lite for Microcontrollers is portable and operation-rich, whereas STM32Cube.AI integrates tightly with STM32 tools and includes on-target testing metrics.TFLite Micro lacks application-code generation and built-in resource measurement, while STM32Cube.AI supports only STM32 hardware and omits 16-bit fixed-point support.
  • Code generation: Generated layer-specific C code avoids the runtime interpreter used by TensorFlow Lite for Microcontrollers and allows compiler optimizations.The conversion module quantizes and converts layer weights when fixed-point representation is enabled.
  • Motivation: MicroAI addresses limitations of existing tools by supporting nonsequential convolutional networks, 16-bit quantization, hardware flexibility, and easier modification.The framework targets gaps involving proprietary or complex implementations, limited hardware families, and missing 16-bit support.
  • MicroAI framework: MicroAI combines training code based on Keras or PyTorch with KerasCNN2C, which converts trained Keras models into portable C inference code.Both components are implemented in Python and can be configured as part of the framework flow.
  • General flow: MicroAI provides an automated experiment flow configured through a TOML file, including training, deployment, and on-target evaluation.The framework includes dataset importation modules and preprocessing support, while its allocator reuses output-buffer pools to reduce RAM usage.

6. Results

Across UCI-HAR, Spoken MNIST, and GTSRB, 16-bit quantization generally preserves float32 accuracy, while 8-bit quantization trades accuracy for lower memory and can perform differently across tasks. Deployment results show that platform and inference engine strongly affect memory overhead, latency, and energy consumption.

  • Quantization accuracy: 16-bit quantization matches the float32 baseline on UCI-HAR, Spoken MNIST, and GTSRB, whereas 8-bit quantization causes dataset-dependent accuracy drops.The reported maximum drops are 0.81% for UCI-HAR, 1.07% for Spoken MNIST, and 1.1% for GTSRB.
  • UCI-HAR memory trade-offs: For UCI-HAR, 8-bit quantization reaches 92.41% accuracy with 3958 parameter bytes at 16 filters, compared with 92.46% and 7916 bytes for 16-bit quantization.Beyond 24 filters, 16-bit quantization has the best accuracy-versus-memory ratio, while above 48 filters 8-bit quantization is worse than the float32 baseline.
  • Spoken MNIST: On Spoken MNIST, 16-bit quantization is best for memory footprint, and 8-bit quantization provides no accuracy-versus-memory advantage even for small networks.The 8-bit accuracy drop reaches 1.07%, then remains around 0.5%–0.6% at higher filter counts.
  • GTSRB: On GTSRB, 8-bit accuracy approaches the float32 baseline as capacity increases, with a 0.33% drop at 64 filters, and remains an interesting option for two-dimensional networks.The maximum reported 8-bit accuracy drop is 1.1%.
  • Deployment performance: STM32Cube.AI with 8-bit inference is fastest at 352 ms per input, compared with 592 ms for TFLite Micro and 1034–1003 ms for MicroAI.For 16-bit inference, MicroAI takes 1223 ms on Nucleo-L452RE-P and 1042 ms on SparkFun Edge; float32 inference is slower than 8- or 16-bit inference across tools and targets.
  • Energy and platforms: SparkFun Edge consumes approximately six times less power than Nucleo-L452RE-P and provides the best power efficiency across the evaluated situations.On SparkFun Edge, MicroAI uses 0.75 µWh with 8-bit and 0.78 µWh with 16-bit inference, while TFLite Micro uses 0.45 µWh with 8-bit inference.

7. Discussion

The discussion finds that fixed-point inference improves embedded efficiency, while accuracy and training stability vary across quantization choices and datasets. Platform and engine implementations materially affect inference-time performance.

  • Accuracy and quantization: High variance across datasets makes definitive accuracy conclusions difficult, although the results reveal experiment-specific performance trends.The authors compare accuracy against model size and identify variability rather than a uniform pattern.
  • Efficiency: Fixed-point 8-bit and 16-bit inference significantly decreases inference time and can reduce average power consumption.Shorter inference may permit lower operating frequency or longer sleep intervals between inferences.
  • Accuracy and quantization: 16-bit fixed-point inference preserves accuracy across test cases without quantization-aware training and reduces both power consumption and memory footprint versus 32-bit floating point.The authors therefore prefer 16-bit fixed-point inference to 32-bit floating-point inference.
  • Accuracy and quantization: 8-bit quantization offers no substantial improvement over 16-bit quantization and requires more difficult, variance-sensitive quantization-aware training.Conservative SGD parameters are preferred over Adam or RAdam to reduce variance, even at lower maximum accuracy.
  • Accuracy and quantization: 8-bit TensorFlow Lite post-training quantization outperformed the framework’s 8-bit quantization-aware training, likely because of finer-grained scaling and truncation optimizations.The authors report that 9-bit post-training quantization can outperform TensorFlow Lite’s 8-bit result.
  • Engine and platform comparisons: MicroAI supports 16-bit quantization, unlike TensorFlow Lite for Microcontrollers and STM32Cube.AI, providing a compromise among accuracy, inference time, and memory footprint.SIMD instructions that could further improve 8-bit performance were not yet implemented.
  • Engine and platform comparisons: STM32Cube.AI was fastest in floating-point and integer inference, while MicroAI was intermediate in floating point and behind both engines for 8-bit inference.MicroAI lacked SIMD optimizations for 8-bit inference during these experiments.
  • Engine and platform comparisons: The Ambiq Apollo3 platform was more energy efficient than the STM32L452RE, showing that hardware can outweigh software performance differences.The comparison identifies substantial effects from both software and hardware platforms on energy efficiency.

8. Conclusions

The paper concludes that MicroAI is a viable alternative for deploying quantized neural networks on microcontrollers. Its 16-bit fixed-point post-training quantization preserves accuracy while reducing resource demands, whereas 8-bit quantization offers further efficiency at added complexity and slight accuracy loss.

  • Conclusions: MicroAI is presented as a viable alternative to STM32Cube.AI and TensorFlow Lite for Microcontrollers for neural-network inference on embedded platforms.The conclusion bases this assessment on inference-time and energy-efficiency measurements across two embedded platforms.
  • Conclusions: MicroAI introduces 16-bit integer post-training quantization unavailable in the two compared frameworks.The method improves over 32-bit floating-point inference and remains competitive with 8-bit quantization-aware training.
  • Conclusions: 16-bit quantization reduces memory footprint by half while maintaining the same accuracy and reducing inference time relative to floating-point inference.The conclusion describes this as a compromise between accuracy and efficiency.
  • Conclusions: Further 8-bit efficiency gains come with a slight accuracy decrease and a more complex implementation.Ongoing work targets per-filter quantization, asymmetric ranges, non-power-of-two scale factors, and SIMD instructions.
  • Future work: Future work includes mixed-precision integer inference, FPGA deployment, automatic PyTorch deployment, and integration into smart glasses for human activity recognition.The smart-glasses application is framed in the context of elder care, with preliminary results already published.
  • Availability: The open-source MicroAI software framework is available online for use and further development.The supplementary-materials passage provides the project repository link.

Appendix A. Comparison of the Inference Times of a Microcontroller, a CPU and a GPU

Appendix A documents comparisons of inference time and platform power across a microcontroller, CPU, and GPU. The measurements use a single-input microcontroller setting and batched, repeated CPU/GPU inference to offset startup overhead.

  • Platform comparison: Table A1 compares STM32L452RE, Intel Core i7-8850H, and Nvidia Quadro P2000M platforms.GPU and CPU power figures are manufacturer TDP values rather than exact device measurements.
  • Inference-time comparison: Table A2 compares 32-bit floating-point inference time for one input on a microcontroller, CPU, and GPU.The neural network varies convolution filters from 16 to 80; CPU and GPU use batch size 512 with the dataset repeated 104 times.

Appendix B. Number of Integer ALU Operations for a Fixed-Point Residual Neural Network

Appendix B specifies the arithmetic and logic operation counts used to analyze fixed-point inference in a residual neural network. The table parameterizes counts by layer dimensions and convolution settings.

  • Operation-count formulation: Table A3 reports arithmetic and logic operations for the main layers of a fixed-point residual neural network.Its variables include filters f, input samples s, input channels c, kernel size k, neurons n, and residual-layer inputs i.
  • Operation-count formulation: The Conv1D operation-count formulation assumes no padding and a stride of 1.This condition defines the convolution setting used in the appendix’s operation analysis.

Appendix C. Comparison of TensorFlow Lite for Microcontrollers and MicroAI Quantizations

Figure A1 compares accuracy across filter counts for four quantization configurations, using a convolutional network whose filters vary from 32 to 48.

  • Figure A1 plots accuracy against the number of filters per convolution layer.The filter count varies from 32 to 48.
  • The comparison includes a float32 baseline, int8 TFLite post-training quantization, int8 MicroAI quantization-aware training, and int9 MicroAI post-training quantization.
  • The evaluated dataset and neural-network architecture are those described in Section 6 and Section 6.1.1.

Appendix D. MicroAI Commands to Run for Automatic Training and Deployment of Deep Neural Networks

MicroAI provides a command sequence for preprocessing data, training a model, preparing deployment code, and deploying and evaluating the resulting binaries.

  • Preprocessing: Data preprocessing serializes the source dataset into an intermediate dataset file.The documented command is `microai <config.toml> preprocess_data`.
  • Training: Training is started with the configured MicroAI training command.The documented command is `microai <config.toml> train`.
  • Deployment preparation: Deployment preparation generates and builds code for the targeted platform.The documented command is `microai <config.toml> prepare_deploy`.
  • Deployment and evaluation: After binaries are generated, they are deployed and evaluated on the target device.The documented command is `microai <config.toml> deploy_and_evaluate`.

Appendix E. Detailed Results of the Evaluation of Frameworks and Embedded Platforms Evaluation

Appendix E reports detailed framework comparisons for ROM footprint, inference time, and energy consumption as the number of filters varies.

  • ROM footprint: ROM footprint is reported against filter count for TFLite Micro, STM32Cube.AI, and MicroAI.
  • Inference time: Inference time for one input is reported against filter count for TFLite Micro, STM32Cube.AI, and MicroAI.
  • Energy consumption: Energy consumption for one input is reported against filter count for TFLite Micro, STM32Cube.AI, and MicroAI.
Loading 2105.13331v2…