PaperlayerView exact revision on arXiv

Source-linked AI summary

Power-Performance Characterization of TinyML Systems

Yujie Zhang, Dhananjaya Wijerathne, Zhaoying Li, Tulika Mitra

arXiv:2608.21646v1cs.LGcs.AI

TL;DR

TinyML lacks quantitative, full-stack evidence about how models, software, operating systems, and hardware affect MCU inference. The paper systematically characterizes these layers and develops a full-stack latency model; it finds abstraction tradeoffs and relationships between model size and latency under controlled model structures, while identifying scope boundaries for those relationships.

  • Problem

    TinyML systems have little quantitative, full-stack analysis of how abstraction layers and hardware affect MCU inference performance and energy.

  • Method

    The paper evaluates TinyML applications across models, libraries, operating systems, and MCU architectures, then develops a full-stack model to estimate inference latency.

  • Results

    For three similarly structured models, MicroNet-KWS-L is 5.75 times the TFLite model size of MicroNet-KWS-S, while inference runtime is 5.86 times higher on MCU-S and 5.71 times higher on MCU-D.

  • Takeaways & Limitations

    Operating systems add considerable memory and energy overhead that can be reduced by removing redundant peripheral support, while the deep learning framework adds little overhead.

  • Takeaways & Limitations

    The proportional relationship between TFLite model size and inference latency holds for similarly structured models but is not obvious across applications with different structures.

Abstract

from arXiv · show

TinyML systems are enabling machine learning (ML) inference at the edge. However, there is little quantitative analysis of such systems. This paper presents a systematic performance and power characterization of diverse TinyML applications on microcontrollers (MCUs), spanning neural network models, software libraries, operating systems, and hardware architectures. We focus on the impact of the multiple layers of abstraction that provide higher programmability at the expense of performance and energy efficiency. We propose a model to estimate the costs of different abstraction layers and make recommendations for minimizing those costs. Our findings can help designers with Neural Architecture Search (NAS) and CNN inference optimization on edge devices.

I. INTRODUCTION

TinyML lacks systematic quantitative analysis across its full stack, even as model, software, and hardware choices jointly determine inference behavior. This paper analyzes those layers to identify bottlenecks and guide optimization and platform-specific NAS.

  • TinyML has little quantitative analysis compared with the extensive study of ML training systems and lacks a full-stack view of MCU inference.
  • Model, framework, compiler/library, and architecture choices interact to determine TinyML inference performance on resource-constrained devices.SIMD-based matrix multiplication can accelerate inference and improve energy efficiency through hardware-specific model optimization.
  • The study systematically analyzes power and performance across neural networks, software libraries, operating systems, and hardware architectures to identify bottlenecks.
  • Five representative TinyML applications are evaluated across MCU platforms for inference latency, power efficiency, and energy consumption.The study also examines abstraction-layer effects, GEMM implementations, hardware characteristics, and full-stack latency prediction.
  • The resulting analysis is intended to inform system optimization and platform-specific neural architecture search.

III. EXPERIMENTAL SETUP

The experimental setup evaluates representative TinyML workloads across diverse MCU platforms and implementation stacks. It combines quantized models, TFLM and CMSIS-NN, operating-system variants, and direct latency, power, energy, and instruction measurements.

  • Four MCU platforms with Cortex-M4F or Cortex-M7F processors are selected to cover differences in compute capability, memory availability, clock speed, and SRAM.The platforms are Arduino Nano 33 BLE Sense, SparkFun Edge Apollo3 Blue, Nucleo-144 STM32F746ZG, and Nucleo-64 STM32L476RG.
  • Five TinyML benchmarks vary in input types, model structures, and computation intensity, enabling fine-grained evaluation of model-structure effects.
  • Quantized int8 models run with TensorFlow Lite Micro and CMSIS-NN on the MCUs.TFLM reduces memory requirements by operating without OS support, while CMSIS-NN uses SIMD kernels for matrix multiplication.
  • Inference latency is measured with Mbed Timer API on Mbed OS and clock registers on bare metal.
  • Average execution power is recorded with SmartPower2, energy is computed by multiplying power by inference time, and J-Trace counts executed instructions.

IV. IMPACT OF ABSTRACTION LAYERS

The study quantifies how operating-system and deep-learning-framework abstractions trade programmability for TinyML inference efficiency. Optimized kernels provide major gains, while carefully reduced operating-system support limits memory and energy overhead.

  • Optimization libraries: CMSIS-NN optimization improves runtime and energy efficiency by around 10x over pure C bare-metal inference.SIMD instructions and matrix multiplication reduce memory access and enable data reuse.
  • Operating-system abstraction: Adding Mbed OS increases SRAM and Flash requirements by approximately 98% and 196%, respectively, while latency changes little.The comparison uses CMSIS-NN-optimized inference on MCU-M.
  • Operating-system abstraction: Mbed OS requires 65% additional energy compared with bare-metal implementation because bare metal avoids extra peripheral settings.The paper frames this as overhead from operating-system support and peripheral configuration.
  • Operating-system abstraction: Simplified Mbed OS improves memory and energy relative to full-fledged Mbed OS, balancing programmability and efficiency.The simplification removes unnecessary peripheral API support.
  • Framework abstraction: TFLM on Mbed OS increases inference time, energy, and memory requirements by around 3% compared with direct C implementation.The paper reports that the framework’s abstraction benefits outweigh this efficiency overhead.
  • Conclusion and recommendation: The results quantify a programmability-efficiency tradeoff: operating-system overhead is substantial, whereas the deep-learning framework adds little overhead and reduces designer effort.The authors recommend removing redundant peripheral support when feasible.

V. OPTIMIZATION LIBRARY LIMITATION

The section examines CMSIS-NN’s GEMM-based optimization and shows that fixed kernel sizes can underutilize MCU resources for layers with inappropriate shapes. Parameter choices that enable matrix-multiplication conversion can therefore improve inference performance.

  • GEMM-based optimization: Matrix multiplication is the most important and computationally intensive neural-network inference kernel, but GEMM conversion is not always suitable.CMSIS-NN and related libraries use matrix multiplication across convolution, depthwise separable convolution, and fully connected layers.
  • GEMM-based optimization: 2-D convolution is lowered to matrix-vector multiplication by replicating input activations into rows representing convolution windows.The input-expanding transform is implemented with IM2COL; CMSIS-NN uses partial IM2COL to reduce memory overhead.
  • Implementation details: CMSIS-NN uses a 2x2 matrix-multiplication kernel designed for Cortex-M registers and based on the SMLAD multiply-and-accumulate instruction.Partial IM2COL limits expansion while preserving the GEMM-based implementation.
  • Optimization limitation: Layer B performs nearly twice as many memory accesses and MACC operations as layer A but has only 0.69x of A’s latency on MCU-B.Layer A’s small input-channel count limits its ability to exploit GEMM conversion and SIMD optimization.
  • Conclusion and recommendation: Fixed matrix-multiplication kernel sizes force computation for inappropriate layer shapes onto normal instructions, increasing latency.Careful parameter setting that enables all model computation to use matrix-multiplication kernels can improve inference performance.

B. Kernel Implementation Overheads

CMSIS-NN kernel transformations introduce layer-dependent latency and operational-intensity costs, especially for depthwise convolution, reducing compute utilization and energy efficiency. Direct SIMD-optimized convolution is proposed as an alternative.

  • Fully connected layers have higher latency than convolutions at equal MACC because their lower operational intensity requires more memory accesses.
  • Depthwise convolutions have higher latency per MACC than convolutions because GEMM conversion creates greater IM2COL overhead.
  • All five applications are computation-bound, but IM2COL and GEMM-conversion overhead prevent especially Person Detection from fully using MCU-S compute capacity.
  • Runtime GEMM-conversion overhead impedes energy efficiency because applications with greater latency consume more energy under near-constant power.
  • Direct convolution with SIMD optimization is recommended to reduce the cost of GEMM conversion in neural-network library kernels.

VI. ARCHITECTURAL IMPACT

The paper characterizes how MCU hardware characteristics influence power and performance during machine-learning inference.

  • The analysis examines the influence of MCU hardware characteristics on inference power and performance.

A. Energy Efficiency

MCU choice creates a latency–energy trade-off: MCU-D is fastest but uses the most energy, while Cortex-M4F platforms provide higher performance per watt than Cortex-M7F MCU-D. Lower-capability MCUs can therefore improve performance per watt for the same model.

  • MCU-D achieves the least model latency but the highest energy, while energy is proportional to inference latency across the measured MCUs.
  • Cortex-M4F provides higher performance per watt than Cortex-M7F MCU-D because it is designed for high energy efficiency.
  • Executing the same model on an MCU with lower computation and memory capability can achieve higher performance per watt.

B. Frequency and memory bandwidth

Hardware bandwidth, frequency, and memory capacity constrain TinyML throughput and deployability. Model structure and runtime memory organization therefore matter alongside raw operation count when selecting MCUs and designing models.

  • Frequency and memory bandwidth: Throughput is proportional to MCU bandwidth and frequency, while MCU-D’s dual-issue capability doubles its throughput increase relative to its frequency increment.
  • Frequency and memory bandwidth: Fully connected and depthwise-separable layers are better suited to MCUs with large memory bandwidth, whereas convolution-heavy models favor high-frequency platforms.
  • Memory limitation: An application fits an MCU only when its working memory resides in SRAM and its executable binary resides in Flash memory.
  • Memory limitation: TFLM arena size is determined by the model’s widest layer and can approach MCU-D’s SRAM limit, making balanced NAS structures important.
  • Memory limitation: After excluding the TFLite model, operator implementation code is about 79 KB on MCU-D for the evaluated convolution, depthwise-separable, and fully connected operations.
  • Memory limitation: For similarly structured models, TFLite model size tracks latency more closely than MACC, but this relationship is not obvious across applications with different structures.
  • Memory limitation: Balanced neural-network structures whose largest layer fits on-chip memory are recommended for memory-constrained devices.

VII. PERFORMANCE PREDICTION

The paper develops an analytical approach for predicting MCU layer latency that accounts for model, software, and hardware effects, rather than relying only on operation count.

  • The approach captures model parameters, software optimizations, and hardware characteristics to improve latency prediction accuracy.

A. Analytical Approach

The analytical method combines model, operator, and architecture specifications with LLVM-based instruction analysis to estimate operator latency. It separates loop overhead, computation time, and data movement time.

  • Inputs: The method takes Model Specification, Operator Implementation, and Architecture Specification as inputs for full-stack latency estimation.LLVM bitcode and intermediate representation support subsequent instruction analysis.
  • Inputs: Model parameters describe layer structure and provide high-level information about compute and memory needs before software optimization.
  • Inputs: CMSIS-NN operator implementations differ by layer type, including IM2COL for convolutions but not fully connected layers.
  • Operator Latency: Operator latency is decomposed into loop execution overhead, computation time, and data movement time.
  • Operator Latency: Computation time uses IR instruction counts, while data movement analysis accounts for model data and other variables transferred through memory.

B. Performance Prediction Results

The full-stack analytical approach closely predicts layer and model latency on MCU-S. Across five applications, it outperforms prediction based only on model operation count.

  • Layer Prediction: 12.24% relative difference and 4.59% MAE separate predicted from actual latency across evaluated layers.The comparison covers FC, Conv, Conv1x1, and DWC layers on MCU-S.
  • Layer Prediction: Predicted latency follows actual latency trends, with lower relative error for larger FC, Conv, and DWC layers.A fully connected example has 5.31% predicted error, with 63 µs actual and 66 µs predicted latency.
  • Model Prediction: 5.12% maximum difference and 3.57% MAE characterize model-latency prediction across five applications.The full-stack prediction considers influences that operation-count-based regression does not capture.
  • Conclusion: The analysis provides insights for NAS and system optimization while enabling accurate, rapid latency estimation for ML models.
Loading 2608.21646v1…