PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning for Microcontroller-Class Hardware: A Review

Swapnil Sayan Saha, Sandeep Singh Sandha, Mani Srivastava

arXiv:2205.14550v5cs.LG

TL;DR

Microcontroller ML is constrained by severe memory, compute, and deployment limits, motivating a systematic alternative to conventional model development. The paper characterizes a closed-loop workflow and evaluates its stages across applications, finding diverse workflow adaptations and identifying challenges for next-generation TinyML frameworks.

  • Problem

    Conventional ML deployment exceeds microcontroller resource limits, while prior surveys lack quantitative comparisons across workflow choices and application-specific case studies.

  • Method

    The paper reviews a closed-loop workflow spanning model selection, NAS, hardware profiling, inference optimization, compression, deployment, and online learning.

  • Results

    Across applications, the review reports distinct workflow adaptations, including 90% or higher accuracy for many simple activity and gesture tasks and 72%–73% top 1% accuracy on ImageNet for MCUNetv2 and AttendNets.

  • Takeaways & Limitations

    The review concludes that current methods can transition state-of-the-art ML models to ultraresource-constrained environments while leaving challenges for next-generation TinyML frameworks.

Abstract

from arXiv · show

The advancements in machine learning opened a new opportunity to bring intelligence to the low-end Internet-of-Things nodes such as microcontrollers. Conventional machine learning deployment has high memory and compute footprint hindering their direct deployment on ultra resource-constrained microcontrollers. This paper highlights the unique requirements of enabling onboard machine learning for microcontroller class devices. Researchers use a specialized model development workflow for resource-limited applications to ensure the compute and latency budget is within the device limits while still maintaining the desired performance. We characterize a closed-loop widely applicable workflow of machine learning model development for microcontroller class devices and show that several classes of applications adopt a specific instance of it. We present both qualitative and numerical insights into different stages of model development by showcasing several use cases. Finally, we identify the open research challenges and unsolved questions demanding careful considerations moving forward.

I. INTRODUCTION

Microcontrollers enable low-cost, low-power onboard intelligence, but their severe resource limits make direct deployment of conventional ML models unsuitable. The paper reviews a closed-loop TinyML workflow, compares workflow choices across applications, and identifies open challenges.

  • Resource constraints: A typical microcontroller has 128-kB RAM and 1 MB of flash, compared with gigabytes of memory and storage in mobile phones.These ultraresource limitations make direct porting of models designed for high-end edge devices unsuitable.
  • Paper contribution: The paper presents a coherent closed-loop workflow with qualitative and numerical insights, application-dependent comparisons, and analysis of tradeoffs.The workflow covers model development and deployment for microcontroller-class IoT nodes.
  • TinyML scope: TinyML combines lightweight first-principles approaches with the richer inferences of neural networks for onboard sensor analytics.Onboard inference also avoids dependence on network connectivity and cloud-based inference.

C. Security and Privacy

TinyML keeps sensitive data and inference on the IoT node, while its workflow prepares, projects, selects, optimizes, and deploys models under hardware constraints. The paper connects this workflow to benchmarked application patterns and data-quality challenges.

  • Security and privacy: Onboard inference confines private data within the IoT node, reducing the probability of privacy leaks and attack surfaces.Federated learning, secure aggregation, and homomorphic encryption provide privacy-preserving alternatives but require expensive network and compute resources.
  • Workflow: The TinyML workflow prepares sensor datasets, optionally reduces input dimensionality, selects lightweight models, and deploys optimized embedded code.Deployment includes compression, inference and operator optimization, code generation, flashing, and possible online fine-tuning.
  • Evaluation: MLPerf Tiny evaluates multiclass and binary image recognition, keyword spotting, and outlier detection using quality targets and model-size goals.The benchmark is intended to compare microcontroller inference workflows across representative tasks.
  • Data engineering: Preparing data that generalizes to unseen scenarios is a major challenge, addressed through augmentation, cleaning, unit tests, and anomaly detection.Specialized frameworks support transformations, oversampling, class balancing, noise addition, and labeled-data quality checks.

A. Linear Methods

Feature projection and compression reduce the computational and memory burden of microcontroller ML, but their suitability depends on the method and hardware support. Linear and domain-specific approaches are more deployable than nonlinear techniques for real-time microcontroller execution.

  • Feature projection: Feature projection reduces input dimensionality so models operate on intrinsic data dimensions and become more computationally tractable.The workflow includes linear, nonlinear, and domain-specific feature-extraction approaches.
  • Nonlinear Methods: Nonlinear methods such as t-SNE and UMAP are generally unsuitable for real-time microcontroller execution, despite revealing manifold and cluster structure.They are usually used to visualize high-dimensional handcrafted features, whereas linear methods have been ported to microcontrollers.
  • Quantization: Posttraining quantization can reduce model size by 4× and speed inference by 2–3×, while quantization-aware training can provide up to 8× size reduction.The latter is recommended to mitigate layerwise quantization error, provided the target hardware supports the selected bitwidth.
  • Structured Sparsity: Unstructured sparsity may waste execution time through irregular processing, so effective acceleration requires specialized hardware and software support.Structured alternatives include differentiable pruning, which provides up to 1.7× speedup while compressing unpruned models by 80×.

VI. LIGHTWEIGHT MACHINE LEARNING BLOCKS

Lightweight ML blocks reduce memory, compute, or latency through sparse projections, efficient convolutions, residual designs, and specialized temporal models. The section also reports application-level examples of these strategies and their measured tradeoffs.

  • Lightweight Convolutional Blocks: Depthwise separable, pointwise, and group convolutions reduce channel or kernel costs while channel shuffle improves cross-channel relations.The figure describes depthwise separable convolution as yielding 7–9× memory savings over vanilla convolution.
  • Sparse Projection: Sparse projection maps high-dimensional inputs onto low-dimensional prototypes, reducing parameter count and improving compute efficiency for Bonsai decision trees and ProtoNN kNN classifiers.The projection matrix can be learned with stochastic gradient descent and iterative hard thresholding to mitigate accuracy loss.
  • Lightweight Convolutional Blocks: Inverted residuals connect bottleneck layers rather than excitation layers, reducing memory and compute while enforcing a linear output at the final convolution.This design is presented alongside squeeze and excitation strategies for compact convolutional networks.
  • Lightweight Multiscale Spatial Convolution: EfficientDet combines bidirectional feature pyramids with hardware-oriented compound scaling to support multiscale object detection.Its topology is optimized through neural architecture search over weight, depth, and resolution.
  • Lightweight Multiscale Spatial Convolution: EfficientDet is 4–9× smaller and uses 13–42× fewer FLOPs than the compared models, while removal or fusion of batch normalization and input downscaling yields 3.6–8.8× faster inference.The comparisons include YOLOv3, RetinaNet, AmoebaNet, ResNet, DeepLabV3, vanilla YOLO, and MobileNetsV1.
  • Recurrent and Temporal Blocks: FastRNN stabilizes long-sequence training with a weighted residual connection, while gated, low-rank, sparse, and quantized RNN designs target lightweight recurrent inference.The passage contrasts these approaches with LSTM, gated recurrent units, and unitary RNNs, which can add overhead or lose accuracy.

E. Temporal Convolutional Networks

The supplied passages describe hardware-aware NAS for microcontrollers, spanning layerwise, cellwise, hierarchical, and execution-level search spaces. They compare search strategies and identify performance and portability tradeoffs across reinforcement learning, gradient-driven, evolutionary, and Bayesian approaches.

  • Search Space Formulation: Hardware-aware NAS balances accuracy, latency, and energy by combining a search space, search strategy, and cost function.The search space defines operators, connection rules, and parameter values available to the optimizer.
  • Search Space Formulation: Layerwise spaces are expressive but computationally expensive, whereas cellwise spaces are more time-efficient and transferable but less flexible for hardware specialization.Cellwise architectures stack repeating directed acyclic graph cells while fixing the global network architecture.
  • Search Space Formulation: Hierarchical spaces recursively optimize larger blocks built from cells, allowing different blocks per layer without increasing search cost and supporting hardware specialization.They feed low-level motif feature maps into higher-level motifs.
  • Search Space Formulation: Execution-level search optimizes loop transformations, data reuse, and in-place operators for applications with extreme memory and energy budgets.iNAS uses reinforcement learning to optimize tiling, loop order, and power-cycle tile retention.
  • Search Space Formulation: iNAS reduced intermittent inference latency by 60% compared with architectural NAS frameworks, with a 7% increase in search overhead.The result applies when appropriate power-cycle energy, memory, and latency constraints are included.
  • Search Strategy: NAS search strategies include reinforcement learning, differentiable gradient descent, evolutionary search with weight sharing, and Bayesian optimization.Training-free NAS seeks to avoid the costly inner-loop training used by many conventional approaches.
  • Search Strategy: Gradient-driven NAS can reduce search and training cost but requires large supernetworks, high GPU memory, and repeated searches for new hardware.Hardware metrics are converted into differentiable objectives, and path binarization can produce the final subnetwork through pruning.
  • Search Strategy: Evolutionary weight-sharing NAS avoids separate searches across hardware platforms but incurs substantial computation and time for supernetwork training, fine-tuning, and evolutionary convergence.Progressive shrinking and knowledge distillation jointly fine-tune subnetworks within a once-for-all supernetwork.

C. Cost Function

NAS cost functions evaluate candidate networks against accuracy and hardware-resource objectives, often balancing multiple constraints. Hardware profiling trades measurement accuracy against search speed, from hardware-in-the-loop execution to analytical estimates.

  • NAS cost functions commonly include accuracy, SRAM usage, flash usage, latency, and energy usage.The search seeks an extrema or Pareto-optimal configuration of network parameters.
  • Hardware constraints can be incorporated as regularizers in constrained single-objective optimization or combined through weighted multiobjective scalarization.
  • Hardware-aware NAS uses hardware-specific cost functions or search heuristics based on profiling the target device.
  • Hardware-in-the-loop profiling is slowest but most accurate, whereas analytical estimation is fastest but least accurate during NAS runtime.Lookup tables, prediction models, silicon-accurate emulators, and architectural heuristics provide intermediate or alternative estimation strategies.

VIII. TINYML SOFTWARE SUITES

TinyML software suites convert trained models into deployable embedded implementations while optimizing operators, memory, execution, and inference engines for microcontroller constraints. Their approaches range from interpreter-based runtimes to generated specialized code, with application-dependent performance tradeoffs.

  • VIII. TINYML SOFTWARE SUITES: TinyML software suites generate embedded code, optimize operators and inference engines, and may provide runtime resource management for deployment.The deployment stage prepares the best model for execution on microcontrollers.
  • A. Operator and Inference Optimizations: Fused and in-place operators reduce memory usage by 1.6× and improve speedup by 1.2–2×.Fusion combines operators into specialized kernels without storing intermediate feature representations in memory.
  • A. Operator and Inference Optimizations: Loop transformations improve spatiotemporal execution by reducing loop overhead and increasing data reuse or locality.Examples include loop reordering, reversal, fusion, distribution, unrolling, and tiling.
  • A. Operator and Inference Optimizations: Input reuse reduces SRAM usage by 2.57× and provides 2× speedup in TF-Net direct buffer convolution.The method reuses unpacked inputs in a buffer across all weight filters.
  • B. Notable TinyML Software Suites: TFLM freezes and converts training graphs into flatbuffer models, then uses a preallocated SRAM arena and runtime bin-packing for memory reuse.Its interpreter-based runtime supports model replacement and upgradability while sacrificing finite memory savings.
  • B. Notable TinyML Software Suites: TFLM provided 2.2× speedup and 1.08× memory and flash savings over CMSIS-NN for image recognition.
  • B. Notable TinyML Software Suites: uTensor generates C++ files targeting models smaller than 2 kB, while Micro-TVM generates C code through graph, intermediate-representation, operator, and loop optimization.
  • B. Notable TinyML Software Suites: X-Cube-AI provided 1.3× memory reduction and 2.2× speedup over TFLM for gesture recognition and keyword spotting.TinyML frameworks also include CMSIS-NN, EdgeML, Edge Impulse EON Compiler, and other open- and closed-source suites.

IX. ONLINE LEARNING

Online learning addresses distribution shifts after deployment, but microcontrollers face communication, privacy, memory, and energy constraints that complicate server-based retraining. The section frames continual adaptation as a distinct TinyML requirement.

  • IX. ONLINE LEARNING: Deployed models may require periodic updates to account for feature-distribution shifts in real-world data.
  • IX. ONLINE LEARNING: Limited bandwidth and privacy concerns can prevent offloading training to a server, while conventional training requires much more memory and energy than inference.

A. On-Device Training

On-device training frameworks detect distribution shifts, adapt models, and select informative samples under severe resource constraints. TinyML federated learning extends this process across non-IID devices, while accuracy, communication, privacy, and compatibility remain important tradeoffs.

  • A. On-Device Training: On-device training divides into detecting when to learn, adapting the model, and selecting samples that maximize learning effect.Detection can use distribution divergence, streaming statistics, or prediction confidence; sample selection can use gradient norms, class balancing, or replay weighting.
  • A. On-Device Training: Last-layer transfer learning is lightweight but performs poorly when new-data distributions differ substantially from the original training set.Limited capacity and catastrophic forgetting constrain this approach.
  • A. On-Device Training: TinyTL refines output feature maps by updating biases instead of weights, while ML-MCU trains only influential one-versus-one classifiers.
  • A. On-Device Training: Quantized continual learning replays stored activation maps, whereas incremental training updates weights one sample at a time through constrained optimization.Both approaches have limited supported network types or application space.
  • A. On-Device Training: None of the on-device training frameworks is directly compatible with popular TinyML software suites because frozen model graphs cannot be unfrozen on board.
  • A. On-Device Training: Federated learning distributes local model updates from edge devices to a server for aggregation without transferring the underlying data.The setting is distributed and non-IID, with devices receiving a common aggregated model.
  • A. On-Device Training: Transfer-learning and knowledge-distillation variants improved human-action-recognition accuracy by 5%–11% over vanilla FedAvg while reducing communication cost by 10–5000×.
  • A. On-Device Training: RL-based model aggregation achieved 1.4%–2.7% higher accuracy than FedAvg for image recognition.

X. KEY APPLICATIONS

TinyML applications span visual, audio, anomaly-detection, and activity-tracking tasks, with workflow variants and model designs adapted to microcontroller constraints.

  • Application-specific workflows: TinyML workflow variants are selected according to the application, with numerical insights reported across several application-specific deployments.The reviewed applications include image recognition, visual wake words, audio keyword spotting, anomaly detection, and activity or gesture tracking.
  • Image recognition and visual wake words: MCUNetv2 and AttendNets achieved 72%–73% top 1% accuracy on ImageNet for microcontrollers.MCUNetv2 combines once-for-all NAS with patch-by-patch inference and receptive field redistribution; AttendNets use a visual attention condenser.
  • Image recognition and visual wake words: RaScaNet achieves the best balance of accuracy and resource usage among the listed visual wake-word detection models.It combines convolutional feature extraction on image patches with recurrent learning of the full-image latent representation.
  • Anomaly detection: Anomaly detection targets industrial malfunctions, physiological disorders, and climate conditions in an unsupervised fashion.It detects outliers or deviations in input streams and is included in the MLPerf Tiny v0.5 benchmark.
  • Activity and gesture tracking: Activity and gesture models commonly achieve 90% or more accuracy for simple macrogestures or macroactivities while remaining 10^0 or 10^1 kB in size.Most models are hand-tuned, with a few automated using NAS.

E. Odometry and Navigation

Odometry and navigation deployments combine compact neural models with sensor fusion and runtime optimizations to fit microcontroller hardware while retaining useful task performance.

  • E. Odometry and Navigation: TinyOdom produces neural inertial odometry models 31–134× smaller than existing neural inertial odometry models for Cortex-M deployment.Its design combines THIN-Bayes, a TCN backbone, magnetometer input, physics, and velocity-centric sequence learning.
  • E. Odometry and Navigation: VNNs achieve 97% accuracy with 7.5–163.5 kMACs, while PULP on GAP8 reduces latency 13× and energy 3.2× over Cortex-M architectures.The reported latency is 0.2–1.2 mS and energy is 3.9–18.9 µJ per inference.
  • E. Odometry and Navigation: Runtime methods including tiling, quantization, parallelization, and signal processing support neural navigation deployments on nano-UAV hardware.The passage connects these techniques with DroNets deployments on nano-UAVs retrofitted with a PULP GAP8 shield.
  • Runtime optimizations: Patch-by-patch inference reduces activation-map regions processed, receptive field redistribution shifts receptive field and FLOPs later, and dual-memory management swaps variables between flash and RAM.These mechanisms target peak memory usage and runtime memory availability.
  • Feature projection: Feature projection can preserve data variance while reducing dimensionality, allowing an MLP gesture pipeline to use 2.5× less flash and run 2.2× faster at equal accuracy.The comparison is between CNN and MLP pipelines in the gesture-recognition example.

B. Compression Versus No Compression

Compression, lightweight architectures, and NAS address different microcontroller constraints, but their benefits depend on hardware support, accuracy tradeoffs, and deployment-specific budgets.

  • Compression versus no compression: LSTM-Prune and LSTM-KP provide an average 12× model-size reduction with only 2.3% accuracy loss relative to FastGRNN and Bonsai.Across the examples, pruned uncompressed models average 13.5× size reduction versus 2.1× for lightweight FastGRNN.
  • Compression versus no compression: Pruning averages 13.6× flash compression versus 3.9× for quantization, while combining both yields a 16× average model-size reduction.Quantization is more likely than pruning to reduce SRAM and inference latency, whereas pruning often incurs greater accuracy loss.
  • Compression versus no compression: Pruning reduces accuracy by 4.9% on average, compared with 0.4% for quantization.The passage attributes this difference to greater information loss from pruning.
  • Compression versus no compression: Quantization benefits depend on compatible microcontroller and software support because some platforms cannot exploit arbitrary or sub-byte bitwidths.TFLM does not support arbitrary bitwidths of weights and activations, and SIMD limits can restrict low-precision acceleration.
  • Lightweight versus vanilla models: Vanilla models can be too large for commodity microcontrollers, forcing lightweight operators, but these may improve one metric while degrading another or generalization.SqueezeNet has lower model size but higher latency and energy than AlexNet, while TCN has a 5% test-accuracy reduction in one activity-detection comparison.
  • NAS versus handcrafted models: NAS consistently provides the most accurate models within device constraints when accuracy, flash, SRAM, latency, and energy form competing objectives.NAS searches architectures under specified bounds, including operator restrictions and target-device budgets.
  • NAS versus handcrafted models: TinyOdom uses NAS to produce models for four microcontrollers with a 1.6–30× model-size reduction and a 1.2× resolution drop versus handcrafted models.The example illustrates architecture adaptation across hardware with distinct compute and memory budgets.
  • NAS versus handcrafted models: NAS can reveal achievable performance and architectural choices when prior expert knowledge does not match deployment needs or hardware conditions.Examples include inertial odometry on microcontrollers and DNN deployment on tighter-budget AVR RISC devices.

E. Using Runtime Optimizations Versus No Optimizations

TinyML deployment requires software suites and workflow choices that fit models within microcontroller memory while meeting latency and energy constraints. The paper contrasts runtime optimization, platform support, and online-learning tradeoffs.

  • Runtime optimizations: 133 kB of working memory lets the CIFAR-10 CNN fit where 332 kB would overflow Cortex-M7 SRAM.Partial im2col in CMSIS also reduces inference latency by 4.6× and energy usage by 4.9×.
  • Runtime optimizations: TinyML software suites generate embedded code and optimize inference engines, operators, and model compression before flashing the model onto the microcontroller.The deployment phase can also include periodic fine-tuning through online-learning frameworks.
  • Workflow choices: Software-suite selection involves application requirements such as memory, latency, energy, platform support, intermittent computing, online learning, and multiple-model deployment.The workflow also asks whether automated schedule exploration and dynamic memory management are necessary.
  • Workflow choices: STM32Cube.AI supports only STM32 Cortex-M microcontrollers but deploys neural and nonneural models, whereas TFLM offers broader platform support but only neural-network deployment.The comparison exposes a tradeoff between platform breadth and model-type support.
  • Online learning: Online learning improves model performance by adapting on board, but TinyML software suites generally lack comprehensive support for on-device learning and its training-memory overhead.The paper notes unresolved integration issues for NAS, software frameworks, and federated learning.
  • Open challenges: TinyML development has progressed from fitting models within microcontroller limits toward emerging challenges requiring further research.The paper characterizes current efforts as a first generation of TinyML.

A. Application Specific Safety and Heuristic Requirements

Microcontroller ML must account for domain safety rules, uncertain sensor streams, changing data distributions, and behavioral fidelity to upstream models. The review identifies these requirements as open challenges alongside hardware–software co-design.

  • Application-specific safety and heuristic requirements: Neural networks cannot assure that learned distributions obey application-specific laws, such as UAV stability limits or event-ordering rules.Real-world IoT applications therefore impose domain bounds, correlations, and heuristic requirements on ML behavior.
  • Sensing uncertainty: Missing data, timestamp misalignment, and window jitter arise from timing delays, clock imperfections, sensor faults, memory overflow, and power constraints.The paper reports that sensing uncertainty can reduce ML performance in complex event processing.
  • Domain shifts: Microcontroller models must balance on-device fine-tuning against over-the-air updates because existing robust adaptation methods may not fit on the platform.The review also identifies uncertainty about porting unsupervised embeddings to microcontrollers.
  • Backward compatibility: Equal aggregate accuracy does not guarantee functional equivalence between upstream and downstream models, because samplewise inconsistencies can create new failures.Robust inference requires both high accuracy and high fidelity to the upstream model.
  • Security: Compressed TinyML models remain vulnerable to adversarial attacks and false-data injection, while physical and microarchitectural side channels can leak chip information.Keeping private data on the IoT node reduces some cloud-associated leakage risks but does not eliminate the platform’s attack surface.
  • Hardware–software co-design: TinyML software and hardware should be optimized jointly as neural-network workloads grow while microcontroller platforms continue shrinking.Proposed hardware directions include systolic arrays, stochastic computing, in-memory computing, near-data processing, and spiking hardware.
  • Conclusion: The review calls for next-generation TinyML frameworks that address challenges receiving limited exploration after first-generation deployment advances.It presents application tradeoffs and open opportunities rather than a completed solution to these challenges.
Loading 2205.14550v5…