PaperlayerView exact revision on arXiv

Source-linked AI summary

Tiny Machine Learning: Progress and Futures

Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Song Han

arXiv:2403.19076v2cs.LGcs.AIcs.CV

TL;DR

TinyML addresses how deep learning can operate on memory-constrained microcontrollers and IoT devices, where cloud- and mobile-designed models do not fit. The review synthesizes TinyML progress, presents system-algorithm co-design through MCUNet, and reports strong MCU results across visual wake words, detection, and ImageNet-scale classification. It also identifies broader modalities and applications as future directions while noting that the review mainly focuses on CNN-based computer vision.

  • Problem

    TinyML must support deep learning on devices with very limited memory, while existing models and deployment approaches are not designed for MCU constraints.

  • Method

    The review surveys TinyML challenges and progress, then presents MCUNet’s system-algorithm co-design and discusses tiny on-device training techniques.

  • Results

    MCUNet achieves over 90% visual wake-word accuracy under 32kB SRAM, reaches 71.8% ImageNet accuracy on commercial MCUs, and improves detection mAP by double digits.

  • Takeaways & Limitations

    TinyML supports customized and private AI by processing sensor data locally and adapting models on edge devices.

  • Takeaways & Limitations

    The review mainly focuses on CNN-based computer vision, leaving audio, language, anomaly detection, and multisensor workloads for future exploration.

Abstract

from arXiv · show

Tiny Machine Learning (TinyML) is a new frontier of machine learning. By squeezing deep learning models into billions of IoT devices and microcontrollers (MCUs), we expand the scope of AI applications and enable ubiquitous intelligence. However, TinyML is challenging due to hardware constraints: the tiny memory resource makes it difficult to hold deep learning models designed for cloud and mobile platforms. There is also limited compiler and inference engine support for bare-metal devices. Therefore, we need to co-design the algorithm and system stack to enable TinyML. In this review, we will first discuss the definition, challenges, and applications of TinyML. We then survey the recent progress in TinyML and deep learning on MCUs. Next, we will introduce MCUNet, showing how we can achieve ImageNet-scale AI applications on IoT devices with system-algorithm co-design. We will further extend the solution from inference to training and introduce tiny on-device training techniques. Finally, we present future directions in this area. Today's large model might be tomorrow's tiny model. The scope of TinyML should evolve and adapt over time.

I. OVERVIEW OF TINY MACHINE LEARNING

TinyML extends efficient machine learning to ultra-low-power devices, but MCU memory constraints prevent directly scaling cloud or mobile models. The review therefore emphasizes system-algorithm co-design and covers inference, training, applications, and field progress.

  • Definition and motivation: TinyML brings efficient machine learning to IoT devices and microcontrollers, extending EdgeML toward ultra-low-power, always-on, on-device AI.On-device processing can improve responsiveness and privacy while reducing wireless communication energy.
  • Applications: TinyML supports applications including personalized healthcare, wearables, smart homes, autonomous vehicles, anomaly detection, and agriculture.The review also describes continuous adaptation and lifelong learning on devices without cloud connectivity.
  • Review scope: The review proposes system-algorithm co-design as a central response and surveys TinyML progress before introducing MCUNet for inference and training.Its stated scope includes definitions, challenges, applications, literature comparisons, MCUNet, on-device training, and future directions.
  • Challenges: MCUs impose severe deployment constraints because they lack DRAM and operating systems, while SRAM can be smaller than 256kB and flash is read-only.These resources are orders of magnitude below those available on mobile platforms.
  • Challenges: 4.2× fewer parameters in MobileNetV2 than ResNet-50 still coincides with 2.3× higher peak inference memory, showing that parameter reduction does not solve TinyML’s activation bottleneck.The supplied experiment also reports that training requires 6.9× more memory than inference.

2) Directly adapting models for inference does not work for tiny training.

Tiny on-device training is harder than inference because backward propagation requires storing intermediate activations on severely memory-limited MCUs. The section motivates new training methods alongside TinyML applications across health, wearables, homes, vehicles, robotics, and agriculture.

  • Training challenge: Training requires intermediate activations for backward gradients, making activation storage a central obstacle for tiny devices.Prior cloud and mobile approaches focused mainly on FLOPs or inference memory rather than this training bottleneck.
  • Training challenge: 6.9× more memory is required for full-backpropagation training than inference, while MobileNet training improves memory by only 10% over ResNet.A typical 256KB MCU therefore has barely enough SRAM for inference, let alone training.
  • Design implication: Co-design is necessary because TinyML hardware constraints require architectures and training frameworks tailored jointly to the device rather than separately optimized for mobile or cloud systems.The passage identifies both unsuitable prior architectures and insufficient support for memory-efficient forward and backward computation.
  • Applications: TinyML applications span healthcare wearables, speech interfaces, smart homes, autonomous transportation, anomaly detection, and ecological or agricultural monitoring.These examples use local sensing for health tracking, recognition, decision-making, and environmental monitoring.
  • Applications: TinyML can support local object detection, lane detection, and decision-making without a cloud connection, alongside anomaly detection and environmental forecasting.The application scope extends beyond vision to robots, sensors, ecology, agriculture, and phenomics.

II. RECENT PROGRESS IN TINYML

Recent TinyML progress combines MCU-specific algorithms, kernels, compilation, memory scheduling, and lightweight runtimes, while benchmarks compare models and frameworks under strict device limits. TinyNAS and TinyEngine exemplify joint adaptation of architectures and inference systems.

  • Hardware and systems: MCU deployment is constrained by limited SRAM for activations, limited flash for models, and substantially lower processor speed than conventional platforms.The STM32F746 example has 320KB SRAM, 1MB flash, and a 216MHz processor.
  • Algorithmic methods: TinyNAS uses a two-step search that first adapts the search space to device resources and then specializes architectures for constraints such as latency, energy, and memory.This addresses the dependence of neural architecture search on an appropriate search space.
  • Software support: Standard training frameworks impose runtime overhead, and lightweight edge frameworks remain too large for many MCUs.The cited frameworks support mobile or edge devices but cannot accommodate IoT devices with sufficiently limited memory.
  • Evaluation landscape: Tables II and III compare TinyML research and model-framework performance under specified microcontroller resource constraints.Table III targets an STM32H743 at 480MHz with 512KB peak memory and 2MB storage.
  • Hardware and systems: Recent work reduces MCU costs through optimized kernels, operator scheduling, convolution-loop transformations, data swapping, partial spatial computation, and mixed-precision libraries.TinyEngine combines memory-efficient inference with a larger model search space.

B. Recent Progress on TinyML Training

TinyML training research addresses the larger memory and computation demands of on-device learning through lightweight training strategies and system-algorithm co-design. MCUNetV3 combines quantization-aware scaling, sparse updates, and a dedicated training engine for constrained microcontrollers.

  • Motivation: On-device training offers personalization and privacy but requires more memory and computation than inference.Training must store intermediate activations and gradients, increasing its resource footprint.
  • Existing approaches: Transfer learning reduces training memory by fine-tuning only the final layer on a task-specific dataset.The network is pretrained on a large dataset and used as a feature extractor.
  • Existing approaches: TinyTL freezes network weights and fine-tunes biases, allowing intermediate activations to be discarded during backpropagation.TinyOL similarly trains only the final-layer weights to keep activation memory small.
  • MCUNetV3: MCUNetV3 combines Quantization-Aware Scaling, sparse updates, and the Tiny Training Engine for memory-constrained on-device learning.QAS stabilizes 8-bit quantized training, sparse updates skip less important gradient computations, and TTE supports both techniques.
  • System-algorithm co-design: MCUNet jointly optimizes neural architecture and inference scheduling to better use microcontroller resources.The framework co-designs TinyNAS and TinyEngine in the same loop.

A. TinyNAS: Automated Tiny Model Design

TinyNAS automates tiny-model design by first optimizing a resource-compatible search space and then specializing a network within it. Its search-space selection uses computational distributions, while co-design with TinyEngine expands feasible architecture choices.

  • Search strategy: TinyNAS first optimizes the search space for tiny and diverse resource constraints, then searches within the optimized space.Optimizing the search space significantly improves the final model’s accuracy.
  • Search-space construction: TinyNAS varies input resolution and width multiplier to cover microcontrollers with different resource constraints.The supported input resolutions span 48 to 224, while width multipliers span 0.2 to 1.0.
  • Automated search-space optimization: TinyNAS evaluates candidate search spaces using FLOPs distributions from randomly sampled satisfying networks instead of training every space fully.This avoids the computational cost of collecting accuracy distributions through extensive training.
  • Automated search-space optimization: A search space averaging 52.0M FLOPs achieved 2.3% better accuracy than one averaging 46.9M FLOPs on ImageNet-100 using STM32F746.Only models satisfying the memory requirement under TinyEngine’s best scheduling were retained.
  • Network specialization: TinyNAS uses one-shot specialization with a weight-shared super network covering 2 × 10^19 possible subnetworks.The search space varies depth-wise kernel sizes, inverted-bottleneck expansion ratios, and stage depths.
  • System-algorithm co-design: TinyEngine’s efficiency affects both inference latency and the accuracy attainable through architecture search.By reducing memory waste, TinyEngine enables a larger search space, making higher-accuracy models more likely.

1) Code generation.

TinyEngine uses compilation and memory-aware scheduling to fit neural inference within microcontroller limits. Code generation, in-place operators, and patch-based execution address binary size, activation memory, and early-layer bottlenecks, while receptive-field redistribution reduces patching overhead.

  • Code generation: TinyEngine uses code-generator compilation instead of interpreter-based execution to avoid runtime interpretation and metadata memory.This frees MCU memory for larger model designs and inference.
  • Code generation: Model-adaptive compilation reduces code size by up to 4.5× versus TF-Lite Micro and 5.0× versus CMSIS-NN.The reduction comes from compilation together with model-specific optimization and specialization.
  • Memory optimization: In-place depth-wise convolution overwrites completed input activations with output activations to reduce peak memory.Because depth-wise convolution does not filter across channels, completed channels can be reused safely.
  • Patched-based inference: MobileNetV2’s first five blocks exceed MCU memory limits, with the third block using 8× more memory than the rest of the network.This imbalanced distribution makes early blocks the primary memory bottleneck.
  • Patched-based inference: Patch-based inference stores one patch rather than the whole feature map, reducing the memory required by initial convolutional layers.It computes the final output buffer patch by patch instead of processing each layer over the full feature map.
  • Receptive-field redistribution: Redistributing receptive fields reduces the initial patch stage’s receptive field and increases the later stage’s receptive field to limit repeated computation.A smaller initial receptive field reduces patch size, while later layers compensate for potential performance loss; the manually tuned MbV2-RD example has negligible overhead.

1) Co-Design Loop

MCUNet co-designs neural architectures and inference scheduling to satisfy tight MCU constraints, enabling stronger image classification and detection across resource-limited hardware. Its results show improved accuracy or mAP while reducing peak SRAM, often by using higher input resolution and patch-based inference.

  • Co-Design Loop: MCUNet jointly searches neural architecture and inference scheduling to balance per-layer execution against larger-model patch-based inference under hardware constraints.The search uses evolutionary optimization over architecture and scheduling choices, including patch count.
  • Co-Design Loop: Joint optimization significantly advances state-of-the-art MCU-based tiny image classification under strict memory and flash constraints.The reported gains span ImageNet classification, visual wake words, Pascal VOC detection, and face detection.
  • Co-Design Loop: 4.6% higher ImageNet accuracy at 18% lower peak SRAM is achieved under 256kB SRAM/1MB Flash, while 71.8% accuracy sets a new record under 512kB SRAM/2MB Flash.The lower-bit or mixed-precision results are described as orthogonal techniques.
  • Co-Design Loop: Over 90% visual wake word accuracy is achieved under 32kB SRAM, with 4.0× smaller memory than per-layer inference.Patch-based inference also expands the search space for better accuracy-versus-latency trade-offs.
  • Co-Design Loop: MCUNet-H7 improves Pascal VOC mAP by 16.9% on STM32H743, while the 256kB-SRAM version improves mAP by 13.2% at 1.9× smaller peak SRAM.The improvement is associated with higher input resolution enabled by patch-based inference.
  • Co-Design Loop: MCUNet-L achieves comparable face-detection mAP at 3.4× smaller peak SRAM and 1.6× smaller computation than RNNPool-Face-C.MCUNet-S also outperforms several alternatives at smaller peak memory.

IV. TINY TRAINING

Tiny on-device training adapts deployed models to newly collected sensory data for continuous learning, customization, and privacy. It remains difficult because tiny devices lack memory and runtime support for backpropagation, requiring joint algorithm-system design.

  • IV. TINY TRAINING: Tiny on-device training adapts pre-trained models to newly collected sensory data after deployment, supporting continuous learning and user customization.Local adaptation can also protect privacy by keeping sensitive data near the sensors.
  • IV. TINY TRAINING: A typical 256KB SRAM budget is barely enough for inference, while training needs extra computation for backward propagation and extra memory for intermediate values.The passage describes a memory gap exceeding 1000× between tiny devices and conventional training requirements.
  • IV. TINY TRAINING: Existing training systems overlook tight edge-device resources, while inference frameworks provide slim runtimes but lack backpropagation support.Existing efficient update schemes also fail to realize theoretical savings in measured memory and can yield low downstream accuracy.
  • IV. TINY TRAINING: Tiny on-device training must address quantized graphs and hardware limits that prevent full backpropagation within available SRAM and computation.Quantized graphs involve mixed-precision tensors and lack Batch Normalization layers, complicating optimization.

A. Quantization Aware Scaling

Quantization-aware scaling addresses distorted weight-gradient ratios in real int8 training, restoring optimization and matching floating-point transfer-learning accuracy without extra memory. The method targets the optimization difficulty introduced by quantized graphs while retaining TinyML memory constraints.

  • Results: Algorithm and system co-design reduces training memory from 303MB to 149KB, a 2077× reduction, while preserving transfer-learning accuracy.The measurement uses MobileNetV2-w0.35 with batch size 1 and 128×128 resolution, fitting within 256KB SRAM.
  • Motivation: Real int8 training can underperform fp32 because mixed precisions and fused layers distort gradient updates.Adaptive optimizers may improve accuracy but can require more memory or remain unstable.
  • Motivation: Quantization changes the weight-gradient norm ratio, causing different tensors to update at mismatched speeds.The distortion arises from scaling factors and differing int8 and int32 tensor types.
  • Quantization-Aware Scaling: QAS compensates for quantization-induced scaling so gradient ratios match the floating-point counterpart.The method uses a hyper-parameter-free scaling rule during quantized training.
  • Results: QAS closes the transfer-learning accuracy gap with floating-point training without extra memory cost.The result is reported for updating the last two blocks of MCUNet on downstream datasets.

B. Memory-Efficient Sparse Update

Sparse update selects only influential layers and tensor portions for fine-tuning, reducing memory while preserving or improving downstream accuracy. Contribution analysis and automated search identify update schemes under a memory constraint.

  • Motivation: Updating the whole model or several final blocks requires too much memory for TinyML, motivating sparse layer and tensor updates.Updating the last two MCUNet blocks can exceed 256KB of extra memory.
  • Sparse Update: Sparse update prunes backpropagation and parameter updates, trading update coverage against memory and expressiveness.Bias updates require less memory, while weight updates are more memory-intensive but more expressive.
  • Automated Selection: Contribution analysis estimates each parameter’s accuracy contribution, and evolutionary search maximizes total contribution under a memory constraint.The search avoids exhaustive enumeration of the roughly 10^30 combinations for MCUNet’s update choices.
  • Results: Sparse update uses partial back-propagation to reduce memory and computation while maintaining comparable accuracy to full update.The comparison includes full, last-only, bias-only, and sparse update schemes.
  • Results: Sparse update achieves higher downstream accuracy with 4.5-7.5× smaller memory overhead than updating the last k layers.Results average performance across eight downstream datasets and can exceed the baseline’s best statistics.

C. Tiny Training Engine (TTE)

Tiny Training Engine translates sparse-update algorithms into optimized execution graphs so theoretical savings become measured memory and speed improvements. Compile-time graph transformation prunes unnecessary computation and reorders updates to reduce tensor lifetimes.

  • Compiler Co-Design: TTE converts sparse-update algorithms into slim binary code through compile-time differentiation and code generation.This bridges the gap between algorithmic memory savings and actual framework-level memory usage.
  • Compiler Co-Design: Graph optimization prunes frozen-weight nodes and interlaces gradient updates with backward computation to reduce memory.Operator reordering enables in-place updates and operator fusion, avoiding large intermediate tensors.
  • Results: 20-21× lower measured peak memory makes sparse-update training feasible on tiny edge devices.The reduction is reported across different models, while graph optimization consistently lowers peak memory across sparse-update schemes.
  • Results: 23-25× faster training than full update with TF-Lite Micro operators is achieved using sparse update and TTE kernels.The speed improvement is attributed to compiler optimizations including loop unrolling and tiling, with associated energy savings.

V. CONCLUSION AND OUTLOOK

The review presents TinyML as a rapidly evolving approach to deep learning on resource-constrained devices, emphasizing co-design for inference and training. It identifies broader modalities, self-supervised learning, and changing hardware boundaries as future directions.

  • Conclusion: TinyML supports customized and private AI by processing sensor data locally on resource-constrained devices.The conclusion links practical inference and continual training to changing edge environments.
  • Conclusion: TinyML requires redesigned model spaces, backpropagation schemes, and algorithm-system co-design for tiny inference and training.The review frames direct adaptation of mobile models and inference methods as insufficient for these settings.
  • More applications and modalities: TinyML’s current focus on CNN-based computer vision leaves audio, language, anomaly detection, and multisensor workloads for future exploration.The review explicitly identifies these modalities and sensor inputs as broader application opportunities.
  • Self-supervised learning: Self-supervised learning is proposed for on-device training because accurately labeled data can be difficult or expensive to obtain.The motivation is especially relevant to domain adaptation for vision tasks such as segmentation and detection.
  • Relationship between TinyML and LargeML: TinyML techniques can also apply to large-scale cloud machine learning, including quantization under resource constraints.The review presents TinyML and LargeML as related efforts targeting efficient models.
  • Relationship between TinyML and LargeML: A model considered large today may become tiny as hardware improves, so TinyML’s scope should evolve over time.The review uses ResNet-50’s changing deployment status as an example of this shifting boundary.
Loading 2403.19076v2…