PaperlayerView exact revision on arXiv

Source-linked AI summary

MCUNet: Tiny Deep Learning on IoT Devices

Ji Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, Song Han

arXiv:2007.10319v2cs.CV

TL;DR

Microcontrollers have extremely limited memory, making deep learning deployment difficult despite the appeal of always-on IoT intelligence. MCUNet jointly designs TinyNAS and TinyEngine to fit models and inference within those constraints, achieving 70.7% ImageNet top-1 accuracy and faster wake-word inference on commercial microcontrollers.

  • Problem

    Microcontrollers have memory budgets far below mobile devices, while existing model and NAS designs do not fit their tight memory and storage constraints.

  • Method

    MCUNet jointly designs TinyNAS, which first optimizes a resource-fitting search space and then searches within it, with TinyEngine, a memory-efficient inference library.

  • Results

    MCUNet achieves 70.7% ImageNet top-1 accuracy on an off-the-shelf commercial microcontroller and runs wake-word inference 2.4-3.4× faster than existing solutions.

  • Takeaways & Limitations

    The results support ImageNet-scale and always-on tiny machine learning on off-the-shelf microcontrollers.

  • Takeaways & Limitations

    Neural architecture search uses subsets of training data for ImageNet and VWW validation, while Speech Commands uses its validation set for search and test set for reporting.

Abstract

from arXiv · show

Machine learning on tiny IoT devices based on microcontroller units (MCU) is appealing but challenging: the memory of microcontrollers is 2-3 orders of magnitude smaller even than mobile phones. We propose MCUNet, a framework that jointly designs the efficient neural architecture (TinyNAS) and the lightweight inference engine (TinyEngine), enabling ImageNet-scale inference on microcontrollers. TinyNAS adopts a two-stage neural architecture search approach that first optimizes the search space to fit the resource constraints, then specializes the network architecture in the optimized search space. TinyNAS can automatically handle diverse constraints (i.e.device, latency, energy, memory) under low search costs.TinyNAS is co-designed with TinyEngine, a memory-efficient inference library to expand the search space and fit a larger model. TinyEngine adapts the memory scheduling according to the overall network topology rather than layer-wise optimization, reducing the memory usage by 4.8x, and accelerating the inference by 1.7-3.3x compared to TF-Lite Micro and CMSIS-NN. MCUNet is the first to achieves >70% ImageNet top1 accuracy on an off-the-shelf commercial microcontroller, using 3.5x less SRAM and 5.7x less Flash compared to quantized MobileNetV2 and ResNet-18. On visual&audio wake words tasks, MCUNet achieves state-of-the-art accuracy and runs 2.4-3.4x faster than MobileNetV2 and ProxylessNAS-based solutions with 3.7-4.1x smaller peak SRAM. Our study suggests that the era of always-on tiny machine learning on IoT devices has arrived. Code and models can be found here: https://tinyml.mit.edu.

1 Introduction

MCUNet addresses the severe memory constraints that make deep learning difficult on microcontrollers by jointly designing the model and inference library. Its TinyNAS–TinyEngine co-design enables ImageNet-scale inference and improves performance on microcontroller tasks.

  • Motivation: Microcontrollers provide 3 orders of magnitude less memory than mobile devices, making standard deep learning deployment difficult.An ARM Cortex-M7 example has 320kB SRAM and 1MB Flash; even int8 MobileNetV2 exceeds the memory limit by 5.3×.
  • Motivation: Existing efficient models and NAS methods target GPUs or smartphones, optimizing FLOPs or latency rather than microcontroller memory and storage.At similar 70% ImageNet accuracy, MobileNetV2 reduces model size by 4.6× versus ResNet-18 but increases peak activation size by 1.8×.
  • Approach: MCUNet jointly optimizes TinyNAS, a two-stage architecture search method, and TinyEngine, an inference library, to fit diverse microcontroller constraints.TinyNAS first optimizes the search space for resource constraints, then searches within the optimized space.
  • Results: TinyEngine reduces peak memory usage by 3.4× and accelerates inference by 1.7-3.3× compared to TF-Lite and CMSIS-NN.The resulting memory efficiency allows TinyNAS to explore a larger architecture design space.
  • Results: MCUNet achieves 70.7% ImageNet top-1 accuracy on an off-the-shelf commercial microcontroller and state-of-the-art wake-word performance.Wake-word inference runs 2.4-3.4× faster with 3.7-4.1× smaller peak SRAM than existing solutions.

2 Background

Microcontroller inference frameworks and neural architecture search face constraints that conventional mobile-oriented designs do not address. MCUNet responds by co-designing inference scheduling and architecture search for limited memory.

  • Inference and model design: Microcontroller frameworks can waste SRAM and Flash through runtime interpretation and layer-level optimization that ignores whole-network structure.Interpreter overhead can consume up to 65% of peak memory and increase latency by 22%.
  • MCUNet response: MCUNet jointly optimizes neural architecture and inference scheduling rather than optimizing either the network or library in isolation.This system-algorithm co-design is intended to better utilize microcontroller resources.
  • Inference and model design: NAS performance depends strongly on search-space quality, but standard mobile search spaces lack established designs for memory-limited microcontrollers.Mobile search spaces commonly inherit MobileNetV2-style settings such as 224 input resolution and similar channel configurations.

3 MCUNet: System-Algorithm Co-Design

MCUNet jointly optimizes neural architecture search and inference scheduling so microcontroller models fit tight memory constraints. TinyNAS specializes architectures within optimized search spaces, while TinyEngine expands feasible designs through memory-aware and specialized execution.

  • System-algorithm co-design: MCUNet jointly optimizes the neural-network architecture and inference scheduling in one co-design loop.This contrasts with methods that search architectures for a fixed library or tune a library for a fixed network.
  • TinyNAS: TinyNAS first optimizes the search space for resource constraints, then searches for a model within the optimized space.The two-stage procedure is designed for tiny and diverse microcontroller memory constraints while keeping search costs low.
  • TinyNAS: TinyNAS evaluates search spaces by sampling feasible networks and comparing their FLOPs distributions instead of training models across every candidate space.For ImageNet-100 on STM32F746, it sampled 1,000 networks per configuration and selected the configuration with the largest average FLOPs among feasible models.
  • TinyEngine: TinyEngine is 3× and 1.6× faster than TF-Lite Micro and CMSIS-NN, respectively, while its in-place depth-wise convolution reduces activation memory from 2N to N+1.Model-adaptive im2col improves inference efficiency by 13%, kernel specialization by 22%, and in-place depth-wise convolution reduces measured memory usage by 1.6×.
  • TinyEngine: TinyEngine enlarges TinyNAS’s feasible design space by reducing memory use and inference overhead through model-adaptive scheduling and specialized execution.Its design includes model-adaptive tiling, operation-specific kernel specialization, operation fusion, and in-place depth-wise convolution.

4 Experiments

Experiments evaluate MCUNet across classification, wake-word, detection, hardware, latency, and search-space settings, showing that system-algorithm co-design improves accuracy while meeting tight MCU constraints.

  • Large-Scale Image Recognition on Tiny Devices: MCUNet achieves the highest ImageNet accuracy among models runnable on a microcontroller and outperforms baselines across latency requirements.The evaluation uses int8 deployment on diverse STM32 microcontrollers, with STM32F746 as the default reporting platform.
  • Large-Scale Image Recognition on Tiny Devices: TinyEngine and TinyNAS each improve performance under latency constraints, with co-design providing additional accuracy gains over scaled MobileNetV2.For MobileNetV2, TinyEngine improves accuracy by 1.9% at 5 FPS and 5.4% at 10 FPS; MCUNet adds 8.3% and 6.4%, respectively.
  • Visual&Audio Wake Words: MCUNet advances the wake-word accuracy–latency and accuracy–memory trade-offs, achieving higher accuracy with 2.4–3.4× faster inference and 3.7–4.1× smaller peak SRAM.On Speech Commands, it is 2.8× faster with 4.1× smaller peak memory; on VWW, it is 2.4× faster than the previous state-of-the-art.
  • Object Detection: MCUNet improves Pascal VOC detection mAP by 20% under a 512kB SRAM constraint by fitting greater model capacity and computation at smaller peak memory.MobileNet-v2 with CMSIS-NN is limited to a 34M-FLOP model whose peak memory slightly exceeds the budget.
  • Large-Scale Image Recognition on Tiny Devices: 70.7% ImageNet top-1 accuracy is achieved on an off-the-shelf STM32H743 MCU, while using 3.5× less memory and 5.7× less Flash than similarly accurate 8-bit baselines.The comparison is against ResNet-18 and MobileNetV2-0.75, both reported at 69.8% accuracy.
  • Analysis: Search-space optimization adapts resolution and width to SRAM and Flash limits, revealing trade-offs that are difficult to discover manually.More SRAM supports higher resolution, whereas more Flash supports larger width; increasing width can require reducing resolution under SRAM limits.
  • Analysis: Evolution search on TinyEngine outperforms random search by 1% in best accuracy, whereas CMSIS-NN produces worse results because memory inefficiency restricts model size.The plotted curves show average accuracy with min–max ranges across the search process.

5 Conclusion

MCUNet jointly designs TinyNAS and TinyEngine to enable deep learning on tiny hardware resources. It reaches record ImageNet accuracy on microcontrollers and accelerates wake-word inference, supporting always-on TinyML applications.

  • Conclusion: MCUNet jointly designs the neural architecture TinyNAS and inference library TinyEngine for deep learning on tiny hardware resources.The framework combines model and system design rather than optimizing them independently.
  • Conclusion: 70.7% ImageNet accuracy is achieved on off-the-shelf microcontrollers, while wake-word inference is accelerated by 2.4–3.4×.The conclusion presents these results as evidence that always-on tiny machine learning on IoT devices is feasible.

Statement of Broader Impacts

MCUNet is positioned as a way to bring affordable, local deep learning to low-cost microcontrollers. This could broaden access to AI applications while keeping sensor data on-device.

  • Broader Impacts: MCUNet is expected to enable tiny-scale deep learning on microcontrollers costing $5 or less, expanding access to AI applications.The paper frames this as a further reduction in inference hardware cost from workstations and mobile phones.
  • Broader Impacts: The large quantity and low cost of commercial microcontrollers could extend AI applications to healthcare, retail, agriculture, factories, and underserved areas.The paper cites 250B commercial microcontrollers and applications for people without reliable Internet or high-end hardware.
  • Broader Impacts: Local processing of raw sensor data can protect privacy because data need not be transmitted to the cloud.This consequence follows from processing data at the source on always-on low-power microcontrollers.

A Demo Video

The demo shows MCUNet improving visual wake-word accuracy and speed over MobileNetV1 on TF-Lite Micro, while reported frame rates include camera capture overhead.

  • 12% higher accuracy and 2.5× faster speed are achieved than MobileNetV1 on TF-Lite Micro.
  • The demo reports actual frame rate rather than inference-only throughput, including approximately 30ms of camera capture latency per frame.
  • Camera latency reduces inference from 10 FPS to 7.3 FPS in the reported demonstration.

B Profiled Model Architecture Details

The profiled models include a compact CIFAR network and scaled ImageNet models whose input resolutions and width multipliers are reduced to run on profiled libraries.

  • SmallCifar: SmallCifar uses 32 × 32 inputs, three convolution–max-pooling stages, and a final 1024×10 linear layer for CIFAR logits.
  • SmallCifar: SmallCifar is mainly used for MicroTVM comparison because most ImageNet models run out of memory with MicroTVM.
  • ImageNet Models: The ImageNet profiles use 64 × 64 inputs for MobileNetV2 and ProxylessNAS and 96 × 96 for MnasNet.
  • ImageNet Models: The profiled width multipliers are 0.35 for MobileNetV2, 0.3 for ProxylessNAS, and 0.2 for MnasNet.

C Design Cost

MCUNet reduces neural architecture design cost so models can be specialized for diverse IoT deployment constraints without expensive repeated searches.

  • Billions of IoT devices have drastically different constraints, requiring different search spaces and model specialization.
  • 300 GPU hours versus 40,000 GPU hours for MnasNet reduces the search cost by 133×.
  • MCUNet reduces CO2 emission from 11,345 lbs to 85 lbs per model.
  • The search-space optimization takes around 2 CPU hours, requires no training or testing, and can be reused across constraints.

D Resource-Constrained Model Specialization Details

MCUNet specializes resource-constrained models through a weight-sharing super network with variable kernels, expansion ratios, and stage depths, then samples sub-networks using fixed indexing rules.

  • The search space supports depth-wise kernels of 3/5/7, inverted-bottleneck expansion ratios of 3/4/6, and variable stage depths.
  • The largest sub-network is trained first, then its weights initialize a super network that supports different sub-networks through weight sharing.
  • Smaller kernels use central windows, smaller expansion ratios use the first channels, and smaller depths keep the first blocks while skipping the rest.

E Training&Testing Details

The super network is trained with SGD and momentum-based regularization, using cosine-annealed learning rates. Training duration varies by dataset size across ImageNet, Speech Commands, and Visual Wake Words.

  • SGD training uses momentum 0.9 and weight decay 5e-5.
  • Cosine annealing starts at a learning rate of 0.05 for every 256 samples.
  • The largest sub-network is trained for 150 epochs on ImageNet, 100 on Speech Commands, and 30 on Visual Wake Words.The durations differ because of dataset sizes.

F Changelog

The camera-ready version adds in-place depth-wise convolution to TinyEngine and updates reported results and memory statistics. It also adds per-block peak-memory comparisons for network-design insights.

  • The v2 camera-ready release adds in-place depth-wise convolution to TinyEngine.The technique is presented in Figure 7 and further reduces peak inference memory.
  • Updated TinyEngine results in Tables 2, 3, and 4 reflect its increased model capacity.
  • Figure 9 reports reduced peak SRAM statistics in the new version.
  • Figure 11 adds per-block peak-memory distribution comparisons for network-design insights.
Loading 2007.10319v2…