PaperlayerView exact revision on arXiv

Source-linked AI summary

MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning

Ji Lin, Wei-Ming Chen, Han Cai, Chuang Gan, Song Han

arXiv:2110.15352v2cs.CV

TL;DR

Tiny deep learning on MCUs is limited by tight SRAM budgets and imbalanced CNN memory usage. MCUNetV2 combines patch-based inference, receptive-field redistribution, and joint neural architecture–schedule search. It reduces peak memory and delivers strong classification and detection results on tiny devices.

  • Problem

    MCUs have tight SRAM budgets, while CNNs concentrate activation memory in early stages, restricting model capacity and input resolution.

  • Method

    MCUNetV2 combines patch-by-patch inference with receptive-field redistribution and joint optimization of network architecture and inference scheduling.

  • Results

    MCUNetV2 achieves 71.8% ImageNet accuracy on an MCU, over 90% VWW accuracy under 32kB SRAM, and 16.9% higher object-detection mAP than the existing state of the art.

  • Takeaways & Limitations

    The study addresses the memory bottleneck in tinyML and extends MCU vision applications beyond image classification.

  • Takeaways & Limitations

    The approach assumes CNN designs without highly complicated branching structures, which are generally less efficient for inference.

Abstract

from arXiv · show

Tiny deep learning on microcontroller units (MCUs) is challenging due to the limited memory size. We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs: the first several blocks have an order of magnitude larger memory usage than the rest of the network. To alleviate this issue, we propose a generic patch-by-patch inference scheduling, which operates only on a small spatial region of the feature map and significantly cuts down the peak memory. However, naive implementation brings overlapping patches and computation overhead. We further propose network redistribution to shift the receptive field and FLOPs to the later stage and reduce the computation overhead. Manually redistributing the receptive field is difficult. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2. Patch-based inference effectively reduces the peak memory usage of existing networks by 4-8x. Co-designed with neural networks, MCUNetV2 sets a record ImageNet accuracy on MCU (71.8%), and achieves >90% accuracy on the visual wake words dataset under only 32kB SRAM. MCUNetV2 also unblocks object detection on tiny devices, achieving 16.9% higher mAP on Pascal VOC compared to the state-of-the-art result. Our study largely addressed the memory bottleneck in tinyML and paved the way for various vision applications beyond image classification.

1 Introduction

MCU inference is constrained by tight SRAM budgets and memory-imbalanced CNNs, motivating patch-based scheduling and joint architecture–schedule design. MCUNetV2 reduces peak memory while improving tiny-device classification and detection results.

  • MCUs commonly provide less than 512kB SRAM, making peak activation memory a central deployment constraint for deep networks.
  • Existing efficiency methods target parameters and FLOPs rather than the activation-memory bottleneck, restricting model capacity and input resolution.
  • Patch-by-patch execution processes small feature-map regions during memory-intensive stages, reducing stored activations and enabling larger input resolutions.
  • 71.8% ImageNet accuracy is achieved on an MCU, while VWW exceeds 90% accuracy under 32kB SRAM and object detection reaches 16.9% higher mAP than the prior state of the art.

2 Understanding the Memory Bottleneck of Tiny Deep Learning

Efficient CNNs concentrate peak activation memory in early stages because spatial downsampling outpaces channel expansion. This imbalance forces whole-network scaling despite later stages fitting MCU memory limits.

  • MobileNetV2’s first five blocks exceed MCU memory constraints, with the third block using 8× more memory than the rest of the network.
  • Hierarchical CNNs reduce pixels 4× after each stage while channels increase only 2× or less, causing activation memory to decline toward later stages.
  • The early memory bottleneck limits executable model capacity and input resolution even though most later network blocks fit within 256kB.

3 MCUNetV2: Memory-Efficient Patch-based Inference

MCUNetV2 schedules early convolutions patch by patch to reduce activation storage, then uses receptive-field redistribution and joint search to control overlap overhead. The resulting design preserves performance while substantially lowering memory and computation costs.

  • Reducing Computation Overhead by Redistributing the Receptive Field: 1372kB to 172kB: per-patch MobileNetV2 inference cuts peak memory by 8×, while redistribution reduces overall computation overhead from 10% to 3%.
  • Breaking the Memory Bottleneck with Patch-based Inference: Patch-based inference stores one spatial patch rather than an entire feature map during the initial memory-intensive stage, while later layers run conventionally.
  • Breaking the Memory Bottleneck with Patch-based Inference: Overlapping input patches preserve outputs across convolutional receptive fields but introduce repeated computation, increasing network computation by 10–17% under optimal settings.
  • Reducing Computation Overhead by Redistributing the Receptive Field: Receptive-field redistribution reduces the initial-stage receptive field and increases the later-stage receptive field to reduce overlap without hurting performance.
  • Automated Joint Design: Joint search selects backbone and scheduling variables, including kernels, expansion ratios, depth, width, resolution, patch count, and patch-stage depth.

4 Experiments

Experiments show that patch-based inference substantially reduces memory on analytic profiles and MCUs, while architecture co-design improves overhead and task performance across classification and detection.

  • Reducing Peak Memory: On-device measurements show 4-6× lower peak SRAM, with latency overhead reducible to 4% through architecture design.The reported measurements use STM32F746, with width and resolution scaled to fit MCU memory.
  • Reducing Peak Memory: Analytic profiling shows 3.7-8.0× lower peak memory with 8-17% computation overhead across efficient CNN backbones.For MobileNetV2, receptive-field redistribution reduces computation overhead from 10% to 3% without hurting accuracy.
  • Tiny Image Classification: MCUNetV2 achieves a record 71.8% ImageNet accuracy on MCUs and improves prior results by 4.6% under 256kB SRAM.Under 512kB SRAM and 2MB Flash, the 71.8% result is reported as a new record under the same quantization policy.
  • Tiny Object Detection: On Pascal VOC, MCUNetV2-H7 improves mAP by 16.9% over the state of the art, while the M4 version improves mAP by 13.2% at 1.9× smaller peak SRAM.The M4 configuration uses similar computation to MCUNet but benefits from larger input resolution and smaller models.
  • Tiny Object Detection: For WIDER FACE, MCUNetV2-L achieves comparable mAP at 3.4× smaller peak SRAM and 1.6× smaller computation than RNNPool-Face-C.MCUNetV2-S also outperforms two referenced methods at 1.8× smaller peak memory.
  • Tiny Image Classification: MCUNetV2 exceeds 90% visual wake word accuracy under 32kB SRAM while achieving better accuracy at 4.0× smaller memory than per-layer inference.The expanded search space also improves the accuracy-versus-latency trade-off.
  • Ablation Studies: Ablations show that more patch-by-patch blocks and patches generally reduce memory but increase computation overhead, while redistribution mitigates the overhead.For MobileNetV2, the optimal index is n*=5, where the feature map is down-sampled by 8×.
  • Search and Co-design: The joint neural architecture and inference-scheduling search automatically discovers the reported designs without human expertise.This co-design is used to navigate the expanded architecture space under MCU constraints.

5 Related Work

Related work addresses efficient models through compression, architecture design, and automated search, while inference scheduling reduces memory through implementation-level computation reordering.

  • Tiny Deep Learning on Microcontrollers: Existing MCU deployment frameworks support per-layer inference, limiting executable model capacity and higher-resolution inputs under small memory budgets.The cited frameworks include TensorFlow Lite Micro, CMSIS-NN, TinyEngine, MicroTVM, and CMix-NN.
  • Efficient Neural Networks: Existing efficient deep-learning methods use pruning, quantization, neural architecture search, or directly designed efficient architectures.These approaches primarily target parameters, FLOPs, or model architecture.
  • Computation Scheduling/Re-ordering: Inference memory can be reduced through convolution-loop optimization, operator reordering, data swapping, or partial spatial computation across layers.System-only optimization can introduce repeated computation or complicated designs.

6 Conclusion

MCUNetV2 combines patch-based inference with neural architecture and scheduling optimization to reduce memory and improve MCU vision performance beyond classification.

  • Conclusion: Patch-based inference reduces tinyML memory usage by up to 8× and expands the design space for IoT vision applications.The method is jointly optimized with neural architecture and inference scheduling.
  • Conclusion: MCUNetV2 achieves record MCU results in image classification and object detection while substantially improving Pascal VOC mAP.The conclusion identifies vision applications beyond classification as the broader supported scope.

A Flow Chart of Contributions

The contribution flow chart organizes MCUNetV2 around diagnosing imbalanced memory, reducing patch overlap overhead, and jointly optimizing architecture with inference scheduling.

  • Analysis and Scheduling: The work first analyzes imbalanced memory distribution and proposes patch-based inference scheduling to reduce peak memory.The scheduling operates as a central response to the identified memory imbalance.
  • Network Redistribution: It redistributes receptive fields to reduce computation overhead caused by overlapping patches.This redistribution complements patch-based execution.
  • Joint Optimization: It jointly optimizes neural architecture and inference scheduling in the same search loop.The flow chart presents joint optimization as the final co-design step.

B Experimental Details

The experiments define a joint search space for neural architectures and inference scheduling, then train, validate, search, fine-tune, and quantize candidate networks.

  • Search space: The neural architecture search space varies kernel size, expansion ratio, stage depth, block width, and input resolution.The choices are k ∈ {3, 5, 7}, e ∈ {3, 4, 6}, d ∈ {2, 3, 4}, w ∈ {0.5, 0.75, 1.0}, and r ∈ {96, 128, 160, 192, 224, 256}.
  • Search space: Inference scheduling additionally searches the number of spatial patches and the number of layers using patch-based inference.The input is split into p × p patches, with p ∈ {1, 2, 3, 4}; the remaining layers use per-layer inference.
  • Training and validation: Super-network training uses a randomly split training set, SGD, batch size 1024, initial learning rate 0.2, weight decay 4e-5, and cosine decay.Training lasts 150 epochs for ImageNet and 30 epochs for VWW, with validation sets of 10,000 and 5,000 examples respectively.
  • Search and post-processing: Selected architectures are fine-tuned for 10 epochs at one-tenth the initial learning rate, batch-normalization statistics are recalibrated, and int8 quantization uses 10 epochs of quantization-aware training.Validation uses super-network weights by indexing and recalibrates batch normalization with 20 batches of size 64.
  • Search and post-processing: Evolutionary search begins with 100 feasible sub-networks and generates later populations through top-20 selection, crossover, and mutation.Crossover and mutation each produce 50 candidates, using a mutation rate of 0.1.

C Memory Distributions of Efficient Models

Efficient CNNs exhibit highly imbalanced memory distributions, while patch-based inference substantially reduces their peak memory usage across models.

  • Memory distribution: All evaluated MnasNet, FBNet, and MCUNet-320kB models have highly imbalanced memory distributions.The imbalance persists even in MCUNet, which is specialized for memory-constrained settings.
  • Patch-based inference: 3.5-6.1× peak memory reduction is achieved by enabling patch-based inference on these efficient models.Figure 11 reports this reduction for MnasNet, FBNet, and MCUNet-320kB.

D Ablation Study on Neural Architecture Search

The study expands the architecture search space to jointly vary width and resolution, and reports improved computation-accuracy trade-offs for tiny networks across budgets.

  • Search-space design: Including width multiplier w and input resolution r lets neural architecture search globally scale networks to fit tight resource budgets.This merges scaling and architecture search rather than choosing w and r in a separate preliminary step.
  • Ablation results: The extended MobileNetV3 search space consistently outperforms existing techniques in computation-accuracy trade-offs under different computation budgets.The comparison evaluates tiny networks after extending the space to support different input resolutions and widths.
  • Ablation results: Under tiny computation settings (<50M), the NAS method reports the best accuracy at different budgets using models derived from one shared search space.Table 6 notes that starred models were re-measured for MACs and parameters with the authors’ profiler.

E Qualitative Results of Face Detection

On WIDER FACE, MCUNetV2-S outperforms RNNPool-Face-Quant in quantitative and qualitative evaluations, including accuracy, memory, pose robustness, and background false positives.

  • Quantitative results: MCUNetV2-S achieves better mAP at 1.3× smaller peak memory than RNNPool-Face-Quant on the WIDER FACE validation set.Table 7 reports the same comparison while following the prior work’s peak-memory calculation.
  • Qualitative results: MCUNetV2-S is more robust to poses and background false positives in qualitative WIDER FACE validation results.Figure 12 filters predictions with confidence threshold 0.5.
Loading 2110.15352v2…