PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Microcompression: Structured Pruning and Bit-packed Quantization for Microcontrollers

Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye Eigbe

arXiv:2609.05081v1cs.LG

TL;DR

Bare-metal microcontrollers impose severe memory, power, and compute constraints that make unoptimized neural-network deployment infeasible. DMC combines structured pruning, quantization-aware training, and fixed-length bit-packing into a dependency-free, hardware-aware pipeline, achieving up to 55.8× compression and enabling documented standard-CNN deployment on an Arduino Uno.

  • Problem

    Bare-metal microcontrollers have severe memory, power, and compute constraints, creating a gap between modern neural-network requirements and deployable hardware.

    Source: §1 · p. 1

  • Method

    DMC integrates structured pruning, quantization-aware training, and hardware-aware fixed-length bit-packing to generate portable, dependency-free C inference code.

    Source: §1 · p. 1, §2 · p. 2, §2.3 · p. 2

  • Results

    DMC achieves up to 55.8× compression for LeNet-5, while benchmarks against TFLite show a smaller footprint and greater portability.

    Source: §4 · p. 4, §3.2.1 · p. 4

  • Takeaways & Limitations

    DMC enables neural-network deployment on bare-metal microcontrollers, including the first documented standard-CNN deployment on an Arduino Uno with 98.17% accuracy.

    Source: §4 · p. 4, §3.2.1 · p. 4

    Baseline and Ultra models exceed ATmega resources, so their metrics were measured on the RP2040 with a software-emulated FPU for fair comparison.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

This paper introduces Deep Microcompression (DMC), a hardware-aware pipeline for deep learning inference on bare-metal microcontrollers. DMC integrates structured pruning, quantization-aware training, and fixed-length bit-packing to achieve a 55.8$\times$ weight compression ratio on LeNet-5 (98.77\% accuracy), generating a dependency-free C library with deterministic latency. On the RP2040 (Cortex-M0+), DMC reduces binary size by 3$\times$ versus TensorFlow Lite while matching its accuracy. Critically, DMC enables the first documented deployment of a standard CNN on the ATmega328P, a device constrained to 2KB SRAM, previously considered infeasible for CNN inference.

1. Introduction

DMC targets the mismatch between neural-network resource demands and bare-metal MCU constraints by combining hardware-aware compression with dependency-free deployment. It is motivated by the need for deterministic, portable inference on devices where variable-length coding and runtime frameworks are impractical.

  • ATmega328P provides just 2 KB of RAM and 32 KB of flash, making full-scale, unoptimized models infeasible on the device.Source: §1 · p. 1
  • Variable-length coding and unstructured pruning introduce irregular access, decoding overhead, and non-deterministic latency that limit bare-metal deployment.Source: §1 · p. 1
  • DMC integrates structured pruning, quantization-aware training, and hardware-aware bit-packing into a framework-independent pipeline that generates minimal C code.Source: §1 · p. 1
  • Fixed-length bit-packing replaces complex Huffman coding with simple bitwise decompression on standard microcontrollers.Source: §1 · p. 1
  • DMC is positioned for legacy 8-bit deployments where low-cost hardware, battery operation, and offline inference are important.Source: §1 · p. 2

2. The Deep Microcompression (DMC) Method

DMC combines structured pruning, quantization-aware training, and compile-time bit-packing to produce integer-only inference code tailored to microcontroller constraints. Its fixed-length representation precomputes tensor layouts and enables constant-time bitwise extraction without runtime decoding machinery.

  • DMC avoids expensive computation, complex data structures, unnecessary memory movement, and specialized decoders through a microcontroller-centric design.Source: §2 · p. 2
  • The development pipeline applies structured pruning, quantization-aware training, and bit-packing before runtime integer inference with low-cost activations.Source: §2 · p. 2
  • Structured channel pruning removes low-magnitude filters and neurons while preserving dense contiguous layouts for standard kernels and deterministic execution.Source: §2.1 · p. 2
  • Quantization-aware training adapts the network to low precision, while static quantization precomputes scaling factors and removes runtime floating-point operations.Source: §2.2 · p. 2

The Hardware-Aware Low-Level Optimization

  • DMC shifts bit-offset computation to compilation, generating kernels whose fixed-length packed weights can be decompressed in O(1) time.Source: §2.3 · p. 2
  • Compile-time constants and bitwise shifts and masks replace runtime division, modulo, lookup tables, and variable-length decoding during unpacking.Source: §2.3.1 · p. 2, §2.3.2 · pp. 2–3

3. Experimental Results

The experiments use LeNet-5 on MNIST to test compression, feasibility below 2 KB SRAM, and deployment efficiency against an uncompressed baseline. Results show strong weight compression and a documented CNN deployment on the ATmega328P, while activation memory remains a key constraint.

  • LeNet-5 is used as a worst-case stress test because its early convolutional activations exceed common 2 KB SRAM limits by nearly 3×.Source: §3 · p. 3
  • The evaluation measures model size, peak SRAM, MACs, BOPs, latency, energy, and Top-1 accuracy against an uncompressed 32-bit baseline.Source: §3.1 · pp. 3–4

Compression Performance

  • DMC-Ultra achieved a 55.8× parameter-size reduction, from 145KB to 2.66KB, with 98.77% accuracy and a 0.65% accuracy drop.Source: §3.2.1 · p. 4
  • DMC-Ultra’s 5.74 KB peak activation memory exceeds the ATmega328P’s 2KB SRAM limit, showing that weight compression alone is insufficient.Source: §3.2.1 · p. 4
  • DMC-Tiny reduces peak activation workspace to 1.27 KB, leaving approximately 700 bytes for the system stack and drivers.Source: §3.2.1 · p. 4
  • DMC-Tiny enabled the first documented deployment of a standard CNN on an Arduino Uno with 98.17% accuracy.Source: §3.2.1 · p. 4
  • Prior implementations trade off accuracy, latency, or generality: LogNNet reports 84% accuracy and 7.11s latency, while Gural reports 99.11% accuracy and 684ms latency with domain-specific optimization.Source: §3.2.2 · p. 4

4. Conclusion

DMC enables neural-network deployment on bare-metal microcontrollers without heavy runtime dependencies, combining compression techniques to reduce model size and improve portability. Its reported compression reaches 55.8× for LeNet-5, while supporting migration from 32-bit to energy-efficient 8-bit devices.

  • 55.8× compression is achieved for LeNet-5.Source: §4 · p. 4
  • DMC integrates structured pruning, quantization-aware training, and hardware-aware bit-packing.Source: §4 · p. 4
  • DMC benchmarks highlight improved footprint and portability, enabling workload migration from 32-bit to energy-efficient 8-bit devices.Source: §4 · p. 4

A. Preliminaries and Related Work

Model compression reduces model footprint and accelerates inference, providing the foundation for the DMC pipeline. The section situates these techniques within TinyML applications.

A.1. Pruning

Pruning removes redundant neural-network parameters, with structured pruning retaining dense submodels that are more hardware-friendly. This creates a practical balance between compression and deployment efficiency.

  • Pruning removes redundant connections or neurons from a neural network.Source: §A.1 · p. 7
  • Structured pruning removes parameter groups such as channels, filters, or layers, producing a smaller dense model.Source: §A.1 · p. 7
  • Structured pruning offers a balance between compression and hardware efficiency through reduced operations.Source: §A.1 · p. 7
  • The resulting structured-pruned network can run on platforms without specialized sparse-matrix libraries.Source: §A.1 · p. 7

A.2. Quantization

Quantization reduces the precision and storage requirements of neural-network parameters, while static quantization precomputes scaling factors for integer arithmetic. Lower bitwidths affect model size and performance.

  • 32-bit floating-point storage has traditionally been used because of precision requirements for convergence and accuracy.Source: §A.2 · p. 7
  • Quantization can store parameters as codebook indices, but lookup-based approaches may retain floating-point computation.Source: §A.2 · p. 7
  • Static quantization precomputes scaling factors to enable pure integer arithmetic without dynamic runtime overhead.Source: §A.2 · p. 7

A.3. Bit-Packing and Low-Level Optimizations

DMC uses fixed-length bit-packing to store low-bitwidth weights in standard memory units and decode them with elementary integer operations. This design avoids variable-length decoding and supports portable, constant-time execution on generic microcontrollers.

  • Unlike bit-serial or Huffman-based approaches, fixed-length packing uses generic integer instructions and avoids specialized kernels or serial decoding.The resulting structure balances compression with execution efficiency on standard microcontroller architectures.Source: §A.3 · pp. 7–8
  • Fixed-length bit-packing maps multiple low-precision weights directly into 8-bit bytes or 32-bit words.For example, four 2-bit weights can occupy one byte.Source: §A.3 · p. 7
  • DMC’s algorithmic design targets portable storage and runtime decoding across microcontroller architectures.The supplied algorithm descriptions specify both compile-time packing and runtime unpacking stages.Source: §A.3 · p. 8
  • Runtime unpacking computes a weight’s byte location and bit offset, then extracts it with shifts and masks.Signed values are sign-extended after extraction.Source: §A.3 · p. 8
  • Compile-time packing combines quantized parameters into packed memory units using fixed-width shifts and masks.The procedure accepts at most 8 ÷ b values for bitwidth b.Source: §A.3 · p. 8

A.4. Impact of Toolchain Optimization

Compiler optimization strongly affects DMC’s deployability on constrained microcontrollers. Size optimization best supports the ATmega328P, while maximum optimization favors latency and energy efficiency.

  • 17.1KB: the unoptimized avr-gcc -O0 build can exceed the available flash memory of 8-bit devices.The result demonstrates that compiler settings directly affect whether the binary fits the target device.Source: §A.4 · p. 8
  • 38.7%: enabling -Os reduced the instruction footprint and produced the best viable configuration for the ATmega328P.The -Os flag also minimized stack depth by 23%.Source: §A.4 · p. 8
  • -O3 gives the best latency and lower energy consumption per inference among the compared optimization levels.This favors computational efficiency rather than the smallest binary.Source: §A.4 · p. 8
  • ≈6KB: static RAM usage remains largely constant across optimization levels and is dominated by the 5.74KB activation workspace.Stack usage, rather than global allocation, varies substantially with compiler optimization.Source: §A.4 · p. 8
  • Peak static-plus-stack memory below 2048B allows the DMC-Tiny model to stay within the ATmega328P’s strict 2KB SRAM budget.The -Os configuration achieves this through a 23% reduction in stack depth.Source: §A.4 · p. 8

A.5. Impact of Packing Operation

Four-bit packing reduces memory usage but imposes substantial software unpacking overhead. The resulting trade-off favors storage efficiency at the cost of latency and energy.

  • 56%: 4-bit packing reduces weight storage relative to the byte-aligned 8-bit variant.Static RAM also decreases by 48%, supporting deployment on more constrained devices.Source: §A.5 · p. 8
  • More than 20 × #MAC additional bit operations arise from on-the-fly decoding in the 4W4A configuration.The overhead is attributed to software-based unpacking.Source: §A.5 · p. 8
  • 46%: inference latency increases with 4-bit packing compared with the 8W8A variant.Energy consumption rises proportionally with the added decoding workload.Source: §A.5 · p. 8
  • Software unpacking is a major bottleneck, motivating more efficient packing methods and specialized hardware support such as barrel shifters.The paper identifies this as a direction for future low-power MCU designs.Source: §A.5 · p. 8

A.6. Benchmarking Against TFLite for Microcontrollers

DMC trades some platform-specific speed for portability and a smaller binary footprint than TFLite Micro. On RP2040, its generic C implementation runs on architectures without CMSIS-NN support while reducing binary size by approximately 3×.

  • DMC generates generic scalar C code, whereas TFLite Micro benefits from CMSIS-NN optimized kernels for superior inference speed.The portability advantage comes with higher latency than platform-tuned libraries.Source: §A.6 · p. 8
  • ≈3×: DMC reduces total binary size relative to TFLite Micro, reaching 168.5KB in the reported comparison.The reduction removes TFLite’s runtime interpreter and library overhead.Source: §A.6 · p. 8
  • 32KB: the ATmega328P flash capacity makes DMC’s binary-footprint advantage relevant to ultra-constrained devices.The comparison frames footprint as a deployment constraint rather than only a storage statistic.Source: §A.6 · p. 8
  • DMC explicitly trades ARM-specific speed for architectural portability, minimal footprint, and zero dependencies.This trade-off is the central comparison between generic DMC code and hardware-specific acceleration.Source: §A.6 · p. 8
  • The benchmark compares DMC with TFLite Micro using LeNet-5 on the RP2040/Pico.Table 3 is the named benchmark context for the reported latency and binary-size comparison.Source: §A.6 · p. 8, Table 3 · p. 9

Related papers

Loading 2609.05081v1…