PaperlayerView exact revision on arXiv

Source-linked AI summary

DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs

Alessio Burrello, Angelo Garofalo, Nazareno Bruschi, Giuseppe Tagliavini, Davide Rossi, Francesco Conti

arXiv:2008.07127v3cs.DCcs.ARcs.NE

TL;DR

Low-cost MCUs have limited on-chip memory and often rely on scratchpads and explicit DMA transfers, while existing automatic end-to-end deployment options can be slow or proprietary. DORY uses constraint-based tiling and generated C code to manage deployment; on GAP-8, it reports higher energy efficiency and performance than STM32H743 and improvement over GWT's proprietary tool.

  • Problem

    Low-cost MCU deployment must manage DNN memory traffic across limited memory hierarchies, while automatic end-to-end solutions are described as slow interpreters or proprietary code generators.

    Source: §1 · pp. 1–2

  • Method

    DORY formulates memory tiling as a Constraint Programming problem and generates ANSI C to orchestrate memory transfers and computation.

    Source: §1 · p. 2

  • Results

    12.6× higher energy efficiency and 7.1× higher performance than STM32H743, with up to 26.6% end-to-end inference improvement over GWT's proprietary tool.

    Source: §7 · p. 13

  • Takeaways & Limitations

    DORY's framework, optimized backend kernels, and PULP heuristics are released as open-source software.

    Source: §1 · p. 2

    The presented deployment targets 8-bit quantized networks and assumes zero offsets for linear inputs and quantization/activation outputs.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

The deployment of Deep Neural Networks (DNNs) on end-nodes at the extreme edge of the Internet-of-Things is a critical enabler to support pervasive Deep Learning-enhanced applications. Low-Cost MCU-based end-nodes have limited on-chip memory and often replace caches with scratchpads, to reduce area overheads and increase energy efficiency -- requiring explicit DMA-based memory transfers between different levels of the memory hierarchy. Mapping modern DNNs on these systems requires aggressive topology-dependent tiling and double-buffering. In this work, we propose DORY (Deployment Oriented to memoRY) - an automatic tool to deploy DNNs on low cost MCUs with typically less than 1MB of on-chip SRAM memory. DORY abstracts tiling as a Constraint Programming (CP) problem: it maximizes L1 memory utilization under the topological constraints imposed by each DNN layer. Then, it generates ANSI C code to orchestrate off- and on-chip transfers and computation phases. Furthermore, to maximize speed, DORY augments the CP formulation with heuristics promoting performance-effective tile sizes. As a case study for DORY, we target GreenWaves Technologies GAP8, one of the most advanced parallel ultra-low power MCU-class devices on the market. On this device, DORY achieves up to 2.5x better MAC/cycle than the GreenWaves proprietary software solution and 18.1x better than the state-of-the-art result on an STM32-F746 MCU on single layers. Using our tool, GAP-8 can perform end-to-end inference of a 1.0-MobileNet-128 network consuming just 63 pJ/MAC on average @ 4.3 fps - 15.4x better than an STM32-F746. We release all our developments - the DORY framework, the optimized backend kernels, and the related heuristics - as open-source software.

1 INTRODUCTION

DORY addresses the memory and data-movement constraints of deploying DNNs on low-cost MCUs with an automatic tiling and code-generation framework. On GAP-8, it reports performance and energy gains over competing MCU tools and releases its framework, kernels, and heuristics as open source.

1 INTRODUCTION

  • MCU deployment must fit DNNs within typically at most 1 MB of on-chip memory, limited compute, and a 100–200 mW peak power envelope.The paper identifies scarce memory as a fundamental limit on embedded DNN system performance.Source: §1 · p. 1
  • Existing end-to-end MCU deployment approaches are described as either slow and inefficient interpreters or proprietary code-generation frameworks.DORY targets memory-starved end-nodes, particularly those using the PULP paradigm, and GAP-8 is its main case study.Source: §1 · p. 2
  • 18.1× better MAC/cycle than STM32-H743 on single-layer execution is reported for DORY; it also achieves up to 2.5× better throughput than GWT AutoTiler on GAP-8.For end-to-end 8-bit networks, DORY reports up to 8.00 MACs/cycle and a 13.2× improvement over the same networks running on STM32-H743 with ST X-CUBE-AI.Source: §1 · p. 2
  • DORY uses Constraint Programming to match memory-hierarchy constraints with DNN tensor geometry, then generates ANSI C for transfers and computation.Its contributions include PULP performance heuristics, triple-buffered DMA orchestration, a bidirectional stack for residual connections, and an optimized depthwise backend.Source: §1 · p. 2
  • 11.9 mJ/classification at 4.3 fps is reported for image recognition on an extreme edge-node.The introduction also reports 2.5× faster average single-core execution on GAP-8 than on the M7 in cycles/cycles.Source: §1 · p. 2
  • DORY, its optimized backend kernels, and the PULP heuristics are released as open-source software.The release is presented as supporting research on real-world deeply embedded DNN applications.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses DNN efficiency through topology reduction, quantization, specialized software and instruction sets, and data-flow scheduling across computing platforms. For MCUs, DORY is positioned as an open-source framework that tackles memory hierarchy management, tiling, loop ordering, and residual-layer fragmentation.

2 RELATED WORK

  • Deployment efficiency also depends on reducing DNN redundancy through topology changes or search and reducing operation and parameter costs through quantization.These approaches aim to limit quality-of-results costs while shrinking networks or using lower-precision computation.Source: DNN algorithm minimization · p. 2
  • DNN software libraries use HWC or CHW layouts to balance channel-wise and spatial data reuse, while ISA specialization supports low-bitwidth convolution computation.CHW favors spatial reuse; HWC favors channel-wise reuse and can require an im2col buffer to exploit spatial reuse partially.Source: Optimized software & ISA for DNN… · pp. 2–3

Memory hierarchy management

  • Data-flow scheduling and tiling work spans high-performance computing, DNN accelerators, and embedded systems, with accelerator research targeting data movement and processing-element utilization.Table 1 surveys methods across supercomputing, ASIC accelerators, and tiny MCUs.Source: Memory hierarchy management · p. 3, Table 1 · p. 3
  • MCU scheduling must account for small memories and register files, as register limits can add load-store operations and make loop sizing important.Embedded platforms combine computing and I/O tasks with more complex memory hierarchies and DMA-based data marshaling.Source: Memory hierarchy management · p. 3, DNN-oriented microcontrollers and… · p. 3
  • Industry tools X-CUBE-AI and AutoTiler provide MCU DNN deployment, but are proprietary and vendor-locked; X-CUBE-AI relies on caches, while AutoTiler generates GAP-8 tiling and transfer code.The paper identifies these as the two most powerful state-of-the-art deployment tools available at the time.Source: DNN-oriented microcontrollers and… · p. 4
  • DORY is presented as the first open-source framework to directly address MCU memory hierarchy management, including tiling, layer-specific loop ordering, and residual-layer data fragmentation.The paper says DORY consistently outperforms the compared competitors on its proposed benchmarks.Source: DNN-oriented microcontrollers and… · p. 4

3 BACKGROUND

The background describes integer-domain QNN operations and the GAP-8 platform used to execute them. It then introduces backend execution phases and tensor tiling across memory levels.

3.1 Quantized Neural Networks

  • QNN layers apply linear operations, optional batch normalization, and quantization/activation; accumulators use higher precision before being requantized.The linear dot product uses weights and inputs, while quantization/activation applies a nonlinearity and reduces accumulator bitwidth.Source: §3.1 · p. 4

3.2 Parallel Ultra-Low-Power computing paradigm

  • GAP-8 combines an I/O core with an eight-core cluster, 64 kB L1 and 512 kB L2 scratchpads, and external L3 storage.Its cluster cores use SIMD instructions for 8-bit operands; DMA transfers data between L1 and L2.Source: §3.2 · p. 5

3.3 QNN Execution Model on GAP-8

  • PULP-NN executes QNN layers through Im2Col, MatMul, and Norm/Qnt phases, with Im2Col omitted for 1×1 convolutions.The MatMul inner loop computes 32 MACs per iteration and can produce two adjacent output pixels in parallel.Source: §3.3 · p. 5

3.4 QNN Tensor Tiling

  • Tensor tiling fragments inputs, outputs, and weights to fit memory levels while preserving tensor relationships and overlapping DMA transfers with computation.The tiling strategy uses double buffering to load the next tile while the current tile is being computed.Source: §3.4 · p. 5

4 DORY: DEPLOYMENT ORIENTED TO MEMORY

DORY supports weight and activation tiling across L3-L2 and L2-L1 and generates code for network execution. Its offline workflow decodes the ONNX graph, analyzes layers, and builds network-level memory and execution orchestration.

4 DORY: DEPLOYMENT ORIENTED TO MEMORY

  • DORY supports L3-L2 and L2-L1 tiling for weights and activations, with weight tiling needed for many non-trivial networks.Activation tiling is typically needed for networks processing high-resolution images with large spatial dimensions.Source: §4 · p. 5
  • The generated L2-L1 loop asynchronously loads next-tile weights and inputs while computing on the current tile, then transfers outputs back to L2.The loop uses double buffering and cluster DMA calls to overlap transfers with computation.Source: §4 · p. 6
  • DORY's layer analyzer generates tiling and backend-call code; its network parser sizes hierarchical buffers and emits an ANSI C file for the full DNN.The network parser combines information from across the network to infer buffer sizes.Source: §4 · p. 6

4.1 ONNX Decoder

DORY decodes an already quantized ONNX graph and reorganizes its nodes into layers, each representing a canonical sequence of operations.

4.1 ONNX Decoder

  • Each DORY layer comprises a linear/add/pooling operation, optional batch normalization, and quantization/activation; temporary data is represented as 32-bit signed integers.Layer inputs, outputs, and weights are quantized.Source: §4.1 · p. 6

4.2 Layer Analyzer

The Layer Analyzer solves each layer’s memory tiling, applies backend-specific constraints and heuristics, then generates code to coordinate transfers and computation. Its tiling process handles both L3-to-L2 and L2-to-L1 memory levels.

4.2 Layer Analyzer

  • The Layer Analyzer treats layers independently, combining a platform-agnostic tiling solver, backend-specific heuristics and constraints, and a software-cache generator.Its first optimization phase uses weight dimensions from the preceding layer.Source: §4.2 · p. 6

4.2.1 DORY Tiling Solver

  • DORY solves L3-L2 tiling before L2-L1 tiling, using a five-stage search that avoids output tiling when possible because it doubles transfers.The stages progressively try input, weight, and output tiling; output tiling is a last resort when other choices cannot fit L2.Source: §4.2.1 · p. 6
  • The CP solver maximizes L1 buffer utilization while satisfying hardware capacity and tensor-geometry constraints.Its capacity constraint includes backend workspace such as PULP-NN im2col storage.Source: §4.2.1 · p. 7

4.2.2 Target-specific Heuristics & Constraints

  • PULP-NN heuristics favor output-channel reuse, height tiles balanced across eight cores, and tile dimensions aligned with its 4x2 matrix multiplication.Reusing the im2col buffer across output pixels also reduces the overhead of constructing it.Source: §4.2.2 · p. 7
  • DORY restricts channel and filter-spatial tiling to avoid storing and transferring 32-bit partial accumulations.The authors report that the excluded solutions are suboptimal.Source: §4.2.2 · p. 7

4.2.3 DORY SW-cache Generator

  • The SW-cache generator automatically emits C code that schedules asynchronous, pipelined transfers and backend-kernel calls with double-buffering at both memory interfaces.The layer loop also manages tile padding, stride, and overlap; the approach can be generalized to three-level memory hierarchies.Source: §4.2.3 · pp. 7–8
  • For depthwise execution, Fig. 4 shows an im2col buffer built from one channel of CHW activations, with quantized outputs stored using the PULP-NN model.The caption refers readers to Table 2 for the buffer naming scheme.Source: Figure 4 · p. 8

4.3 DORY Hybrid Model

DORY retains HWC data layout for general convolutions and uses a hybrid layout for depthwise layers. Its depthwise execution reorders activations on L1 to exploit spatial reuse and computes different channels in parallel.

4.3 DORY Hybrid Model

  • General HWC convolutions exploit channel reuse, but depthwise convolutions have no such reuse because each input channel maps independently to one output channel.Depthwise layers are described as typically accounting for 10% or less of overall operations, motivating a hybrid rather than universal layout change.Source: §4.3 · p. 8
  • DORY keeps activations in HWC in L2/L3 but reorders them to CHW on L1 for depthwise convolutions, enabling reuse along spatially adjacent pixels.The im2col buffer is a contiguous vertical stripe of width Kw, and the inner loop computes one output pixel per iteration.Source: §4.3 · p. 8
  • Multiple cores process different depthwise channels in parallel, reducing memory contention while supporting varied filter shapes and strides.The outputs are quantized and stored in HWC layout for transfer back to L2.Source: §4.3 · p. 8

4.4 Network Parser

The Network Parser builds a callable layer graph and generates network-level execution code that transfers weights, invokes layers, and updates buffer offsets. Its bidirectional buffer stack manages layer and residual buffers with statically determined capacity.

4.4 Network Parser

  • DORY builds a network graph from the layer tiling results and generates a callable execution loop that transfers following-layer weights, runs layers, and updates buffer offsets.The generated function accepts externally allocated L1 and L2 memory buffers and their maximum sizes.Source: §4.4 · p. 8

4.4.1 Buffer allocation stack & Residual connections

  • The bidirectional allocation stack uses two LIFO stacks growing in opposite directions to reduce fragmentation across differently sized layers.Without residual connections, its size is bounded by that of two concurrent same-direction stacks.Source: §4.4.1 · p. 9
  • For residual networks, the allocator tracks activation lifetimes, prioritizes the branch with the most nodes, and computes stack size offline from adjacent-layer buffers and live residuals.Activation buffers are deallocated when their lifetime counters reach zero.Source: §4.4.1 · p. 9

5 RESULTS

DORY is evaluated on GAP-8 for single layers and end-to-end networks, with comparisons to MCU deployment frameworks and analysis of execution behavior. The reported results include advantages over AutoTiler and STM32H7 baselines, with performance varying by layer type.

5.1 Single layer performance & SoA comparison

  • Depth-wise layers spend much of backend time in SIMD-leftover loops, while point-wise layers spend nearly all execution time in MatMul's innermost loop.DORY's tiling overhead does not impair compute-bound point-wise layers, where computation can overlap memory transfers; the overhead is more pronounced for smaller depth-wise layers.Source: §5.1 · p. 9, Figure 5 · p. 9
  • 2.9× to 229.6×: DORY-generated GAP-8 layers outperform TFLite Micro and X-CUBE-AI in MAC/cycle, while DORY is 1.6× faster than AutoTiler for point-wise and 1.9× slower for depth-wise convolutions.The comparisons use different platforms for TFLite Micro and X-CUBE-AI versus DORY; AutoTiler targets the same GAP-8 platform.Source: §5.1 · p. 10, Table 3 · p. 10

5.2 End-to-end network performance

  • End-to-end GAP-8 runs of the benchmark networks were checked against a PyTorch-based bit-accurate QNN model for intermediate and final results.This validates the functionality of DORY and the PULP-NN backend for the reported network runs.Source: §5.2 · p. 10

5.2.1 End-to-end MobileNet-v1 and -v2 & SoA comparison

  • 20.5%: DORY performs better on average than GWT AutoTiler on GAP-8; with an identical network including batch-normalization folding, the gain reaches 26.6%.DORY keeps batch normalization explicit by default, while AutoTiler folds it into the convolution transformation.Source: §5.2.1 · p. 10
  • 7.1×: DORY achieves up to this throughput versus STM32H7 (1.78 vs. 0.25 GMAC/s), and up to 12.6× better energy efficiency at different operating points.The comparison covers MobileNet-v1 variants and uses different frequencies and power consumption on the two platforms.Source: §5.2.1 · p. 10, Table 4 · p. 11

6 ABLATION STUDY

The ablation study tests how tiling heuristics, depth-wise kernels, clock settings, and memory sizes affect DORY's deployment performance. Its results show that larger tiles alone do not guarantee faster execution, and that memory and frequency choices involve trade-offs.

6.1 Single tile performance

  • 29%, 38%, and 28%: violating MATMUL_W/CH, HIDE_IM2COL, and PAR_BALANCE respectively causes these maximum performance losses in the example layer.The losses are cumulative because the heuristics target different sections of the PULP-NN kernel.Source: §6.1 · p. 11
  • 80.6% lower: maximizing tile size alone yields 2.78 MAC/cycles, versus 14.37 MAC/cycles with the reported βi values; border tiles can account for up to 50% of workload.The example uses a 28 kB L1 constraint and executes both the main and border tiles 32 times.Source: §6.1 · p. 11

6.2 Hybrid optimization for Depthwise layers

  • 4.3 FPS: the optimized DORY kernel library raises MobileNet-v1-1.0 at 128×128 from 3.0 FPS with HWC kernels, while reaching 0.54× Greenwaves' depth-wise performance.The library is 2.5× faster than CHW on classical convolutions; its depth-wise kernel is 1.5×/2.0× faster than the HWC one.Source: §6.2 · pp. 11–12

6.3 Voltage and frequency scaling

  • 93.9 ms at 24.6 mJ per frame and 244 ms at 12.5 mJ per frame are the reported minimum-latency and minimum-energy operating points, respectively.Increasing fabric-controller frequency improves performance by speeding L3-L2 transfers, while raising average dynamic power.Source: §6.3 · p. 12, Figure 10 · p. 12

6.4 Memory hierarchy sizing

  • L1 is swept from 22 to 400 kB and L2 from 256 kB to 4 MB to explore memory-hierarchy sizing on an FPGA-based PULP emulator.The 22 kB lower L1 bound supports the smallest tile and its im2col buffer; the 400 kB upper bound is where no further performance improvement had been observed.Source: §6.4 · p. 12

L1 [KB]

The memory-sizing analysis reports that a smaller L1/L2 configuration can approach the performance of the largest tested configuration. The single-core comparison also distinguishes software caching from architectural performance on GAP-8 and STM32H7.

L1 [KB]

  • 8%: an 80 kB L1 and 384 kB L2 configuration reaches 9.74 MAC/cycles versus 10.57 MAC/cycles with 400 kB L1 and 4 MB L2, reducing total L1/L2 sizes by 80%/91%.The larger configuration eliminates external-memory transfers and L2-L1 tiling for the targeted network.Source: Figure 11 · p. 12, L1 [KB] · p. 12
  • Up to 3.2 FPS is lost when L2 is halved from 512 kB to 256 kB; reducing L1 from 30 kB to 28 kB can cause a 20% performance drop across 13 layers.Reducing L1 from 70 kB to 22 kB generally reduces performance through smaller tiles and more iterations; increasing it from 70 kB to 400 kB brings minimal gain.Source: L1 [KB] · p. 12

6.5 Single core performance on different architectures

  • 9.1 ± 2.1% and 3.9 ± 3.8%: DORY software caching on STM32H7 DTCM is faster on average than D-Cache for depth-wise and normal convolution layers, respectively.With D-Cache disabled, STM32H7 performance degrades by 58.5 ± 5.5% over the benchmark layers.Source: Figure 12 · p. 13, §6.5 · p. 13
  • 2.5 ± 0.9×: single-core GAP-8 is faster than the STM32H7 baseline in cycle/cycle, with multi-core execution disabled.The reported speedup is attributed mainly to the more specialized architecture, particularly DSP extensions used by PULP-NN.Source: §6.5 · p. 13, Figure 12 · p. 13

7 CONCLUSION

DORY automates DNN deployment on end-nodes; in the GAP-8 case study, it improves energy efficiency and performance over the STM32H743 and improves end-to-end inference over GWT AutoTiler. The framework, kernels, and heuristics are released as open source, with future work aimed at stronger quantization, hardware-accelerated primitives, and emerging memory technologies.

7 CONCLUSION

  • 12.6× higher energy efficiency and 7.1× higher performance versus STM32H743, plus up to 26.6% end-to-end inference improvement versus GWT AutoTiler, are reported for the GAP-8 case study.Future work targets stronger quantization, hardware-accelerated primitives, and emerging memory technologies for higher-accuracy networks on sub-10 mW edge platforms.Source: §7 · p. 13
Loading 2008.07127v3…