PaperlayerView exact revision on arXiv

Source-linked AI summary

PULP-NN: Accelerating Quantized Neural Networks on Parallel Ultra-Low-Power RISC-V Processors

Angelo Garofalo, Manuele Rusci, Francesco Conti, Davide Rossi, Luca Benini

arXiv:1908.11263v1cs.NE

TL;DR

IoT edge inference must fit deep-learning workloads within severe power, memory, and computation limits. PULP-NN addresses this problem with low-bitwidth QNN kernels optimized for DSP-enabled, parallel RISC-V processors. Its GAP-8 evaluation reports large performance and energy-efficiency gains over MCU-based CMSIS-NN baselines.

  • Problem

    Deep-learning inference on extreme-edge devices is constrained by limited power, memory, and computation resources.

  • Method

    PULP-NN provides an optimized multicore library for 8-, 4-, 2-, and 1-bit QNN inference on parallel RISC-V edge platforms.

  • Results

    PULP-NN delivers up to 63x performance improvement over a corresponding RV32IMC implementation and 19.49× fewer cycles than STM32H7 CMSIS-NN inference.

  • Takeaways & Limitations

    The results support fully programmable parallel ultra-low-power MCUs as a platform for extreme-edge QNN inference.

Abstract

from arXiv · show

We present PULP-NN, an optimized computing library for a parallel ultra-low-power tightly coupled cluster of RISC-V processors. The key innovation in PULP-NN is a set of kernels for Quantized Neural Network (QNN) inference, targeting byte and sub-byte data types, down to INT-1, tuned for the recent trend toward aggressive quantization in deep neural network inference. The proposed library exploits both the digital signal processing (DSP) extensions available in the PULP RISC-V processors and the cluster's parallelism, achieving up to 15.5 MACs/cycle on INT-8 and improving performance by up to 63x with respect to a sequential implementation on a single RISC-V core implementing the baseline RV32IMC ISA. Using PULP-NN, a CIFAR-10 network on an octa-core cluster runs in 30x and 19.6x less clock cycles than the current state-of-the-art ARM CMSIS-NN library, running on STM32L4 and STM32H7 MCUs, respectively. The proposed library, when running on GAP-8 processor, outperforms by 36.8x and by 7.45x the execution on energy efficient MCUs such as STM32L4 and high-end MCUs such as STM32H7 respectively, when operating at the maximum frequency. The energy efficiency on GAP-8 is 14.1x higher than STM32L4 and 39.5x higher than STM32H7, at the maximum efficiency operating point.

I. INTRODUCTION

IoT growth increases data, capacity, security, and reliability pressures while extreme-edge devices remain constrained by power, memory, and computation. PULP-NN addresses this gap with a multicore, low-bitwidth QNN library optimized for parallel RISC-V processors and reports substantial performance and energy gains.

  • IoT nodes generate rapidly increasing data volumes, intensifying network capacity, security, and reliability challenges.
  • Edge processing uses ML and DL to compress raw sensor data into semantically denser information before wireless transmission.
  • Extreme-edge design must balance the high computation and memory demands of DL against the scarce resources of battery- or energy-harvested embedded systems.
  • Quantization compresses weights or activations into 8-bit or smaller types, reducing QNN memory requirements relative to 32-bit floating-point models.
  • PULP-NN is a multicore library supporting 8-, 4-, 2-, and 1-bit QNN operations on programmable edge devices.
  • 9× speedup comes from exploiting DSP extensions relative to a plain RV32IMC implementation, while eight-core execution increases kernel throughput by up to 7.5×.
  • 19.5× higher performance than a CMSIS-NN-based ARM Cortex-M7 MCU and 14.1× better energy efficiency than an ARM Cortex-M4 MCU are reported for CIFAR-10 inference.

II. RELATED WORK

Related work spans programmable processors, heterogeneous platforms, FPGAs, and ASIC accelerators, exposing trade-offs among flexibility, performance, energy efficiency, cost, and power envelope. PULP-NN targets the underdeveloped software support for parallel, programmable ultra-low-power RISC-V edge platforms.

  • GPUs and FPGAs support CNN inference but generally exceed the tight power envelope required by IoT end-nodes.
  • ASIC accelerators can deliver very high throughput and energy efficiency, but their application specificity limits flexibility.
  • Software-programmable processors offer the highest flexibility, while high-performance GPUs and multicore application processors are typically unsuitable for IoT power or cost constraints.
  • Low-power STM32L4 and STM32H7 microcontrollers target edge deep learning but remain within the resource-constrained MCU class.
  • Helium is a DSP-oriented ISA extension for inference, but the cited discussion states that no device supports it yet.
  • Heterogeneous MCU platforms combine processors with CNN accelerators, but proprietary, platform-specific designs have limited software-flow support and developer acceptance.
  • Table I summarizes trade-offs among the CNN computing platforms covered in the related-work discussion.

Optimized Software Libraries:

The paper targets the computational gap between aggressively quantized neural networks and resource-constrained MCUs with an open-source library for parallel ultra-low-power processors. It combines low-bitwidth support with DSP and parallel execution to improve performance and energy efficiency.

  • Optimized Software Libraries:: 8-, 4-, 2-, and 1-bit quantized data types are supported for QNN inference on programmable edge devices.The work focuses on computationally efficient inference for reduced-precision networks.
  • Optimized Software Libraries:: INT-8 convolution uses high-precision INT-32 accumulation, while INT-4, INT-2, and INT-1 use INT-16 accumulation before compression to Q-bit outputs.INT-8 compression uses scaling and clamping; lower-bit cases use thresholding-based compression.
  • Optimized Software Libraries:: 1-bit convolution reduces to logical XNOR and bit-count operations, followed by threshold-based compression.The bit-count operator is popcount.
  • Optimized Software Libraries:: Aggressive quantization reduces neural-network memory requirements and computational cost for MCU-class devices, with limited accuracy loss reported for fixed-point networks.Prior work cited in the paper reports accuracy close to floating-point at 4-bit quantization and limited drops for lower-bit configurations.

B. Dataflow Schedule and Data Layout

The convolution dataflow uses CMSIS-NN’s layout and computes output feature-map values as dot products between filter banks and unrolled input regions. The inner dot product is implemented as a 2×2 matrix-multiplication kernel.

  • B. Dataflow Schedule and Data Layout: A convolution output value is computed as the dot product of a weight filter bank and the corresponding C × kw × kh input region.The input region is represented in an im2col buffer.
  • B. Dataflow Schedule and Data Layout: The convolution inner loop is realized as a matrix-multiplication kernel that computes s output features from r activation outputs.The cited example uses s=2 and r=2.
  • B. Dataflow Schedule and Data Layout: The 2×2 configuration computes two spatially adjacent pixels across two consecutive channels inside the convolution inner loop.This configuration is identified as 2×2 in the paper.
  • B. Dataflow Schedule and Data Layout: Height-Width-Channel layout stores channel values with stride 1 and width values with stride C.This layout introduces minor overhead when constructing the im2col buffer compared with Channel-Height-Width layout.

C. Target Architectures

The target platform is an eight-core GAP8 cluster of in-order RISC-V processors implementing RV32IMC with Xpulp extensions. SIMD, dot-product, bit-manipulation, and packing instructions support efficient low-precision computation.

  • C. Target Architectures: GAP8 contains eight RISC-V cores with four-stage in-order single-issue pipelines and RV32IMC plus Xpulp extensions.The cluster is the commercial embodiment used for experiments.
  • C. Target Architectures: v4s packs four INT-8 values into a 32-bit register, while v2s packs two INT-16 values.These vectorial data types support sub-word parallel processing.
  • C. Target Architectures: sdotp4 computes four 8-bit products and accumulates their sum in one cycle, while sdotp2 handles two 16-bit products.The output accumulator is INT-32 for the built-in operation.
  • C. Target Architectures: bextract, bitinsert, pack4, and popcnt provide bit extraction, bit insertion, INT-8 packing, and one-cycle bit counting.These instructions support sub-byte data manipulation and binary kernels.

IV. PULP-NN LIBRARY

PULP-NN is optimized for an eight-processor PULP cluster and focuses on high-performance, energy-efficient QNN kernels. Its convolution and fully connected implementations reuse the same matrix-multiplication strategy while ancillary kernels exploit DSP instructions.

  • IV. PULP-NN LIBRARY: The library optimizes QNN kernels and matrix multiplication for the RV32IMCXpulp ISA on a parallel eight-processor cluster.The implementation assumes data resides in the cluster’s L1 memory.
  • IV. PULP-NN LIBRARY: INT-8 convolution uses hardware loops, post-increment loads and stores, and 8-bit SIMD instructions to accelerate the regular inner loop.The kernel begins from a 2×2 matrix-multiplication implementation.
  • IV. PULP-NN LIBRARY: Each INT-8 inner-loop iteration loads four consecutive elements from two im2col buffers and two weight banks.The four loads use pointers associated with the two input buffers and two weight banks.
  • IV. PULP-NN LIBRARY: The fully connected kernel reuses the matrix-by-vector methodology without constructing an im2col buffer.This follows from the fully connected operation’s matrix-by-vector form.
  • IV. PULP-NN LIBRARY: A 2×2 INT-8 matrix-multiplication kernel forms the basic low-level computation used by the convolution implementation.The figure documents the kernel’s organization for INT-8 operands.
  • IV. PULP-NN LIBRARY: ReLU and max pooling use hardware loops, post-increment memory operations, and SIMD max4; max pooling proceeds across width and then height.The width pass operates destructively in situ on the input buffer.

Sub-byte Extensions:

PULP-NN supports sub-byte QNN computation by unpacking compact operands into INT-8 for matrix multiplication, then compressing accumulators back to the target precision. Its multicore schedule partitions convolution work across output-map spatial locations, while fully connected layers partition neurons across cores.

  • Sub-byte data handling: INT-4 operands are unpacked into INT-8 during im2col construction or the matrix-multiplication inner loop, while INT-1 uses native binary operations.INT-1 convolution uses XNOR and popcount without casting or unpacking.
  • Sub-byte data handling: 16-bit accumulators are compressed back to INT-4 through threshold comparison and balanced-tree quantization, with two output values packed per byte.The procedure uses 2^4 − 1 threshold values and bit insertion to reduce memory footprint.
  • Multicore execution: Convolution work is split along output-feature-map spatial locations so each core computes all output features for assigned coordinates.This strategy follows the HWC data layout and requires private im2col buffers.
  • Multicore execution: The eight-core schedule requires additional private im2col storage, reaching about 9% of total memory in the stated worst-case configuration, while weights remain shared.For a 2×2 kernel, each core allocates and loads two private im2col buffers.
  • Multicore execution: Fully connected layers distribute balanced groups of output neurons across cores because their outputs have no spatial dimension.ReLU and max-pooling instead divide balanced groups of pixels across the input feature map.

C. Matrix Multiplication Kernel Size Exploration

The matrix-multiplication kernel increases throughput by reusing operands across multiple output computations, raising the MAC-to-load ratio. Kernel-size scaling is constrained by register-file capacity, and the evaluation compares resulting convolution speedups against RV32IMC and CMSIS-NN baselines.

  • C. Matrix Multiplication Kernel Size Exploration: Register-level data reuse reduces operand-loading cost in the memory-intensive matrix-multiplication kernel.The direct implementation requires separate loads for an im2col element and a weight, causing a load stall.
  • C. Matrix Multiplication Kernel Size Exploration: Reusing weights across adjacent outputs enables two dot products with three loads, while a 2×2 kernel performs four sdotp4 operations with four loads.The 2×2 kernel computes two features for two adjacent output pixels and reaches a MAC-to-load ratio of 4.
  • C. Matrix Multiplication Kernel Size Exploration: Kernel-size upscaling is limited by register-file resources needed to store operands and accumulators.The explored design space seeks the register-file reuse condition that maximizes throughput.
  • C. Matrix Multiplication Kernel Size Exploration: The comparison figure reports PULP-NN and CMSIS-NN convolution speedups relative to the RV32IMC ISA.PULP-NN measurements use single-core GAP8 execution, while CMSIS-NN measurements use STM32H7 and STM32L4.

V. EXPERIMENTAL RESULTS AND DISCUSSION

The evaluation runs PULP-NN on GAP8 and compares it with RV32IMC and ARM CMSIS-NN implementations across convolution kernels and quantization levels. Results show strong gains from DSP-supported execution, especially for INT-8, while sub-byte unpacking reduces speedup.

  • V. EXPERIMENTAL RESULTS AND DISCUSSION: GAP8 evaluation uses an eight-core PULP cluster, with experiments also replicable on the open-source PULP platform through RTL simulation.GAP8 is treated as an embodiment of the target parallel ultra-low-power RISC-V architecture.
  • V. EXPERIMENTAL RESULTS AND DISCUSSION: The convolution evaluation benchmarks a 16×16×32 HWC input with 64×3×3×32 filters and compares PULP-NN with RV32IMC and CMSIS-NN on STM32 MCUs.Convolution represents about 96% of CIFAR-10 QNN workload in the stated evaluation.
  • A. Comparison with RV32IMC ISA: INT-8 achieves the best speedup, outperforming STM32H7 by 2.54× and STM32L4 by 4.51×.The reported advantage is attributed to 8-bit SIMD sdotp instructions, hardware loops, and differences in ARM vector support.
  • V. EXPERIMENTAL RESULTS AND DISCUSSION: Figure 8 compares cycles/MAC for PULP-NN on one or eight GAP8 cores with CMSIS-NN on STM32L4 and STM32H7.The comparison spans single-core and multicore PULP-NN execution against both MCU baselines.
  • A. Comparison with RV32IMC ISA: Sub-byte speedup declines from 8.8× for INT-8 to 3.69× for INT-4 and 4.22× for INT-2 relative to RV32IMC.The reduction is associated with extra instructions for unpacking and casting sub-byte operands to INT-8.

B. Multicore Execution Results

PULP-NN’s multicore kernels achieve near-linear scaling and substantial throughput on the eight-core GAP-8 cluster. The best INT-8 kernel reaches 15.5 MACs/cycle, while register pressure limits larger kernel configurations.

  • Multicore scaling: 7.16× speedup is achieved for the convolution kernel on eight cores, with overall multicore speedups reaching up to 63× over sequential RV32IMC execution.Parallel execution provides near-linear scaling, while sub-byte kernels require more cycles per MAC than INT-8.
  • Multicore overheads: 67% of convolution overhead comes from instruction-cache non-idealities, with additional losses from load stalls and TCDM contention.Shared cache-bank conflicts increase with core count, while the smaller fully connected kernel experiences fewer instruction-cache stalls.
  • Kernel throughput: 15.5 MACs/cycle is reached with a 4×2 matrix multiplication kernel on eight cores, using 1.01 LD/ST per MAC.This corresponds to 49% MAC utilization and is close to the 32 MACs/cycle theoretical cluster peak.

D. Comparison with GAP8 Native Library

PULP-NN is compared with GWT-NN on INT-8 convolution workloads and with CMSIS-NN on complete CIFAR-10 inference. Its advantage depends on workload shape and is especially pronounced when channel counts are high.

  • GWT-NN comparison: PULP-NN outperforms GWT-NN on all small images and most larger-image cases in the single-core comparison.The comparison uses a representative 3×3 convolution and varies channel counts for PULP-NN and spatial dimensions for GWT-NN.
  • GWT-NN comparison: +89% speedup is achieved by PULP-NN over GWT-NN when input and output channel counts are high.GWT-NN remains mostly competitive when feature-map spatial dimensions greatly exceed channel counts.
  • CIFAR-10 inference: 28.6 ms is required for single-core GAP-8 inference at 170 MHz, while two and four cores provide 1.99× and 3.79× speedups.With eight cores, the speedup is slightly below 7×.
  • CIFAR-10 inference: 1.07 GMAC/s is achieved on GAP-8 at 170 MHz and 1.2 V, corresponding to 241 fps and 0.27 mJ/frame.At this operating point, performance is 7.45× higher than STM32H7 and 36.8× higher than STM32L4.

F. Discussion

The discussion concludes that PULP-NN combines optimized software with a parallel ultra-low-power platform to support programmable QNN inference at the IoT edge. It also identifies sub-byte performance overheads caused by limited hardware support.

  • Discussion: Sub-byte kernels lose performance relative to INT-8 because the target architecture supports SIMD instructions only for 8-bit data.Additional packing and unpacking functions create the stated overhead, although sub-byte execution remains more than one order of magnitude better than MCU-based state-of-the-art solutions.
  • Discussion: PULP-NN targets INT-8, INT-4, INT-2, and INT-1 QNN operands while using SIMD extensions and bit-manipulation instructions.The library improves kernel performance by up to 63× over corresponding RISC-V IMC implementations in an eight-core cluster.
Loading 1908.11263v1…