PaperlayerView exact revision on arXiv

Source-linked AI summary

FANN-on-MCU: An Open-Source Toolkit for Energy-Efficient Neural Network Inference at the Edge of the Internet of Things

Xiaying Wang, Michele Magno, Lukas Cavigelli, Luca Benini

arXiv:1911.03314v3cs.LGeess.SPstat.ML

TL;DR

Battery-powered IoT MCUs have limited energy and computational resources, and few neural-network implementations and measurements exist for them. FANN-on-MCU deploys FANN-trained networks on Cortex-M and PULP processors; parallel RI5CY reaches up to 13.5× speedup over Cortex-M4.

  • Problem

    Battery-powered IoT MCUs have limited energy and computation, and few neural-network implementations and measurement results exist for these devices.

    Source: §I · p. 1, §VIII · p. 13

  • Method

    FANN-on-MCU facilitates deployment of FANN-trained neural networks on ARM Cortex-M and RISC-V-based PULP processors, with and without floating-point units.

    Source: §VIII · p. 13

  • Results

    Up to 13.5× speedup is reported for parallel RI5CY cores over ARM Cortex-M4, while parallel execution reaches up to 7.7× speedup over a single RI5CY core.

    Source: B. Experimental Results and Discussion · p. 9

  • Takeaways & Limitations

    The framework supports neural-network inference comparisons across ARM Cortex-M and PULP processors, including processors with and without floating-point units.

    Source: §VIII · p. 13

    The application measurements do not include data acquisition or preprocessing, so they do not evaluate the complete system chain.

    Source: §VII · p. 13

Abstract

from arXiv · show

The growing number of low-power smart devices in the Internet of Things is coupled with the concept of "Edge Computing", that is moving some of the intelligence, especially machine learning, towards the edge of the network. Enabling machine learning algorithms to run on resource-constrained hardware, typically on low-power smart devices, is challenging in terms of hardware (optimized and energy-efficient integrated circuits), algorithmic and firmware implementations. This paper presents FANN-on-MCU, an open-source toolkit built upon the Fast Artificial Neural Network (FANN) library to run lightweight and energy-efficient neural networks on microcontrollers based on both the ARM Cortex-M series and the novel RISC-V-based Parallel Ultra-Low-Power (PULP) platform. The toolkit takes multi-layer perceptrons trained with FANN and generates code targeted at execution on low-power microcontrollers either with a floating-point unit (i.e., ARM Cortex-M4F and M7F) or without (i.e., ARM Cortex M0-M3 or PULP-based processors). This paper also provides an architectural performance evaluation of neural networks on the most popular ARM Cortex-M family and the parallel RISC-V processor called Mr. Wolf. The evaluation includes experimental results for three different applications using a self-sustainable wearable multi-sensor bracelet. Experimental results show a measured latency in the order of only a few microseconds and a power consumption of few milliwatts while keeping the memory requirements below the limitations of the targeted microcontrollers. In particular, the parallel implementation on the octa-core RISC-V platform reaches a speedup of 22x and a 69% reduction in energy consumption with respect to a single-core implementation on Cortex-M4 for continuous real-time classification.

I. INTRODUCTION

Battery-powered IoT devices have limited energy and computational resources, making neural-network inference on low-power MCUs challenging. FANN-on-MCU deploys FANN-trained networks across ARM Cortex-M and PULP processors and evaluates their performance, including wearable applications.

I. INTRODUCTION

  • Edge computing moves IoT processing closer to sensors, addressing cloud workload, communication cost, transmission energy, reliability, latency, and privacy concerns.Devices may preprocess data locally and send intermediate results to the fog.Source: §I · p. 1
  • Small batteries constrain energy and computation on MCUs, and the paper notes few neural networks running on these common edge devices.Parallelism, accelerators, memory-access optimization, and near-threshold technology are proposed approaches to improve ultra-low-power processor performance.Source: §I · pp. 1–2
  • Fully connected networks can be effective for low-bandwidth sensor time series, while efficient MCU inference requires implementations optimized for processor architecture.The paper identifies special instructions, parallelism, scratchpad memories, and hardware accelerators as optimization opportunities.Source: §I · p. 2
  • The evaluation compares ARM Cortex-M and PULP performance across network sizes and memory footprints, and reports wearable applications including gesture recognition, fall detection, and activity classification.The wearable uses an ARM Cortex-M4 and a PULP processor; measurements include memory usage, accuracy, feasible network sizes, and power consumption.Source: §I · p. 2
  • 22× runtime speedup and 69% lower energy consumption are reported for parallel PULP versus single-core Cortex-M4 continuous real-time classification.The reported values are maxima.Source: §I · p. 2
  • FANN-on-MCU is an open-source framework for deploying FANN-trained neural networks on ARM Cortex-M processors with or without an FPU and on RISC-V-based PULP processors.The paper also reports releasing the toolkit as open-source software.Source: §I · p. 2

II. RELATED WORK

The section introduces multilayer perceptrons, FANN and FANNTool, then reviews frameworks for deploying neural networks on MCUs.

II. RELATED WORK

  • The related-work discussion covers MLPs, FANN and FANNTool, and existing approaches to deploying neural networks on MCUs.It previews these topics as the section’s scope.Source: §II · p. 3

A. Multi-Layer Perceptrons

The section introduces MLP structure and training, describes FANN and its graphical companion, and positions FANN-on-MCU among MCU deployment tools.

A. Multi-Layer Perceptrons

  • An MLP is a feed-forward network with input, hidden, and output layers; its nodes combine weighted inputs and bias, then apply a nonlinearity.The network is trained end-to-end using backpropagation and stochastic gradient descent; its architecture specifies hidden layers, node counts, and nonlinearities.Source: A. Multi-Layer Perceptrons · p. 3

B. Fast Artificial Neural Network (FANN) Library and FANNTool

  • FANN is a C framework for MLP training and inference, with fixed-point optimization and cascade training that adds neurons as training proceeds.FANNTool provides a graphical interface for modifying architectures and training options, monitoring progress, and automatically selecting hyperparameters.Source: B. Fast Artificial Neural Network… · p. 3

C. Frameworks for Deploying NNs on MCUs

  • FANN-on-MCU extends FANNCortexM with fixed-point support, detailed measurements, and an optimized PULP backend, alongside existing MCU deployment frameworks.The paper also describes Cortex-M code generation built on CMSIS.Source: C. Frameworks for Deploying NNs on MCUs · pp. 3–4

III. LOW-POWER PROCESSORS

The paper introduces low-power processors as targets for its MLP toolkit, focusing on the ARM Cortex-M family’s memory, compute, and instruction-set constraints. It also describes CMSIS functions used to accelerate Cortex-M computations.

A. Low-Power Embedded Processing: ARM Cortex-M Family

  • The Cortex-M family spans milliwatt-power processors, from 16 MHz M0 cores to M7 cores reaching 300 MHz.Most lack a floating-point unit; only Cortex-M4F and M7 have one.Source: A. Low-Power Embedded Processing: ARM… · p. 4
  • Cortex-M MCUs have limited SRAM and flash, so memory footprint and operations per second constrain embedded workloads.The passage reports SRAM of 256–512 kB and maximum flash sizes of 1–2 MB.Source: A. Low-Power Embedded Processing: ARM… · p. 4
  • Cortex-M3, M4, and M7 include DSP instructions, while M4 and M7 also offer SIMD and multiply-accumulate instructions relevant to neural-network computation.These instructions may accelerate signal processing and neural-network operations.Source: A. Low-Power Embedded Processing: ARM… · p. 4
  • 36% lower execution time was measured using CMSIS arm_dot_prod; arm_fill and arm_copy each improved execution time by about 30%.The toolkit uses optimized CMSIS floating-point and fixed-point multiplication functions.Source: A. Low-Power Embedded Processing: ARM… · p. 4

B. Parallel Ultra-Low-Power Platform: PULP Processors

PULP is an open-source RISC-V multicore platform designed for tunable, energy-efficient processing, and Mr. Wolf combines a small controller core with an eight-core cluster. The InfiniWolf wearable testbed brings Mr. Wolf and an ARM Cortex-M4 together with sensors and dual-source energy harvesting.

B. Parallel Ultra-Low-Power Platform: PULP Processors

  • PULP is an open-source RISC-V multicore platform designed for tunable performance and energy efficiency within a power envelope of a few milliwatts.Its intended applications include processing sensor data streams in IoT systems.Source: B. Parallel Ultra-Low-Power Platform:… · p. 4
  • Mr. Wolf pairs an IBEX core in the fabric controller with an eight-core RI5CY cluster, connected through hierarchical memory and DMA transfers.The FC handles I/O and can offload compute-intensive work to the cluster, which is activated on demand.Source: B. Parallel Ultra-Low-Power Platform:… · p. 5
  • 2× speedup over baseline RV32IMC comes from post-incremental loads and stores with hardware loops; packed SIMD instructions add up to approximately 10× speedup.The comparison concerns cycle reduction from PULP ISA extensions.Source: B. Parallel Ultra-Low-Power Platform:… · p. 5, Figure 3 · p. 5

C. Testbed Platform: InfiniWolf

  • InfiniWolf is a wearable, battery-operated multisensor platform with an ARM Cortex-M4 and Mr. Wolf for local processing.The Nordic SoC handles communications and power management; the dual-processor design supports on-board classification.Source: C. Testbed Platform: InfiniWolf · pp. 5–6
  • InfiniWolf harvests energy from solar and thermoelectric sources, with sensors for motion, pressure, audio, biomedical signals, and skin response.The wrist-worn device periodically and opportunistically acquires sensor data according to available energy.Source: C. Testbed Platform: InfiniWolf · p. 6

IV. OPEN-SOURCE FANN-ON-MCU FRAMEWORK

The framework automates deployment of FANN-trained neural networks as optimized C code for resource-constrained MCUs. Its workflow covers application development, model conversion, platform-aware memory placement, and, on PULP, DMA-assisted use of the memory hierarchy.

A. Embedded Machine Learning System Development

  • Embedded system development proceeds from application and platform selection through data collection, preprocessing, model training, accuracy checks, and device deployment.Deployment includes implementation optimization, sensor and preprocessing integration, and measurement of performance and power.Source: A. Embedded Machine Learning System… · p. 6

B. The FANN-on-MCU Deployment Framework

  • A single command generates optimized, dependency-free C code for the selected platform, using architecture-specific instructions and embedding network parameters in the generated files.The framework also provides a test method for verification and benchmarking with dataset samples.Source: B. The FANN-on-MCU Deployment Framework · p. 6
  • The deployment workflow converts data to FANN format, trains and saves a network, optionally converts it to fixed-point, then generates code for integration and evaluation.The converter supports fixed- and floating-point models, including processors without a floating-point unit.Source: B. The FANN-on-MCU Deployment Framework · pp. 6–7
  • The memory estimate accounts for input buffers, neurons, weights, and layer metadata to determine the network’s required storage.The estimate includes double buffering for input samples and storage for neuron and layer indices and other neuron parameters.Source: B. The FANN-on-MCU Deployment Framework · p. 7
  • The framework places network parameters in the memory level closest to the processor that can still contain the network.For example, a 30 kB estimated network is loaded into RAM when 96 kB is available on an ARM Cortex-M.Source: B. The FANN-on-MCU Deployment Framework · p. 7
  • On PULP, the framework selects private or shared L2 for FC execution and uses L1 for cluster execution when the network fits.When cluster execution exceeds L1 capacity, it uses shared L2 and DMA double buffering; transfers can be layer-wise or neuron-wise depending on layer size.Source: B. The FANN-on-MCU Deployment Framework · p. 7
  • The PULP FC can run a small onset-detection network while the cluster is activated for a deeper classification network when onset is detected.The framework stores the small network in private L2 and transfers the classification network to L1 for cluster computation.Source: B. The FANN-on-MCU Deployment Framework · p. 7

V. PERFORMANCE EVALUATION

The performance evaluation examines networks with varied architectures and sizes, then presents application showcases in Section VI.

V. PERFORMANCE EVALUATION

  • The evaluation covers a wide variety of network architectures before the application showcases in Section VI.The showcases demonstrate efficiency for specific applications.Source: §V · pp. 7–8
  • Network architectures vary in input and output sizes, hidden-layer counts, and hidden-unit counts.Source: §V · pp. 7–8
  • The section sets up a broad network evaluation, with specific application showcases deferred to Section VI.Source: §V · pp. 7–8

A. Methods

The evaluation profiles FANN-based networks on Cortex-M4 and Mr. Wolf, then measures single-layer and whole-network performance across varying network sizes.

A. Methods

  • The initial profile used a 5-input, two-hidden-layer network with 100 neurons per hidden layer and three output classes on Cortex-M4 and Mr. Wolf.Each neuron used a hyperbolic-tangent activation; Mr. Wolf was tested in floating- and fixed-point.Source: A. Methods · p. 8
  • The toolkit was implemented for ARM Cortex-M and PULP, with performance evaluated on STM32L475VG and Mr. Wolf.The evaluation used the single-layer and whole-network procedures described above.Source: A. Methods · p. 8
  • Single-layer runtime was measured while varying the input-feature vector length and the number of layer neurons.Whole-network measurements fixed inputs at 100 features and outputs at eight classes, while varying hidden-layer count and units.Source: A. Methods · p. 8
  • Hidden-layer sizes follow a growth rule controlled by d, which determines how quickly the network grows toward platform memory limits.Measurements used onboard cycle counts without normalization; the chosen d was selected after considering memory regions, transfer modes, and parallel efficiency.Source: A. Methods · p. 8

B. Experimental Results and Discussion

The experiments compare fixed- and floating-point inference across Cortex-M4 and PULP cores, including memory-transfer regimes. Results show faster execution with RI5CY and parallel PULP, with performance depending on network size and memory placement.

B. Experimental Results and Discussion

  • 3.1% and 7.7% runtime improvements followed elimination of unnecessary initialization in floating- and fixed-point Cortex-M4 implementations.The fixed-point implementation was around 15% faster, while weight-matrix computation without activation functions occupied approximately 88% of compute time for the example network.Source: B. Experimental Results and Discussion · p. 8
  • Only every fifth RI5CY instruction used the FPU, reaching 80% utilization without creating a performance bottleneck.Cortex-M inner loops required 8 floating-point or 7 fixed-point cycles, while both RI5CY implementations required five one-cycle instructions.Source: B. Experimental Results and Discussion · p. 8
  • Fixed- and floating-point implementations had similar RI5CY execution times despite two FPUs being shared among eight cores.On Cortex-M4, fixed point was around 15% faster for the profiled example network.Source: B. Experimental Results and Discussion · p. 8
  • The single-layer cycle measurements varied input and output counts and marked when data moved beyond RAM or private L2.Figure 8 provides Cortex-M4 and IBEX reference measurements; its table companion gives assembly code for the inner-loop dot product, but no cycle values are supplied here.Source: B. Experimental Results and Discussion · p. 8, Figure 8 · p. 9, Table I · p. 9
  • 2.2× single-core speedup over IBEX and 7.7× parallel speedup over one RI5CY core were measured for single-layer execution.The speedup from DMA-based neuron-wise transfer was higher for larger input sizes, where DMA activation overhead became less significant.Source: B. Experimental Results and Discussion · pp. 8–9, Figure 9 · p. 10
  • 13.5× speedup was reached by parallel RI5CY cores over Cortex-M4 in single-layer comparisons.A single RI5CY core reached almost 2× speedup over Cortex-M4; Figure 10 varies layer inputs and outputs.Source: B. Experimental Results and Discussion · p. 9, Figure 10 · p. 11
  • d = 8 was selected for whole-network sweeps spanning 1 hidden layer with 8 nodes to 24 layers with 1,248 hidden units.Networks fit in Mr. Wolf L1 up to 12 hidden layers; larger networks use piece-wise DMA transfers, and beyond 21 layers transfer is neuron-wise. Figure 11 reports cycles and Figure 12 speedups.Source: B. Experimental Results and Discussion · p. 10, Figure 11 · p. 11
  • 11.1× speedup over Cortex-M4 was reached when its network was stored in flash and Mr. Wolf transferred data from L2 to L1 using DMA.For whole networks, a single RI5CY core was almost twice as fast as Cortex-M4; Figure 12 compares network speedups across sizes.Source: B. Experimental Results and Discussion · p. 11

VI. APPLICATION SHOWCASES

The paper demonstrates FANN-on-MCU using three MLP application showcases, reproducing their networks on Cortex-M4 and Mr. Wolf within the InfiniWolf wearable system.

VI. APPLICATION SHOWCASES

  • Three MLP application showcases were selected from prior work to demonstrate framework usability and power efficiency on supported MCUs.They cover hand gestures, fall-risk assessment, and human activity classification.Source: §VI · p. 11
  • Figure 12 presents whole-network speedups on Mr. Wolf and compares Mr. Wolf with Cortex-M4 across hidden-layer counts and hidden-unit sizes.The caption identifies the comparisons but does not state a result.Source: Figure 12 · p. 12
  • The reproduced networks ran on the InfiniWolf Nordic nRF52832 with Cortex-M4 and on PULP Mr. Wolf.The framework was used to reproduce the architectures described for the showcases.Source: §VI · p. 12

A. Hand Gesture Recognition

The showcases reproduce three MLP applications on wearable-system processors and measure their runtime and power. The results distinguish the trade-offs for individual classifications and continuous operation.

A. Hand Gesture Recognition

  • Gesture recognition uses fused EMG and nine-axis IMU features from 76 time-domain measurements to classify 10 hand gestures.The cited model uses three hidden layers of 300, 200, and 100 units and reported 85.58% accuracy.Source: A. Hand Gesture Recognition · p. 12

B. Fall Detection for Elderly People

  • The fall-risk model uses 117 inputs, 20 hidden nodes, and two outputs; the best reported accuracy was 84%.Its selected inputs combined pressure sensors with accelerometers at the head, pelvis, and left shank.Source: B. Fall Detection for Elderly People · p. 12

C. Human Activity Classification

  • The activity-classification MLP uses seven input features, six hidden nodes, and five outputs; the cited FPGA implementation reported 270 ns execution and 241 mW power.The task classifies five activity classes from waist-worn accelerometer data.Source: C. Human Activity Classification · p. 12

D. Experimental Evaluation and Results

  • For application A, multi-RI5CY execution was 7.1× faster than one RI5CY core; the reproduced gesture model has 76 features and hidden layers of 300, 200, and 100 units.The original gesture task classified 10 hand gestures and reported 85.58% accuracy.Source: D. Experimental Evaluation and Results · p. 12, A. Hand Gesture Recognition · p. 12
  • For one classification, multi-RI5CY speedup was 5.7× over IBEX and 8.8× over Cortex-M4; with more classifications per cluster activation, it approached 14.3× and 22×.Parallel energy use was 54 µJ per classification plus about 13 µJ fixed overhead; for application B it paid off beyond six classifications, and continuous execution was 4× as energy-efficient as one IBEX core.Source: D. Experimental Evaluation and Results · p. 13, Figure 9b · p. 13

VII. DISCUSSION

The discussion contrasts parallel speedup with its power cost and notes that acquisition and preprocessing are excluded; the conclusion reports inference gains and low-power deployment measurements.

VII. DISCUSSION

  • Eight-core parallelization reaches up to 7.1× speedup versus one core, but adding cores increases power consumption (Fig. 13).Fig. 13 measures end-to-end power for one classification of application A on Mr. Wolf with RI5CY cores.Source: §VII · p. 13, Figure 13 · p. 13
  • Data acquisition and preprocessing are excluded from the evaluation; the authors plan to implement the full chain in future work.The stated limitation means the measurements do not cover the complete acquisition-and-processing system.Source: §VII · p. 13

VIII. CONCLUSION

  • A network with 103800 multiply-accumulate operations runs in 17.6 ms using 183 µJ on a Nordic nRF52832 MCU.The conclusion presents this as evidence that ANNs can fit ultra-low-power MCUs in memory use, compute time, and energy consumption.Source: §VIII · p. 13
  • Cluster activation, initialization, and deactivation add around 1~1.3 ms at an average power consumption of 11.88 mW.The footnote notes that some runtimes are below or close to the measuring instrument’s precision and are calculated from cycle counts.Source: §VIII · p. 14
  • On average, Mr. Wolf’s parallel implementation provides 14× speedup and 80% energy reduction versus ARM Cortex-M4 inference.The conclusion also reports that the same inference takes less than 1 ms and consumes around 50 µJ on eight RI5CY cores.Source: §VIII · p. 14, Table II · p. 14
  • FANN-on-MCU is released open-source for deploying neural networks on low-power embedded systems.The framework supports both ARM Cortex-M and RISC-V-based PULP processors.Source: §VIII · pp. 13–14
Loading 1911.03314v3…