PaperlayerView exact revision on arXiv

Source-linked AI summary

Intelligence Beyond the Edge: Inference on Intermittent Embedded Systems

Graham Gobieski, Nathan Beckmann, Brandon Lucia

arXiv:1810.07751v2cs.DC

TL;DR

Energy-harvesting IoT systems need local inference because communication is expensive, yet accurate DNNs exceed the computational, memory, and intermittent-execution capabilities of these devices. The paper combines network compression and intermittence-aware execution, demonstrating lower inference energy while guaranteeing correct execution across power systems.

  • Problem

    Energy-harvesting systems need local inference because communication is expensive, but accurate DNNs are computationally demanding and intermittent power failures can corrupt results and impede progress.

  • Method

    The paper combines GENESIS for accuracy–energy network compression with SONIC’s loop continuation and idempotence techniques, alongside TAILS for SIMD-based efficiency.

  • Results

    Across three neural networks on a commercially available microcontroller, SONIC and TAILS reduce inference energy by 6.9× and 12.2×, respectively, over the state-of-the-art.

  • Takeaways & Limitations

    The paper demonstrates efficient DNN inference on commodity energy-harvesting systems while guaranteeing correct intermittent execution across different power systems.

  • Takeaways & Limitations

    Current energy-harvesting MCUs have efficiency-limiting architectural constraints, including small SRAM buffers and incomplete hardware support.

Abstract

from arXiv · show

Energy-harvesting technology provides a promising platform for future IoT applications. However, since communication is very expensive in these devices, applications will require inference "beyond the edge" to avoid wasting precious energy on pointless communication. We show that application performance is highly sensitive to inference accuracy. Unfortunately, accurate inference requires large amounts of computation and memory, and energy-harvesting systems are severely resource-constrained. Moreover, energy-harvesting systems operate intermittently, suffering frequent power failures that corrupt results and impede forward progress. This paper overcomes these challenges to present the first full-scale demonstration of DNN inference on an energy-harvesting system. We design and implement SONIC, an intermittence-aware software system with specialized support for DNN inference. SONIC introduces loop continuation, a new technique that dramatically reduces the cost of guaranteeing correct intermittent execution for loop-heavy code like DNN inference. To build a complete system, we further present GENESIS, a tool that automatically compresses networks to optimally balance inference accuracy and energy, and TAILS, which exploits SIMD hardware available in some microcontrollers to improve energy efficiency. Both SONIC & TAILS guarantee correct intermittent execution without any hand-tuning or performance loss across different power systems. Across three neural networks on a commercially available microcontroller, SONIC & TAILS reduce inference energy by 6.9x and 12.2x, respectively, over the state-of-the-art.

1 Introduction

Energy-harvesting IoT systems need local DNN inference because communication is costly, but DNN computation and intermittent operation strain constrained devices. The paper addresses these challenges with GENESIS and SONIC, while analyzing accuracy and energy trade-offs.

  • 480× or more end-to-end application performance improvement can result from local inference on energy-harvesting systems.
  • DNNs improve inference accuracy but require substantial computation and memory, with typical networks using tens of millions of weights and billions of operations.
  • Intermittent devices typically operate for around 100,000 instructions per power period, so existing DNN implementations do not tolerate intermittent operation.
  • Prior intermittent-execution systems impose significant runtime overhead, slowing DNN inference by an average of 10× in the authors’ experiments.
  • The paper presents the first demonstration of intermittent DNN inference on real-world networks running on a widely available energy-harvesting system.
  • GENESIS automatically compresses networks to balance inference energy and accuracy, while SONIC specializes intermittent execution for DNN inference.

2 Background

Intermittent energy-harvesting systems combine frequent power failures with severe resource constraints, complicating correct and efficient DNN execution. Prior models guarantee consistency but incur overheads, while intermittent operation remains unaddressed by efficient DNN techniques.

  • Energy-harvesting devices operate intermittently because harvested power is unavailable continuously, accumulating energy in a buffer before each operating burst.
  • Power failures clear volatile state while non-volatile memory persists, potentially leaving memory inconsistent and impeding progress.
  • Task-based systems restore consistent memory but incur redo-logging, duplication, and task-transition costs that increase with written data or frequent transitions.
  • Tasks longer than the device’s energy buffer can support may never complete, causing non-termination.
  • SONIC addresses correct intermittent DNN execution with loop continuation and idempotent loop iterations, reducing redo-logging, task transitions, and wasted work.
  • Efficient DNN techniques reduce storage, computation, or power, but intermittent operation remains unaddressed.

3 Motivation for intermittent inference

Energy-harvesting devices must perform local inference because communication costs far more energy than local computation, while inference accuracy strongly affects application performance. The model and wildlife-monitoring case study show that selective communication can greatly improve useful events per joule, but inaccurate inference erodes those gains.

  • The need for inference beyond the edge: Communication costs orders of magnitude more energy than local computation, making cloud or edge offloading impractical for energy-harvesting devices.An eight-byte packet can require a 900mF capacitor bank in the cited OpenChirp example.
  • The need for inference beyond the edge: Local inference filters sensor readings so devices communicate only potentially interesting events, improving the number of useful readings sent per fixed harvested energy.The paper uses interesting messages sent per Joule (IMpJ) as its figure of merit.
  • Why accuracy matters: Inference benefits deteriorate as accuracy declines, and qualitatively similar trends persist when the interesting-event rate p varies.The magnitude of benefit increases as p becomes smaller.
  • Why accuracy matters: 98× lower communication energy in the wildlife-monitoring example makes inference energy non-negligible and exposes the value of efficient local inference.Before filtering, the system is dominated by the energy required to send results.
  • Why accuracy matters: 480× more events can be detected and communicated by SONIC & TAILS than by the baseline without local inference.This result applies when only inference results are sent in the wildlife-monitoring example.
  • Why accuracy matters: 4.6× is the SONIC & TAILS advantage over naïve local inference when only inference results are communicated.The remaining gap between SONIC & TAILS and the ideal system is 2.2×.

4 System overview

The system converts a high-level DNN description into an efficient executable for resource-constrained, intermittently powered devices. GENESIS optimizes the network, while SONIC and TAILS provide intermittence-safe execution with optional hardware acceleration.

  • System overview: GENESIS automatically compresses each network layer to optimize IMpJ while meeting resource constraints.It takes a programmer’s high-level network description as input and produces an optimally compressed network.
  • System overview: SONIC is an intermittence-safe, task-based API and runtime with specialized DNN support that improves performance while remaining compatible with existing task-based frameworks.Its specialized support safely relaxes rules used by existing task-based systems.
  • System overview: TAILS provides an alternative runtime that combines SONIC’s optimizations with hardware-accelerated vector operations, including convolutions, on TI LEA hardware.Using TAILS requires linking the compiled binary to the TAILS-enabled runtime.
  • System overview: GENESIS, SONIC, and TAILS produce resource-constrained DNN applications that are robust to intermittent operation and can leverage widely available hardware acceleration.The workflow starts from a high-level network description and ends with an efficient intermittent application.

5 Optimal DNN compression with GENESIS

GENESIS compresses neural networks to fit energy-harvesting microcontrollers while balancing inference accuracy, energy, and application performance. It evaluates feasible configurations and selects the one maximizing estimated end-to-end performance.

  • Compression approach: GENESIS automatically explores compressed neural-network configurations using separation and pruning to reduce memory and computation requirements.It compresses both fully connected layers, which typically dominate memory, and convolutional layers, which typically dominate compute.
  • Evaluation networks: The evaluation considers MNIST, human activity recognition, and Google keyword spotting as image, wearable, and audio applications.MNIST is used instead of ImageNet because ImageNet images do not fit resource-constrained device memory; competing binary and SVM models performed poorly on the device.
  • Compression approach: Each GENESIS configuration is evaluated across true positive rate, true negative rate, memory size, and compute or energy.The tool builds a Pareto frontier while retraining networks after compression to recover accuracy.
  • Fitting networks: Original configurations are infeasible for all three networks because their parameters do not fit the device’s memory.Figure 4 marks feasible configurations as green circles and infeasible configurations as grey ×s.
  • Compression approach: Pruning is generally more effective than separation, although the two compression techniques are complementary.GENESIS shows the Pareto frontier for each technique.
  • Configuration selection: GENESIS selects the feasible configuration that maximizes estimated IMpJ rather than simply choosing the most accurate network.The model incorporates application sensing, communication, per-operation energy, inference accuracy, and true positive and negative rates.

6 Efficient intermittent inference with SONIC

SONIC specializes intermittent execution for loop-heavy DNN inference. Its loop continuation mechanism resumes long loops after power failures while preserving correctness with idempotent iterations and substantially lower overhead than prior task-based systems.

  • SONIC runtime: SONIC is an intermittence-aware software system optimized for DNN inference on resource-constrained devices.It exposes common DNN operations through an API and uses a runtime that guarantees correct intermittent operation.
  • Performance: 25%–75% overhead replaces prior task-based systems’ 10× average and up to 19× slowdowns relative to a standard non-intermittent baseline.The comparison concerns correct intermittent inference implementations.
  • Loop continuation: Loop continuation directly persists loop indices and allows computation to resume from its prior position instead of re-executing completed work.It eliminates major overheads from redo-logging, task transitions, and wasted work in long-running loop nests.
  • Task-tiling limitations: Task-tiling can waste completed iterations or fail to make progress when a fixed tile exceeds the device’s available energy.Tile-5 redoes work after failures, while Tile-12 cannot complete on the illustrated energy buffer.
  • Loop continuation: In the Figure 6 example, SONIC resumes on the ninth iteration after interruption, rather than restarting the entire loop or every fifth iteration.This demonstrates the reduced re-execution cost of loop continuation compared with fixed task tiling.
  • Idempotence: SONIC guarantees safe resumption by making loop iterations idempotent through loop-ordered buffering or sparse undo-logging.These mechanisms prevent inconsistent partial updates without statically privatizing or dynamically checkpointing all data.

7 Hardware acceleration with TAILS

TAILS extends SONIC with DMA and SIMD-style LEA hardware acceleration for intermittent DNN inference. It adapts hardware parameters to the device’s energy buffer while retaining software execution for operations that LEA handles inefficiently.

  • Hardware acceleration: TAILS uses DMA and the TI Low-Energy Accelerator to move data efficiently and execute DNN operations in parallel.It can be linked optionally with the SONIC runtime system.
  • Adaptive configuration: TAILS calibrates DMA block size and LEA vector width at runtime so operations complete without exceeding the fixed energy buffer.If a tile fails to complete before power loss, calibration halves its size and retries.
  • Hardware constraints: LEA accelerates dense convolutions and matrix multiplications, but it can access only the device’s 4KB SRAM rather than its 256KB FRAM.TAILS therefore transfers inputs and results between SRAM and FRAM using DMA.
  • Sparse operations: TAILS handles sparse operations selectively because padding filters can add unnecessary work and sparse fully connected layers lack sufficient filter reuse.Sparse fully connected layers remain in software, while factored convolutional layers use LEA’s dot-product operation.
  • Evaluation: The measurement setup is presented as a diagram for evaluating the TAILS system.

8 Methodology

The evaluation runs SONIC and TAILS on a commodity energy-harvesting microcontroller, measuring inference across continuous and intermittent power systems against naïve and tiled baselines.

  • The prototype runs on a TI-MSP430FR5994 at 16MHz with a Powercast harvester and three capacitor sizes: 1mF, 50mF, and 100µF.Configurations run under both continuous and intermittent power.
  • Measurements automate compilation, flashing, power switching, and energy collection using charge cycles between GPIO pulses.The setup also supports finer-grained energy measurement across code regions.
  • Inference time is evaluated for three networks across six implementations and four power systems, separating recharge, convolution, fully connected, and other time.Convolution layers dominate live execution time.
  • The comparison includes a naïve implementation that does not tolerate intermittence and three Alpaca-based implementations that split loops by tiling iterations.The tiled implementations provide task-based intermittent baselines.

9 Evaluation

Across three networks and multiple power systems, SONIC and TAILS provide correct intermittent execution while substantially reducing time and energy relative to tiled baselines.

  • Correctness: SONIC and TAILS guarantee correct execution for every network on every power system, unlike the naïve baseline and most tiled implementations.Only Tile-8 reliably executes correctly among the other implementations; Tile-32 and Tile-128 fail on specified 100µF configurations.
  • Inference time: 13.4× is Tile-8’s geometric-mean slowdown versus the naïve baseline, compared with 1.45× for SONIC and 1.2× faster for TAILS.SONIC improves performance 6.9× over tiled Alpaca, while TAILS improves it 12.2×.
  • Inference time: 5.2× and 9.2× are SONIC’s and TAILS’s average speedups over Tile-128 on continuous power.Larger tilings are unattractive because they fail on some networks or capacitor sizes.
  • Hardware acceleration: 1.4× is the performance improvement from LEA, while DMA improves performance by 14% on average.These effects come from comparing hardware-supported configurations with software emulations.
  • Overhead sources: SONIC’s overhead mainly comes from intermittence-related control, while TAILS accelerates kernels but incurs software bit-shift overheads for LEA.Tile-32 spends more time in both control and kernels because redo-logging buffers and commits every write.
  • Energy: SONIC and TAILS achieve high performance because they require less energy, and recharge dead time dominates inference time.Inference energy is directly proportional to recharge dead time.
  • Energy: 26% of SONIC’s energy goes to control instructions, and 14% comes from FRAM writes to loop indices.The energy profile also separates memory, arithmetic, task-transition, and remaining energy categories.

10 Future intermittent architecture research

The paper identifies hardware opportunities for intermittent inference, emphasizing energy-efficient programmable architectures with built-in intermittent support and better accelerator capabilities.

  • Architecture opportunities: Current microcontrollers waste substantial energy on instruction fetch and decode because they are sequential, single-cycle processors.After deducting nop energy, SONIC’s authors estimate 40% of energy goes to instruction fetch and decode.
  • Architecture opportunities: LEA’s 4KB SRAM, non-overlapped DMA, unsupported access patterns, and missing vector left-shift and scalar multiply limit inference efficiency.These limitations force frequent transfers and software fallbacks.
  • Correctness support: Correctness requires a full-stack approach because architecture alone misses I/O, data timeliness, and interrupt effects.Software can identify the state needed for correctness more precisely than conservative per-instruction architectural backup.
  • Energy efficiency: 14% of system energy could be saved by eliminating frequent FRAM writes to loop indices alone.Future intermittent architectures must also eliminate or amortize wasted energy in fetch, decode, register files, and FRAM.
  • Programmability: Programmability remains essential because dominant applications in intermittent computing are not yet determined.Highly specialized ASICs offer efficiency but sacrifice programmability.

11 Conclusion

The paper presents efficient DNN inference on commodity energy-harvesting systems and combines network compression with intermittence-aware execution support.

  • SONIC and TAILS specialize intermittence support to guarantee correct execution regardless of the power system.The conclusion reports overhead reductions of up to 6.9× and 12.2× over the state-of-the-art, respectively.
  • GENESIS automatically compresses networks to maximize end-to-end application performance by balancing inference energy and accuracy.
Loading 1810.07751v2…